A self-healing system and method for transmission errors in computer networks based on policy learning

Abstract

This invention discloses a computer network transmission error self-healing system and method based on policy learning, relating to the field of computer network transmission technology. It includes an error detection module for real-time acquisition of link quality indicators of the network transmission link, including signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and latency jitter; a security rule layer storing a set of hard constraint rules preset based on expert experience, used to generate action security masks according to the link quality indicators. The action security mask is a binary vector, where each element corresponds to a self-healing action, with a value of 1 allowing the action to be executed and a value of 0 prohibiting the action. The security rule layer of this application encodes expert experience into hard constraint rules, generating action security masks to prohibit illegal actions from the source. The policy network can only make decisions within a subset of actions allowed by the mask; a deterministic fallback layer takes over when the policy network times out or the mask fails, forming a dual security defense.

Description

Technical Field

[0001] This invention relates to the field of computer network transmission technology, specifically to a self-healing system and method for computer network transmission errors based on policy learning. Background Technology

[0002] During data transmission, computer networks inevitably experience transmission errors such as bit errors, packet loss, and latency jitter due to factors such as link aging, electromagnetic interference, load fluctuations, and equipment failures. Traditional error handling methods include forward error correction (FEC), automatic repeat request (ARQ), adaptive modulation and coding (AMC), and fast rerouting (FRR). These technologies have played important roles in their respective applicable scenarios, but they still have the following shortcomings.

[0003] In error detection and localization, traditional detection methods such as Cyclic Redundancy Check (CRC) only trigger fixed rule responses after detecting bit errors, and cannot dynamically select recovery strategies based on the real-time network status. Although knowledge graph-based intelligent diagnostic solutions improve root cause analysis capabilities, knowledge graph construction relies on a large amount of manual annotation, resulting in high cold-start costs. Furthermore, the computational overhead of graph reasoning is difficult for edge devices to bear. Probe packet schemes (such as TTL expiration-triggered probes) can only identify specific fault types such as routing loops and lack the ability to detect latent faults such as gradual degradation of link quality.

[0004] In terms of transmission self-healing, although Topology Independent Fast Rerouting (TI-LFA) can achieve 100% coverage of backup paths, the pre-computed backup paths do not guarantee global optimality and increase control plane overhead. Adaptive modulation and coding rely on channel state information feedback, resulting in significant decision lag in high mobility or fast fading scenarios. Centralized control schemes in Software-Defined Networking (SDN) become single-point bottlenecks when controllers fail, and the fixed hierarchy of multi-level repair strategies makes them ill-suited for complex fault scenarios. Regarding the application of policy learning and reinforcement learning to network optimization, existing technologies have explored deep reinforcement learning (DRL) for routing optimization, policy gradient methods for resource scheduling, and multi-agent reinforcement learning for distributed network control. However, reinforcement learning faces unique challenges in network transmission error self-healing scenarios: First, the online exploration process may output action decisions that violate security constraints (such as choosing to reduce the coding rate when the bit error rate is already high), leading to a sharp deterioration in network performance. Second, parameter drift during policy updates may cause decision oscillations, and there is a lack of smooth switching mechanisms between old and new policies. Third, in multi-agent collaborative scenarios, independent decisions by each node may lead to action conflicts (such as adjacent nodes simultaneously switching to the same backup link), and there is a lack of effective conflict resolution solutions. Fourth, in resource-constrained scenarios such as insufficient network device computing power or communication interruptions, existing solutions lack a smooth degradation mechanism from policy learning modes to deterministic rule modes, failing to guarantee even the most basic error recovery capabilities.

[0005] Therefore, this application proposes a self-healing system and method for computer network transmission errors based on policy learning to overcome the above-mentioned defects. Summary of the Invention

[0006] To address the aforementioned technical problems, a self-healing system and method for computer network transmission errors based on policy learning is provided. This technical solution solves the problem of not being able to guarantee error recovery capability.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A self-healing system for computer network transmission errors based on policy learning includes: an error detection module for real-time acquisition of link quality indicators of network transmission links, wherein the link quality indicators include signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and delay jitter; The security rule layer stores a set of hard constraint rules preset based on expert experience, which is used to generate an action security mask according to the link quality index. The action security mask is a binary vector, where each element corresponds to a self-healing action. A value of 1 indicates that the action is allowed to be executed, and a value of 0 indicates that the action is prohibited from being executed. The action security mask determines the value of each element according to the judgment result of the hard constraint rule set on the current link quality index. The policy learning layer is equipped with a lightweight policy network. It takes the link quality index and the one-hot encoding of the self-healing action executed in the previous time step as state input, and outputs the self-healing action decision within the action subset constrained by the action security mask. A deterministic fallback layer stores preset threshold triggering rules. When the policy learning layer does not output an action decision within a preset configurable time period, or when the interval between the generation time of the action security mask and the current time exceeds the preset configurable mask validity period, the deterministic self-healing action corresponding to the threshold triggering rule is executed. The decision arbitration module is connected to the security rule layer, the policy learning layer, and the deterministic fallback layer, respectively. It is used to select the policy learning layer to output the self-healing action when the action security mask is valid and the policy learning layer outputs the action decision within a set time period; otherwise, it selects the deterministic fallback layer to output the self-healing action.

[0008] Preferably, the policy learning layer further includes a constraint penalty unit, used to add a constraint violation penalty term based on the set of hard constraint rules to the loss function of the policy network, so that the policy network tends to satisfy all the constraint conditions in the set of hard constraint rules during the training process. The decision arbitration module employs a double buffering mechanism during the policy parameter update of the policy learning layer. The updated policy parameters are written to a backup buffer. After the policy parameters in the backup buffer are verified, they are switched to the active buffer; otherwise, the policy parameters in the original active buffer remain unchanged.

[0009] A self-healing method for transmission errors in computer networks based on policy learning includes the following steps: S1. Real-time collection of link quality indicators of network transmission links, including signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and delay jitter; S2. Input the link quality index into the security rule layer, and generate an action security mask based on the preset hard constraint rule set. The action security mask is a binary vector that identifies the subset of self-healing actions that are currently allowed to be executed. S3. Input the link quality index, the one-hot encoding of the self-healing action executed in the previous time step and the action safety mask into the policy network of the policy learning layer. The policy network outputs the self-healing action decision within the action subset constrained by the action safety mask. S4. Determine whether the policy network outputs a self-healing action decision within the set time and whether the action security mask is still valid. If the policy network fails to output the decision within the time limit or the action security mask has expired, proceed to step S5. Otherwise, execute the self-healing action decision and proceed to step S6. S5. The deterministic fallback layer executes the corresponding deterministic self-healing action according to the preset threshold trigger rules; S6. Observe the changes in network state after the self-healing action is executed, calculate the reward value, and update the policy parameters of the policy network.

[0010] Preferably, the set of hard constraint rules in step S2 includes: When the bit error rate exceeds 1%, the action of maintaining the current encoding rate is prohibited, and forward error correction enhancement is forcibly enabled or an automatic retransmission request is triggered. When the bit error rate exceeds 5%, actions to reduce the coding rate are prohibited, and a forced switch to the backup link is initiated. When the link switching delay exceeds 10ms, the link switching action is prohibited and the forward error correction enhancement action is performed instead.

[0011] Preferably, the policy parameter update of the policy network in step S6 includes: Add a constraint violation penalty term based on the set of hard constraint rules to the loss function of the policy network, so that the policy network tends to satisfy all the constraints in the set of hard constraint rules during the training process; A double-buffering mechanism is used to switch policy parameters: the updated policy parameters are written to the backup buffer. After the policy parameters in the backup buffer pass the preset verification conditions, the backup buffer is switched to the active buffer; otherwise, the policy parameters in the original active buffer remain unchanged.

[0012] Preferably, the preset verification conditions include: Under the policy parameters of the backup buffer, the simulation decisions of the most recent N time steps are evaluated. If the proportion of the simulation decisions that violate the set of hard constraint rules is greater than the preset allowable violation ratio threshold, the verification is deemed to have failed. If the average reward value of the simulation decision is lower than the average reward value of the original active buffer policy parameters, and the difference exceeds the preset degradation threshold, then the verification is deemed unsuccessful.

[0013] Preferably, the self-healing action decision of the policy network in step S3 includes at least one of the following actions: maintaining the current transmission parameters, reducing the coding rate by level 1, reducing the coding rate by level 2, enabling forward error correction enhancement, switching to a backup link, and triggering an automatic retransmission request. The reward value mentioned in step S6 is calculated according to the following formula: in, As a reward value, For the current throughput, For the current round-trip delay, The current packet loss rate, , , , For preset weighting coefficients, As a policy switching penalty, when the currently executed self-healing action is different from the self-healing action of the previous time step. Select 1 if the value is 1, otherwise select 0. The one-hot encoding of the self-healing action executed in the previous time step is used as part of the policy network state input, enabling the policy network to directly compute in the current state. The value of is chosen to ensure that the reward function is fully observable with respect to the current state, satisfying the Markov property condition required by the policy gradient algorithm.

[0014] Preferably, the method further includes a multi-agent cooperation step: The self-healing system instances deployed on different network nodes are treated as agents. Each agent synchronizes its own link quality indicators and self-healing action decisions through the gNMI or gNOI protocol. When a conflict is detected between the self-healing action decision of this node and the self-healing action decision of neighboring agents, negotiation is carried out according to a preset priority rule. The priority rule is as follows: each agent determines its priority in order of comprehensive degradation score from high to low. The agent with the highest comprehensive degradation score performs the switching action first, and the other agents adjust to maintain the current transmission parameters or reduce the coding rate. The overall degradation score Calculate using the following formula: in, For intelligent agents The overall degradation score, This represents the number of categories for link quality metrics. For the first Preset weights for class indicators For intelligent agents No. Normalized degradation ratio of class indicators; When the When the value of a category indicator is higher, it indicates lower quality. ; When the When the value of a category indicator is higher, it indicates better quality. ; in, For intelligent agents No. The current measured value of the category indicator. For the first Preset alarm thresholds for similar indicators; The fault contribution weighting factor Calculated by the fault contribution ranking step, when the fault contribution ranking step is not initiated. Use the default value of 1.

[0015] Preferably, the method further includes a fault contribution ranking step: When multiple link quality indicators are abnormal at the same time, the conditional mutual information between each indicator is calculated based on the historical change sequence of each link quality indicator, and a fault correlation diagram is constructed. Perform a limited-depth reverse causal tracing on the fault correlation graph to calculate the contribution score of each fault node to the current transmission error. The reverse causal tracing starts from the current transmission error symptom node and propagates the contribution value layer by layer along the edges of the fault correlation graph to its upstream nodes, setting a maximum tracing depth. and layer-by-layer decay factor ; in the In layer propagation, upstream nodes From its downstream nodes The increment of the received contribution score is , in, For nodes In the The incremental contribution score received by the layer For nodes and The conditional mutual information between them is the edge weight. The propagation coefficient of this edge; when a propagation path passes through a sequence of nodes with repeated nodes, the path is truncated to eliminate the circular dependency; the final contribution score of each faulty node. ; Based on the contribution score Calculate the fault contribution weighting factor : ,in This is the contribution amplification factor. This represents the maximum contribution score among all indicators. Based on the contribution scores from highest to lowest, the corresponding faulty nodes are prioritized as the input items for the policy network state.

[0016] Preferably, the method further includes a resource-constrained degradation step: When it is detected that the available computing resources of the current device are lower than a preset resource threshold, the policy parameter update of the policy learning layer is frozen, and only the deterministic fallback layer performs self-healing action according to the threshold triggering rule. When the communication connection between the policy learning layer and the external network is interrupted, the policy parameter update of the policy learning layer is frozen and the multi-agent collaboration step is stopped. The local deterministic fallback layer independently performs the self-healing action. After the communication connection is restored, the state information of the neighboring agents is resynchronized.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: The security rule layer of this application encodes expert experience into hard constraint rules and generates an action security mask to prohibit illegal actions from the source. The policy network can only make decisions within the subset of actions allowed by the mask. The deterministic fallback layer takes over when the policy network times out or the mask fails, forming a double security defense.

[0018] By introducing a constraint violation penalty term into the training loss function through a constraint penalty unit, the policy network continuously tends to satisfy hard constraints during parameter updates; the double buffering mechanism ensures that the active policy is replaced only after the new policy parameters have been verified, thus avoiding performance degradation caused by unverified parameters.

[0019] A priority negotiation mechanism employing a comprehensive degradation score ranking system, implemented through multi-agent collaboration, prioritizes actions for each node based on quantifiable degradation levels, avoiding link congestion and resource contention caused by simultaneous switching. A resource-constrained degradation mechanism automatically freezes policy learning when computing power is insufficient or communication is interrupted, reverting to purely deterministic rule-driven approaches to ensure basic self-healing capabilities are not lost; upon communication recovery, the state is automatically resynchronized without manual intervention. The fault contribution ranking step quantifies the contribution of each faulty node based on conditional mutual information and limited-depth reverse causal tracing. The tracing depth is constrained by the maximum number of layers and layer-by-layer decay factors, and cyclic dependencies are eliminated through path truncation, ensuring the deterministic and reproducible calculation process of contribution scores. This allows the policy network's state input to focus on key fault factors, improving decision-making efficiency. Attached Figure Description

[0020] Figure 1 This is an architectural block diagram of the self-healing system for computer network transmission errors based on policy learning according to the present invention. Figure 2 This is a flowchart of the self-healing method for computer network transmission errors based on policy learning according to the present invention. Figure 3 This is a schematic diagram illustrating the generation of security rule layer masks and the constraint of violation penalty items in this invention; Figure 4 This is a schematic diagram of the double buffering mechanism of the present invention; Figure 5 This is a schematic diagram illustrating the multi-agent collaboration and fault contribution ranking of the present invention; Figure 6 This is a flowchart of the resource-constrained degradation process of the present invention; Figure 7Fault correlation diagram and contribution weighting factors Calculation flowchart. Detailed Implementation

[0021] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0022] Reference Figure 1 As shown, a computer network transmission error self-healing system based on policy learning includes: an error detection module, a security rule layer, a policy learning layer, a deterministic backoff layer, and a decision arbitration module.

[0023] The error detection module is used to collect link quality indicators of network transmission links in real time. The link quality indicators include signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and latency jitter. The error detection module obtains the real-time values ​​of the above indicators by polling the performance monitoring interface of network devices (such as SNMP MIB, gNMI telemetry stream). The collection period is configured according to business requirements, with a typical value of 100ms to 1s. The error detection module outputs the collected link quality indicators to the security rule layer and the policy learning layer respectively.

[0024] The security rule layer stores a set of hard constraint rules pre-set based on expert experience, used to generate action security masks according to link quality indicators. The action security mask is a binary vector, where each element corresponds to a self-healing action; a value of 1 indicates that the action is allowed, and a value of 0 indicates that the action is prohibited. The mask determines the value of each element based on the hard constraint rule set's judgment of the current link quality indicator. The security rule layer regenerates the mask each time a new link quality indicator is received and outputs the mask to the policy learning layer and the decision arbitration module. Simultaneously, the security rule layer records the mask's generation time for the decision arbitration module to determine whether the mask is still within its valid duration.

[0025] The policy learning layer deploys a lightweight policy network that takes the link quality index and the one-hot encoded self-healing action executed in the previous time step as state inputs. It outputs the self-healing action decision within a subset of actions constrained by the action safety mask. The policy network employs a fully connected neural network architecture. The input layer receives a concatenated vector of the link quality index vector and the one-hot encoded vector of the self-healing action executed in the previous time step. The inclusion of the one-hot encoded self-healing action from the previous time step in the state input is due to the policy switching penalty term in the reward function. This depends on a comparison between the current action and the action at the previous time step. If the state does not contain information about the action at the previous time step, then... Hidden variables that are unobservable to the policy network do not satisfy the Markov property requirement, leading to biased policy gradient estimation and affecting algorithm convergence. By incorporating the action from the previous time step into the state, the policy network can fully compute the reward value in the current state, the state transition satisfies the Markov property, and the policy gradient is calculated correctly.

[0026] The hidden layer consists of two layers, each with 64 neurons, and uses the ReLU activation function. The output layer outputs the probability distribution of each candidate action. During the inference phase, the policy network sets the probability of actions with a value of 0 in the mask to zero, normalizes the remaining action probabilities, and then samples or takes the action with the highest probability as the decision output. The action decision output by the policy network is sent to the decision arbitration module.

[0027] The deterministic fallback layer stores preset threshold triggering rules. When the policy learning layer fails to output an action decision within a set time (i.e., policy network inference timeout), or when the interval between the generation time of the action safety mask and the current time exceeds the preset mask validity period (i.e., the mask is invalid and cannot reflect the latest constraint state), the deterministic self-healing action corresponding to the threshold triggering rule is executed. The threshold triggering rule is a set of "indicator condition-action" mapping pairs. For example: when the bit error rate exceeds 1%, forward error correction enhancement is enabled; when the bit error rate exceeds 5%, switching to the backup link is initiated; when the round-trip latency exceeds 200ms, the coding rate is reduced. The deterministic fallback layer outputs the execution result to the decision arbitration module.

[0028] The decision arbitration module connects to the security rule layer, policy learning layer, and deterministic fallback layer. It selects the policy learning layer to output the self-healing action when the action security mask is valid and the policy learning layer outputs an action decision within a set time period; otherwise, it selects the deterministic fallback layer. The specific decision logic is as follows: the decision arbitration module checks two conditions: first, whether the policy learning layer returned an action decision within the set time period; and second, whether the interval between the current time and the mask generation time does not exceed the mask's valid duration. When both conditions are met, the action decision output by the policy learning layer is selected; when either condition is not met, the action output by the deterministic fallback layer is selected. The decision arbitration module also records the self-healing action executed at the end of each decision cycle and outputs the self-healing action executed at the previous time step to the policy learning layer in the next decision cycle, allowing the policy network to construct a state vector containing historical action information.

[0029] Both the set duration and the mask validity duration are configurable parameters. They are set according to the characteristics of the network environment during system deployment and remain fixed during operation without dynamic adjustment. The set duration is configured based on the maximum time consumed by a single inference attempt in the policy network: on edge servers with sufficient computing resources, the inference time of the policy network is usually no more than 10ms, and the set duration can be configured to 10ms to 20ms; on embedded devices with limited computing power, the inference time can reach 30ms to 50ms, and the set duration should be configured accordingly to 50ms. The mask validity duration is configured based on the rate of change of link quality indicators: in high mobility or rapid fading scenarios, link quality can change significantly within 1 second, and the mask validity duration should be configured to 1s to 2s; in wired or fixed wireless link scenarios, link quality changes more slowly, and the mask validity duration can be configured to 3s to 5s.

[0030] Building upon the above embodiments, the learning layer further includes a constraint penalty unit and a double-buffered switching mechanism. The constraint penalty unit adds a constraint violation penalty term based on the set of hard constraint rules to the loss function of the policy network, causing the policy network to tend to satisfy all constraints in the set of hard constraint rules during training. Loss function The calculation method is as follows: in, This is the basic loss term for policy gradient algorithms (such as the clip loss in the PPO algorithm). To constrain the penalty coefficient, Constraint violation metric. The calculation method is as follows: For each hard constraint rule, calculate the violation magnitude of the constraint after the current policy is executed in the simulation environment, and sum the violation magnitudes of all constraints. For example, if a hard constraint stipulates that reducing the coding rate is prohibited when the bit error rate exceeds 5%, but the policy network still chooses to reduce the coding rate in a simulation state with a bit error rate of 6%, then the violation magnitude of this constraint is the portion of the bit error rate exceeding the threshold (i.e., This is multiplied by the probability of the violation action being selected. By introducing this term into the loss function, the direction of parameter updates in the policy network is guided by constraints, enabling the learned policy to maximize cumulative reward while satisfying hard constraints.

[0031] The decision arbitration module employs a double-buffering mechanism during policy parameter updates at the policy learning layer. Updated policy parameters are written to a backup buffer; once verified, the backup buffer becomes the active buffer; otherwise, the original active buffer's policy parameters remain unchanged. The introduction of constraint penalties constrains policy behavior during training, while action safety masks constrain policy output during inference. These two elements form a collaborative mechanism of soft constraints during training and hard constraints during inference, enhancing policy security.

[0032] like Figure 2 As shown, this embodiment of the invention provides a self-healing method for computer network transmission errors based on policy learning, including the following steps: S1. Real-time acquisition of network transmission link quality indicators, including signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and jitter. The acquisition method is the same as that of the error detection module described above.

[0033] S2. Input the link quality indicators into the security rule layer, and generate an action security mask based on the preset set of hard constraint rules. The action security mask is a binary vector that identifies the subset of self-healing actions that are currently allowed to be executed.

[0034] S3. Input the link quality index, the one-hot encoding of the self-healing action executed in the previous time step, and the action safety mask into the policy network of the policy learning layer. The self-healing action executed in the previous time step is recorded by the decision arbitration module and fed back to the policy learning layer in the next decision cycle. When there is no action from the previous time step in the first decision cycle, the one-hot encoding is initialized to an all-zero vector. The policy network outputs the self-healing action decision within the action subset constrained by the action safety mask. The policy network sets the probability of actions with a value of 0 in the mask to zero, normalizes the probabilities of the remaining actions to obtain the effective action distribution, and samples or selects the action with the highest probability from the effective action distribution as the decision output.

[0035] S4. Determine whether the policy network outputs a self-healing action decision within the set time and whether the action safety mask is still valid. If the policy network fails to output the decision within the timeout period or the action safety mask has expired, proceed to step S5; otherwise, execute the self-healing action decision output by the policy network and proceed to step S6. The determination in step S4 is performed by the decision arbitration module, and the determination logic is consistent with the description in the decision arbitration module.

[0036] S5. The deterministic fallback layer executes the corresponding deterministic self-healing action according to the preset threshold triggering rules. The threshold triggering rules select the action directly based on the comparison result between the current link quality index and the preset threshold, without relying on the policy network.

[0037] S6. Observe the changes in network state after the self-healing action is executed, calculate the reward value, and update the policy parameters of the policy network.

[0038] The steps S1 to S6 above constitute a self-healing decision cycle. During operation, this cycle is continuously executed in a set collection cycle, enabling the system to respond to changes in network transmission status in real time.

[0039] In this embodiment, as Figure 3 As shown, the set of hard constraint rules in step S2 includes the following three rules: Rule 1: When the bit error rate exceeds 1%, maintaining the current coding rate is prohibited; forward error correction enhancement must be forcibly enabled or an automatic retransmission request must be triggered. The rationale for this rule is that a bit error rate exceeding 1% indicates that the error correction capability at the current coding rate is insufficient to cope with channel degradation. Continuing to maintain the coding rate will lead to bit error accumulation, necessitating the introduction of additional error correction or retransmission mechanisms. Rule 2: When the bit error rate exceeds 5%, reducing the coding rate is prohibited, and a forced switch to a backup link is required. The rationale for this rule is that a bit error rate exceeding 5% indicates severe degradation of the current link quality. While reducing the coding rate can improve the error correction capability of a single link, it will sacrifice throughput. Furthermore, under severe degradation conditions, the reduction in coding rate is insufficient to restore reliable transmission, and a direct switch to a better-quality backup link is necessary. Rule 3: When the link switching delay exceeds 10ms, link switching actions are prohibited; instead, forward error correction enhancement actions are performed. The rationale for this rule is that a link switching delay exceeding 10ms indicates that the target backup link is currently unreachable or has establishment delays. Performing a switching action at this time would cause service interruption, and it is necessary to temporarily replace it with enhanced error correction capabilities on the current link.

[0040] The process of generating the action safety mask is as follows: Let the set of candidate self-healing actions be... These correspond to "Maintain current transmission parameters", "Reduce coding rate by 1 level", "Reduce coding rate by 2 levels", "Enable forward error correction enhancement", "Switch to backup link", and "Trigger automatic retransmission request", respectively. Initialize mask vector. Then check each hard constraint rule one by one: If the current link quality metric triggers rule one, then the mask element corresponding to "keep current transmission parameters" will be set to 0; If rule two is triggered, the mask elements corresponding to "reduce coding rate by 1 level" and "reduce coding rate by 2 levels" will be set to 0. If rule three is triggered, the mask element corresponding to "switch to backup link" will be set to 0. The final generated mask vector... This is the action safety mask at the current moment.

[0041] The above three hard constraint rules can be adjusted according to the actual network configuration, including the threshold value and the scope of prohibited actions. Modification of the set of hard constraint rules does not affect the interaction logic between the security rule layer, the policy learning layer, the deterministic fallback layer, and the decision arbitration module.

[0042] In this embodiment, as Figure 3 and Figure 4 As shown, the policy parameter update of the policy network in step S6 includes two parts: adding a constraint violation penalty term and a double buffering mechanism. The constraint violation penalty term is added as described above, by adding it to the loss function. In online update scenarios, after each action decision is made and the reward value is observed, the policy network uses a policy gradient algorithm (such as PPO) to calculate the parameter update amount. The calculation of the parameter update amount includes the gradient contribution of the constraint violation penalty term, so that the update direction can increase the cumulative reward while avoiding increasing the risk of constraint violation.

[0043] like Figure 4 As shown, the process of policy parameter switching using the double-buffering mechanism is as follows: The system maintains two parameter buffers. The active buffer stores the policy parameters currently used for inference, and the standby buffer stores new policy parameters to be verified. When policy parameters are updated, the calculated new parameters are written to the standby buffer without directly replacing the active buffer. Then, preset verification conditions are run under the parameters in the standby buffer. If the verification passes, the standby buffer is switched to the active buffer, and the original active buffer becomes the standby buffer for the next update. If the verification fails, the parameters in the standby buffer are discarded, the active buffer remains unchanged, and the policy network continues to use the currently active parameters for inference. The double-buffering mechanism ensures that the parameters used by the policy network at any time are verified and stable parameters, avoiding the degradation of decision quality caused by unverified new parameters.

[0044] The preset verification conditions include the following two items: Verification Condition 1: Under the policy parameters of the backup buffer, evaluate the simulation decisions of the most recent N time steps. If the proportion of decisions violating the set of hard constraints in the simulation decisions is greater than the preset allowable violation ratio threshold, the verification is deemed unsuccessful. A typical value for N is 100 to 500 time steps, and a typical value for the allowable violation ratio threshold is 0.01 to 0.05. The purpose of this condition is to ensure that the constraint satisfaction rate of the new policy is not lower than an acceptable level. An allowable violation ratio threshold greater than 0 allows the policy to make a small number of violations during the exploration process, avoiding frequent policy rollbacks due to individual noisy samples that could affect learning efficiency.

[0045] Verification condition 2: If the average reward value of the simulated decision is lower than the average reward value of the original active buffer policy parameters, and the difference exceeds the preset degradation threshold, then the verification is deemed unsuccessful. The typical value of the degradation threshold is 5% to 10% of the original average reward value. The purpose of this condition is to ensure that the performance of the new policy is not lower than that of the old policy and to prevent performance degradation caused by policy parameter updates.

[0046] When both verification conditions one and two pass, the policy parameters of the backup buffer are deemed to have passed verification, and a buffer switch is executed. When either verification condition fails, the verification is deemed to have failed, and the original active buffer remains unchanged.

[0047] The evaluation method for simulation decisions is as follows: using the policy parameters of the spare buffer, the historical state sequence of the most recent N time steps is re-inferred in the offline simulation environment to obtain the action decision for each time step. Then, the violation rate and average reward value are calculated according to the hard constraint rules and reward function respectively. The offline simulation environment uses the same state space, action space and hard constraint rule set as the online environment, but does not execute actions on the real network and does not affect online services.

[0048] In this embodiment, the self-healing action decision of the policy network in step S3 includes at least one of the following actions: maintaining the current transmission parameters, reducing the coding rate by 1 level, reducing the coding rate by 2 levels, enabling forward error correction enhancement, switching to a backup link, and triggering an automatic retransmission request. The above six actions constitute a discrete action space, and the policy network outputs a six-dimensional probability vector, which is filtered and normalized by an action security mask to obtain the effective action distribution.

[0049] The reward value in step S6 is calculated according to the following formula: in, As a reward value, This represents the current throughput (in Mbps). This represents the current round-trip time (in milliseconds). The current packet loss rate (value from 0 to 1). This is the throughput weighting coefficient. This is the time delay weighting coefficient. This is the packet loss rate weighting coefficient. The penalty weight coefficient for strategy switching As a policy switching penalty, the currently executed self-healing action is different from the self-healing action in the previous time step. Select 1 if the value is 1, otherwise select 0.

[0050] The weighting coefficients reflect the service's preference for network performance metrics. For throughput-sensitive services, Take the larger value; for latency-sensitive services, Take the larger value; for reliability-sensitive services, Take the larger value. The introduction of the suppression strategy prevents the network from frequently switching actions between adjacent time steps, thus avoiding network instability caused by action oscillations.

[0051] A typical weight configuration is as follows: , , , This configuration prioritizes packet loss penalty the highest, followed by throughput, then latency, and finally action switching penalty the lowest.

[0052] The data required for calculating the reward value comes from the link quality indicators obtained by the error detection module in the next collection cycle after the self-healing action is executed.

[0053] It should be pointed out that, The value of depends on whether the current action is the same as the action at the previous time step. If the state of the policy network does not contain the action from the previous time step, then The current state is unobservable for the current decision, and the reward function does not satisfy computability based on the current state, violating the Markov property condition required by the policy gradient algorithm. This invention addresses this by including a one-hot encoding of the action from the previous time step in the state input of the policy network, enabling the policy network to determine the current state upon observation. The value of is determined to ensure the state observability of the reward function and the unbiasedness of the policy gradient estimation.

[0054] like Figure 5 As shown, the method in this embodiment of the invention also includes a multi-intelligent collaborative step, where self-healing system instances deployed on different network nodes are treated as agents. Each agent synchronizes its own link quality indicators and self-healing action decisions through the gNMI (gRPC Network Management Interface) or gNOI (gRPC Network Operations Interface) protocol. gNMI and gNOI are open network management protocols that support efficient structured data transmission and are suitable for real-time state synchronization scenarios. Each agent sends its local link quality indicators and current decisions to neighboring agents at a set synchronization period, typically ranging from 1 second to 5 seconds.

[0055] When a conflict is detected between the self-healing action decision of this node and the self-healing action decision of neighboring agents, negotiation is conducted according to a preset priority rule. The conditions for determining a conflict are: both this node and the neighboring node choose to switch to the same backup link within the same time window, or the execution of this node's action will affect the link quality of the neighboring nodes (e.g., after this node switches to the backup link, the load on the backup link increases, affecting the neighboring nodes already on that link).

[0056] The priority rule is as follows: each agent is prioritized according to its overall degradation score, from highest to lowest. The agent with the highest overall degradation score executes the handover action first, while the remaining agents adjust their actions to maintain the current transmission parameters or reduce the coding rate. Overall degradation score Calculate using the following formula: in, For intelligent agents The overall degradation score, The number of categories of link quality indicators (in this embodiment) (corresponding to five categories of indicators: signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and latency jitter). For the first Preset base weights for class indicators, For the first Fault contribution weighting factor of class indicators For intelligent agents No. The normalized degradation ratio of the class of indicators. When the fault contribution ranking step is not initiated, The default value is set to 1, at which point the overall degradation score degenerates into a fixed weighted sum.

[0057] Normalized degradation ratio The calculation distinguishes between two cases: when the first... A higher value for a metric indicates lower quality (e.g., bit error rate, packet loss rate, round-trip time, and latency jitter). When the first When a higher value for a metric indicates better quality (such as signal-to-noise ratio), the metric is considered to have higher quality. .in For intelligent agents No. The current measured value of the category indicator. For the first Preset alarm thresholds for similar indicators.

[0058] A typical weighting configuration is as follows: bit error rate weight 0.3, packet loss rate weight 0.3, round-trip time weight 0.2, latency jitter weight 0.1, and signal-to-noise ratio weight 0.1. Typical alarm threshold values ​​are: bit error rate 1%, packet loss rate 0.1%, round-trip time 100ms, latency jitter 50ms, and signal-to-noise ratio 15dB.

[0059] The overall degradation score incorporates a fault contribution weighting factor. The significance lies in this: when multiple link quality indicators are abnormal simultaneously, some indicators may be direct manifestations of the root cause failure, while others may simply be collateral effects triggered by the root cause. If the root cause and collateral effects are not distinguished, and scores are calculated using only fixed weights, collateral effect indicators may dominate the scoring results due to their larger values, leading to a priori bias in symptom assessment rather than root cause evaluation. By increasing the weight of root cause indicators and suppressing the weight of cascading effect indicators, root cause-driven agents can obtain higher scores in priority negotiation, ensuring that resources are allocated to the faulty nodes that need the most attention.

[0060] like Figure 5 and Figure 7As shown, the method also includes a fault contribution ranking step. When multiple link quality indicators are abnormal simultaneously, the conditional mutual information between the indicators is calculated based on the historical change sequences of each link quality indicator, and a fault correlation graph is constructed. The conditional mutual information measures the statistical dependence of two indicators under the given condition of other indicators. The calculation method is as follows: in, and The first The and the first Historical change sequence of link quality indicators The sequence formed by the remaining indicators, Representing the probability distribution, the larger the conditional mutual information value, the stronger the direct statistical dependence between the two indicators, and the larger the corresponding edge weight in the fault association graph.

[0061] Perform reverse causal tracing on the fault correlation graph to calculate the contribution score of each fault node to the current transmission error. Reverse causal tracing starts from the current symptom node of the transmission error and propagates the contribution score layer by layer along the edges of the fault correlation graph to its upstream nodes. The propagation process is constrained by the maximum tracing depth and the layer-by-layer decay factor, and cyclic dependencies are truncated. The specific steps are as follows: 1) Initialization: Set the current transmission error symptom node. Contribution score increment at level 0 The cumulative contribution score of all other faulty nodes .

[0062] 2) Layer-by-layer propagation: For the first... layer( ), traversing the first Each node in the layer that receives the contribution score increment For the fault association graph from the node Pointing to node Each edge compute nodes In the Increment of contribution score received by the layer: in For nodes In the The incremental contribution score received by the layer (note that it is not the node) The cumulative score only includes the newly added portion of that layer. For nodes and The conditional mutual information between edges is the weight of that edge. Let be the propagation coefficient of that edge. This is the layer-by-layer attenuation factor. Propagation coefficient. The propagation coefficient is determined based on the time sequence of the edges: edges where the upstream indicator's anomaly occurs earlier than the downstream indicator are assigned a higher propagation coefficient (typical value 0.8 to 1.0), while edges where the time sequence is unclear are assigned a lower propagation coefficient (typical value 0.3 to 0.5).

[0063] 3) Circular Dependency Truncation: Maintain the sequence of nodes along the current propagation path during propagation. If a node... Already exists from symptom nodes To the node If a propagation path forms a loop (e.g., A causes B, and B then feeds back to A), then the propagation path is cut off, and the propagation continues along this path to the node. Upstream propagation, but nodes The accumulated contribution score remains unaffected. This truncation rule ensures that each propagation path is a loop-free path, preventing contribution scores from accumulating infinitely in loops.

[0064] 4) Termination Condition: Propagation terminates when any of the following conditions are met: ① Current layer number Reaching maximum traceability depth ;②th Increment of all contribution scores in the layer All are below the preset cutoff threshold (Typical value 0.01), meaning that the impact of continued propagation on the final contribution score is negligible.

[0065] 5) Contribution Score Summary: The final contribution score of each faulty node is the sum of the increments accumulated across all propagation layers, i.e. .

[0066] Maximum traceability depth The typical value is 3, which is the layer-by-layer decay factor. The typical value ranges from 0.5 to 0.8. This limits the scope of propagation and avoids excessive tracing in large fault correlation graphs, which could lead to unreasonable contribution scores for remote weakly correlated nodes. This causes the contribution of the more distant upstream node to the received data to decrease more significantly, and the causal effect decreases with the propagation distance.

[0067] Based on contribution score Calculate the fault contribution weighting factor : in, This is the contribution amplification factor (typically ranging from 0.5 to 2.0). This represents the maximum contribution score among all indicators. This formula maximizes the contribution score of the indicator with the highest contribution. Indicators with zero contribution The remaining indicators are interpolated according to their contribution ratio. The value of determines the degree to which the root cause index is amplified relative to the associated index in the overall deterioration score.

[0068] Fault Contribution Weighting Factor Directly substitute into the comprehensive degradation scoring formula of Example 8 In this process, the weights corresponding to the root cause indicators are amplified, while the weights corresponding to the cascading effect indicators are suppressed, thereby enabling root cause-driven agents to obtain higher scores in multi-agent priority negotiation.

[0069] Furthermore, based on the contribution scores from highest to lowest, the corresponding faulty nodes are prioritized as input items to the policy network. In the input features of the policy network, fault indicators with high contribution are given greater weight, causing the policy network to prioritize responding to the root cause indicators most likely to cause transmission errors during decision-making. This step allows the policy network to focus on the key indicators with the greatest causal contribution, rather than equally distributing attention to all abnormal indicators. This effectively reduces state space complexity and improves decision-making efficiency in scenarios with many indicator dimensions.

[0070] like Figure 6 As shown, the method in this embodiment of the invention further includes a resource-constrained degradation step. When the available computing resources of the current device are detected to be lower than a preset resource threshold, the policy parameter updates of the policy learning layer are frozen, and only the deterministic fallback layer performs self-healing actions according to the threshold triggering rules. Available computing resources include CPU utilization, memory usage, and inference accelerator load. When CPU utilization exceeds 80% or memory usage exceeds 90%, computing resources are considered insufficient. Freezing policy parameter updates means that the policy network stops online learning, but the trained policy network can still continue to be used for inference. When the computing resources recover to above the threshold, the freeze is automatically lifted, and online updates of policy parameters resume.

[0071] When a communication interruption is detected between the policy learning layer and the external network, the policy parameter updates of the policy learning layer are frozen and the multi-agent collaborative steps are stopped. The local deterministic fallback layer then independently performs the self-healing action. The conditions for determining a communication interruption are: no response is received from the peer after three consecutive heartbeat checks, or five consecutive timeouts in the gNMI / gNOI session. During the communication interruption, the local node cannot obtain the state information of neighboring agents. Therefore, multi-agent collaboration is stopped to avoid conflicting decisions based on outdated information. Instead, the local deterministic fallback layer independently performs the self-healing action based on locally collected link quality indicators and preset threshold triggering rules.

[0072] Once the communication connection is restored, the state information of neighboring agents is resynchronized. The synchronized information includes the decision logs of neighboring agents during the communication interruption and the current link state. After synchronization is complete, the online updates of multi-agent collaborative steps and policy parameters are restarted.

[0073] The resource-constrained degradation mechanism ensures that even under extreme conditions such as insufficient network device computing power or communication interruption, the system can still provide basic error self-healing capabilities through the deterministic fallback layer, and will not lose its self-healing function due to the unavailability of the policy learning layer. Both degradation and recovery processes are executed automatically without manual intervention.

[0074] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A self-healing system for computer network transmission errors based on policy learning, characterized in that, include: The error detection module is used to collect link quality indicators of the network transmission link in real time. The link quality indicators include signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and delay jitter. The security rule layer stores a set of hard constraint rules preset based on expert experience, which is used to generate an action security mask according to the link quality index. The action security mask is a binary vector, where each element corresponds to a self-healing action. A value of 1 indicates that the action is allowed to be executed, and a value of 0 indicates that the action is prohibited from being executed. The action security mask determines the value of each element according to the judgment result of the hard constraint rule set on the current link quality index. The policy learning layer is equipped with a lightweight policy network. It takes the link quality index and the one-hot encoding of the self-healing action executed in the previous time step as state input, and outputs the self-healing action decision within the action subset constrained by the action security mask. A deterministic fallback layer stores preset threshold triggering rules. When the policy learning layer does not output an action decision within a preset configurable time period, or when the interval between the generation time of the action security mask and the current time exceeds the preset configurable mask validity period, the deterministic self-healing action corresponding to the threshold triggering rule is executed. The decision arbitration module is connected to the security rule layer, the policy learning layer, and the deterministic fallback layer, respectively. It is used to select the policy learning layer to output the self-healing action when the action security mask is valid and the policy learning layer outputs the action decision within a set time period; otherwise, it selects the deterministic fallback layer to output the self-healing action.

2. The computer network transmission error self-healing system based on policy learning according to claim 1, characterized in that: The policy learning layer also includes a constraint penalty unit, which is used to add a constraint violation penalty term based on the set of hard constraint rules to the loss function of the policy network, so that the policy network tends to satisfy all the constraint conditions in the set of hard constraint rules during the training process. The decision arbitration module employs a double buffering mechanism during the policy parameter update of the policy learning layer. The updated policy parameters are written to a backup buffer. After the policy parameters in the backup buffer are verified, they are switched to the active buffer; otherwise, the policy parameters in the original active buffer remain unchanged.

3. A self-healing method for transmission errors in computer networks based on policy learning, characterized in that: Includes the following steps: S1. Real-time collection of link quality indicators of network transmission links, including signal-to-noise ratio, bit error rate, packet loss rate, round-trip time, and delay jitter; S2. Input the link quality index into the security rule layer, and generate an action security mask based on the preset hard constraint rule set. The action security mask is a binary vector that identifies the subset of self-healing actions that are currently allowed to be executed. S3. Input the link quality index, the one-hot encoding of the self-healing action executed in the previous time step and the action safety mask into the policy network of the policy learning layer. The policy network outputs the self-healing action decision within the action subset constrained by the action safety mask. S4. Determine whether the policy network outputs a self-healing action decision within the set time and whether the action security mask is still valid. If the policy network fails to output the decision within the time limit or the action security mask has expired, proceed to step S5. Otherwise, execute the self-healing action decision and proceed to step S6. S5. The deterministic fallback layer executes the corresponding deterministic self-healing action according to the preset threshold trigger rules; S6. Observe the changes in network state after the self-healing action is executed, calculate the reward value, and update the policy parameters of the policy network.

4. The method for self-healing computer network transmission errors based on policy learning according to claim 3, characterized in that: The set of hard constraint rules mentioned in step S2 includes: When the bit error rate exceeds 1%, the action of maintaining the current encoding rate is prohibited, and forward error correction enhancement is forcibly enabled or an automatic retransmission request is triggered. When the bit error rate exceeds 5%, actions to reduce the coding rate are prohibited, and a forced switch to the backup link is initiated. When the link switching delay exceeds 10ms, the link switching action is prohibited and the forward error correction enhancement action is performed instead.

5. The self-healing method for computer network transmission errors based on policy learning according to claim 3, characterized in that: The policy parameter update of the policy network in step S6 includes: Add a constraint violation penalty term based on the set of hard constraint rules to the loss function of the policy network, so that the policy network tends to satisfy all the constraints in the set of hard constraint rules during the training process; A double-buffering mechanism is used to switch policy parameters: the updated policy parameters are written to the backup buffer. After the policy parameters in the backup buffer pass the preset verification conditions, the backup buffer is switched to the active buffer; otherwise, the policy parameters in the original active buffer remain unchanged.

6. The method for self-healing computer network transmission errors based on policy learning according to claim 5, characterized in that: The preset verification conditions include: Under the policy parameters of the backup buffer, the simulation decisions of the most recent N time steps are evaluated. If the proportion of the simulation decisions that violate the set of hard constraint rules is greater than the preset allowable violation ratio threshold, the verification is deemed to have failed. If the average reward value of the simulation decision is lower than the average reward value of the original active buffer policy parameters, and the difference exceeds the preset degradation threshold, then the verification is deemed unsuccessful.

7. The method for self-healing computer network transmission errors based on policy learning according to claim 3, characterized in that: The self-healing action decision of the policy network in step S3 includes at least one of the following actions: maintaining the current transmission parameters, reducing the coding rate by level 1, reducing the coding rate by level 2, enabling forward error correction enhancement, switching to the backup link, and triggering an automatic retransmission request. The reward value mentioned in step S6 is calculated according to the following formula: in, As a reward value, For the current throughput, For the current round-trip delay, The current packet loss rate, , , , For preset weighting coefficients, As a policy switching penalty, when the currently executed self-healing action is different from the self-healing action of the previous time step. Select 1 if the value is 1, otherwise select 0. The one-hot encoding of the self-healing action executed in the previous time step is used as part of the policy network state input, enabling the policy network to directly compute in the current state. The value of is chosen to ensure that the reward function is fully observable with respect to the current state, satisfying the Markov property condition required by the policy gradient algorithm.

8. The self-healing method for computer network transmission errors based on policy learning according to claim 3, characterized in that: The method also includes a multi-agent cooperative step: The self-healing system instances deployed on different network nodes are treated as agents. Each agent synchronizes its own link quality indicators and self-healing action decisions through the gNMI or gNOI protocol. When a conflict is detected between the self-healing action decision of this node and the self-healing action decision of neighboring agents, negotiation is carried out according to a preset priority rule. The priority rule is as follows: each agent determines its priority in order of comprehensive degradation score from high to low. The agent with the highest comprehensive degradation score performs the switching action first, and the other agents adjust to maintain the current transmission parameters or reduce the coding rate. The overall degradation score Calculate using the following formula: in, For intelligent agents The overall degradation score, This represents the number of categories for link quality metrics. For the first Preset weights for class indicators For intelligent agents No. Normalized degradation ratio of class indicators; When the When the value of a category indicator is higher, it indicates lower quality. ; When the When the value of a category indicator is higher, it indicates better quality. ; in, For intelligent agents No. The current measured value of the category indicator. For the first Preset alarm thresholds for similar indicators; The fault contribution weighting factor Calculated by the fault contribution ranking step, when the fault contribution ranking step is not initiated. Use the default value of 1.

9. The self-healing method for computer network transmission errors based on policy learning according to claim 3, characterized in that: The method also includes a fault contribution ranking step: When multiple link quality indicators are abnormal at the same time, the conditional mutual information between each indicator is calculated based on the historical change sequence of each link quality indicator, and a fault correlation diagram is constructed. Perform a limited-depth reverse causal tracing on the fault correlation graph to calculate the contribution score of each fault node to the current transmission error. The reverse causal tracing starts from the current transmission error symptom node and propagates the contribution value layer by layer along the edges of the fault correlation graph to its upstream nodes, setting a maximum tracing depth. and layer-by-layer decay factor ; in the In layer propagation, upstream nodes From its downstream nodes The increment of the received contribution score is , in, For nodes In the The incremental contribution score received by the layer For nodes and The conditional mutual information between them is the edge weight. The propagation coefficient of this edge; when a propagation path passes through a sequence of nodes with repeated nodes, the path is truncated to eliminate the circular dependency; the final contribution score of each faulty node. ; Based on the contribution score Calculate the fault contribution weighting factor : ,in This is the contribution amplification factor. This represents the maximum contribution score among all indicators. Based on the contribution scores from highest to lowest, the corresponding faulty nodes are prioritized as the input items for the policy network state.

10. The method for self-healing computer network transmission errors based on policy learning according to claim 3, characterized in that: The method also includes a resource-constrained degradation step: When the available computing resources of the current device are detected to be lower than a preset resource threshold, the policy parameter update of the policy learning layer is frozen, and only the deterministic fallback layer performs the self-healing action according to the threshold triggering rule. When the communication connection between the policy learning layer and the external network is interrupted, the policy parameter update of the policy learning layer is frozen and the multi-agent collaboration step is stopped. The local deterministic fallback layer independently performs the self-healing action. After the communication connection is restored, the state information of the neighboring agents is resynchronized.

Images