CDN fault node isolation method and system based on operator network management data

Abstract

The invention discloses a CDN fault node isolation method and system based on telecom operator network management data, and relates to the technical field of CDN fault management and operation and maintenance, and the method comprises the steps that an operator network management center collects multi-stage network equipment and link data, and a CDN health center collects indexes such as node packet loss rate; comparing an index and a threshold value quality index degree which exceeds a threshold value, and marking a suspected node; querying network management data to locate a fault category; calculating a score by combining a fault position weight, and isolating if the score exceeds a threshold value and the fault position is a node or a direct connection link; eliminating nodes and scheduling traffic; and quickly triggering the alarm, subscribing the network management alarm, directly marking the suspected node and triggering the process. According to the method, the isolation accuracy of the CDN fault node is improved, and misjudgment caused by an upstream network fault is reduced; the isolation response time delay is shortened; congestion aggravation is inhibited; a quantifiable and configurable isolation decision system is formed, the operation and maintenance intelligence is improved, and the service quality and the user experience are optimized.

Description

Technical Field

[0001] This invention relates to the field of CDN fault management and operation and maintenance technology, and in particular to a CDN fault node isolation method and system based on operator network management data. Background Technology

[0002] With the continuous expansion of internet business, content delivery networks (CDNs) have become a critical infrastructure for internet services such as government and enterprise websites, video-on-demand, live streaming, and file downloads. The service quality of CDN nodes depends not only on the node's own hardware resources and software status, but is also significantly affected by the network status of multiple levels of operators, including access networks, metropolitan area networks, and backbone networks.

[0003] Existing CDN fault node isolation solutions typically rely solely on probe data from the CDN operator's internal health center. This involves periodically collecting network quality metrics such as packet loss rate and latency, as well as some node operational metrics, and comparing them to preset thresholds to determine node health. In real-world large-scale network environments, these solutions exhibit the following main problems: First, the accuracy of isolation is significantly insufficient in certain scenarios. Due to the lack of support from carrier network management data, existing solutions struggle to distinguish whether a fault originates from the CDN node itself or its direct link, or from upstream access equipment, metropolitan area networks, or backbone networks. When upstream carrier network links or equipment experience failures or congestion, it often leads to one or more CDN nodes within the same access domain simultaneously exhibiting abnormal network quality indicators. If isolation is based solely on CDN-side detection results, it is easy to misjudge upstream network faults as node faults, resulting in incorrect isolation of a large number of nodes, leading to misjudgments and over-isolation in isolation decisions, and further deteriorating the user's access experience.

[0004] Secondly, the timeliness of isolation affects the duration of the fault's impact. Traditional solutions often employ detection cycles of 30 seconds or even minutes, requiring multiple consecutive cycles of exceeding limits before triggering isolation to avoid misjudgments caused by occasional fluctuations. This low-frequency detection + multi-cycle confirmation mechanism often results in a delay of several minutes between the occurrence of a fault and the effective isolation. During this period, various user access services cannot be promptly redirected to other nodes, impacting user experience. Effectively shortening the fault diagnosis and isolation time can reduce the scope of the fault's impact on users.

[0005] Third, incorrect or delayed isolation can easily trigger increased congestion or even a chain reaction of congestion, specifically: On the one hand, in the case of upstream network failure, if error isolation is performed on a large number of nodes, the traffic that was originally concentrated in the faulty link or domain will be forcibly migrated to other paths or nodes in a short period of time. These paths or nodes may be close to saturation and can easily form new congestion bottlenecks, thereby inducing congestion to worsen.

[0006] On the other hand, when the isolation action lags behind the fault evolution, some areas of the network have already experienced queuing and load imbalance. Subsequent large-scale traffic migration is more likely to cause cascading overload of links and nodes on a larger scale, forming a chain of congestion, which multiplies the scope of the fault's impact and significantly increases the difficulty of operation and maintenance recovery.

[0007] In summary, existing technologies have significant shortcomings in terms of isolation accuracy and timeliness, and are prone to exacerbating congestion in complex operator network environments. There is an urgent need for a new CDN fault node isolation method and system that can combine operator network management data for refined fault location and explicitly suppress large-scale false isolation and cascading congestion. Summary of the Invention

[0008] The present invention proposes a CDN fault node isolation method and system based on operator network management data to solve the problems mentioned in the prior art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a CDN fault node isolation method based on operator network management data, comprising the following steps: First, during the cross-domain data collection and correlation phase, the status and performance data of devices and links in multiple network domains managed by the telecom operator's network management center, including access networks, metropolitan area networks, and backbone networks, are collected. Meanwhile, the network quality indicators of each CDN node are periodically collected by the CDN health center. These network quality indicators include at least packet loss rate and latency, and their judgment thresholds, detection cycles, and triggering conditions are configurable parameters. This invention aligns and correlates the operator's network management data and the CDN node's network quality indicators within the same monitoring period, enabling subsequent isolation decisions to comprehensively utilize observation information from both the operator and CDN sides under the same time benchmark.

[0010] Secondly, during the suspected fault node screening phase, the CDN Health Center compares the network quality indicators of each CDN node with the corresponding thresholds. When any indicator meets the preset abnormal triggering conditions within the monitoring period, the corresponding CDN node is marked as a suspected fault node, triggering isolation judgment instead of direct isolation. This effectively filters out minor fluctuations or instantaneous jitters, avoiding repeated triggering of the isolation process for atypical anomalies.

[0011] Secondly, in the fault location determination stage based on network management data, for CDN nodes marked as suspected faults, the CDN health center initiates a network management data query to the operator's network management center through the API gateway to obtain the network topology, link operating status, and upstream device operating status associated with the suspected fault node. Based on the above network management data and the network quality indicators of the suspected fault node, a fault location score is calculated, and the fault location is determined into categories such as "node itself," "direct link between the node and the access switch," "access switch," and "metropolitan area network core equipment or backbone network equipment" according to preset fault location classification rules. This invention, by clearly distinguishing between "node / direct link / device faults" and "upstream network faults," fundamentally corrects misjudgments caused by solely relying on CDN-side detection, thereby improving the location accuracy of isolation decisions.

[0012] Subsequently, in the isolation score calculation and isolation decision-making stage, this invention constructs a CDN anomaly score. And scores are given based on the fault points. Fault location weights are pre-configured for fault location categories. The weighting is as follows: faults within the CDN node itself have the highest weighting, followed by faults in the direct link between the node and the access switch, then faults in the access switch itself have a lower weighting, and faults in core metropolitan area network (MAN) and backbone network equipment have the lowest weighting. The final isolation score for suspected faulty nodes is calculated. .

[0013] when When the fault location is greater than or equal to the preset isolation threshold and the fault location is located on the CDN node itself or its directly connected network link, isolation is performed on the node; when When the fault is below the preset isolation threshold, or when the fault location is determined to be located in the core equipment of the metropolitan area network or the backbone network equipment, the fault is determined to be an upstream network fault. The explicit constraint "does not perform isolation on any single CDN node" suppresses large-scale false isolation from both numerical and policy mechanisms, reducing the risk of inducing "congestion aggravation".

[0014] During the isolation execution and traffic scheduling phase, the CDN scheduling center receives isolation decision results from the CDN health center, removes nodes requiring isolation from the traffic scheduling candidate list, and migrates user access requests to other healthy nodes through DNS resolution policy adjustments or HTTP redirection policies. On the one hand, by orchestrating DNS resolution records and HTTP redirection responses, traffic migration can be completed quickly without changing the business entry domain name, shortening the overall latency from fault discovery to traffic migration. On the other hand, during the migration process, the traffic weight of each healthy node is gradually increased in a preset step size based on the real-time load of each healthy node. When the network quality index or load index of a healthy node is detected to be close to the preset safety limit, traffic to that node is restricted and the new traffic is distributed to more healthy nodes, avoiding the artificial creation of new bottlenecks during the migration process, thereby reducing the secondary impact of the isolation action itself on the network.

[0015] Regarding the triggering mechanism, this invention adds a subscription mechanism for operator network management fault alarms to the traditional periodic detection. The CDN health center subscribes to fault alarm information of devices or links associated with CDN nodes through the API gateway. When the operator network management center issues a fault alarm for a relevant device or link, the CDN health center can skip the multi-period confirmation wait and directly mark the associated CDN node as a suspected fault node, immediately initiating the fault location determination and isolation score calculation process. This upgrades the isolation decision from a single reliance on detection to a hybrid mode of "alarm-driven + periodic detection". This mechanism significantly shortens the time window for fault identification and handling while keeping false positives controllable, improving the timeliness of isolation, and is especially suitable for latency-sensitive business scenarios.

[0016] In terms of system architecture, this invention also proposes a CDN fault node isolation system based on operator network management data to implement the above method. The system includes: a network management center module, an API gateway module, a health center data acquisition module, a health center intelligent isolation decision module, and a traffic scheduling module. Specifically, the network management center module is responsible for collecting and managing device and link status and performance data across multiple network domains, and providing query and alarm interfaces; the API gateway module provides the CDN health center with unified network management data query and alarm subscription capabilities; the health center data acquisition module is responsible for periodically collecting network quality indicators of CDN nodes and aligning them with network management data in the time dimension; the health center intelligent isolation decision module is responsible for calculating the network quality indicators of each location. Perform fault location classification and assign fault location weights. And calculate the isolation score. It generates a decision to "isolate a node" or "not isolate a single node"; in addition, it provides a way to directly trigger a rapid judgment process when a network management alarm is received; the traffic scheduling module is responsible for executing node removal, DNS resolution adjustment and HTTP redirection according to the isolation decision, and distributing the migration traffic among multiple healthy nodes according to a progressive traffic scheduling strategy, reducing the probability of the isolation operation itself causing increased congestion.

[0017] Compared with existing technologies, the beneficial effects of this invention are: 1. Improved Isolation Accuracy: This invention introduces operator network management data to uniformly model the device and link status of multi-level network domains such as access network, metropolitan area network, and backbone network, and integrates and analyzes it with the network quality indicators of CDN nodes. This enables a fine-grained distinction between fault locations in "node / directly connected link / device" and "upstream network". Node-level isolation is only performed when the fault is determined to be in the node itself or in the case of a fault in the directly connected link / device. In upstream network fault scenarios such as metropolitan area network and backbone network, it is explicitly stated that "isolation is not performed on a single node", which significantly reduces the false isolation rate caused by upstream network faults.

[0018] 2. Improved isolation timeliness: This invention adds a rapid triggering mechanism driven by operator network management fault alarms on the basis of periodic detection. When a critical device or link fails, the fault location determination and isolation score calculation process can be started immediately. This upgrades the isolation response from a passive mode that simply relies on 30-second or even minute-level detection to an active mode of "alarm-driven + periodic detection supplement". This significantly shortens the time delay from the occurrence of the fault to the isolation decision and reduces the duration of the fault's impact.

[0019] 3. Suppressing congestion escalation: This invention uses fault location classification and fault location weighting. The joint design, from a mechanism perspective, limits the error isolation of a large number of CDN nodes in the event of upstream network failure, avoiding a large-scale passive migration of traffic caused by a single failure. At the same time, a progressive traffic scheduling strategy is adopted during the isolation process, gradually increasing the traffic weight and distributing it across multiple nodes according to the real-time load level of healthy nodes, reducing the formation of new bottlenecks caused by a sudden increase in traffic at a single point, and comprehensively suppressing the "congestion exacerbation" caused by incorrect isolation and delayed isolation.

[0020] 4. Establishing an interpretable and configurable isolation decision-making system: This invention calculates anomaly scores for both the CDN itself and the upstream network location. Fault location weight Finally, the isolation score was calculated. A quantifiable and configurable isolation decision model is built, so that isolation actions no longer rely on a single threshold or empirical rules, but are completed based on multi-dimensional quantitative assessments. This makes it easier for operation and maintenance personnel to understand, optimize, and audit, thereby improving the intelligence and controllability of the CDN fault management and operation and maintenance system. Attached Figure Description

[0021] Figure 1 This is a schematic block diagram of the CDN fault node isolation method based on operator network management data proposed in this invention; Figure 2 This is a schematic block diagram of the CDN fault node isolation system based on operator network management data proposed in this invention. Figure 3 A comparison chart of false isolation rates for CDN nodes under different fault scenarios; Figure 4 A bar chart comparing isolation scores at different fault locations; Figure 5 A bar chart comparing the isolation response delay of the traditional method and the present invention; Figure 6 Line graph showing the load change of healthy nodes during gradual traffic migration. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0025] Reference Figures 1 to 6 A CDN fault node isolation method based on operator network management data includes the following steps: S1 Cross-Domain Data Collection and Association Steps: The operator's network management center uses SNMPv3, NETCONF, and COBRA protocols to collect CPU utilization, memory utilization, link bandwidth utilization, bit error rate, and fault identification data from routers and switches in the access network, metropolitan area network, and backbone network. The CDN health center periodically collects the average packet loss rate and average latency (from the detection results of all network detection nodes to this node). The fault threshold values ​​for these indicators are configurable, the detection period is configurable in the range of 10-60 seconds, and the trigger condition supports the configuration of "m data points exceeding limits within n consecutive periods" (n is generally ≥5, m is generally ≥3 and ≤m). The operator's network management data and CDN node indicators are associated with minute-level timestamps at a 5-minute monitoring cycle to ensure data time sequence consistency.

[0026] S2 suspected fault node screening steps: The CDN health center compares the network quality indicators of each CDN node with the corresponding preset thresholds. When any indicator among the sliding n monitoring points meets the over-limit condition for more than m consecutive monitoring cycles, the node is judged to be suspected of being faulty.

[0027] S3's fault location determination steps based on carrier network management data: For CDN nodes marked as suspected faults, the CDN Health Center initiates a network management data query to the carrier network management center through the carrier API gateway to obtain the three-level network topology associated with the suspected fault node, the operating status of direct links and access switch equipment, and the operating status and performance indicators of core equipment in the metropolitan area network and backbone network; based on the above network management data and the network quality indicators of the suspected fault node, isolation scores are calculated and isolation decisions are made.

[0028] S4 Isolation Score Calculation and Isolation Decision Steps: For suspected faulty nodes, start the node. Fraction calculation, Fault location weights are pre-configured for each type of fault location according to the fault location category. Among them, the failure of the CDN node itself has the highest weight, followed by the failure of the direct link between the node and the access switch, and the failure of the metropolitan area network core equipment and backbone network equipment has the lowest weight; the calculation principles for the score of each type of equipment and link are as follows: Calculated based on the node server failure rate: =Number of faulty servers in the cluster / Total number of servers in the cluster * 100. For example, if a node server cluster has 20 servers, and 4 of them have device failures, 2 have CPU utilization exceeding 80%, and 2 have link failures, meaning a total of 8 devices in the cluster are faulty or have exceeded the threshold, then... =8 / 20*100=40.

[0029] Calculate based on access switch fault conditions: When the entire device fails, or CPU utilization is >80%, or outbound link bandwidth utilization is >90%, etc. (This can be configured based on experience and alarm categories). =100. If none of the above situations occur, then =Number of failed switch outbound links / Total number of switch outbound links * 100. For example, if there are 2 outbound links and 1 of them is failed, then... =1 / 2*100=50.

[0030] and Calculations based on the failure status of metropolitan area network (MAN) and backbone network routers: When the entire device fails or the CPU utilization rate is >80%, the bandwidth utilization rate of the MAN egress link is >90%, or the overall bandwidth utilization rate of the backbone network is >80%, etc. (configurable based on experience and alarm categories). and It equals 100. If none of the above occurs, then... =Number of outbound link failures / Total number of outbound links * 100. =Number of failed links / Total number of links * 100. For example, if there are a total of 4 outbound links, and 1 of them is failed, then... =1 / 4*100=25.

[0031] CDN anomaly score is calculated based on suspected faulty nodes. Pre-configure fault location weights for each type of fault location according to their location category. Among them, the weight of CDN node failures is the highest, followed by access switch failures, and the weight of failures of metropolitan area network core equipment and backbone network equipment is the lowest; according to the formula Calculate the final isolation score of the suspected faulty node. When the final isolation score If the final isolation score is greater than or equal to the preset isolation threshold, the node is determined to be a node that needs to be isolated; when the final isolation score is... If the value is less than the preset isolation threshold, no isolation operation will be performed on any individual CDN node.

[0032] S5 Isolation Execution and Traffic Scheduling Steps: The CDN scheduling center receives the isolation decision results from the CDN health center and removes the nodes that need to be isolated from the traffic scheduling candidate list; through DNS resolution policy adjustment (modifying the A record corresponding to the node) and HTTP 302 redirection, user access requests are scheduled to other healthy nodes; during the traffic migration process, based on the real-time load level of the healthy nodes, the traffic weight of one or more target healthy nodes is gradually increased in steps of 5%-15%, and the CPU utilization rate of each healthy node is monitored in real time to be ≤70% and the bandwidth utilization rate is ≤85%. When the above thresholds are approached, traffic to be migrated to that node is restricted, and the new traffic is distributed to other healthy nodes with lower load.

[0033] The S6 alarm-driven rapid triggering mechanism steps are as follows: The CDN Health Center subscribes to the operator's network management center for device fault alarms and link interruption alarms through the operator's API gateway; when it receives a device or link fault alarm associated with the target CDN node, it skips the multi-cycle confirmation process (5 preset monitoring cycles), directly marks the relevant CDN node as a suspected fault node, and immediately triggers the S3 fault location determination step and the S4 isolation score calculation step.

[0034] In this invention, the screening of suspected faulty nodes is carried out during the monitoring period. Within, N sampling times are obtained at a preset sampling interval. The monitoring data, for each sampling time Collect average packet loss rate Average latency Two types of network quality indicators; and fault threshold values ​​for each type of indicator. , (all greater than 0), , This is a configurable threshold parameter based on operational experience; when there are m points out of n consecutive data points during the sliding process. or m points When the fault threshold is exceeded, the operator's network management data query and fault location determination are triggered.

[0035] In this invention, the fault location classification rules include: When determining a node's own fault, the following conditions must be met simultaneously: "average CPU utilization of the server ≥ 70%" and "number of available servers within the node ≤ 4". When determining an access network device fault, the following conditions must be met: "uplink fault of the access switch or access network device is obtained through the network management system". When determining an upstream metropolitan area network and backbone core network fault, the following conditions must be met: "uplink topology metropolitan area network and backbone core network device fault (number of faults ≥ 1), egress link fault (number of faults ≥ 1) and fault ratio are obtained through the network management system".

[0036] In this invention, the fault location weight The configuration satisfies: the fault location weight corresponding to the CDN node's own fault. Maximum value; access network equipment fault weight Less than Fault location weights corresponding to faults in metropolitan area network core equipment and backbone network equipment and It is the minimum value.

[0037] In this invention, the isolation execution and traffic scheduling further include: before traffic migration, it is necessary to obtain the current load data (CPU utilization, memory utilization, bandwidth utilization) of each healthy node, and prioritize selecting nodes with loads below the threshold of 50% as traffic migration targets; when the loads of all healthy nodes are in the range of 50%-70%, traffic is allocated according to the principle of "the lower the load, the higher the priority"; during the traffic migration process, the load data of healthy nodes is updated every 30 seconds to ensure that no node load exceeds the threshold.

[0038] This invention also discloses a system applying the aforementioned CDN fault node isolation method based on operator network management data, characterized by comprising the following modules: Network Management Center Module: Deployed on the telecom operator's side, it supports SNMPv3 agent, NETCONF client and COBRA interface services; it can collect CPU utilization (measurement accuracy ±1%), memory utilization (measurement accuracy ±1%), link bandwidth utilization (measurement accuracy ±0.5%), and bit error rate (measurement accuracy ±0.1%) of devices in access network, metropolitan area network and backbone network, as well as related device alarms and link interruption alarms; it provides network management data query interface and alarm push interface with HTTP / HTTPS protocol, supports 1000+ concurrent query requests, and the interface response latency is ≤300ms.

[0039] API Gateway Module: As an intermediate communication node between the Network Management Center Module and the CDN Health Center, it supports the conversion between SNMPv3, NETCONF and RESTful protocols, and provides the CDN Health Center with a unified network management data query interface and fault alarm subscription interface; the interface call logs must be stored for more than 30 days, and support multi-dimensional retrieval by "call time, calling module, interface type"; the interface abnormal retry mechanism supports 3 automatic retries with a retry interval of 500ms.

[0040] Health Center Data Acquisition Module: Deployed on the CDN side, it includes a node data acquisition unit and a data association unit. The node data acquisition unit obtains the network quality indicators of CDN nodes every 10-30 seconds via HTTP POST requests, supporting simultaneous acquisition from 1000+ CDN nodes. Data transmission uses TLS 1.3 encryption. The data association unit monitors at a 1-5 minute cycle, aligning operator network management data and CDN node data with minute-level timestamps. The associated data is stored in the database, and hot data within 24 hours is cached in memory, supporting a data retrieval speed of 100,000 data points per second.

[0041] The intelligent isolation decision module of the health center includes a suspected fault node judgment unit, a fault location judgment unit, an isolation score calculation unit, and an alarm triggering unit. The fault location judgment unit has four types of fault judgment logic built in and supports visual editing and saving of fault judgment rules. The isolation score calculation unit supports online adjustment of isolation threshold and fault location weight, and the adjustment results take effect in real time. After receiving an alarm from the operator's network management system, the alarm triggering unit triggers the fault location judgment and isolation score calculation process within 1-2 seconds.

[0042] Traffic scheduling module: includes node management unit and traffic allocation unit; the node management unit supports the maintenance of scheduling list for 10,000+ CDN nodes, and the response time for node removal operation is ≤1 second; the traffic allocation unit supports DNS resolution adjustment and HTTP 302 redirection, the progressive traffic scheduling step size supports configuration of 5%-15%, and real-time monitoring of the CPU utilization and bandwidth utilization of healthy nodes. When the indicators exceed the threshold, traffic transfer is triggered, and the transfer response time is ≤10 seconds.

[0043] In this invention, the intelligent isolation decision module of the health center is also configured to generate a decision instruction of "upstream network failure, do not perform isolation on single node" when the fault is determined to be a metropolitan area network or backbone network equipment failure.

[0044] The following two examples further illustrate the specific implementation of this system: Example 1

[0045] Decision not to isolate suspected faulty CDN nodes This embodiment is applied to a CDN node in a metropolitan area network of a first-tier city. The node carries video-on-demand and file download traffic for the metropolitan area network. The node has one access switch with two 100Gbps links, which are connected to two core routers of the metropolitan area network. The node server cluster contains 20 business servers. The main focus is on verifying the accuracy of network management data-assisted isolation decision-making and fully covering the entire process of the technical solution.

[0046] 1. Cross-domain data collection and associated execution The operator's network management center deploys SNMPv3 agents, NETCONF clients, and COBRA interface services. Real-time status of access network and metropolitan area network (MAN) devices is collected via SNMPv3: every 5 minutes, the network management system collects data on CPU utilization, uplink bandwidth utilization, bit error rate, and operational status of devices and links from access switches, MAN and backbone core routers. When device failures or link interruptions occur, access switches, MAN and backbone core routers report the faults to the network management center in real time via the SNMPTRAP protocol; and the COBRA interface retrieves planned maintenance information for the next 24 hours at 3:00 AM daily, including the device ID to be maintained, maintenance time period, and affected link range.

[0047] The CDN health center is configured with a node server proactive reporting mechanism. All other probe nodes across the network send 10 data packets every 30 seconds to the probe nodes within this node, for a total of 100 nodes. Each probe node reports its probe results to the health center. Upon receiving all probe results, the health center aggregates them based on destination and time dimensions, calculates the average packet loss rate and average latency for each time period, and stores this data in a database. Hot data from the past hour is stored in an in-memory database, while cold data older than one hour is compressed and stored in a disk database. Backups are performed every hour.

[0048] During the data association phase, the operator's network management data and CDN node network quality indicators are aligned according to a 1-minute monitoring cycle. Using timestamps as a benchmark, this ensures that the data from both sides correspond one-to-one within the same cycle, providing fused data with a unified time benchmark for subsequent decision-making.

[0049] 2. Screening of suspected fault nodes The CDN Health Center has preset fault thresholds for detection indicators: packet loss rate 0.5%, latency 100ms, CPU utilization 80%, and memory utilization 70%. During the monitoring period from 9:00 to 9:03, the detection data for this node showed anomalies: Over five consecutive periods from 9:00 to 9:03, the average packet loss rates from each node to this node were 4.2%, 3.8%, 0.5%, 4%, and 3%, respectively, with average latencies of 60ms, 108ms, 70ms, 50ms, and 65ms, respectively. Four data points exceeded the fault threshold of 0.5% for packet loss rate, and one data point exceeded the average latency threshold. During this period, CPU utilization was less than 75% and memory utilization was less than 70%, not exceeding the thresholds.

[0050] According to the rule that a suspected fault node is identified if more than 3 data points exceed the limit within 5 cycles, and since 3 data points lost packets in the first 4 data points, the rule for a suspected fault node is met in the 4th data set. Therefore, the fault location determination process based on operator network management data is initiated.

[0051] 3. Fault location determination and isolation decision The CDN Health Center initiates a query to the operator's network management center through the operator's API gateway to obtain the operational status of the access switches, metropolitan area network (MAN), and backbone network devices and links associated with the node. The network management data shows that the access switches, MAN, and backbone network routers are in normal status, but two of the four links from the MAN to the backbone network are interrupted, therefore it is determined to be a MAN link failure.

[0052] Configure fault location weights: =0.5 (node ​​fault) =0.3 (Access network equipment failure) =0.1 (Metropolitan Area Network equipment failure) =0.1 (backbone network equipment failure). CDN isolation score is calculated using the formula. Calculation yields =0*0.6+0*0.2+2 / 4*100*0.1+0*0.1=5. The preset isolation threshold is 30.0. The value is 5, which is less than the isolation threshold of 30, so it is determined that no isolation node is required.

[0053] 4. Effectiveness verification data Table 1 compares the effectiveness of the traditional method and the present invention in a metropolitan area network (MAN) fault scenario:

[0054] Table 1 shows that traditional methods, lacking network management data support, exacerbate metropolitan area network (MAN) link congestion due to isolated nodes. This invention achieves accurate fault location through network management data fusion, identifying MAN faults and demonstrating that isolated nodes are ineffective for service recovery. This fully validates the advantages of this invention in isolation decision-making. Example 2

[0055] CDN node isolation implementation driven by network management alarms This embodiment is applied to a CDN node in a certain city. This node, together with two other nodes, serves the city's video-on-demand services. It is connected to one access switch, and the access switch is connected to two metropolitan area network routers, each with a 100Gbps link. The node server cluster contains 30 business servers. The focus is on verifying the alarm-driven rapid triggering mechanism and the effect of gradual traffic scheduling.

[0056] 1. Cross-domain data collection and associated execution The operator's network management center collects status data from access switches, metropolitan area networks, and backbone core routers via SNMPv3: CPU utilization, memory utilization, link traffic, and link connectivity status are collected at 5-minute granularities. Real-time reporting of TRAP events is configured on switches and routers via SNMPv3 TRAP, reporting event types including device failure, LINKDOWN, and LINKUP events, and then reporting them to the network management center's data acquisition module. The COBRA interface retrieves planned maintenance information daily at 2:00 AM.

[0057] The CDN health center is configured to report node data every 30 seconds, collecting metrics including packet loss rate, latency, CPU utilization, memory utilization, and bandwidth utilization. The inter-node probe pinging interval is also 30 seconds. Collected data is time-aligned and correlated on a minute-by-minute basis to ensure synchronization between carrier network management data and CDN node data.

[0058] 2. Screening of suspected fault nodes and determination and isolation of fault location At 2:00, the network management center received two LINKDOWN events from the city's metropolitan area network routers. The network management center then pushed the events to the CDN Health Center, a subscriber to these events.

[0059] The CDN health center determined that the faulty link was between the access switch where the CDN node resides and the metropolitan area network router. The CDN node was restarted. The calculation. According to

[0060] Because the detection period has not yet expired, the node itself did not trigger a quality detection alarm. =0; Based on the link failure events of the metropolitan area network router, determine that the two outgoing links of the access switch are faulty. =2 / 2*100=100; Although the metropolitan area network router has a link failure, it is not a failure of its outbound link, therefore =0. The backbone router is not faulty. =0. Configure fault location weights: =0.5、 =0.3、 =0.1、 =0.1. Therefore:

[0061] The value is 30, which equals the isolation threshold of 30, thus generating an isolation node instruction.

[0062] 4. Isolation Execution and Traffic Scheduling After receiving the instruction, the traffic scheduling module adjusts the global scheduling policy, removes the resolved IP of the node through DNS resolution, and immediately redirects the traffic that was originally flowing to the node to the service IPs of the other two healthy nodes.

[0063] 5. Effectiveness verification data Table 2 compares the effectiveness of the traditional method and the present invention in a switch egress link failure scenario:

[0064] Table 2 shows that traditional methods rely solely on inter-node probes to detect faults, resulting in corresponding fault diagnosis and processing cycles. In contrast, the alarm-driven rapid triggering mechanism reduces the isolation response latency to 8 seconds, validating the stability and effectiveness of this invention in complex network environments.

[0065] refer to Figure 3 This figure highlights the core advantage of this invention: accurately identifying fault locations through network management data. Traditional methods cannot identify the source of faults, with false isolation rates reaching 25%-28% in metropolitan area networks and backbone networks, and even 18% in mixed scenarios. This invention determines fault locations by querying network management data, achieving a false isolation rate of 0% for node-specific faults and ≤3% for upstream network faults, both below the 5% threshold. This effect stems from the fault location classification rules and the isolation score calculation mechanism. For example, in Example 1, with a metropolitan area network link fault, the false isolation rate of this invention is only 2%, while traditional methods exacerbate congestion due to misjudging isolated nodes, fully validating the breakthrough of this invention in improving isolation accuracy.

[0066] refer to Figure 4The chart accurately matches the core logic of "weight configuration + score calculation" in the embodiment. In Embodiment 1, the fault location weights are set to θ=0.5, θ=0.3, θ=0.1, and θ=0.1, with an isolation threshold of 30. By analyzing the code data, the fault score of the node itself (45) and the fault score of the directly connected link (35) both exceed the threshold, indicating that CDN node isolation needs to be performed; while the fault scores of the access network device (12), the metropolitan area network device (8), and the backbone network device (5) are all below the threshold of 30, indicating upstream network faults, and single-node isolation is not performed according to the rules. This mechanism accurately verifies the key logic in Embodiment 1: when the metropolitan area network link fails, Score_iso=8<30, and isolation is not performed according to the rules, avoiding the congestion aggravation problem caused by mis-isolation in traditional methods; while the fault score of the node itself (45) exceeds the threshold, the faulty node can be quickly isolated, highlighting the core advantage of the present invention of "accurately distinguishing fault locations" and effectively avoiding over-isolation of upstream network faults.

[0067] refer to Figure 5 This chart quantifies the advantages of the hybrid triggering method of "alarm-driven + periodic detection" in the embodiment. By comparing response latency data under different fault location scenarios: In the metropolitan area network scenario, the traditional method relies on 30-second periodic detection + multiple periodic confirmations, resulting in a latency of up to 120 seconds; the latency of the present invention is compressed to 15 seconds, a reduction of 87.5%. In the backbone network scenario, the traditional method has a latency of 180 seconds; the present invention, through instant alarm triggering, reduces the latency to only 8 seconds, a reduction of 95.5% compared to the traditional method. In the switch link fault scenario (corresponding to direct link fault), the traditional method relies on periodic detection, resulting in a latency of 100 seconds; the present invention triggers the process within 1-2 seconds, with a latency of only 8 seconds. This low-latency characteristic perfectly adapts to the latency-sensitive service requirements of real-time live streaming, video on demand, etc. in the embodiment, avoiding the service interruption and disconnection problems caused by the minute-level latency of the traditional method, and significantly improving the user experience.

[0068] refer to Figure 6 This diagram aligns with the core design of the progressive traffic scheduling in this embodiment. In Embodiment 2, the traffic migration step size is set to 15%, and the healthy node load thresholds are CPU ≤ 70% and bandwidth ≤ 85%. During the migration process, the load of the three healthy nodes gradually increases from an initial level of around 40%, growing by 7-8% every 30 seconds, reaching a maximum of 70% after 120 seconds, without exceeding the safety limit. If a traditional one-time migration were performed, the load would instantly surge to over 85%, triggering new congestion. In this embodiment, by monitoring the load in real time, when the node bandwidth utilization approaches 85%, new traffic allocation is immediately stopped, and the traffic is distributed to other nodes to avoid secondary congestion. The diagram verifies the effectiveness of this strategy, which both quickly migrates traffic from faulty nodes and ensures the stable operation of healthy nodes, demonstrating the advantages of the "isolation and scheduling coordination" of this invention.

[0069] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A CDN fault node isolation method based on operator network management data, characterized in that, The method comprises the following steps: Collecting, by an operator network management center, state and performance data of devices and links in each network domain of an access network, a metropolitan area network, and a backbone network managed by the operator network management center; periodically collecting, by a CDN health center, network quality indexes of each CDN node; and time-aligning and correlating the operator network management data and the network quality indexes of the CDN nodes in the same monitoring period; The CDN health center compares the network quality indexes of each CDN node with corresponding threshold values, and marks a corresponding CDN node as a suspected faulty node when any index meets a preset abnormal triggering condition in the monitoring period; For a CDN node marked as a suspected faulty node, the CDN health center initiates a network management data query to the operator network management center through an operator API gateway, and obtains network topology structure, link operation state and performance indexes, and upstream device operation state associated with the suspected faulty node; Based on the network management data and the network quality indexes of the suspected faulty node, a preset fault location classification rule is followed; For suspected fault nodes, start the node Score calculation, first according to the location of each fault device and link, give the fault point score through the empirical formula ; Then according to the fault location category, pre-configure the fault location weight for each type of fault location ; According to the formula Calculate the final isolation score of the suspected fault node , wherein: Indicates the final isolation score; pos indicates the fault location; Indicates the score corresponding to the fault location pos; When the final isolation score Is greater than or equal to the preset isolation threshold; When the final isolation score Is less than the preset isolation threshold, do not perform isolation operation on any single CDN node; The CDN scheduling center receives the isolation decision result of the CDN health center, removes the node to be isolated from a traffic scheduling candidate list, and adjusts a DNS resolution strategy or performs HTTP redirection to schedule user access requests to other healthy nodes; The CDN health center subscribes to fault alarm information of the operator network management center through the operator API gateway, and when receiving a device or link fault alarm associated with a target CDN node, directly marks the related CDN node as a suspected faulty node and triggers a fault location judgment and isolation score calculation process without waiting for multi-period confirmation of periodic detection.

2. The CDN faulty node isolation method based on operator network management data according to claim 1, characterized in that, The suspected judgment rule is: in a monitoring period , monitoring data at N sampling time points are acquired at a preset sampling interval, two types of network quality indexes, i.e., average packet loss rate and average time delay, are collected for each sampling time point , and fault threshold values are configured for the two types of indexes respectively , , , , ​ The node suspected fault is judged according to the following principles: in N collection periods, M collection points are greater than the fault threshold value; n, m, , are parameters configurable according to operation experience; when in the sliding N periods, M or M are greater than the fault threshold value, the operator network management data query and fault location determination are triggered. 3.The CDN fault node isolation method based on operator network management data according to claim 1, characterized in that, The fault location classification rule comprises: If the operator network management data shows that the upstream devices are all normal and only the target node is abnormal, it is determined that the node itself is faulty; If the operator network management data shows that an access switch associated with the target node is faulty, or there is packet loss, error code or interruption on the egress link, it is determined that the upstream access device is faulty; If the operator network management data shows that a metropolitan area network core device or a backbone network device is faulty or a link is congested, it is determined that the upstream metropolitan area network or backbone network is faulty, and a decision is generated not to perform separate isolation on each abnormal node.

4. The CDN faulty node isolation method based on operator network management data according to claim 1, characterized in that, The fault location weight The configuration satisfies: the fault location weight corresponding to the CDN node self fault is the maximum value; the fault location weight corresponding to the node associated access switch fault is less than ; the fault location weight corresponding to the metropolitan area network core equipment and the backbone network equipment fault is the minimum value. ​​​​ 5. The CDN faulty node isolation method based on operator network management data according to claim 1, characterized in that, Isolation execution and traffic scheduling further comprise: during the traffic migration process for the node to be isolated, gradually increasing the traffic weight of the target healthy node according to a preset step based on the real-time monitored load level of other healthy nodes; when it is detected that the network quality index or the load index of a certain healthy node approaches a preset safety upper limit, limiting the continued migration of traffic to the node, and distributing the newly added traffic to other healthy nodes.

6. A system for applying the CDN faulty node isolation method based on operator network management data according to any one of claims 1-5, characterized in that, The method comprises the following modules: A network management center module for collecting state and performance data of devices and links in each network domain of an access network, a metropolitan area network, and a backbone network managed by the network management center module, and providing a network management query interface and a fault alarm publishing interface; An API gateway module in communication with the network management center module, providing a unified network management data query interface and a fault alarm subscription interface for the CDN health center; and A CDN health center module for periodically collecting network quality indexes of each CDN node, time-aligning and correlating the operator network management data and the network quality indexes of the CDN nodes in the same monitoring period, comparing the network quality indexes of each CDN node with corresponding threshold values, marking a corresponding CDN node as a suspected faulty node when any index meets a preset abnormal triggering condition in the monitoring period, initiating a network management data query to the operator network management center through the operator API gateway for a suspected faulty node, obtaining network topology structure, link operation state and performance indexes, and upstream device operation state associated with the suspected faulty node, following a preset fault location classification rule based on the network management data and the network quality indexes of the suspected faulty node, receiving an isolation decision result of the CDN health center, removing the node to be isolated from a traffic scheduling candidate list, and adjusting a DNS resolution strategy or performing HTTP redirection to schedule user access requests to other healthy nodes. Health Center Data Collection Module: Using PING, TCP access and other methods, periodically collect network quality indicators between each CDN node, and align the indicators with the operator network management data obtained through the API gateway module in the time dimension, to realize cross-domain data correlation; Health Center Intelligent Isolation Decision Module: used to determine whether the node indicators are deteriorating within the period according to the network quality indicators provided by the Health Center Data Collection Module, and to screen suspected faulty nodes; The API gateway module queries the operator's network management data associated with suspected faulty nodes, determines the fault location category according to preset fault location classification rules, and assigns fault location weights to each fault location. And scores based on fault location. and fault location weight Calculate the final isolation score ; Decisions to generate isolation nodes or not to perform isolation on individual nodes based on isolation thresholds; When a fault alarm is received from the network management center module, the fault location determination and isolation decision process is directly triggered; Traffic scheduling module: communicate with the Health Center Intelligent Isolation Decision Module, used to remove the nodes to be isolated from the traffic scheduling candidate list according to the isolation decisions issued by the Health Center Intelligent Isolation Decision Module, and adjust or HTTP redirect user access requests to healthy nodes through DNS resolution; According to the gradual traffic scheduling strategy provided by the Health Center Intelligent Isolation Decision Module, the migration traffic is distributed among multiple healthy nodes.

7. The system of claim 6, wherein, The health center intelligent isolation decision module is further configured to: when the suspected node queries the operator network management data, obtain When the value is less than the isolation threshold, it is determined that the metropolitan area network core device or the backbone network device is faulty, an upstream network fault is uniformly generated, and no separate isolation decision is performed on each abnormal node.

Images