A device failure quick feedback and diagnosis log processing system

Abstract

The present application relates to the technical field of IT operation and fault diagnosis, in particular to a device fault rapid feedback and diagnosis log processing system to solve the technical problems that the centralized software alarm processing scheme in the prior art has high processing delay under the alarm storm scene, cannot suppress derived alarms in real time, and causes the root cause alarm to be submerged; the technical scheme comprises an edge collection node, a convergence gateway node and a cloud diagnosis control node; the edge collection node completes preliminary deduplication and topology feature label packaging of logs, the convergence gateway node compares and matches derived alarms in parallel through a hardware matching engine, the root cause alarm is uploaded through a high-priority channel, the cloud builds a full-network topology to complete fault tracing and dynamically update the interception flow table and the deduplication strategy.The present application can realize two-stage hardware acceleration noise reduction of alarms, improve the fault feedback speed and diagnosis accuracy, and is suitable for fault diagnosis scenes of distributed device clusters.

Description

Technical Field

[0001] This invention relates to the field of IT operations and maintenance and fault diagnosis technology, and more specifically, to a system for rapid feedback and diagnostic log processing of equipment faults. Background Technology

[0002] In complex distributed systems such as large data centers, cloud computing platforms, and industrial IoT, in order to ensure business continuity, it is necessary to monitor the operating status of massive physical devices and virtual services in real time and respond to faults quickly. When the underlying infrastructure (such as core switches and storage arrays) fails, the impact of the failure will spread rapidly up and down the chain of dependencies between devices, triggering tens of thousands of alarms from different devices and different levels within seconds.

[0003] Existing alarm handling solutions generally adopt centralized, pure software event correlation engines. Their typical workflow is as follows: monitoring probes located in various places report all raw alarms to a central alarm management server. The software on this server compresses, filters, suppresses, and performs causal analysis on the alarms through a preset rule engine. For example, the system maintains a set of static rules: if the core switch A fails, network interruption alarms of all its downstream servers B, C, and D will be suppressed.

[0004] However, this solution has a fundamental technical flaw: the mismatch between the concurrent scale of the alarm storm and the software's serial / limited parallel processing capabilities. A single core device jitter can trigger tens of thousands of derivative alarms within milliseconds. The software rule engine of the central server needs seconds or even minutes to complete the analysis and convergence. During this period, the massive number of alarms has caused management network congestion, central platform overload, and user interface paralysis. Operation and maintenance personnel are unable to filter out the real root cause fault alarms from the alarm storm in real time, causing the rapid feedback mechanism to completely fail.

[0005] Therefore, a rapid feedback and diagnostic log processing system for equipment faults is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a rapid feedback and diagnostic log processing system for equipment faults, in order to solve the technical problems of high processing latency, inability to suppress derivative alarms in real time, and overwhelmed root cause alarms in the case of alarm storms in the prior art.

[0007] To address the aforementioned technical problems, the present invention aims to provide a rapid feedback and diagnostic log processing system for equipment faults, comprising: an edge acquisition node, an aggregation gateway node, and a cloud-based diagnostic control node; The edge acquisition node is used for: Obtain raw device logs from physical devices and collect network topology connection data in real time to reflect device status; Extract the feature identifiers from the original device logs and perform frequency comparison within a locally configured time sliding window to perform preliminary deduplication; simultaneously, convert the network topology connection data into topology feature labels to characterize the physical and logical locations of devices; The topology feature tags are encapsulated into the deduplicated log data to generate tagged alarm messages, which are then output to the aggregation gateway node. At the same time, the network topology connection data is reported to the cloud diagnostic control node. The aggregation gateway node is used for: Receive tagged alarm messages from the edge acquisition nodes; The topology feature tags in the alarm message are parsed and extracted. The topology feature tags are used as retrieval keys and compared in parallel with the pre-stored alarm interception flow table in the internal hardware matching engine to determine whether the alarm message belongs to a derivative alarm caused by an upstream node failure. If the comparison matches, it indicates a derived alarm, and a physical drop operation is directly triggered on the data forwarding plane of the aggregation gateway node; if it does not match, it indicates a potential root cause fault, and it is marked as a root cause alarm message and sent to the cloud diagnostic control node through a high-priority channel. The cloud-based diagnostic control node is used for: Receive network topology connection data reported by the edge acquisition node and root cause alarm messages reported by the aggregation gateway node; A network-wide device topology relationship diagram is constructed based on the network topology connection data; the propagation path of the root cause alarm message is traced and analyzed based on the network-wide device topology relationship diagram to determine the fault diagnosis result; and newly discovered fault derivative paths are mined by combining global historical alarm data, thereby generating an updated alarm interception flow table and an edge deduplication optimization strategy. The updated alarm interception flow table is sent to the corresponding aggregation gateway node, and the edge deduplication optimization strategy is sent to the corresponding edge acquisition node.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This equipment fault rapid feedback and diagnosis log processing system moves the alarm suppression function from the central software to the hardware gateway of the network aggregation layer, and introduces tags injected by the edge acquisition nodes that accurately reflect the topological location of the equipment. This enables the system to perform parallel matching and physical discarding at hardware line speed during alarm message transmission, solving the problem of high processing latency in the central software solution. 2. In the device fault rapid feedback and diagnostic log processing system, through the cloud diagnostic control node's fusion analysis of the entire network topology data and the bidirectional policy distribution to the aggregation gateway node and edge acquisition node, the system establishes a closed-loop self-optimization mechanism of edge, aggregation, and cloud. This makes the alarm suppression rules no longer static, but can automatically adapt to network topology changes, automatically discover new fault derivative paths, and adjust edge deduplication parameters. Attached Figure Description

[0009] Figure 1 This is an overall system block diagram of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] like Figure 1 As shown, this embodiment provides a rapid feedback and diagnostic log processing system for device faults, including: an edge acquisition node, an aggregation gateway node, and a cloud diagnostic control node; The edge acquisition node is used to: acquire the original device logs of physical devices and collect network topology connection data reflecting device status in real time; extract the feature identifiers of the original device logs and perform frequency comparison within a locally configured time sliding window to perform preliminary deduplication; simultaneously, convert the network topology connection data into topology feature tags that characterize the physical and logical locations of devices; encapsulate the topology feature tags into the deduplicated log data to generate tagged alarm messages and output them to the aggregation gateway node, while also reporting the network topology connection data to the cloud diagnostic control node; The aggregation gateway node is used to: receive tagged alarm messages from the edge acquisition node; parse and extract the topology feature tags from the alarm messages, use the topology feature tags as retrieval keys, and perform parallel comparison with the pre-stored alarm interception flow table in the internal hardware matching engine to determine whether the alarm message belongs to a derivative alarm caused by an upstream node failure; if the comparison matches, it indicates that it is a derivative alarm, and a physical drop operation is directly triggered on the data forwarding plane of the aggregation gateway node; if it does not match, it indicates that it is a potential root cause failure, and it is marked as a root cause alarm message and sent to the cloud diagnostic control node through a high-priority channel; The cloud-based diagnostic control node is used to: receive network topology connection data reported by the edge acquisition nodes and root cause alarm messages reported by the aggregation gateway nodes; construct a network-wide device topology relationship diagram based on the network topology connection data; perform source tracing analysis on the propagation path of the root cause alarm messages based on the network-wide device topology relationship diagram to determine the fault diagnosis result; and combine global historical alarm data to mine newly discovered fault derivative paths, thereby generating an updated alarm interception flow table and an edge deduplication optimization strategy; and distribute the updated alarm interception flow table to the corresponding aggregation gateway nodes and the edge deduplication optimization strategy to the corresponding edge acquisition nodes.

[0012] The entire system consists of three types of nodes deployed in a distributed manner: edge acquisition nodes, aggregation gateway nodes, and cloud diagnostic control nodes. Edge acquisition nodes are located close to the data source and are deployed on each physical server or device that needs to be monitored. Aggregation gateway nodes are deployed at the critical aggregation layer of the network, and all alarm messages reported by edge acquisition nodes must pass through them. Cloud diagnostic control nodes are deployed in a centralized management and control center, providing a global perspective. The specific configuration and working mechanism of these three types of nodes are explained in detail below.

[0013] The working process of the edge acquisition node is as follows.

[0014] First, raw logs and topology data are collected. Raw device logs are obtained from physical devices via Syslog, SNMP, or device-specific log interfaces. Log content includes device ID, fault time, fault type, and fault description. The purpose of collecting network topology connection data is that single-device logs alone cannot determine upstream and downstream dependencies of a fault; a comprehensive understanding of the entire network's physical and logical connections is essential to identify derived alarms. The process of collecting network topology connection data is specifically divided into two parts: physical connection data collection and logical service dependency data collection. Physical connection data is obtained by periodically sending link layer discovery protocol probe packets. The probe period is set to 30 seconds by default and can be configured within the range of 10 to 120 seconds. The information obtained includes the interface correspondence between this device and adjacent devices, link speed, and link status. Logical service dependency data is obtained by listening to the trace identifiers of microservice call chains, parsing the service call relationships in the call chain, and clarifying the dependency paths of downstream services on upstream services. The collected network topology connection data is reported to the cloud diagnostic control node every 60 seconds, and incremental reporting is triggered immediately when the topology changes.

[0015] Secondly, preliminary log deduplication is performed. The purpose of this preliminary deduplication is that the same device will repeatedly generate identical alarm logs during the duration of the fault. Uploading all of them directly would create meaningless redundant data, increasing the processing pressure on subsequent stages. Specifically, a time-sliding window mechanism is used. The default window duration is 30 seconds, and the sliding step is 3 seconds, meaning the window time range is updated every 3 seconds, retaining all log data from the most recent 30 seconds. The initial deduplication threshold is set according to the device type. For sensor devices with low alarm frequency, the initial threshold is set to 2 alarms per window. (Server and switching equipment...) Alarms are prone to repeated reporting, with an initial threshold of 5 times per window; the threshold can be configured from 1 to 20 times. Feature identifiers are extracted from the original device logs. These identifiers are generated by concatenating the device ID, fault type code, and fault level code, serving as the basis for determining log uniqueness. Logs with the same feature identifier within the sliding window are cumulatively counted. When the count exceeds the deduplication threshold, local suppression is triggered, and subsequent identical logs will no longer generate new alarm messages; only the frequency count of the corresponding feature identifier in memory is updated. This filters out a large number of duplicate alarms from a single device at the edge, significantly reducing the amount of uplink data.

[0016] Next is the generation of topology feature tags and message encapsulation. The purpose of generating topology feature tags is to encode the topology dependency information of the device into fixed-length data and place it in the message header, so that the hardware of the subsequent aggregation gateway node can directly extract and match it without parsing the log payload content, thus improving the processing speed. Specifically, the physical connection information and logical dependency information are encoded and mapped to generate binary format topology feature tags. The tags are used as network layer extended message headers and inserted before the original log payload to form tagged alarm messages. The encapsulated alarm messages are sent to the corresponding aggregation gateway node via the UDP protocol.

[0017] The working process of the aggregation gateway node is as follows.

[0018] As an intermediate processing layer for alarm data, the aggregation gateway node's core function is to intercept derived alarms at the forwarding level, preventing redundant data from being uploaded to the cloud. Upon receiving tagged alarm messages, it directly extracts topology feature tags from the message extension header without parsing the payload content. The hardware matching engine uses dedicated hardware circuitry to implement parallel comparison of the alarm interception flow table, improving comparison speed compared to software. The comparison is based on the alarm interception flow table, which stores the topology prefix codes of downstream nodes known to generate derived alarms. The comparison uses the longest prefix matching principle: if the topology tag prefix of the alarm message matches an entry in the flow table, it indicates that the device corresponding to the alarm is downstream of the faulty upstream node, belonging to a derived alarm, and is directly physically discarded at the data forwarding plane, without entering the gateway's operating system protocol stack or consuming CPU and memory resources. If no match is found, it indicates that the alarm may be a root cause fault, and it is marked as a root cause alarm message and uploaded to the cloud through a high-priority channel. Through the above settings of the aggregation gateway node, it filters out a large number of cross-device derived alarms, further reducing the processing pressure on the cloud.

[0019] The working process of the cloud-based diagnostic control node is as follows.

[0020] First, the system receives network topology connection data reported by edge nodes across the entire network, constructing a network-wide device topology graph. This graph includes both physical connections and logical dependencies. Upon receiving root cause alarm messages from the aggregation gateway node, the system traces upstream along the dependency path of the topology graph, starting from the device node corresponding to the alarm, to check the alarm status of all upstream nodes, locate the originating node of the fault, and generate a fault diagnosis result. Simultaneously, global historical alarm data is stored in the cloud. Correlation analysis is used to uncover new fault derivative paths, identifying which downstream nodes will inevitably generate alarms when an upstream node fails. An updated alarm interception flow table is generated based on these newly discovered derivative paths and distributed to the corresponding aggregation gateway node. Simultaneously, based on alarm frequency statistics for each edge node, an edge deduplication optimization strategy is generated and distributed to the corresponding edge acquisition nodes. Therefore, the longer the system runs, the higher the accuracy of alarm interception, eliminating the need for continuous manual rule maintenance.

[0021] Furthermore, considering that the software processing speed of general-purpose CPUs cannot meet the line-speed alarm processing requirements of gigabit or even 10-gigabit networks, high-speed processing of the forwarding plane must be achieved through dedicated hardware. Since rule management and policy configuration at the control plane are more flexible when implemented by the CPU, the aggregation gateway node specifically includes a data forwarding plane submodule and a control plane submodule. The data forwarding plane submodule integrates a field-programmable gate array (FPGA) and a tri-state content-addressable memory (TRM) to receive tagged alarm messages at network line speed. The FPGA extracts the topology feature tag from the message header and sends it to the TRM, performing a full parallel comparison with the alarm interception flow table within one clock cycle. For messages that do not match the target, the FPGA allows them to proceed to the control plane submodule. The central processing unit within the control plane submodule modifies the Differentiated Services Code Point (DSC) field in the message header to an accelerated forwarding identifier, and sends the tagged message to the cloud through a network device scheduling policy that supports the accelerated forwarding identifier. The specific implementation is as follows.

[0022] The field-programmable gate array (FPGA) of the data forwarding submodule uses a Xilinx Artix-7 series FPGA chip, and the tri-state content-addressable memory uses an independent TCAM chip with a capacity of 16K rows and a bit width of 40 bits. The system operating clock is 100MHz, and the single clock cycle is 10 nanoseconds. After the alarm message enters the physical network port, it is directly sent to the FPGA for processing. The FPGA parses the network layer extension header of the message, extracts the 64-bit topology feature tag, and uses the 32-bit hierarchical topology prefix code as the search keyword, which is then sent to the TCAM chip. The TCAM internally stores the alarm interception flow table. Each flow table entry contains a 32-bit topology prefix code, a prefix length bit, and a drop instruction bit. The TCAM completes the parallel comparison of all flow table entries within a single clock cycle and returns the matching result to the FPGA.

[0023] If the match result is a hit, the FPGA directly sends a drop command to the physical network interface. The current packet is dropped at the physical layer and will not enter the subsequent memory buffer and protocol stack. If the match result is a miss, the FPGA forwards the packet to the central processing unit of the control plane submodule. The control plane submodule uses an embedded CPU with an ARM architecture. After receiving the packet, the CPU modifies the Differentiated Services Code Point field in the IP header and sets it to the identifier corresponding to accelerated forwarding. Then, the packet is sent back to the data forwarding plane and sent to the cloud through the uplink port. The network devices of the uplink are configured with corresponding QoS (Quality of Service) scheduling policies. Packets with the accelerated forwarding identifier are placed in the highest priority queue, and a strict priority scheduling algorithm is used to ensure that root source alarm messages are forwarded first when congestion occurs.

[0024] By setting up the above hardware architecture, the aggregation gateway node can complete alarm comparison and interception at gigabit network line speed. The single packet processing latency is short and will not cause additional delay to normal business traffic. At the same time, derived alarms are directly discarded at the physical layer without occupying system CPU and memory resources, and the overall operating load of the gateway is kept at a low level.

[0025] Furthermore, considering that only by integrating both physical topology and logical dependency information can the fault propagation path of the device be fully reflected, and that the information is encoded into binary tags and placed in the extended message header to meet the needs of subsequent rapid hardware matching, the topology feature tag generation mechanism of the edge acquisition node specifically includes: obtaining physical connection information between devices by sending link layer discovery protocol probe packets, and obtaining logical service dependency information by listening to the trace identifier of the microservice call chain; encoding and mapping the physical connection information and the logical service dependency information as the network topology connection data to generate binary data composed of service domain identifier and hierarchical topology prefix code, which serves as the topology feature tag; inserting the topology feature tag as the network layer extended message header before the payload of the original device log to generate the tagged alarm message. The specific implementation method is as follows.

[0026] The generation of topological feature labels consists of three steps: information collection, encoding mapping, and message encapsulation.

[0027] Physical connection information collection adopts the standard LLDP protocol. The edge collection node simulates an LLDP proxy, sending LLDP probe frames to all directly connected devices every 30 seconds and receiving LLDP response frames returned by the devices. From these frames, information such as peer device ID, interface number, and interface type is extracted to form the physical adjacency relationship between devices. Logical service dependency information collection is achieved through the service call traffic of the mirrored device. The trace identifiers in the HTTP or RPC calls, namely TraceID and SpanID, are parsed to reconstruct the upstream and downstream relationships of service calls. For example, the order service depends on the inventory service, and the inventory service depends on the database service, thus forming a logical dependency chain.

[0028] During the encoding and mapping phase, each device is first assigned a service domain identifier, and then a unique hierarchical topology prefix code is assigned to each device based on the network topology. The service domain identifier and the hierarchical topology prefix code are then combined into binary data.

[0029] After encoding, the message encapsulation stage begins, where the topology feature label is encapsulated into an IPv4 option extension header or an IPv6 hop-by-hop option extension header, located after the IP header and before the transport layer header, i.e. before the original log payload. The advantage of this encapsulation method is that standard network devices can forward the message normally, while the FPGA can directly extract the label from the IP extension header without parsing the upper-layer protocol content.

[0030] With the above settings, the topology feature tag fully carries the physical location and logical affiliation information of the device. The upstream and downstream dependencies of the device can be determined by the tag alone, which provides a basis for the identification of subsequent gateway-derived alarms. At the same time, the encapsulation method is compatible with standard network protocols and does not require modification of existing network equipment.

[0031] Considering that only interception rules generated based on real topology dependencies and historical fault patterns can accurately identify derived alarms and avoid falsely intercepting normal alarms, the flow table generation mechanism of the cloud diagnostic control node specifically includes: when receiving network topology connection data reported by edge acquisition nodes across the entire network; fusing and generating a network-wide device topology relationship graph with a directed acyclic graph structure; by assigning an encoding that reflects its upstream dependency path to each node in the directed acyclic graph, expanding the mesh-like topology relationship between nodes into a hierarchical topology prefix code in a tree structure; when it is determined that the fault of a downstream node is necessarily caused by the fault of an upstream node, combining the hierarchical topology prefix code corresponding to the downstream node with a discard instruction to generate the updated alarm interception flow table. The specific implementation method is as follows.

[0032] After receiving the topology data reported by all edge nodes, the cloud first performs data cleaning and fusion, removing duplicate connections, supplementing missing node information, and generating a directed acyclic graph (DAG) structure of the entire network device topology. Each node in the graph represents a physical device or a logical service, and directed edges point from upstream nodes to downstream nodes, indicating that the operation of downstream nodes depends on upstream nodes. The specific process of assigning hierarchical topology prefix codes to each node is as follows: the core node of the network is selected as the root node, and the prefix code of the root node is set to 0; a breadth-first traversal is performed starting from the root node, and each first-level child node is assigned one hierarchical code, for example, the first first-level child node is 001, the second is 002; the prefix code of each second-level child node is the prefix code of the parent node followed by its own hierarchical code, and so on, until the prefix code of each node completely contains the encoding information of all its upstream dependent nodes; for nodes with multiple upstream dependencies, the prefix code is generated by the main upstream dependency path, and the remaining dependencies are stored in an auxiliary association table.

[0033] The method for determining fault propagation relationships is as follows: statistically analyze alarm data from the past 30 days. When an upstream node A fails, the probability that a downstream node B simultaneously generates an alarm reaches a preset value (e.g., 95%) or higher, and is manually confirmed to be a derivative alarm caused by fault propagation, then it is determined that the fault of node B must have been caused by the fault of node A. At this time, the hierarchical topology prefix code of node B is added to the alarm interception flow table, and the corresponding action is to discard it. The generated flow table is split according to the jurisdiction of the aggregation gateway node and distributed to the corresponding aggregation gateway node. Each aggregation gateway node only stores the interception flow table of the nodes within its jurisdiction, saving TCAM storage space.

[0034] With the above settings, the hierarchical topology prefix code naturally supports longest prefix matching, adapts to the hardware lookup characteristics of TCAM, and ensures the accuracy of the interception rules based on the derivation relationship judgment of historical data, thereby reducing the false interception rate.

[0035] Furthermore, considering that TCAM cannot perform a normal lookup operation during the writing of flow table entries, directly updating the working storage area would cause a comparison function interruption of tens of milliseconds, during which generated alarms would be missed, affecting the interception effect. Therefore, the aggregation gateway node adopts a ping-pong caching mechanism for lossless updating of the flow table. When the control plane submodule receives the updated alarm interception flow table issued by the cloud diagnostic control node, the physical storage space of the tri-state content-addressable memory is divided into a first storage area and a second storage area. When the updated alarm interception flow table is received, the first storage area is in an active state, and the second storage area is in a silent state. While maintaining the continuous execution of alarm comparison in the first storage area, the system writes the updated alarm interception flow table to the silent second storage area. After the write verification is successful, the control plane submodule executes a control pointer state toggling instruction to switch the second storage area to an active state to execute the interception task within one clock cycle, and simultaneously switches the first storage area to a silent state. The specific implementation method is as follows.

[0036] Under normal operating conditions, one storage area is in an active state, and the FPGA's comparison and retrieval points to this storage area; the other storage area is in a silent state and does not participate in the comparison work. When the control plane submodule receives a new flow table from the cloud, it first verifies the integrity of the flow table data. After the verification is successful, the new flow table is written to the silent storage area one by one through the TCAM's write interface. During the writing process, the active storage area continues to work normally, and the comparison of all alarm messages is unaffected. After the writing is completed, the control plane CPU reads all flow table entries in the silent storage area and compares them one by one with the issued flow table to verify the accuracy of the written data and ensure that there are no writing errors. After the verification is successful, the CPU sends a control pointer toggle instruction to the FPGA. At the end of the current clock cycle, the FPGA switches the comparison and retrieval pointer from the original active storage area to the original silent storage area. The original silent storage area becomes the new active state, and the original active storage area becomes the new silent state, waiting for the next flow table update. The entire switching process is completed within one clock cycle, with a preset duration (e.g., 10 nanoseconds), and will not cause any message loss or missed judgment.

[0037] With the above settings, the flow table update process does not affect the normal operation of the alarm interception service, ensuring high system availability and avoiding security risks caused by the update window.

[0038] Furthermore, considering that fixed-threshold deduplication methods cannot adapt to the alarm characteristics of different devices or the changes in fault scenarios, dynamically adjusting the threshold can maximize the deduplication effect while ensuring the effectiveness of alarms. Therefore, the frequency comparison preliminary deduplication mechanism of the edge acquisition node specifically includes: setting a fixed-duration time sliding window and monitoring the original device logs entering the window in real time; receiving the edge deduplication optimization strategy issued by the cloud diagnostic control node to update the locally preset deduplication threshold; extracting the feature identifiers of the original device logs, accumulating the count of logs with the same feature identifiers, and comparing it with the updated deduplication threshold; when the count value is greater than the updated deduplication threshold, triggering a local suppression strategy to prevent the generation of tagged alarm messages from subsequent identical original device logs, and only updating the frequency status bit value maintained in memory. The specific implementation is as follows.

[0039] The time-based sliding window is implemented using a forward sliding mechanism. The default window duration is 30 seconds, with a sliding step of 3 seconds. The system performs a window sliding operation every 3 seconds, removing log records that are outside the time range and adding newly generated log records. A hash table is maintained in memory, with the key being the log's feature identifier and the value being the cumulative count of that feature identifier within the current window. Upon receiving a raw log, the feature identifier is extracted, the corresponding count value is retrieved from the hash table, the count value is incremented by 1, and then compared with the current deduplication threshold. If the count value is less than or equal to the threshold, a tagged alarm message is generated and sent. If the count value is greater than the threshold, no new alarm message is generated; only the count value in the hash table is updated.

[0040] The initial value of the deduplication threshold is set by the local configuration file. After the device goes online, the cloud will generate an optimized threshold based on the device's historical alarm data and send it to the edge acquisition nodes through the management channel. Upon receiving the data, the edge nodes will directly update their local threshold parameters. The cloud calculates the optimized threshold by counting the single-window frequency of all alarms with the same characteristics of the device in the past 7 days, taking the 90th percentile as the optimized threshold, and setting boundary limits of a minimum threshold of 1 and a maximum threshold of 20 to avoid false suppression or deduplication failure caused by extreme values. For example, if 90% of the duplicate alarms of a device in history occur no more than 4 times in a single window, then the deduplication threshold of the device will be optimized to 4 times.

[0041] With the above settings, the edge deduplication accuracy is high, it will not miss key alarms and can effectively filter duplicate and redundant logs. Compared with the fixed threshold scheme, the deduplication efficiency is improved.

[0042] Furthermore, considering that the information from the alarm itself is insufficient to support accurate root cause analysis, and that manually logging into devices to obtain logs is inefficient and time-consuming, automatic tracing can quickly obtain fault context and improve diagnostic efficiency. Therefore, the cloud-based diagnostic control node is also configured with a context log tracing mechanism: upon receiving the root cause alarm message filtered by the aggregation gateway node; extracting the fault type feature field from the root cause alarm message, querying the preset fault troubleshooting instruction set, automatically matching and generating a diagnostic probe request for extracting surrounding operating environment data; and sending the diagnostic probe request to the target device that generated the root cause alarm through the reverse control channel to obtain the operating status context log of the target device before and after the root cause fault occurred. The specific implementation method is as follows.

[0043] After receiving a root cause alarm message, the cloud first extracts the fault type feature code from the topology feature tag of the message. Based on the fault type feature code, it matches the corresponding detection command in a preset fault troubleshooting command set. The fault troubleshooting command set is pre-configured. For example, the detection command corresponding to a CPU utilization exceeding limit fault is: obtain the CPU utilization curve, process CPU usage ranking, and system load value for 5 minutes before and after the fault. The detection command corresponding to a link interruption fault is: obtain the interface traffic statistics, packet error count, neighbor status, and optical module power for 3 minutes before and after the fault. After matching is completed, a diagnostic detection request message is generated, which contains the target device ID, the detection command set, and time range parameters.

[0044] Diagnostic probe requests are sent through the reverse control channel, which uses an independent TCP long connection and is physically isolated from the alarm uplink data channel to avoid uplink traffic congestion affecting the issuance of control commands. The probe request is first sent to the edge acquisition node corresponding to the target device, and then forwarded to the target device by the edge acquisition node. After the target device executes the probe command, it uploads the collected operating status context log to the cloud through the edge node. The cloud stores the context log in association with the root cause alarm for use in fault diagnosis and analysis.

[0045] With the above settings, complete device operating context data can be automatically obtained within a short time after a fault occurs, shortening the average time for fault diagnosis and reducing the amount of manual operation required by maintenance personnel.

[0046] Furthermore, considering that fixed-length data is easier for hardware circuits to extract and parse directly without needing to process the offset calculation of variable-length fields, which can further improve hardware processing speed and reduce the complexity of FPGA logic design, the topology feature tag data structure generation configuration in the tagged alarm message is as follows: obtain the various tag parameters to be encapsulated into the original device log; limit the topology feature tag to a fixed-length 64-bit continuous binary data structure, and allocate and map it from front to back as follows: a service domain identifier field occupying 8 data bits, a hierarchical topology prefix code field occupying 32 data bits, a fault type feature code field occupying 16 data bits, and a frequency status bit record field occupying 8 data bits; combine the tag parameters according to the above structure to generate the topology feature tag, and the specific implementation is as follows.

[0047] The 64-bit label is divided into four fields from the most significant bit to the least significant bit.

[0048] The first to eighth bits are the business domain identifier field, a total of 8 bits, which can represent 256 different business domains. This is used to distinguish devices in different business partitions. For example, 0x01 corresponds to the production business domain, 0x02 corresponds to the office business domain, and 0x03 corresponds to the management business domain. Alarms from different business domains can be handled separately. The ninth to fortyth bits are the hierarchical topology prefix code field, a total of 32 bits, which is used to encode the hierarchical topology of the storage device. Every 6 bits correspond to one level of topology, and a maximum of 5 hierarchical structures are supported. The remaining 2 bits are reserved. This field is the core keyword for TCAM matching and is used to determine the upstream and downstream dependencies of devices.

[0049] Bits 41 to 56 are the fault type feature code field, which consists of 16 bits and can represent 65,536 different fault types. For example, 0x0001 represents a power failure, 0x0002 represents a link failure, and 0x0003 represents excessive CPU utilization. This field is used to quickly identify the fault category during cloud-based fault diagnosis and can also be used for fine-grained matching of flow tables.

[0050] Bits 57 to 64 are frequency status record fields, totaling 8 bits, with values ​​ranging from 0 to 255. These fields record the cumulative frequency of this type of alarm within the edge sliding window. If the frequency exceeds 255, the value saturates to 255. This field is used by the gateway and the cloud to reference the frequency of alarms and assist in determining the severity of the fault. All fields are concatenated in sequence into a continuous 64-bit binary data, with the byte order adopting network byte order, i.e., big-endian mode, to ensure consistent parsing across devices with different architectures.

[0051] With the above settings, a single tag can carry four core types of information: alarm classification, topology location, fault type, and frequency status. The fixed-length structure is very suitable for hardware to extract and process quickly, allowing all fields to be parsed within a single clock cycle.

[0052] Furthermore, considering that root cause alarms are core data for fault diagnosis, it is necessary to ensure that they can reach the cloud first even when the network is congested, so as to avoid being blocked by a large amount of ordinary traffic and thus increasing the fault feedback delay. Therefore, the establishment mechanism of the high-priority channel is as follows: when the control plane submodule is about to send a root cause alarm message to the cloud diagnostic control node; the central processing unit in the control plane submodule establishes a priority allocation strategy for the transmission channel through the software-defined network southbound control protocol; the accelerated forwarding identifier is specifically implemented by the central processing unit forcibly rewriting the decimal value of the differentiated service code point of the IP datagram in the packet header to 46, and the specific implementation is as follows.

[0053] The establishment of high-priority channels is divided into a configuration phase and an operation phase. In the configuration phase, the control plane CPU of the aggregation gateway node uses the OpenFlow 1.3 protocol as the southbound control protocol to issue QoS flow table rules to all switching devices along the way. The rule content is: to map packets with a DSCP value of 46 in the IP header to the highest priority queue of the port. The queue scheduling method adopts strict priority scheduling, that is, as long as there are packets in the highest priority queue, the packets in that queue are sent first, until the queue is empty and then the lower priority queues are processed. This configuration only needs to be executed once during network initialization and will be automatically updated when the network topology changes.

[0054] During operation, when the aggregation gateway node determines that a packet is a root source alarm packet, the control plane CPU directly modifies the DSCP field of the IP packet header, setting its decimal value to 46, which corresponds to the standard accelerated forwarding PHB. After the modification is completed, the packet enters the uplink network, and all switches along the way will put it into the highest priority queue for forwarding. Even if network congestion occurs, the transmission latency of the root source alarm can be maintained at a low level.

[0055] The above settings reduce the end-to-end transmission latency of root cause alarms, ensuring timely delivery of root cause alarms even in scenarios where network bandwidth utilization is congested, thus providing a guarantee for rapid fault handling.

[0056] Furthermore, considering that only by discarding packets before they enter system memory can we truly avoid generating alarms that consume system resources, if packets are discarded by software after entering memory, they will still consume bus bandwidth and CPU resources, failing to achieve the optimal offloading effect. Therefore, the physical discarding operation of the data forwarding surface submodule specifically follows the following timing sequence: the tri-state content addressing memory receives the incoming topology feature tag within a single clock cycle; performs a full table search based on the longest prefix matching principle to determine whether it matches the pre-stored alarm interception flow table; when the search result is a match, the physical network interface is directly driven to discard the data frame of the packet within the same current clock cycle, preventing the data of the matched packet from flowing into or occupying the operating system memory buffer pool. The specific implementation method is as follows.

[0057] The specific timing sequence is as follows: At the rising edge of the clock, the topology feature tag data of the message is latched into the input register of the TCAM, and the TCAM simultaneously starts parallel comparison of all memory units; during the high level of the clock, the TCAM completes a full table search, using the longest prefix matching principle, that is, when multiple flow table entries are matched at the same time, the one with the longest prefix is ​​selected as the matching result to ensure matching accuracy; before the falling edge of the clock arrives, the TCAM outputs the matching result signal, including the hit flag and the index of the matching item; when the FPGA receives the hit flag as high, it immediately sends a discard enable signal to the physical network interface chip, and the physical network interface chip directly terminates the reception and forwarding of the current data frame, discards the frame data from the physical layer, and does not write any data to the subsequent FIFO buffer and memory.

[0058] The entire process, from tag input to drop command output, is completed within one clock cycle (e.g., within 10 nanoseconds). The message will not enter the gateway's operating system memory, nor will it trigger any interrupts or protocol stack processing. For a missed message, the FPGA outputs a forwarding command within the same clock cycle, sending the message into the subsequent processing path.

[0059] With the above settings, the interception of derived alarms does not consume any system CPU or memory resources. The alarm processing capability of the gateway depends only on the line speed capability of the physical network port, and can easily support alarm traffic processing at the 10 Gigabit level.

[0060] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A rapid feedback and diagnostic log processing system for equipment faults, characterized in that, include: Edge acquisition nodes, aggregation gateway nodes, and cloud-based diagnostic and control nodes; The edge acquisition node is used for: Obtain raw device logs from physical devices and collect network topology connection data in real time to reflect device status; The feature identifiers of the original device logs are extracted, and frequency comparison is performed within a locally configured time sliding window to perform preliminary deduplication. Simultaneously, the network topology connection data is converted into topology feature tags characterizing the physical and logical locations of devices. Specifically, this includes: obtaining physical connection information between devices by sending link layer discovery protocol probe packets, and obtaining logical service dependency information by listening to the trace identifiers of microservice call chains; the physical connection information and the logical service dependency information are used as the network topology connection data, encoded and mapped to generate binary data composed of a service domain identifier and a hierarchical topology prefix code, which serves as the topology feature tag. Encapsulating the topology feature tags into the deduplicated log data to generate tagged alarm messages and outputting them to the aggregation gateway node specifically includes: inserting the topology feature tags as network layer extended message headers before the payload of the original device logs to generate the tagged alarm messages; and simultaneously reporting the network topology connection data to the cloud diagnostic control node. The aggregation gateway node is used for: Receive tagged alarm messages from the edge acquisition nodes; The topology feature tags in the alarm message are parsed and extracted. The topology feature tags are used as retrieval keys and compared in parallel with the pre-stored alarm interception flow table in the internal hardware matching engine to determine whether the alarm message belongs to a derivative alarm caused by an upstream node failure. If the comparison matches, it indicates a derived alarm, and a physical drop operation is directly triggered on the data forwarding plane of the aggregation gateway node; if it does not match, it indicates a potential root cause fault, and it is marked as a root cause alarm message and sent to the cloud diagnostic control node through a high-priority channel. The cloud-based diagnostic control node is used for: Receive network topology connection data reported by the edge acquisition node and root cause alarm messages reported by the aggregation gateway node; A network-wide device topology graph is constructed based on the network topology connection data. Based on this graph, the propagation path of the root cause alarm message is traced and analyzed to determine the fault diagnosis result. Newly discovered fault-derived paths are mined using global historical alarm data, and an updated alarm interception flow table and edge deduplication optimization strategy are generated accordingly. Specifically, when network topology connection data reported by edge acquisition nodes is received, a network-wide device topology graph with a directed acyclic graph structure is generated. Each node in the directed acyclic graph is assigned an encoding that reflects its upstream dependency path, and the mesh-like topology relationship between nodes is expanded into a tree-like hierarchical topology prefix code. When it is determined that the fault of a downstream node is inevitably caused by the fault of an upstream node, the hierarchical topology prefix code corresponding to the downstream node is combined with a discard instruction to generate the updated alarm interception flow table. The updated alarm interception flow table is sent to the corresponding aggregation gateway node, and the edge deduplication optimization strategy is sent to the corresponding edge acquisition node.

2. The equipment fault rapid feedback and diagnosis log processing system according to claim 1, characterized in that, The aggregation gateway node specifically includes a data forwarding plane submodule and a control plane submodule: The data forwarding face module integrates a field-programmable gate array and a tri-state content-addressable memory, which is used to receive the tagged alarm message in the network line speed state; The field-programmable gate array extracts the topology feature tag from the packet header and sends it to the tri-state content addressing memory, completing a full table parallel comparison with the alarm interception flow table within one clock cycle; For packets that are not hit, the field-programmable gate array allows them to pass to the control plane submodule; the central processing unit in the control plane submodule modifies the Differentiated Services Code Point field in the packet header to an accelerated forwarding identifier, and sends the marked packet to the cloud through the network device scheduling policy that supports the accelerated forwarding identifier.

3. The equipment fault rapid feedback and diagnosis log processing system according to claim 2, characterized in that, The aggregation gateway node uses a ping-pong caching mechanism for lossless updates of the flow table: When the control plane submodule receives the updated alarm interception flow table issued by the cloud diagnostic control node; The tri-state content addressing memory is divided into a first storage area in the active state and a second storage area in the silent state; while the system maintains the continuous execution of alarm comparison in the first storage area, it writes the updated alarm interception flow table into the second storage area in the silent state. After the write verification is successful, the control plane submodule executes the control pointer state toggling instruction to switch the second storage area to the active state to execute the interception task within one clock cycle, and simultaneously switches the first storage area to the silent state.

4. The equipment fault rapid feedback and diagnosis log processing system according to claim 1, characterized in that, The preliminary deduplication mechanism for frequency comparison of edge acquisition nodes specifically includes: Set a fixed-duration sliding window to monitor the original device logs entering the window in real time; and receive the edge deduplication optimization strategy issued by the cloud diagnostic control node to update the local preset deduplication threshold; Extract the feature identifiers from the original device logs, accumulate the counts of logs with the same feature identifiers, and compare them with the updated deduplication threshold; When the count value is greater than the updated deduplication threshold, a local suppression strategy is triggered to prevent subsequent identical original device logs from generating tagged alarm messages, and only the frequency status bit value maintained in memory is updated.

5. The equipment fault rapid feedback and diagnosis log processing system according to claim 1, characterized in that, The cloud-based diagnostic control node is also configured with a context log tracing mechanism: Upon receiving the root cause alarm message filtered by the aggregation gateway node; Extract the fault type feature field from the root cause alarm message, query the preset fault troubleshooting instruction set, and automatically match and generate a diagnostic detection request for extracting data from the surrounding operating environment. The diagnostic detection request is sent to the target device that generated the root cause alarm through a dedicated reverse control channel to retrieve the target device's operating status context log before and after the root cause failure occurred.

6. The equipment fault rapid feedback and diagnosis log processing system according to claim 1, characterized in that, The data structure configuration for generating the topological feature labels in the tagged alarm message is as follows: Obtain the tag parameters to be encapsulated into the original device log; The topology feature label is defined as a fixed-length 64-bit continuous binary data structure, which is allocated and mapped sequentially from front to back as: a service domain identifier field occupying 8 data bits, a hierarchical topology prefix code field occupying 32 data bits, a fault type feature code field occupying 16 data bits, and a frequency status bit record field occupying 8 data bits. The topological feature labels are generated by combining the label parameters according to the above structure.

7. The equipment fault rapid feedback and diagnosis log processing system according to claim 2, characterized in that, The mechanism for establishing the high-priority channel is as follows: When the control plane submodule is preparing to send a root cause alarm message to the cloud diagnostic control node; The central processing unit within the control plane submodule establishes a priority allocation strategy for transmission channels through the software-defined network southbound control protocol. The accelerated forwarding identifier is specifically defined as the central processing unit forcibly rewriting the decimal value of the Differentiated Services Code Point (DSC) of the IP datagram in the packet header to 46.

8. The equipment fault rapid feedback and diagnosis log processing system according to claim 2, characterized in that, The physical discard operation of the data forwarding face module specifically follows the following timing sequence: The tri-state content-addressable memory receives the incoming topological feature tag within a single clock cycle; Perform a full table search based on the longest prefix matching principle to determine whether it matches the pre-stored alarm interception flow table; When the search result is a match, the physical network interface is directly driven to discard the data frame of the packet within the same clock cycle, preventing the data of the matched packet from flowing into or occupying the operating system memory buffer pool.

Images