Network equipment operation monitoring log integration system based on multi-source data acquisition and analysis

Abstract

The invention relates to the technical field of digital information monitoring, in particular to a network equipment operation monitoring log integration system based on multi-source data acquisition and analysis. Comprising a multi-source data acquisition unit; a pretreatment unit; the intelligent analysis unit is used for constructing a normal operation behavior baseline of the network equipment, identifying abnormal operation of the equipment and mining a log association relationship among multiple pieces of equipment; a log integration unit; and a visualization unit. According to the invention, an intelligent analysis unit adopts an attention mechanism enhanced time sequence modeling technology to construct an equipment normal operation behavior baseline, sub-fragments are divided according to equipment operation periodicity, and differentiated weights are given according to key index fluctuation characteristics, so that the baseline is more focused on a time period with significant influence on the equipment state; and meanwhile, an improved association mining algorithm is combined to introduce an equipment topology association degree pruning low association item set, so that the rationality of a behavior baseline and the comprehensiveness of multi-equipment log association analysis are improved.

Description

Technical Field

[0001] This invention relates to the field of digital information monitoring technology, and more specifically, to a network device operation monitoring log integration system based on multi-source data acquisition and analysis. Background Technology

[0002] With the large-scale deployment of node devices such as routers, switches, and firewalls in enterprise networks, device operation logs, as core data for monitoring status and troubleshooting, face challenges such as inconsistent formats and difficulties in adapting to multiple protocols (including public standards and private customizations). At the same time, raw logs contain null values ​​and duplicate records due to transmission errors, and the timestamps of logs from multiple devices are not synchronized, and the cross-device correlations are hidden, which directly affects the accuracy of analysis and the efficiency of operation and maintenance. Therefore, a full-process log management solution is urgently needed.

[0003] For example, Chinese patent CN202411970343.0 discloses a device status monitoring method and system based on multimodal data fusion. It collects numerical and textual data from the device, achieves time-series alignment through an index table, extracts features using Fourier transform and LSTM, and fuses them through multi-head cross-attention. It then uses CatBoost to construct an evaluation model and introduces KL divergence for dynamic updates, solving the problems of difficult alignment of multi-source data, difficult feature fusion, and difficulty in adapting the model to data changes, thus improving the accuracy of status assessment. Another example is Chinese patent CN202510179871.2, which discloses a big data-driven network monitoring and management method and system. It establishes a distributed database to store multimodal data, constructs an anomaly correlation graph through time synchronization and semantic association, identifies anomalies through a multi-level collaborative network, and generates early warnings through prediction, solving the problems of difficult integration of multimodal data, low anomaly detection accuracy, and delayed early warnings, achieving accurate detection and proactive early warning.

[0004] Despite the design advantages of the aforementioned technical solutions, they also suffer from the following technical shortcomings: First, the targeting of network device log collection is insufficient: Chinese patent CN202411970343.0 focuses on physical parameters such as device temperature and vibration acceleration as the core collection objects, failing to consider the unique protocol diversity of network device logs (especially vendor-specific log protocols), and thus cannot adapt to the log acquisition needs of different brands and models of network devices; while CN202510179871.2 involves network device data storage, it does not solve the protocol adaptation problem when adding new network devices. For non-standard protocol devices, hardware adjustments or redevelopment of parsing programs are required, resulting in poor flexibility and scalability. Second, the depth of multi-device log analysis is insufficient: neither of these solutions analyzes the hierarchical relationships and physical connections of network devices, making it difficult to discover implicit correlations between cross-device logs. Furthermore, they do not specifically address the periodic fluctuations of long-series logs, making it impossible to distinguish different time periods in the logs that affect device status, thus limiting the comprehensiveness and accuracy of device operating status analysis. Thirdly, there is a lack of log management and operation and maintenance collaboration capabilities: Chinese patent CN202411970343.0 does not address the design for long-term storage and efficient retrieval of log data, and cannot support large-scale log backtracking queries; although CN202510179871.2 adopts distributed storage, it does not optimize the storage structure for the log's time attributes and frequently queried fields, making the retrieval efficiency insufficient to meet operation and maintenance needs. Furthermore, neither of these patents establishes a standardized data interaction channel with external operation and maintenance systems, making it impossible to achieve cross-system sharing and collaborative processing of monitoring data. Therefore, we propose a network device operation monitoring log integration system based on multi-source data acquisition and analysis. Summary of the Invention

[0005] The purpose of this invention is to provide a network device operation monitoring log integration system based on multi-source data collection and analysis, so as to solve the problems mentioned in the background art, such as insufficient targeting of network device log collection, insufficient depth of multi-device log analysis, and lack of log management and operation and maintenance collaboration capabilities.

[0006] To address the aforementioned technical problems, the present invention aims to provide a network device operation monitoring log integration system based on multi-source data acquisition and analysis, comprising: A multi-source data acquisition unit is used to collect the operation logs and real-time status data of network node devices. It realizes physical layer connection of devices and extraction of original log fields through hardware interfaces and standardized communication protocols, so as to achieve the adaptation and standardized acquisition of log data of different types of network node devices. The preprocessing unit is used to perform noise filtering, preliminary anomaly screening, and structure transformation on the raw log data. It uses an adaptive algorithm based on data traffic statistical characteristics and a string matching algorithm to process the log data, thereby removing invalid data interference and improving the structure of the log data to support subsequent analysis. The intelligent analysis unit is used to construct a baseline of normal network device operation behavior, identify device operation anomalies, and mine the log correlation between multiple devices. It generates a behavior baseline through time-series modeling technology enhanced by attention mechanism, identifies anomalies using dynamic weighted distance calculation method, and analyzes the log correlation of multiple devices by combining an improved correlation mining algorithm, thereby improving the accuracy of long-series log baseline, the accuracy of nonlinear anomaly identification, and the efficiency of large-scale log correlation mining. The log integration unit is used to realize the time synchronization of logs from multiple devices, the matching of related events, and the storage and querying of integrated data. It calibrates log timestamps based on time synchronization protocols, associates logs from different devices through similarity matching algorithms, and uses distributed storage technology to store integrated data, thereby achieving unified fusion management and efficient retrieval of multi-source logs. The visualization unit is used to display the equipment's operating status, abnormal events, and analysis results, and to realize data interaction with third-party operation and maintenance systems. It generates a visualization interface through web front-end technology and provides a data interaction interface based on standardized network protocols, so as to realize the intuitive presentation of monitoring results and support cross-system collaborative operation and maintenance.

[0007] As a further improvement to this technical solution, the multi-source data acquisition unit includes a hardware interface adaptation module, a protocol parsing module, and a field extraction module, wherein: The hardware interface adapter module uses physical interfaces such as RJ45 electrical port and multiple types of high-speed optical ports to achieve physical layer connection with different network node devices, and completes electrical signal transmission and optical signal to electrical signal conversion. The protocol parsing module is based on a variety of standardized network communication protocols, parses the protocol frame structure of log data, and identifies the frame boundaries and protocol types of log data. The field extraction module extracts original log fields (such as timestamps, device identifiers, event types, etc.) from the parsed protocol frame data based on the offset and length information of the original log fields in each protocol.

[0008] As a further improvement to this technical solution, the multi-source data acquisition unit also includes a dynamic adaptation module, which comprises a protocol rule storage submodule and a rule loading submodule, wherein: The protocol rule storage submodule is used to store the private protocol parsing rules of the new network node device; The rule loading submodule is used to load private protocol parsing rules into the protocol parsing module and the field extraction module, so that the multi-source data acquisition unit can adapt to the log data of new network node devices and extract the original fields without modifying the hardware.

[0009] As a further improvement to this technical solution, the preprocessing unit includes a noise filtering module, an anomaly preliminary screening module, and a structured conversion module, wherein: The noise filtering module, based on the statistical characteristics of data traffic (including field integrity and data repetition frequency), dynamically adjusts the filtering threshold through an adaptive algorithm to identify and remove null fields and consecutive duplicate records caused by transmission errors in the original log. The preliminary anomaly screening module constructs the normal fluctuation range of each field based on the statistical distribution of field values ​​in historical normal logs, and identifies abrupt field records that exceed the normal fluctuation range, thus completing the preliminary marking of obviously abnormal logs. The structured conversion module uses a string matching algorithm to extract key information from unstructured log text and map it to preset fields based on preset log field templates (covering fixed categories such as device status parameters, event descriptions, and operation results) to generate structured data.

[0010] As a further improvement to this technical solution, the noise filtering module dynamically adjusts the filtering threshold through an adaptive algorithm, including the following steps: Define the following parameters: initial threshold (Calibrated based on the device's factory default configuration or statistical results from the most recent 30-day historical stable period (a continuous period in which logs show no abnormal fluctuations); Preset lag ratio The range of values ​​is Used for lag correction during threshold callback (to avoid oscillation); preset interval coefficient , , Define the fluctuation range boundary of the threshold callback; A sliding time window is constructed, with the window duration dynamically configured within a preset variable range based on log traffic density, to calculate the duplicate record density of logs within the window in real time. (The ratio of the frequency of identical content log occurrences to the window duration) and the penetration rate of empty fields. (The percentage of logs containing null values ​​out of the total number of logs in the window); Based on the statistical distribution of normal logs from the same historical period, a baseline threshold for duplicate record density is generated. And the baseline threshold for penetration rate of empty fields ; When real-time repeated recording density Exceed Or, the penetration rate of empty fields Exceed When calculating the deviation. (The deviation ratio between the real-time value and the corresponding benchmark threshold), and according to Dynamically lower the filter trigger threshold, and the threshold reduction amount is the same as... Positive correlation, and a single adjustment does not exceed the preset upper limit of the original threshold; When a preset number of consecutive windows meet the requirements Below and Below When this happens, the filtering trigger threshold will be reverted to the initial threshold. fluctuation range And retains the lag correction amount during the callback process. To avoid threshold oscillations.

[0011] As a further improvement to this technical solution, the intelligent analysis unit includes a baseline construction module, an anomaly identification module, and a correlation mining module, wherein: The baseline construction module generates a baseline for normal device operation behavior through attention-enhanced temporal modeling technology, including a temporal segmentation submodule and an attention weight calculation submodule, wherein: The time-series segmentation submodule divides the long time-series log into continuous sub-segments according to the periodic characteristics of the device operation (such as daily / weekly / monthly cycles). Each sub-segment contains the device status parameters and event records within that time period. The attention weight calculation submodule assigns differentiated weights to different sub-segments based on the fluctuation characteristics of key indicators (such as resource utilization, port status, and protocol interaction frequency) in the sub-segments, making the baseline more focused on the time periods that have a significant impact on the device status. The anomaly identification module uses a dynamic weighted distance calculation method to identify equipment malfunctions, and includes a field weight configuration submodule and a dynamic distance calculation submodule, wherein: The field weight configuration submodule assigns initial weights to different fields based on the historical association between log fields and device faults (such as the frequency of association between error code fields and hardware faults), and dynamically updates the weight coefficients based on real-time fault feedback. The dynamic distance calculation submodule generates an anomaly score by calculating the weighted deviation between real-time log features and behavioral baseline features. When the score exceeds a preset threshold, an anomaly alarm is triggered. The association mining module analyzes the association relationships of multi-device logs using an improved association mining algorithm, including a multi-device log time-series alignment submodule and an association pattern optimization submodule, wherein: The multi-device log timing alignment submodule calibrates the log timestamps of different devices based on the network time protocol, eliminates time synchronization deviations between devices, and achieves timing consistency of logs across devices. The association pattern optimization submodule improves association mining efficiency by introducing device topology association (such as physical connection relationship and data forwarding path) to prune log item sets between devices with low association.

[0012] As a further improvement to this technical solution, the attention weight calculation submodule assigns differentiated weights to different sub-segments, including the following steps: Extract key state features for each time segment, including the fluctuation range of device resource utilization within the segment, the frequency of occurrence of abnormal event precursor fields, and stability indicators of protocol interaction. Calculate the first based on historical log data State correlation of each segment (Characterizing the correlation between this segment and the normal operating status of the equipment), and The value increases as the stability of the key state characteristics improves; according to Assign initial weights to each segment , and They are positively correlated, and the sum of the weights of all segments satisfies the normalization condition; via sliding window The weights are updated in real time, and the weights within the window are dynamically adjusted according to the changing trends of the current device operating status, improving adaptability to segments containing sudden changes in status. ; The weight percentage exceeds the preset threshold. The segments are marked as key segments, and the generation of behavioral baselines is mainly based on the state characteristics of the key segments, and The value is dynamically adapted based on the device type and operating scenario.

[0013] As a further improvement to this technical solution, the association pattern optimization submodule prunes the log item sets between devices with low association, including the following steps: Construct a network device topology diagram, and determine the devices based on their physical connection links (such as fiber optic links and network cable connections) and data interaction frequency (the number of cross-device data packet forwardings per unit time). With equipment topological dependency coefficient ; Set the initial association threshold according to the network architecture layers (core layer, aggregation layer, access layer). , The value is dynamically adjusted according to the importance of the device's layer (the threshold between core layer devices is lower than the threshold between access layer devices). For device pairs in a multi-device log association set Calculate its actual topological dependency coefficient. ,when When this happens, the associated itemset is marked as a low-association itemset; Prune the marked low-relevance itemsets, and only retain log records in the itemsets that contain critical events (such as device offline or link interruption). Recalculate at preset intervals (based on the frequency of network topology changes). And update This ensures that the pruning rules are adapted to the actual operating state of the network.

[0014] As a further improvement to this technical solution, the log integration unit includes a time synchronization module, an event association module, and a storage and retrieval module, wherein: The time synchronization module calibrates the timestamps of multi-device logs based on time synchronization protocols (including Network Time Protocol NTPv4 or Precision Time Protocol PTP), and corrects the log time offset through periodic time deviation detection (the detection period is dynamically adjusted according to the number of devices), ensuring that the timestamp deviation of logs from different devices is controlled within a preset threshold. The event association module associates logs from different devices using a similarity matching algorithm. Specifically, it includes: extracting key event fields from the logs (including event ID, device identifier, and operation behavior description), calculating the text similarity of key fields from different device logs, and determining that the similarity exceeds a dynamically adjusted association threshold (the threshold is optimized based on the matching accuracy of historical associated events). The storage and retrieval module employs distributed storage technology to achieve hierarchical management and efficient retrieval of integrated data. It includes a hierarchical storage submodule, an index building submodule, and a retrieval scheduling submodule, wherein: The hierarchical storage submodule stores the original integrated log in shards based on the log time series characteristics, with hourly or daily time granularity. It dynamically configures the replica redundancy strategy according to the importance of the device, and the shard storage rules are linked with the log index construction logic. The index building submodule builds inverted indexes for key fields (such as timestamp, event type, device number, etc.), and the log write operation triggers index updates in real time (the index update delay for a single log is ≤ a preset threshold), and adds secondary hash indexes for high-frequency query fields (such as event type); When the retrieval scheduling submodule receives a query request, it quickly locates the target data shard through the index, schedules at least 3 storage nodes to perform data reading operations in parallel, and combines a hot query result caching mechanism (the cache validity period is dynamically adjusted according to the query frequency) to achieve millisecond-level retrieval response for the integrated log, and the increase in retrieval latency with the growth of log size is ≤ a preset ratio (the threshold and ratio are dynamically calibrated based on the performance of the distributed storage cluster).

[0015] As a further improvement to this technical solution, the visualization unit includes a status display module, an anomaly presentation module, and a data interaction module, wherein: The status display module generates a visual interface through Web front-end technology, which displays the core operating status of the device in real time (including CPU utilization, port traffic, and protocol interaction frequency). It supports filtering by device type, and the interface data update frequency is synchronized with the analysis cycle of the intelligent analysis unit. The anomaly presentation module associates the anomaly judgment results of the intelligent analysis unit, matches the anomaly events with differentiated visual labels according to their severity level, and simultaneously displays the anomaly-related log fragments and preliminary investigation suggestions. The data interaction module provides a data interaction interface based on HTTP and HTTPS protocols, supporting the push of real-time monitoring data to third-party operation and maintenance systems, receiving and responding to query requests, and the interface call implements permission verification through a token authentication mechanism.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention sets up a dynamic adaptation module through a multi-source data acquisition unit, which can load the private protocol parsing rules of new network node devices into the protocol parsing and field extraction module without modifying the hardware. This solves the problems of insufficient adaptation of private protocols of network devices in existing solutions and the need to adjust hardware or redevelop programs when adding new devices. It improves the adaptability and scalability of log collection for multiple types of network node devices and realizes the standardized acquisition of logs from different manufacturers and different protocols. 2. This invention constructs a baseline for normal equipment operation behavior by employing attention-enhanced temporal modeling technology through an intelligent analysis unit. It divides the equipment operation into sub-segments based on the periodicity of the equipment operation and assigns differentiated weights according to the fluctuation characteristics of key indicators, making the baseline more focused on the time periods that significantly affect the equipment status. At the same time, it introduces an improved association mining algorithm to prune low-association itemsets of equipment topology association, which solves the problems of existing solutions that fail to distinguish key time periods in long-term logs and have difficulty in mining implicit cross-device association events, thereby improving the rationality of the behavioral baseline and the comprehensiveness of multi-device log association analysis. 3. This invention calibrates the timestamps of logs from multiple devices based on a time synchronization protocol through a log integration unit. It adopts distributed storage technology combined with hierarchical storage, inverted index and secondary hash index design, which not only achieves unified time synchronization of logs from multiple devices, but also improves the storage management efficiency and retrieval response speed of the integrated log data. It solves the problems of large log time deviation and low efficiency of large-scale log retrieval in existing solutions, and facilitates log backtracking queries for operation and maintenance personnel. 4. This invention uses an adaptive algorithm based on data flow statistical characteristics to filter noise in the preprocessing unit and combines it with a string matching algorithm to complete the structured transformation of unstructured logs. This can remove invalid data such as null fields and consecutive duplicate records in the original logs, and extract key information from the unstructured text and map it to preset fields. This reduces the interference of invalid data on subsequent analysis, improves the structuring degree of log data, and provides a reliable data foundation for intelligent analysis. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system framework of the present invention; The meanings of the labels in the diagram are as follows: 100. Multi-source data acquisition unit; 110. Hardware interface adaptation module; 120. Protocol parsing module; 130. Field extraction module; 140. Dynamic adaptation module; 141. Protocol rule storage submodule; 142. Rule loading submodule; 200. Preprocessing unit; 210. Noise filtering module; 220. Preliminary anomaly screening module; 230. Structured conversion module; 300. Intelligent Analysis Unit; 310. Baseline Construction Module; 311. Time-Sequence Segmentation Submodule; 312. Attention Weight Calculation Submodule; 320. Anomaly Detection Module; 321. Field Weight Configuration Submodule; 322. Dynamic Distance Calculation Submodule; 330. Association Mining Module; 331. Multi-Device Log Time-Sequence Alignment Submodule; 332. Association Pattern Optimization Submodule; 400. Log Integration Unit; 410. Time Synchronization Module; 420. Event Association Module; 430. Storage and Retrieval Module; 431. Hierarchical Storage Submodule; 432. Index Building Submodule; 433. Retrieval Scheduling Submodule; 500. Visualization Unit; 510. Status Display Module; 520. Anomaly Presentation Module; 530. Data Interaction Module. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, this embodiment provides a network device operation monitoring log integration system based on multi-source data acquisition and analysis, including: Multi-source data acquisition unit 100 is used to collect the operation logs and real-time status data of network node devices. It realizes the physical layer connection of devices and the extraction of raw log fields through hardware interfaces and standardized communication protocols, so as to achieve the adaptation and standardized acquisition of log data of different types of network node devices. In this embodiment, the multi-source data acquisition unit 100 includes a hardware interface adaptation module 110, a protocol parsing module 120, and a field extraction module 130, wherein: The hardware interface adapter module 110 uses physical interfaces such as RJ45 electrical port and multiple types of high-speed optical ports to achieve physical layer connection with different network node devices, and completes electrical signal transmission and optical signal to electrical signal conversion. As a further explanation of this embodiment, the hardware interface adapter module 110 of this embodiment integrates 8 RJ45 electrical ports (compatible with 10 / 100 / 1000BASE-T standards) and 4 SFP+ optical ports (supporting 1G / 10G rates). The PHY chip (model RTL8211F) processes the encoding and decoding of electrical port signals and rate negotiation, and the optical module interface circuit adapts to SFP+ optical modules from different manufacturers. In the signal processing flow, the signal received by the physical interface is filtered for noise by the signal shaping circuit and then transmitted to the protocol parsing module 120 through an 8-bit parallel bus (rate 1Gbps). At the same time, the "link status signal" is output to the system status register. In addition, the connection management mechanism has built-in link detection logic, which sends an Ethernet OAM detection frame every 30 seconds. If there is no response after 3 consecutive attempts, the link is reset. If the reset fails, the interface is marked as "faulty" and reported.

[0020] The protocol parsing module 120 parses the protocol frame structure of log data based on a variety of standardized network communication protocols, and identifies the frame boundaries and protocol types of the log data. As a further explanation of this embodiment, the protocol parsing module 120 of this embodiment specifically includes protocol parsing layers, frame boundary processing, and exception handling, wherein: The protocol parsing layer adopts an architecture of "link layer - network layer - application layer". The link layer extracts Ethernet frame header information and strips the frame header. The network layer parses the IPv4 packet protocol fields and transport layer information. The application layer combines the port number and protocol signature to determine the log protocol. Frame boundary processing uses a preset length field or terminator to divide boundaries for TCP streaming protocols, UDP packet protocols are parsed in units of data packets, and unknown protocols extract the first 16 bytes of frame feature code and trigger dynamic adaptation. During exception handling, data that fails to be parsed is marked as "parsing failed" and stored in a local circular cache (capacity 1GB). At the same time, the dynamic adaptation module 140 is queried for rules. If no valid rules are found within 10 seconds, the data is forwarded to the field extraction module 130 in "transparent transmission mode".

[0021] The field extraction module 130 extracts the original log fields (such as timestamp, device identifier, event type, etc.) from the parsed protocol frame data based on the offset and length information of the original log fields in each protocol.

[0022] As a further explanation of this embodiment, the field template management, extraction and verification process of the field extraction module 130 in this embodiment specifically includes: The field template management system includes a built-in standardized protocol field template library (with field attributes defined in XML format) for Syslog, SNMPTrap, and other protocols, while also supporting the loading of private protocol templates from the dynamic adaptation module 140. During extraction and verification, the protocol parsing module 120 first receives "protocol type + frame data" and calls the corresponding template. The field values ​​are extracted according to the template and the format of key fields is verified. Fields that fail verification are marked as "invalid" and the original bytes are retained. Finally, the data is output to the preprocessing unit 200 in key-value pair format.

[0023] In this embodiment, the multi-source data acquisition unit 100 further includes a dynamic adaptation module 140, which includes a protocol rule storage submodule 141 and a rule loading submodule 142, wherein: Protocol rule storage submodule 141 is used to store the private protocol parsing rules of new network node devices; As a further explanation of this embodiment, the storage medium and format of the protocol rule storage submodule 141 and the rule update mechanism of this embodiment specifically include: The storage uses an SQLite database (filenamed protocol_rules.db), which contains three tables: metadata (records rule metadata), feature_codes (stores protocol feature codes), and parse_rules (defines frame structure and field rules). It also supports local updates (importing .rule format files via USB and verifying signatures) and remote updates (pulling incremental update packages via HTTPS at 0:00 every day, and backing up the three most recent old versions after the update).

[0024] The rule loading submodule 142 is used to load private protocol parsing rules into the protocol parsing module 120 and the field extraction module 130, so that the multi-source data acquisition unit 100 can adapt to the log data of new network node devices and extract the original fields without modifying the hardware.

[0025] As a further explanation of this embodiment, the triggering conditions, rule matching and loading, and matching failure handling of the rule loading submodule 142 in this embodiment specifically include: Triggering conditions are divided into automatic triggering (the protocol parsing module 120 reports "unknown protocol" 5 times in a row and the signature is consistent) and manual triggering (operation and maintenance personnel actively load it through the Web console). During rule matching and loading, the "protocol feature code + device MAC prefix" is first extracted to query the rule base. If a match is found, the frame structure rule is pushed to the protocol parsing module 120 and the field template is pushed to the field extraction module 130. Once loading is complete, the system will send a "successful adaptation" message. When a matching failure occurs, the "signature code + device information" is recorded in the log, a prompt message is pushed to the operation and maintenance terminal, and the protocol parsing module 120 is triggered to enter the "transparent transmission mode".

[0026] The preprocessing unit 200 is used to perform noise filtering, preliminary anomaly screening, and structure transformation on the raw log data. It uses an adaptive algorithm based on data flow statistical characteristics and a string matching algorithm to process the log data, thereby removing invalid data interference and improving the structure of the log data to support subsequent analysis. In this embodiment, the preprocessing unit 200 includes a noise filtering module 210, an anomaly preliminary screening module 220, and a structured conversion module 230, wherein: The noise filtering module 210 dynamically adjusts the filtering threshold based on the statistical characteristics of data traffic (including field integrity and data repetition frequency) through an adaptive algorithm to identify and remove null fields and consecutive duplicate records caused by transmission errors in the original log. The anomaly preliminary screening module 220 constructs the normal fluctuation range of each field based on the statistical distribution of field values ​​in historical normal logs, and identifies jump field records that exceed the normal fluctuation range, thus completing the preliminary marking of obviously abnormal logs. As a further explanation of this embodiment, the anomaly preliminary screening module 220 in this embodiment identifies records of field jumps that exceed the normal fluctuation range based on the statistical distribution of field values ​​in historical normal logs, specifically including: Historical normal log selection: Select logs from the past 90 days that are not associated with any clear abnormal events such as "hardware failure" or "link interruption", and classify and statistically analyze their distribution characteristics according to field type (numerical: CPU utilization, memory usage, etc.; discrete: event type, operation result, etc.). Construction of standard fluctuation ranges: For numerical fields, the range is constructed using "mean ± 3 standard deviations" (formula: ),in This represents the average value of the historical normal log field. (The standard deviation is used); for discrete fields, values ​​with a historical frequency greater than 1% are considered as the normal range. Jump field record identification: A single jump exceeding the upper limit of the fluctuation range by 20% (formula: ), or two consecutive jumps with an amplitude greater than 15% (formula: If a record is identified as a jump record, a "pre_abnormal" tag is added to the log, and the jump field name, current value and fluctuation range are recorded simultaneously for easy traceability later.

[0027] The structured conversion module 230 uses a string matching algorithm to extract key information from unstructured log text and map it to preset fields based on preset log field templates (covering fixed categories such as device status parameters, event descriptions, and operation results) to generate structured data.

[0028] As a further explanation of this embodiment, the structured conversion module 230 in this embodiment uses a string matching algorithm to extract key information from unstructured log text based on a preset log field template, generating structured data, specifically including: String matching algorithm selection: When the text length is ≤100 characters, the KMP algorithm (single-field exact matching, time complexity) should be selected. , For text length, (For template keyword length); when text length > 100 characters, use the AC automaton (multi-field batch matching, time complexity...). Automatically switches based on log text length; Pre-defined log field template design: Defined in XML format, covering three types of fields: "Device status parameters" (such as CPU utilization, memory usage, etc., including matching keywords and data types), "Event description" (such as port UP, link disconnection, etc., including value range), and "Operation result" (such as configuration success, authentication failure, etc., including extraction rules). Each field clearly defines the matching keywords and extraction logic. Key information extraction and structured output: The template keywords are located through matching algorithms, and the effective data after the keywords is extracted according to the extraction rules (such as extracting "65%" from "memory usage: 65%"); the structured data is output in JSON format, and missing fields are marked as "null" to ensure complete correspondence with the preset template fields.

[0029] In this embodiment, the noise filtering module 210 dynamically adjusts the filtering threshold using an adaptive algorithm, including the following steps: Define the following parameters: initial threshold (Calibrated based on the device's factory default configuration or statistical results from the most recent 30-day historical stable period (a continuous period in which logs show no abnormal fluctuations); Preset lag ratio The range of values ​​is Used for lag correction during threshold callback (to avoid oscillation); preset interval coefficient , , Define the fluctuation range boundary of the threshold callback; A sliding time window is constructed, with the window duration dynamically configured within a preset variable range based on log traffic density, to calculate the duplicate record density of logs within the window in real time. (The ratio of the frequency of identical content log occurrences to the window duration) and the penetration rate of empty fields. (The percentage of logs containing null values ​​out of the total number of logs in the window); As a further explanation of this embodiment, the sliding time window duration is dynamically configured based on traffic: within the range of 10-60 seconds, 10 seconds are used when the traffic density is >100 messages / second, 30 seconds are used when the traffic density is 20-100 messages / second, and 60 seconds are used when the traffic density is <20 messages / second. The window data is statistically analyzed in real time through a memory counter and reset every cycle.

[0030] Based on the statistical distribution of normal logs from the same historical period, a baseline threshold for duplicate record density is generated. And the baseline threshold for penetration rate of empty fields ; When real-time repeated recording density Exceed Or, the penetration rate of empty fields Exceed When calculating the deviation. (The deviation ratio between the real-time value and the corresponding benchmark threshold), and according to Dynamically lower the filter trigger threshold, and the threshold reduction amount is the same as... Positive correlation, and a single adjustment does not exceed the preset upper limit of the original threshold; When a preset number of consecutive windows meet the requirements Below and Below When this happens, the filtering trigger threshold will be reverted to the initial threshold. fluctuation range And retains the lag correction amount during the callback process. To avoid threshold oscillations.

[0031] As a further explanation of this embodiment, the adaptive threshold adjustment in this embodiment is divided into three steps, specifically including: First, calculate the deviation. When real-time repeated recording density Exceed Or, the penetration rate of empty fields Exceed When, calculate according to the following formula ; Secondly, the threshold is lowered, and the new threshold is calculated using the following formula: The single downward adjustment must be ≤10% of the original threshold; Third, threshold callback, three consecutive windows satisfying Below and Below When the threshold falls after the callback and superimposed Lag correction (e.g.) , (When corrected, the range is 7.6%-10%), to avoid fluctuations.

[0032] It should be added that the anomaly feedback mechanism and fault tolerance handling in this embodiment specifically include: Anomaly feedback mechanism: If the threshold reduction of the noise filtering module 210 is greater than 8% for 5 consecutive windows, a "log transmission anomaly warning" will be pushed to the system management terminal, along with real-time feedback. , value; Fault tolerance: Logs of noise filtering failures (such as counter failures) are marked as "filter_error", and logs of structure conversion failures (such as no matching template) are marked as "struct_error". Both types of logs are temporarily stored in a local circular cache (capacity 2GB, overwritten cyclically by timestamp), and maintenance personnel can manually retry them through the web console.

[0033] The intelligent analysis unit 300 is used to construct a baseline of normal network device operation behavior, identify device operation anomalies, and mine the log correlation between multiple devices. It generates a behavior baseline through time-series modeling technology enhanced by attention mechanism, identifies anomalies using dynamic weighted distance calculation method, and analyzes the log correlation of multiple devices by combining an improved correlation mining algorithm, thereby improving the accuracy of long-series log baseline, the accuracy of nonlinear anomaly identification, and the efficiency of large-scale log correlation mining. In this embodiment, the intelligent analysis unit 300 includes a baseline construction module 310, an anomaly identification module 320, and a correlation mining module 330, wherein: The baseline construction module 310 generates a baseline for normal device operation behavior through attention-enhanced temporal modeling technology, including a temporal segmentation submodule 311 and an attention weight calculation submodule 312, wherein: The time-series segmentation submodule 311 divides the long time-series log into continuous sub-segments according to the periodic characteristics of the equipment operation (such as daily / weekly / monthly cycles). Each sub-segment contains the equipment status parameters and event records within that time period. As a further explanation of this embodiment, the time segment division in this embodiment is based on the daily / weekly / monthly periodicity of device operation, and the time granularity is dynamically adapted according to the period type to ensure that each sub-segment can fully reflect the device status characteristics within the time period: Daily cycle division: Applicable to devices with short-cycle fluctuations (such as access switches), it is divided into 24 consecutive sub-segments with "1 hour / segment". Each segment contains data such as CPU utilization, port traffic, and event records (such as "port UP / DOWN") within that hour. Weekly cycle division: Applicable to devices with medium-cycle fluctuations (such as aggregation routers), it is divided into 7 consecutive sub-segments according to "1 day / segment", and each segment summarizes the average status parameters and key events of each hour of the day; Monthly cycle division: Applicable to long-term stable equipment (such as core switches), it is divided into 4-5 consecutive sub-segments according to "1 week / segment", and each segment summarizes the daily statistical characteristics of the week (such as the maximum memory usage rate and average protocol interaction frequency within the week).

[0034] After segmentation, each sub-segment is associated with the original log through a unique time identifier (such as "2024-05-20 09:00-10:00") to ensure traceability.

[0035] The attention weight calculation submodule 312 assigns differentiated weights to different sub-segments based on the fluctuation characteristics of key indicators (such as resource utilization, port status, and protocol interaction frequency) in the sub-segments, making the baseline more focused on the time periods that have a significant impact on the device status. The anomaly detection module 320 uses a dynamic weighted distance calculation method to identify equipment malfunctions. It includes a field weight configuration submodule 321 and a dynamic distance calculation submodule 322, wherein: The field weight configuration submodule 321 assigns initial weights to different fields based on the historical association between log fields and device faults (such as the frequency of association between error code fields and hardware faults), and dynamically updates the weight coefficients based on real-time fault feedback. As a further explanation of this embodiment, the field weight configuration submodule 321 in this embodiment specifically includes weight allocation and update logic based on fault association history, wherein: Initial weights Calculate using the following formula: , For fields Frequency of association with faults Total number of fields Total association frequency ); Real-time weight updates use exponential smoothing, and the calculation formula is as follows: , The weight at the current moment, Associating frequency at the current moment The smoothing coefficient and This ensures that the weight of fault-sensitive fields is dynamically increased.

[0036] The dynamic distance calculation submodule 322 generates an anomaly score by calculating the weighted deviation between real-time log features and behavioral baseline features. When the score exceeds a preset threshold, an anomaly alarm is triggered. As a further explanation of this embodiment, the anomaly identification process of the dynamic distance calculation submodule 322 in this embodiment includes the following steps: First, calculate the weighted deviation for each field. , For real-time feature values, Baseline characteristic value, Assign field weights; Subsequently, anomaly scores were calculated using weighted Euclidean distance. , The score is dimensionless and only reflects the degree of deviation. Then, preset the anomaly threshold. (Taking abnormal events from the past 90 days) (95th percentile of the value); Next, compare with real time. and ,like This will trigger an abnormal alarm; Finally, record the exception field ( The top 3 largest fields), deviation details ( and The difference between the values ​​and related fragment information can be used to assist in subsequent tracing.

[0037] The correlation mining module 330 analyzes the correlation relationships of multi-device logs using an improved correlation mining algorithm, including a multi-device log time-series alignment submodule 331 and a correlation pattern optimization submodule 332, wherein: The multi-device log timing alignment submodule 331 calibrates the log timestamps of different devices based on the network time protocol, eliminates time synchronization deviations between devices, and achieves timing consistency of logs across devices. As a further explanation of this embodiment, the multi-device log timing alignment submodule 331 in this embodiment specifically includes time synchronization and calibration logic based on the network time protocol, wherein: The time synchronization cycle is once per hour. The device sends a synchronization request to the NTP server and records the local timestamp. With server timestamp deviation ;; Log timestamp correction according to formula ,in This is the original timestamp. To calibrate the timestamp, if Then immediately restart the NTP service; Timing consistency checks need to ensure cross-device associated logs If the difference is ≤1s (the upper limit of network transmission delay), it will be marked as "time sequence abnormal" and the original timestamp will be retained.

[0038] The association pattern optimization submodule 332 improves the efficiency of association mining by introducing device topology association (such as physical connection relationship and data forwarding path) to prune log item sets between devices with low association.

[0039] In this embodiment, the attention weight calculation submodule 312 assigns differentiated weights to different sub-segments, including the following steps: Extract key state features for each time segment, including the fluctuation range of device resource utilization within the segment, the frequency of occurrence of abnormal event precursor fields, and stability indicators of protocol interaction. Calculate the first based on historical log data State correlation of each segment (Characterizing the correlation between this segment and the normal operating status of the equipment), and The value increases as the stability of the key state characteristics improves; according to Assign initial weights to each segment , and They are positively correlated, and the sum of the weights of all segments satisfies the normalization condition; via sliding window The weights are updated in real time, and the weights within the window are dynamically adjusted according to the changing trends of the current device operating status, improving adaptability to segments containing sudden changes in status. ; The weight percentage exceeds the preset threshold. The segments are marked as key segments, and the generation of behavioral baselines is mainly based on the state characteristics of the key segments, and The value is dynamically adapted based on the device type and operating scenario.

[0040] As a further explanation of this embodiment, the differential weight assignment process of the attention weight calculation submodule 312 in this embodiment includes the following steps: First, key state features of each time series segment are extracted, including the fluctuation range of resource utilization. Frequency of occurrence of the abnormal event precursor field (Direct counting acquisition), protocol interaction stability indicators ; The calculation formula is: ,in: For fragments Inner Resource utilization of log entries For fragments The mean, For fragments The number of log entries; The calculation formula is: , The standard deviation of TCP connection success rate. (mean) Subsequently, the state correlation degree of the i-th segment is calculated. The calculation formula is: ; in: The largest in nearly 30 days , The largest in nearly 30 days , Furthermore, the value increases as the feature stability improves; Then, according to Assign initial weights The calculation formula is: , Let be the total number of segments, satisfying ); Next, through the sliding window The weights are updated in real time, and the segments are adaptively adjusted to respond to sudden changes in conditions. The formula for calculating the current window weight is: ; in: For smoothing coefficients, The current window weight. For the current window ; Finally, the weight percentage will exceed the preset threshold. (core equipment) Convergence equipment Access devices The segments marked as key segments are used to generate behavioral baselines using the following formula: ,in Key segments The Each feature value.

[0041] In this embodiment, the association pattern optimization submodule 332 prunes the log item sets between devices with low association, including the following steps: Construct a network device topology diagram, and determine the devices based on their physical connection links (such as fiber optic links and network cable connections) and data interaction frequency (the number of cross-device data packet forwardings per unit time). With equipment topological dependency coefficient ; Set the initial association threshold according to the network architecture layers (core layer, aggregation layer, access layer). , The value is dynamically adjusted according to the importance of the device's layer (the threshold between core layer devices is lower than the threshold between access layer devices). For device pairs in a multi-device log association set Calculate its actual topological dependency coefficient. ,when When this happens, the associated itemset is marked as a low-association itemset; Prune the marked low-relevance itemsets, and only retain log records in the itemsets that contain critical events (such as device offline or link interruption). Recalculate at preset intervals (based on the frequency of network topology changes). And update This ensures that the pruning rules are adapted to the actual operating state of the network.

[0042] As a further explanation of this embodiment, the low-association itemset pruning process of the association pattern optimization submodule 332 in this embodiment includes the following steps: First, construct a network device topology diagram. ;in For device nodes, The edges are physical connections, and their weights are the frequency of data interactions. ; Subsequently, computing devices and topological dependency coefficient , Physical connection weights and , The physical connection coefficient (if there is a connection) If not ), This is the frequency normalized value. Maximum interaction frequency; Then, the initial association threshold is adapted according to the device level. (Core layer) Convergence layer Access layer ); Next, for the device pairs in the associated item set ( ),like Then it is marked as a low-association itemset; Finally, low-relevance itemsets are pruned (only logs containing key events are retained), and the results are recalculated at a preset period (24 hours / time when the topology is stable, and 6 hours / time when it fluctuates). and Ensure that the rules are adapted to the network conditions.

[0043] It should be added that the fault tolerance mechanism in this embodiment is as follows: Baseline construction failed: if real-time If a calculation error occurs (such as a missing log), the baseline features from the previous window will be automatically reused. At the same time, mark "baseline to be updated"; Abnormal score fluctuations: If Five consecutive times in [0.8] 1.2 Range fluctuations (no clear anomalies), automatic adjustment. (From 0.6 to 0.4), enhancing the weights' responsiveness to real-time data; pruning errors related to association. If critical event logs are lost after pruning, the original logs are retrieved from log integration unit 400 and recalculated. With pruning rules.

[0044] Log integration unit 400 is used to realize multi-device log time synchronization, related event matching, and storage and query of integrated data. It calibrates log timestamps based on time synchronization protocol, associates logs from different devices through similarity matching algorithm, and uses distributed storage technology to store integrated data, thereby realizing unified fusion management and efficient retrieval of multi-source logs. In this embodiment, the log integration unit 400 includes a time synchronization module 410, an event association module 420, and a storage retrieval module 430, wherein: The time synchronization module 410 calibrates the timestamps of multi-device logs based on time synchronization protocols (including Network Time Protocol NTPv4 or Precision Time Protocol PTP), and corrects the log time offset through periodic time deviation detection (the detection period is dynamically adjusted according to the number of devices), ensuring that the timestamp deviation of logs from different devices is controlled within a preset threshold. As a further explanation of this embodiment, the selection of the time synchronization protocol in the time synchronization module 410 needs to be combined with the device type and accuracy requirements, as follows: Core layer devices (such as core routers) adopt the Precision Time Protocol (PTP) (following the IEEE 1588-2008 standard) to achieve nanosecond-level synchronization through hardware timestamps (such as the built-in timestamp function of the PHY chip); Aggregation layer and access layer devices (such as aggregation switches and access switches) adopt the Network Time Protocol (NTPv4) to achieve millisecond-level synchronization through software timestamps. Both protocols support client-server mode, and synchronization requests are transmitted through UDP ports (NTP uses port 123, and PTP uses ports 319 / 320). Furthermore, the periodic time deviation detection period is dynamically adjusted according to the number of devices. Specifically: when the number of connected devices is ≤50, the detection period is set to 5 minutes; when the number of devices is 50-200, the period is set to 3 minutes; when the number of devices is >200, the period is set to 1 minute. The detection is performed by sending a "time query frame" to each device and recording the difference between the local timestamp returned by the device and the module's own timestamp, which is the initial time deviation. Furthermore, the time offset correction process needs to first determine the type of deviation, specifically: If it is a fixed deviation (the deviation value fluctuates by ≤5% for 3 consecutive detections), the deviation compensation value is directly added to the log timestamp (e.g., if the deviation is 200ms, the original log timestamp + 200ms is the calibration value). If the deviation is fluctuating (deviation value fluctuation > 5%), the device will be resynchronized (NTP device re-initiates synchronization request, PTP device renegotiates master and slave clock), and the device will be checked again after correction until the deviation stabilizes. Meanwhile, the preset threshold for timestamp deviation control is determined based on the synchronization protocol type: the threshold for PTP synchronization is ≤100ns, and the threshold for NTP synchronization is ≤10ms. If the deviation continues to exceed the threshold, a "time synchronization abnormality" alarm is pushed to the system management terminal, and the abnormal device identifier and deviation value are recorded.

[0045] The event association module 420 associates logs from different devices using a similarity matching algorithm. Specifically, it extracts key event fields from the logs (including event ID, device identifier, and operation behavior description), calculates the text similarity of key fields from different device logs, and determines that the same associated event is when the similarity exceeds a dynamically adjusted association threshold (the threshold is optimized based on the matching accuracy of historical associated events). As a further explanation of this embodiment, the log association process of the event association module 420 in this embodiment includes the following steps: First, extract key event fields (event ID is formatted as "device abbreviation + type code + serial number", device identifier is a unique MAC / number, and operation description retains core words); Then, cosine similarity is used to calculate text similarity, with the following formula: , For key field word vectors; The value ranges from [0,1], with a larger value indicating a higher similarity. Then, determine the dynamically adjusted association threshold: the initial threshold is set based on the matching accuracy of historical association events (e.g., if the historical matching accuracy is ≥95%, the initial threshold is set to 0.7). Subsequently, every 24 hours, the "false match rate" (the proportion of events judged as related but actually unrelated) and "missed match rate" (the proportion of events actually related but not judged) of the association events of the day are calculated. If the false match rate is >5%, the threshold is increased by 0.05; if the missed match rate is >5%, the threshold is decreased by 0.05. The threshold is always kept in the range of 0.5-0.9. Next, compare the similarity. With the associated threshold, if If the threshold is exceeded, it is determined to be the same related event, and a unique related event ID is assigned to it. Finally, the associated ID is written to the "Association Identifier" field in the log, recording the similarity and threshold.

[0046] The storage and retrieval module 430 employs distributed storage technology to achieve hierarchical management and efficient retrieval of integrated data. It includes a hierarchical storage submodule 431, an index building submodule 432, and a retrieval scheduling submodule 433, wherein: The tiered storage submodule 431 stores the original integrated log in shards based on the log time series characteristics, with hourly or daily time granularity. It dynamically configures the replica redundancy strategy according to the importance of the device, and the shard storage rules are linked with the log index construction logic. As a further explanation of this embodiment, the log sharding storage in the hierarchical storage submodule 431 is based on time series characteristics: the original integrated logs are sharded at the "hourly" or "daily" granularity. Devices with a log generation frequency > 1000 logs / hour (such as core routers) use hourly sharding, and the sharding naming format is "YYYYMMDDHH-device type-sharding number" (such as "2024052009-CORE-001"); devices with a log generation frequency ≤ 1000 logs / hour (such as access switches) use daily sharding, and the naming format is "YYYYMMDD-device type-sharding number" (such as "20240520-ACCESS-005"). The sharding file format is uniformly Parquet (column-based storage, supporting compression), and the storage path is hierarchically layered according to "year / month / date". Furthermore, the equipment importance classification standard is determined based on the role of the equipment in the network architecture. Specifically: core layer equipment is level one, aggregation layer equipment is level two, and access layer equipment is level three. The replica redundancy configuration strategy is tied to the equipment importance: level one equipment log sharding is configured with 3 replicas, which are stored on storage nodes in different racks (to avoid data loss due to rack failure); level two equipment is configured with 2 replicas, which are stored on different nodes in the same rack; level three equipment is configured with 1 replica, which is stored on a single node. Furthermore, the linkage logic between sharding and indexing is as follows: when sharding is generated, "shard metadata" (including sharding time range, device list, and key field statistics) is automatically generated synchronously, and the metadata is pushed to the index building submodule 432 to ensure that the field range and time granularity of index building and shard storage are completely aligned, thus avoiding the index pointing to an invalid shard.

[0047] The index building submodule 432 builds inverted indexes for key fields (such as timestamp, event type, device number, etc.), and the log write operation triggers index updates in real time (the index update delay for a single log is ≤ preset threshold), and adds secondary hash indexes for high-frequency query fields (such as event type); As a further explanation of this embodiment, the inverted index in the index construction submodule 432 is constructed for key fields (timestamp, event type, device number), with "field value - shard ID - log offset" as the core structure: the timestamp field is grouped by "hour / day" granularity (consistent with the shard time granularity), such as the timestamp "2024-05-2009:15:30" corresponding to the group "2024052009", which is associated with all shard IDs and log offsets in the shard under this group; Furthermore, the event type field is hierarchically structured as "major event category - minor event category". For example, "fault event - port fault" corresponds to all shards and log offsets containing this type. The device number field is directly associated with all shards and log offsets of the device. The index data is stored in both memory and disk (highly accessed index items are stored in memory, and the full index is stored on disk). Meanwhile, the real-time index update mechanism adopts a "log write-index callback" mode: when log data is written to a shard, a callback function is triggered to synchronously update the inverted index of the corresponding key field. The index update delay of a single log is ≤ a preset threshold (this threshold is determined based on the storage cluster IO performance; for example, when IO throughput > 100MB / s, the threshold is set to 100ms; when IO throughput ≤ 100MB / s, the threshold is set to 200ms). If the update fails, a retry mechanism is triggered (up to 3 retries, with an interval of 50ms each time). If the retry fails, the index is marked as "index to be repaired" and recorded in the log. Furthermore, the secondary hash index adds the following for high-frequency query fields (such as event types, whose query frequency accounts for more than 30% of the total query volume): the field value is used as the hash key, the hash value is calculated by the hash function, and mapped to the bucket in the hash table. The bucket stores the inverted index entry address corresponding to the field value, realizing direct location of "field value - index entry", reducing the traversal time of the inverted index. The hash table supports dynamic expansion.

[0048] When the retrieval scheduling submodule 433 receives a query request, it quickly locates the target data shard through the index, schedules at least 3 storage nodes to perform data reading operations in parallel, and combines a hot query result caching mechanism (the cache validity period is dynamically adjusted according to the query frequency) to achieve millisecond-level retrieval response of the integrated log, and the increase in retrieval latency with the growth of log size is ≤ a preset ratio (the threshold and ratio are dynamically calibrated based on the performance of the distributed storage cluster).

[0049] As a further explanation of this embodiment, the log retrieval process of the retrieval scheduling submodule 433 in this embodiment includes the following steps: First, receive search requests from users or the system, and parse the key fields in the requests; Subsequently, the target data shard is located by indexing: first, the inverted index of the timestamp field is queried to filter out the shard IDs within that time range; then, the secondary hash index of the event type is queried to locate the index entry corresponding to "port failure" and further filter out the shards containing that event type; finally, the final target shard is determined by combining the device type filter. Then, the storage nodes are scheduled to perform data reading in parallel: based on the node information of the sharded storage, at least 3 storage nodes are selected (preferably nodes with a load rate of <70% or nodes with a network latency of <50ms to the retrieval initiator), and a read request of "shard ID + log filtering conditions" is sent to the nodes. The nodes extract the target log from the shard according to the conditions and return the result in the form of a data stream. Next, a hot query result caching mechanism is implemented: It is determined whether the current search request is a hot request (if the request appears ≥5 times within the last 10 minutes, it is considered a hot request). If so, the search result is stored in the cache (the caching medium uses an in-memory database, such as Redis). The cache validity period is dynamically adjusted based on the query frequency: when the query frequency is ≥1 time / minute, the validity period is set to 10 minutes; when the query frequency is 0.5-1 time / minute, the validity period is set to 5 minutes; when the query frequency is <0.5 times / minute, it is not cached, and when the cache is full, the "Least Recently Used (LRU)" strategy is used to evict old data. Finally, control the increase in retrieval latency: The increase in retrieval latency is calculated as "(current-scale retrieval latency - baseline-scale retrieval latency) / baseline-scale retrieval latency". The baseline scale is set to 10 million logs. For every additional 5 million logs, the storage cluster is calibrated (testing retrieval latency at different scales) to ensure that the increase is ≤ a preset ratio (this ratio is determined based on the calibration results; for example, if the baseline latency is 100ms, the latency after adding 5 million logs is ≤120ms, and the increase is ≤20%). If the increase exceeds the preset ratio, the index structure is optimized (e.g., splitting high-frequency field indexes) or the storage nodes are expanded.

[0050] It should be added that the abnormal collaborative processing in this embodiment includes: when the time synchronization module 410 detects that the device time deviation exceeds the threshold, it synchronously notifies the event association module 420 to suspend the log association of the device to avoid time discrepancies leading to association errors; when the retrieval scheduling submodule 433 finds that the index is missing, it triggers the index building submodule 432 to regenerate the index of the corresponding shard, and at the same time confirms the timestamp accuracy of the shard with the time synchronization module 410 to ensure the consistency between the index and the log data.

[0051] The visualization unit 500 is used to display the equipment's operating status, abnormal events, and analysis results, and to realize data interaction with third-party operation and maintenance systems. It generates a visualization interface through web front-end technology and provides a data interaction interface based on standardized network protocols, so as to realize the intuitive presentation of monitoring results and support cross-system collaborative operation and maintenance.

[0052] In this embodiment, the visualization unit 500 includes a status display module 510, an anomaly presentation module 520, and a data interaction module 530, wherein: The status display module 510 generates a visual interface through Web front-end technology, which displays the core operating status of the device in real time (including CPU utilization, port traffic, and protocol interaction frequency). It supports filtering by device type, and the interface data update frequency is synchronized with the analysis cycle of the intelligent analysis unit 300. As a further explanation of this embodiment, the status display module 510 of this embodiment specifically includes Web visualization construction and dynamic interaction logic: The front-end interface is developed using the Vue.js framework, and the ECharts visualization library is integrated to implement three core components: Device topology diagram: Node size maps to CPU utilization, colors distinguish device types (core / aggregation / access), and physical connection relationships are synchronized with the topology data of the log integration unit 400; Core performance dashboard: Real-time display of CPU utilization, peak port traffic, and average protocol interaction frequency, with numerical accuracy consistent with the feature calculation accuracy of the intelligent analysis unit 300; Timeline line chart: Supports switching between 1-hour, 6-hour, and 24-hour time granularities, displaying the trend of indicator fluctuations. The data source is the historical shards of the log integration unit 400.

[0053] Meanwhile, it supports filtering by combination of device type (core / aggregation / access), network level (backbone / regional / access layer), and geographical region (data center A / B). The filtering conditions are synchronized to the retrieval interface of the log integration unit 400 in real time. The interface data update frequency is strictly synchronized with the analysis cycle of the intelligent analysis unit 300, and a real-time channel is established through WebSocket to ensure that the display delay is less than or equal to the analysis cycle.

[0054] The anomaly presentation module 520 associates the anomaly judgment results of the intelligent analysis unit 300, matches the anomaly events with differentiated visual labels according to their severity level, and simultaneously displays the anomaly-related log fragments and preliminary investigation suggestions. As a further explanation of this embodiment, the exception display of the exception presentation module 520 in this embodiment specifically includes: Severity level mapping, associated with the anomaly threshold of the intelligent analysis unit 300, matches abnormal events with differentiated visual labels, specifically including: Fatal anomalies (device offline, link interruption, corresponding to...) ), using flashing red indicators; Critical anomaly (CPU utilization) 80% port error rate 5%, corresponding ), using an orange bouncing logo; Warnings and exceptions (configuration changes, protocol timeouts, corresponding) ), using yellow static signage; Related log display: Clicking the exception icon will load the related log fragments from the log integration unit 400 (sorted by timestamp, displaying device ID, event description, and original fields). The log source is the related event marked by the event association module 420. Troubleshooting suggestion generation: The system calls a preset knowledge base to output a preliminary troubleshooting path. The knowledge base can be manually updated by operations and maintenance personnel via the web interface. Data traceability: Abnormal entries are accompanied by an "Analysis Parameters" jump button, which allows you to trace back the abnormality calculation process of the intelligent analysis unit 300.

[0055] The data interaction module 530 provides a data interaction interface based on HTTP and HTTPS protocols, supporting the push of real-time monitoring data to third-party operation and maintenance systems, receiving and responding to query requests, and the interface call implements permission verification through a token authentication mechanism.

[0056] As a further explanation of this embodiment, the data interaction module 530 of this embodiment provides a RESTful interface (data format is JSON) based on the HTTP / HTTPS protocol, which supports third-party systems to pull real-time indicators (such as CPU utilization), subscribe to abnormal events (including severity level and associated log identifiers) and query historical data according to device type and time range; Meanwhile, JWTToken authentication is used (admin permissions support read and write, viewer permissions support read only, the token is valid for 24 hours, and can be refreshed via the interface 30 minutes before expiration), and the Authorization:Bearer header must be included when calling the interface. <token>Verification of rights; Then, the data interaction adopts the "active push + passive response" mode: real-time indicators are pushed every 5 minutes, abnormal events are pushed immediately, and historical data query requests are forwarded to the retrieval scheduling submodule 433 of the log integration unit 400 for execution; In addition, the fault-tolerant design supports retrying after an interface call fails (up to 3 times, with a 10-second interval), and retains the old interface for a 6-month transition period to ensure cross-system integration stability.

[0057] Those skilled in the art will understand that the process of implementing all or part of the steps of the above embodiments can be carried out by hardware or by a program instructing the relevant hardware.

[0058] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.< / token>

Claims

1. A network device operation monitoring log integration system based on multi-source data acquisition and analysis, characterized in that, include: Multi-source data acquisition unit (100) is used to collect the operation logs and real-time status data of network node devices. It realizes the physical layer connection of the device and the extraction of the original fields of the log through the hardware interface and standardized communication protocol, so as to realize the adaptation and standardized acquisition of log data of different types of network node devices. The preprocessing unit (200) is used to perform noise filtering, preliminary anomaly screening and structure transformation on the raw log data. It uses an adaptive algorithm based on data traffic statistical characteristics and a string matching algorithm to process the log data, thereby removing invalid data interference and improving the structure of the log data to support subsequent analysis. The intelligent analysis unit (300) is used to construct a baseline of normal network device operation behavior, identify device operation anomalies and mine the log correlation between multiple devices. It generates a behavior baseline through time-series modeling technology enhanced by attention mechanism, identifies anomalies by dynamic weighted distance calculation method, and analyzes the log correlation of multiple devices by combining an improved correlation mining algorithm, thereby improving the accuracy of long-series log baseline, the accuracy of nonlinear anomaly identification and the efficiency of large-scale log correlation mining. Log integration unit (400) is used to realize multi-device log time synchronization, related event matching and storage and query of integrated data. It calibrates log timestamps based on time synchronization protocol, associates logs of different devices through similarity matching algorithm, and uses distributed storage technology to store integrated data, so as to realize unified fusion management and efficient retrieval of multi-source logs. The visualization unit (500) is used to display the operating status of the equipment, abnormal events and analysis results, and realize data interaction with the third-party operation and maintenance system. It generates a visualization interface through Web front-end technology and provides a data interaction interface based on standardized network protocols to realize the intuitive presentation of monitoring results and support for cross-system collaborative operation and maintenance.

2. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 1, characterized in that, The multi-source data acquisition unit (100) includes a hardware interface adaptation module (110), a protocol parsing module (120), and a field extraction module (130), wherein: The hardware interface adapter module (110) achieves physical layer connection with different network node devices through physical interfaces of RJ45 electrical port and multiple types of high-speed optical ports, and completes electrical signal transmission and optical signal to electrical signal conversion. The protocol parsing module (120) parses the protocol frame structure of log data based on a variety of standardized network communication protocols, and identifies the frame boundaries and protocol types of log data; The field extraction module (130) extracts the original log fields from the parsed protocol frame data based on the offset and length information of the original log fields in each protocol.

3. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 2, characterized in that, The multi-source data acquisition unit (100) further includes a dynamic adaptation module (140), which includes a protocol rule storage submodule (141) and a rule loading submodule (142), wherein: The protocol rule storage submodule (141) is used to store the private protocol parsing rules of the new network node device; The rule loading submodule (142) is used to load private protocol parsing rules into the protocol parsing module (120) and the field extraction module (130), so that the multi-source data acquisition unit (100) can adapt to the log data of new network node devices and extract the original fields without modifying the hardware.

4. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 3, characterized in that, The preprocessing unit (200) includes a noise filtering module (210), an anomaly preliminary screening module (220), and a structured conversion module (230), wherein: The noise filtering module (210) dynamically adjusts the filtering threshold based on the statistical characteristics of data traffic through an adaptive algorithm to identify and remove null fields and consecutive duplicate records caused by transmission errors in the original log. The anomaly preliminary screening module (220) constructs the normal fluctuation range of each field based on the statistical distribution of field values ​​in historical normal logs, and identifies jump field records that exceed the normal fluctuation range, thus completing the preliminary marking of obviously abnormal logs. The structured conversion module (230) uses a string matching algorithm to extract key information from unstructured log text and map it to preset fields based on preset log field templates to generate structured data.

5. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 4, characterized in that, The noise filtering module (210) dynamically adjusts the filtering threshold through an adaptive algorithm, including the following steps: Define the following parameters: initial threshold Preset lag ratio Used for hysteresis correction during threshold callback; preset interval coefficient , , Define the fluctuation range boundary of the threshold callback; A sliding time window is constructed, with the window duration dynamically configured within a preset variable range based on log traffic density, to calculate the duplicate record density of logs within the window in real time. and empty field penetration rate ; Based on the statistical distribution of normal logs from the same historical period, a baseline threshold for duplicate record density is generated. And the baseline threshold for penetration rate of empty fields ; When real-time repeated recording density Exceed Or, the penetration rate of empty fields Exceed When calculating the deviation. , and according to Dynamically lower the filter trigger threshold, and the threshold reduction amount is the same as... Positive correlation, and a single adjustment does not exceed the preset upper limit of the original threshold; When a preset number of consecutive windows meet the requirements Below and Below When this happens, the filtering trigger threshold will be reverted to the initial threshold. fluctuation range And retains the lag correction amount during the callback process. To avoid threshold oscillations.

6. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 5, characterized in that, The intelligent analysis unit (300) includes a baseline construction module (310), an anomaly identification module (320), and a correlation mining module (330), wherein: The baseline construction module (310) generates a baseline for normal device operation behavior through attention-enhanced temporal modeling technology, including a temporal segmentation submodule (311) and an attention weight calculation submodule (312), wherein: The time-series segmentation submodule (311) divides the long time-series log into continuous sub-segments according to the periodic characteristics of the equipment operation. Each sub-segment contains the equipment status parameters and event records within that time period. The attention weight calculation submodule (312) assigns differentiated weights to different sub-segments based on the fluctuation characteristics of key indicators in the sub-segments, so that the baseline is more focused on the time period that has a significant impact on the equipment status; The anomaly identification module (320) uses a dynamic weighted distance calculation method to identify equipment malfunctions, including a field weight configuration submodule (321) and a dynamic distance calculation submodule (322), wherein: The field weight configuration submodule (321) assigns initial weights to different fields based on the historical association between log fields and device faults, and dynamically updates the weight coefficients based on real-time fault feedback. The dynamic distance calculation submodule (322) generates an anomaly score by calculating the weighted deviation between real-time log features and behavioral baseline features, and triggers an anomaly alarm when the score exceeds a preset threshold. The association mining module (330) analyzes the association relationship of multi-device logs using an improved association mining algorithm, including a multi-device log time-series alignment submodule (331) and an association pattern optimization submodule (332), wherein: The multi-device log timing alignment submodule (331) calibrates the log timestamps of different devices based on the network time protocol, eliminates time synchronization deviations between devices, and achieves timing consistency of logs across devices. The association pattern optimization submodule (332) improves the association mining efficiency by introducing device topology association degree and pruning the log item sets between devices with low association degree.

7. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 6, characterized in that, The attention weight calculation submodule (312) assigns differentiated weights to different sub-segments, including the following steps: Extract key state features of each time segment, including the fluctuation range of device resource utilization within the segment, the frequency of occurrence of abnormal event precursor fields, and stability indicators of protocol interaction. Calculate the first based on historical log data State correlation of each segment ,and The value increases as the stability of the key state characteristics improves; according to Assign initial weights to each segment , and They are positively correlated, and the sum of the weights of all segments satisfies the normalization condition; via sliding window The weights are updated in real time, and the weights within the window are dynamically adjusted according to the changing trends of the current device operating status, improving adaptability to segments containing sudden changes in status. ; The weight percentage exceeds the preset threshold. The segments are marked as key segments, and the generation of behavioral baselines is mainly based on the state characteristics of the key segments, and The value is dynamically adapted based on the device type and operating scenario.

8. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 7, characterized in that, The association pattern optimization submodule (332) prunes the log item sets between devices with low association, including the following steps: Construct a network device topology diagram, and determine the devices based on their physical connection links and data interaction frequency. With equipment topological dependency coefficient ; Set the initial association threshold according to the network architecture level. , The value is dynamically adjusted according to the importance of the device's hierarchy. For device pairs in a multi-device log association set Calculate its actual topological dependency coefficient. ,when When this happens, the associated itemset is marked as a low-association itemset; Prune the marked low-relevance itemsets, retaining only the log records in the itemsets that contain key events; Recalculated every preset period. And update This ensures that the pruning rules are adapted to the actual operating state of the network.

9. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 8, characterized in that, The log integration unit (400) includes a time synchronization module (410), an event association module (420), and a storage retrieval module (430), wherein: The time synchronization module (410) calibrates the timestamps of the logs of multiple devices based on the time synchronization protocol, and corrects the log time offset by periodic time deviation detection to ensure that the timestamp deviation of the logs of different devices is controlled within a preset threshold. The event association module (420) associates logs from different devices through a similarity matching algorithm, specifically including: extracting key event fields from the logs, calculating the text similarity of key fields from different device logs, and determining that the similarity exceeds the dynamically adjusted association threshold when they are the same associated event; The storage and retrieval module (430) uses distributed storage technology to achieve hierarchical management and efficient retrieval of integrated data, including a hierarchical storage submodule (431), an index building submodule (432), and a retrieval scheduling submodule (433), wherein: The hierarchical storage submodule (431) stores the original integrated log in segments based on the log time series characteristics, with hourly or daily time granularity. It dynamically configures the replica redundancy strategy according to the importance of the device, and the segmentation storage rules are linked with the log index construction logic. The index building submodule (432) builds an inverted index for key fields, the log write operation triggers index updates in real time, and adds a secondary hash index for high-frequency query fields; When the retrieval scheduling submodule (433) receives a query request, it quickly locates the target data shard through the index, schedules at least 3 storage nodes to perform data reading operations in parallel, and combines the hot query result caching mechanism to achieve millisecond-level retrieval response of the integrated log, and the increase in retrieval latency with the growth of log size is ≤ preset ratio.

10. The network device operation monitoring log integration system based on multi-source data acquisition and analysis according to claim 9, characterized in that, The visualization unit (500) includes a status display module (510), an anomaly presentation module (520), and a data interaction module (530), wherein: The status display module (510) generates a visual interface through Web front-end technology, displays the core operating status of the device in real time, supports filtering by device type, and the interface data update frequency is synchronized with the analysis cycle of the intelligent analysis unit (300). The anomaly presentation module (520) associates the anomaly judgment results of the intelligent analysis unit (300), matches the anomaly events with differentiated visual labels according to their severity level, and simultaneously displays the anomaly-related log fragments and preliminary investigation suggestions. The data interaction module (530) provides a data interaction interface based on HTTP and HTTPS protocols, supports pushing real-time monitoring data to third-party operation and maintenance systems, receiving and responding to query requests, and the interface call implements permission verification through a Token authentication mechanism.

Images