AI-based weblog analysis processing method

Abstract

The invention discloses an AI-based network log analysis processing method, which comprises the following steps of: firstly, collecting network equipment log information through a collection script supporting multiple protocol modes, and writing the collection script and deploying and starting after determining equipment and protocol types; thirdly, carrying out preliminary classification, data supplementation and formatting processing on the original logs; and then, sending the processed log data to an AI large model for semantic analysis, including deep understanding, basic classification, secondary analysis and providing of processing suggestions. An analysis result is stored in a data think tank for historical comparison, and an inconsistent result is manually confirmed and corrected; when the network equipment triggers a fault alarm, calling the data think tank to supplement alarm content; and meanwhile, equipment related data handbooks provided by manufacturers are uploaded regularly, the analysis accuracy of the AI model is corrected, verification and testing are carried out, it is ensured that the scheme can comprehensively and efficiently collect and process logs, accurately analyze faults and continuously optimize the model, and effective support is provided for network fault processing.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an AI-based network log analysis and processing method. Background Art

[0002] Network managers usually need to use logs to analyze network failures during daily network monitoring and operation and maintenance management. Different network equipment manufacturers do not have a unified specification for the format of network logs. Operation and maintenance personnel need to use the documents provided by different manufacturers to find possible problems in the logs, which makes the learning cost very high for operation and maintenance personnel. The present invention uses a large AI model to analyze network logs, which can automatically process large amounts of log data, significantly reduce the workload and time cost of manual processing, and improve work efficiency. Compared with manual processing, the precise analysis of log data through machine learning algorithms can greatly reduce the error rate and improve the analysis accuracy. It can also learn historical log data and behavior patterns, predict possible log events in the future, and monitor log data in real time. When abnormal behavior is discovered, it can issue early warnings in time, which helps to prevent potential risks. Natural language processing technology can also be used. AI can extract key information hidden behind unstructured logs to help better understand user needs and improve service quality. Summary of the Invention

[0003] This application provides an AI-based network log analysis and processing method, aiming to efficiently and accurately analyze network logs and provide troubleshooting solutions.

[0004] This application provides an AI-based network log analysis and processing method, including: S1, collects log information of network devices through collection scripts that support multiple protocols; S2, performs preliminary classification and data supplement operations on the collected original logs; S3 sends the preliminarily processed log data to the AI big model for semantic analysis; S4, stores the analysis results of the AI model in the data think tank and performs historical comparison; S5: When a network device triggers a fault alarm, the data intelligence library is called to supplement the alarm content. S6, regularly upload equipment-related manuals provided by manufacturers to correct the analysis accuracy of the AI model.

[0005] Preferably, the S3 specifically includes: S3a sends the preliminarily processed log data to the input port of the AI large model; S3b: The AI model performs deep semantic understanding of log content and performs basic classification of logs. In step 3c, the AI model performs secondary analysis on the classification results, correlating operation logs with fault logs to identify human factors or runtime status changes. S3d, the AI model provides treatment solution recommendations based on the analysis results.

[0006] Preferably, the S4 specifically includes: S4a, stores the analysis results of the AI model in the data think tank according to the predetermined format; S4b, when storing new analysis results, first check whether there are analysis results of the same log in the think tank; S4c, if there are existing historical analysis results, compare the old and new analysis results to analyze the differences and fault causes; S4d, when the analysis results are inconsistent, the different analysis results are recorded and sent to the interface for manual confirmation for confirmation; S4e, manually confirm and correct inconsistent analysis results.

[0007] Preferably, the S3 further includes: S31, dividing the timing pattern of device failure occurrence according to the timestamp of the log data based on the location of the device in the network; S32, performing fault-specific analysis on each event description in the time series pattern to identify the time distribution characteristics of the fault-specific event; S33, associate the upstream and downstream of the fault project and divide the fault propagation path; S34, constructs a fault impact range map based on the fault propagation path and timing pattern; S35, adds key influencing nodes to the AI big model for semantic analysis to predict the fault development trend of key influencing nodes.

[0008] Preferably, the S33 specifically includes: S33a, analyzing the causal and correlation relationships between different fault items; S33b, builds a fault propagation chain based on the upstream and downstream association relationships of the fault items; S33c, use a topology map to visualize the fault propagation path, and clearly mark the direction of fault propagation and the relationship between each fault item in the topology map.

[0009] Preferably, the S34 includes: S34a, combining the fault propagation path and timing pattern, analyzes the impact of the fault on various devices and business functions at different time points.

[0010] S34b, using devices and business functions as nodes and fault propagation relationships as edges, constructs a fault impact range map; S34c, mark the importance, impact degree, and fault propagation direction and intensity information of each node in the graph.

[0011] Preferably, in said S33, it further includes: S331, using a machine learning algorithm to build a dynamic fault special classification model, and inputting the log data after upstream and downstream association of the dynamic fault special into the dynamic classification model; S332, the model classifies log records into different dynamic fault categories in real time based on the learned rules; S333, collecting network traffic data and device performance indicator data as multi-source data; S334, uses data fusion technology to correlate and integrate multi-source data; S335, analyzes the correlation between different dynamic fault special projects, the time series pattern of propagation, and the key node characteristics on the propagation path; S336, select the graph neural network to predict the fault propagation path after the dynamic fault special project. When a new fault occurs, the relevant information of the fault is input into the trained prediction model; among which, the relevant information of the fault is the fault special project, occurrence time, involved equipment, etc. The model should output the possible propagation path and propagation probability of the fault.

[0012] Preferably, the S332 specifically includes: for each newly input log record, the model calculates the probability that the log belongs to each dynamic fault special item based on the classification rules learned in the training phase; when using a deep learning model, the score of each category of the output layer is calculated through forward propagation, and the softmax function is used to convert these scores into a probability distribution: ;in, Represents log records Belong to category The probability of is the score of the i-th category in the model output layer, and K is the total number of categories.

[0013] Preferably, in S336, the relevant information of the fault is represented as: , G is the graph structure, X is the feature matrix of nodes and edges, W is the parameter of the model, and f is the prediction function of the graph neural network.

[0014] Preferably, the risk score in S337 specifically includes: determining risk assessment indicators, assigning weights to each indicator, and calculating the risk score of the node; the risk score formula is: , is the weight of the i-th indicator, is the score of the i-th indicator. The risk scores of key influencing nodes, fault special items, propagation paths and other information are integrated into a feature vector, and the feature vector is input into the AI big model for semantic analysis.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: By supporting multiple protocol collection scripts to collect network device log information, first determine the device and protocol type and classify them, then write the corresponding protocol collection script, configure the parameters, and then deploy and start it; then perform preliminary classification, data supplementation, and formatting on the collected original logs; then send the processed log data to the AI big model for semantic analysis, including deep understanding, basic classification, secondary analysis, and providing processing suggestions; the analysis results are then stored in the data think tank for historical comparison, and manual confirmation and correction of inconsistent results; when the network device triggers a fault alarm, the think tank is called to supplement the alarm content; at the same time, the manufacturer's equipment manual is regularly uploaded to correct the AI model analysis accuracy and verification test to ensure that network logs can be analyzed accurately and efficiently and effective fault handling solutions can be provided, with comprehensive and efficient collection and processing of logs, precise fault analysis, and continuous optimization of models.

[0016] By parsing and sorting log data timestamps, and combining device location identifiers to identify the time series patterns of device failures, we conduct a fault-specific analysis of events within the time series patterns to identify their temporal distribution characteristics. We then associate the fault-specific upstream and downstream paths to divide the propagation paths. Based on the propagation paths and time series patterns, we construct a fault impact map, assess risks, and identify key impact nodes. Finally, we incorporate these key impact nodes into the AI big model for semantic analysis to predict fault development trends. This series of operations enables a comprehensive and in-depth analysis of the occurrence, propagation, and impact of network device failures, accurately locates key impact nodes, and predicts fault development trends in advance. This provides a more targeted and forward-looking basis for decision-making in network device troubleshooting and maintenance, effectively improving the efficiency and reliability of network device operation and maintenance.

[0017] By building a dynamic fault classification model, combined with multi-source data fusion and graph neural network technology, we achieve real-time fault classification, propagation path prediction, and risk assessment. The dynamic fault classification model dynamically adjusts classification criteria based on equipment operating status and environmental changes, improving classification accuracy. The graph neural network analyzes fault propagation paths, predicts the scope of fault impact, and calculates node risk scores based on risk assessment indicators, providing precise support for operation and maintenance decision-making. This solution significantly improves the real-time and accuracy of fault analysis, reduces operation and maintenance costs, and enhances network stability and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1The figure is a flowchart of an AI-based network log analysis and processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To facilitate understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains; the terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0021] Example 1: Figure 1 This is a flowchart of an AI-based network log analysis and processing method according to an embodiment of the present invention.

[0022] like Figure 1 As shown, an AI-based network log analysis and processing method includes the following steps: S1, collects log information of network devices through collection scripts that support multiple protocols.

[0023] Specifically, in S1a, identify the network devices for which logs need to be collected and the protocols they support (such as SNMP, SSH, and syslog). List all network devices for which logs need to be collected (such as routers, switches, and firewalls), carefully confirm the log protocol types supported by each device, and categorize the devices by protocol type.

[0024] S1b, write a collection script for each protocol type. The script can obtain log data by pulling (pull) or pushing (push) the device.

[0025] Pulling means the system periodically sends requests to network devices to obtain the latest log information. Pulling involves writing scripts that periodically send requests to devices (via SNMPGet or SSH commands). Pushing means network devices proactively push log information to the system when specific events occur. Pushing involves writing scripts that automatically push logs from receiving devices (Syslog servers). The scripts must support real-time log collection and storage and be capable of handling exceptions (such as device connection failures and log format errors).

[0026] In S1c, configure the log storage path and unified log format for the collection script, and set parameters such as the collection frequency as needed. When setting these parameters, consider factors such as log storage capacity planning, format specifications, and the impact of the collection frequency on system performance.

[0027] S1d: Deploy and start the collection script on the target server, begin collecting network device log information, and monitor the script's running status in real time to ensure its proper operation. During deployment, pay attention to the server's environmental configuration to ensure the script's proper operation. During monitoring, set appropriate monitoring indicators and alarm mechanisms to promptly detect and address script operation anomalies.

[0028] S2, performs preliminary classification and data supplement operations on the collected original logs.

[0029] Specifically, S2a performs a preliminary screening of the collected raw logs, removing duplicate, invalid, or redundant log information based on timestamps and log content. During this screening process, clear criteria must be established. For example, logs with identical timestamps and content are considered duplicates, while logs with content that clearly does not conform to normal business logic or is formatted incorrectly are considered invalid.

[0030] S2b: Initially categorize logs based on information such as manufacturer, type, device model, and software version, and assign corresponding tags. Detailed classification rules should be established to ensure that each log entry is accurately categorized. Labeling should be accurate and standardized to facilitate subsequent querying and analysis.

[0031] S2c: Supplement business information, such as the equipment's location and role, to better understand the log content during subsequent analysis. When supplementing business information, obtain accurate information from relevant business systems and correlate it with the logs.

[0032] S2d formats the initially processed log data to ensure it meets the AI model input requirements. This formatting includes unifying the log's time format and field format, and requires detailed processing based on the AI model's input specifications.

[0033] S3 sends the preliminarily processed log data to the AI big model for semantic analysis.

[0034] Specifically, S3a sends the preliminarily processed log data to the input port of the AI model. During the transmission process, to ensure the stability and integrity of data transmission, mechanisms such as data verification can be used.

[0035] In step S3b, the AI big model performs deep semantic understanding of log content and performs basic log classification (such as operation logs, fault logs, and status logs). This semantic understanding and classification process relies on extensive training data and advanced algorithms to improve classification accuracy.

[0036] In step 3c, the AI model performs secondary analysis on the classification results, correlating operation logs with fault logs to identify human errors or runtime status changes. This secondary analysis requires establishing reasonable correlation rules, such as through time correlation or event correlation, to accurately identify the cause of the fault and status changes.

[0037] In step 3d, the AI model provides treatment recommendations based on the analysis results and sends the analysis results and recommendations to the subsequent processing module. When providing treatment recommendations, historical data and expert experience are combined to ensure the feasibility and effectiveness of the recommendations.

[0038] S4, stores the analysis results of the AI model in the data think tank and conducts historical comparison.

[0039] Specifically, S4a stores the AI model's analysis results in a predefined format in a data think tank. A reasonable database structure should be established during storage to facilitate data query and management.

[0040] S4b, when storing new analysis results, first check whether there are analysis results of the same log in the think tank. When checking, use an efficient query algorithm to improve query efficiency.

[0041] S4c: If historical analysis results are available, compare the old and new analysis results to analyze the differences and fault causes. During this comparative analysis, carefully record the differences and analyze the fault causes based on the business situation and device status.

[0042] S4d: When the analysis results are inconsistent, the different analysis results are recorded and sent to the interface for manual confirmation. The integrity of the information must be ensured when recording so that it can be easily referenced during manual confirmation.

[0043] S4e: Manually confirm and correct inconsistent analysis results to ensure the final analysis results are correct and valid. Manual confirmation should combine professional knowledge and practical experience to accurately judge and correct the analysis results.

[0044] S5: When a network device triggers a fault alarm, the data think tank is called to supplement the alarm content.

[0045] Specifically, in S5a, when a network device triggers a fault alarm, the alarm center receives and processes the alarm information. The alarm center must have the ability to receive and process alarm information in real time to ensure timely processing of alarm information.

[0046] S5b: The alarm center calls the data intelligence library to search for fault information and analysis results related to the alarm information. A fast data retrieval mechanism should be established during the call to improve search efficiency.

[0047] S5c: Based on the found information and analysis results, the alarm content is supplemented, including the meaning of the fault, the cause of the fault, and the handling suggestions. When supplementing the alarm content, the accuracy and completeness of the information must be ensured to facilitate understanding by the operation and maintenance personnel.

[0048] S5d: Send the supplemented alarm information to the relevant operation and maintenance personnel so that they can quickly understand the fault situation and take appropriate measures. When sending the alarm, choose an appropriate communication method to ensure that the operation and maintenance personnel can receive the alarm information in time.

[0049] S6, regularly upload equipment-related manuals provided by manufacturers to correct the analysis accuracy of the AI model.

[0050] Specifically, S6a regularly collects equipment-related manuals provided by manufacturers, including technical specifications, troubleshooting solutions, etc. During this collection process, a good communication mechanism must be established with the manufacturer to ensure the timeliness and accuracy of the information.

[0051] S6b: Organize and analyze the collected manuals to extract key information relevant to AI model analysis. Scientific methods should be used during the organization and analysis to ensure that the extracted key information is accurate and effective.

[0052] In step S6c, the extracted key information is input into the AI model to correct and optimize the model's analytical accuracy. When inputting key information, the AI model's input specifications must be followed to ensure the model can correctly receive and process the information.

[0053] S6d: Verify and test the revised AI model to ensure it can accurately and efficiently analyze network logs and provide effective troubleshooting solutions. Verification and testing should utilize comprehensive test cases to evaluate the model's performance under various circumstances.

[0054] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By supporting multiple protocol collection scripts to collect network device log information, first determine the device and protocol type and classify them, then write the corresponding protocol collection script, configure the parameters, and then deploy and start it; then perform preliminary classification, data supplementation, and formatting on the collected original logs; then send the processed log data to the AI big model for semantic analysis, including deep understanding, basic classification, secondary analysis, and providing processing suggestions; the analysis results are then stored in the data think tank for historical comparison, and manual confirmation and correction of inconsistent results; when the network device triggers a fault alarm, the think tank is called to supplement the alarm content; at the same time, the manufacturer's equipment manual is regularly uploaded to correct the AI model analysis accuracy and verification test to ensure that network logs can be analyzed accurately and efficiently and effective fault handling solutions can be provided, with comprehensive and efficient collection and processing of logs, precise fault analysis, and continuous optimization of models.

[0055] Example 2: In Example 1, the log data after preliminary processing is sent to the AI big model for semantic analysis. Although the AI big model can perform basic classification and correlation analysis on the logs and provide treatment solution recommendations, it is insufficient to deeply explore the propagation impact, timing pattern and key influencing nodes of equipment failures in the network based solely on the semantic analysis of the logs. Network equipment failures often have complex propagation paths and timing characteristics. The correlation between different failures and the degree of impact of the failure on equipment and business functions vary. Relying solely on the conventional semantic analysis of logs by the AI big model is difficult to fully and accurately grasp the full picture and potential risks of the failure. This may lead to inaccurate predictions of the development trend of the failure and untimely identification of key influencing nodes, thereby affecting the efficiency and effectiveness of fault handling. In order to more deeply analyze the timing pattern, propagation path and impact range of network equipment failures, accurately identify key influencing nodes, and effectively predict the development trend of the failure, so as to take more targeted fault handling measures, further optimization and improvement of Example 1 is needed.

[0056] In some embodiments, the preliminarily processed log data is sent to the AI big model for semantic analysis, and step S3 further includes: S31, divide the timing pattern of device failure occurrence according to the timestamp of the log data based on the location of the device in the network.

[0057] The timestamp of the log data is used to arrange the log data in chronological order or in the order of fault occurrence.

[0058] Specifically, S31a parses the timestamps in the log data and converts them into a unified time format.

[0059] S31b, sorting the log data according to the timestamps to facilitate subsequent analysis of the time sequence of device failures.

[0060] S31c, add the location identifier of the device in the network (such as the device's IP address, MAC address, or device number, etc.) to each log record to clarify the device source of the log record.

[0061] S31d: Traverse the sorted log data and identify the time when the device failure occurred based on log records related to the device failure (such as logs containing failure keywords and error codes). The system analyzes the time intervals and sequence of failures across different devices, classifying failure events with similar temporal distribution characteristics as belonging to the same temporal pattern. For example, if multiple devices experience the same type of failure within a short period of time (e.g., network connection interruptions within 10 minutes), these failure events can be classified as belonging to the same temporal pattern.

[0062] S32, performing a fault-specific analysis on each type of event description in the time series pattern to identify the time distribution characteristics of the fault-specific event.

[0063] Specifically, S32a further analyzes the log records in the time series pattern and classifies the event descriptions into different fault special items according to the log contents (such as keywords and error descriptions in the logs).

[0064] S32b, for each fault item, statistics such as the time point of the fault occurrence, the time interval (such as the time difference between two adjacent faults), and the duration (such as the length of time from the start to the end of a fault).

[0065] S32c analyzes the frequency of specific faults during different time periods (e.g., weekdays, weekends, and specific time periods such as peak business hours). It then plots time distribution charts (e.g., bar charts or line graphs) to visually display the temporal distribution characteristics of these faults. Based on historical data and the temporal distribution characteristics of normal operating conditions, thresholds (e.g., fault frequency thresholds, time interval thresholds, etc.) are set.

[0066] S32d, detect whether there is any abnormality in the time distribution of the current fault item, such as a sudden increase or decrease in the frequency of faults in a certain time period, and determine whether there is a potential problem or risk.

[0067] Among them, fault items are different fault types, such as hardware failure, network congestion, configuration error, and interface failure.

[0068] S33 associates the upstream and downstream of the fault project and divides the fault propagation path.

[0069] Specifically, S33a analyzes the causal and correlation relationships between different fault types. For example, a hardware failure may cause software to malfunction, which in turn leads to a network failure. By examining the event sequence (e.g., the chronological order of fault occurrence), dependencies (e.g., the operation of one service depends on the normal operation of another service), and related error codes in the log data, the upstream and downstream relationships between fault types can be determined.

[0070] S33b: Based on the upstream and downstream relationships of the fault-specific items, a fault propagation chain is constructed. Starting from the initial fault source, the fault propagation path is gradually sorted along the propagation chain to clarify how the fault propagates from one device or component to another.

[0071] S33c, use a topology map to visualize the fault propagation path, and clearly mark the direction of fault propagation and the relationship between each fault item in the topology map.

[0072] S34, based on the fault propagation path and timing pattern, build a fault impact range map.

[0073] Specifically, S34a combines the fault propagation path and timing pattern to analyze the fault's impact on various devices and business functions at different time points. This identifies directly affected devices (such as the fault source device and directly associated devices, which can be determined through the fault propagation chain) and indirectly affected business functions (such as business systems that become inaccessible due to the device failure, which can be determined based on business dependencies).

[0074] S34b, with devices and business functions as nodes and fault propagation relationships as edges, constructs a fault impact range map.

[0075] S34c, annotate each node in the graph with information such as its importance (e.g., assessed based on business criticality, equipment value, etc.), its degree of impact (e.g., business interruption duration and data loss caused by the fault), and the direction and intensity of fault propagation (e.g., propagation speed, impact range, etc.).

[0076] S35: Conduct risk assessment on the fault impact range map to determine key impact nodes.

[0077] Specifically, S35a determines indicators for assessing node risk, such as business importance, failure impact, recovery difficulty, and failure frequency.

[0078] S35b, assign a corresponding weight to each indicator and adjust the weight value according to business needs and actual conditions (for example, for nodes related to core business, the weight of the business importance indicator can be set higher).

[0079] S35c, for each node in the fault impact range map, calculate the risk score of the node according to the risk assessment index and weight (the weighted average method can be used for calculation).

[0080] S35d, sorting the nodes according to the risk scores and setting a risk threshold (eg, determined based on historical data and acceptable risk levels for the business).

[0081] S35e, determining nodes whose risk scores exceed a threshold as key impact nodes.

[0082] S35, adds key influencing nodes to the AI big model for semantic analysis to predict the fault development trend of key influencing nodes.

[0083] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By parsing and sorting log data timestamps, and combining device location identifiers to identify the time series patterns of device failures, we conduct a fault-specific analysis of events within the time series patterns to identify their temporal distribution characteristics. We then associate the fault-specific upstream and downstream paths to divide the propagation paths. Based on the propagation paths and time series patterns, we construct a fault impact map, assess risks, and identify key impact nodes. Finally, we incorporate these key impact nodes into the AI big model for semantic analysis to predict fault development trends. This series of operations enables a comprehensive and in-depth analysis of the occurrence, propagation, and impact of network device failures, accurately locates key impact nodes, and predicts fault development trends in advance. This provides a more targeted and forward-looking basis for decision-making in network device troubleshooting and maintenance, effectively improving the efficiency and reliability of network device operation and maintenance.

[0084] Example 3: In Example 2, equipment faults are classified by constructing a dynamic fault-specific classification model, and based on the classification results, fault propagation paths are divided, impact range maps are constructed, and risk assessments are performed. However, there are obvious shortcomings in using only a single dynamic fault-specific classification model based on log data. The generation and propagation of equipment faults is a complex process involving many factors. Although log data can provide certain fault information, multi-source data such as network traffic data and equipment performance indicator data also contain a large amount of key information related to faults. This information plays an important role in accurately classifying faults, deeply understanding the fault propagation mechanism, and evaluating the scope of fault impact. It is difficult for a single classification model to fully integrate and utilize these multi-source data, resulting in limited classification accuracy in complex and changeable equipment operating environments, which in turn affects the reliability of fault propagation path prediction and risk assessment. In order to analyze equipment faults more comprehensively and accurately, improve the accuracy of fault classification and the reliability of subsequent analysis, it is necessary to comprehensively consider multi-source data and further optimize and improve the fault-specific classification method.

[0085] In some embodiments, in step S33, the upstream and downstream of the fault-specific items are associated to divide the fault propagation path, further comprising: S331, using machine learning algorithms to build a dynamic fault special classification model, and inputting the log data after upstream and downstream association of the dynamic fault special into the dynamic classification model.

[0086] Dynamic fault classification refers to fault categories with specific characteristics and implications, derived from the classification and identification of faults during equipment operation using intelligent algorithms such as machine learning, based on real-time data, environmental changes, and the dynamic evolution of equipment status. These fault categories are not fixed but are dynamically adjusted and optimized based on equipment operating conditions, updated fault characteristics, and continuous algorithm learning.

[0087] S332, the model classifies log records into different dynamic fault special items in real time according to the learned rules.

[0088] Specifically, for each newly input log record, the model calculates the probability that the log belongs to each dynamic fault category based on the classification rules learned during the training phase. For example, when using a deep learning model, the scores of each category in the output layer are calculated through forward propagation, and then the softmax function is used to convert these scores into a probability distribution:

[0089] in, Represents log records Belong to category The probability of is the score of the i-th category in the model output layer, K is the total number of categories, and the category with the highest probability is selected as the dynamic fault special classification result of the log record.

[0090] S333, collect network traffic data and device performance indicator data as multi-source data.

[0091] S334 uses data fusion technology to associate and integrate multi-source data.

[0092] For example, through timestamp alignment, device ID matching, and other methods, dynamic fault data can be integrated with network traffic data, device performance data, etc. to build a unified data view.

[0093] S335, analyze the correlation between different dynamic fault special projects, the time series pattern of propagation and the key node characteristics on the propagation path.

[0094] S336, select the graph neural network to predict the fault propagation path after the dynamic fault specialization. When a new fault occurs, the relevant information of the fault is input into the trained prediction model.

[0095] The relevant information of the fault includes the fault type, occurrence time, involved equipment, etc. The model should output the possible propagation path and propagation probability of the fault, for example:

[0096] G is the graph structure, X is the feature matrix of nodes and edges, W is the parameter of the model, and f is the prediction function of the graph neural network.

[0097] S337, based on the predicted fault propagation path, calculates the risk score of each node in the fault impact range map, integrates the information of key impact nodes to form a feature vector, and inputs it into the AI big model for semantic analysis.

[0098] Specifically, determine the risk assessment indicators (business importance, failure impact, recovery difficulty, and failure frequency), assign weights to each indicator, and calculate the node's risk score.

[0099] Risk score formula:

[0100] is the weight of the i-th indicator, is the score of the i-th indicator. The risk scores of key influencing nodes, fault special items, propagation paths and other information are integrated into a feature vector, and the feature vector is input into the AI big model for semantic analysis.

[0101] Among them, the path characteristics are path length, propagation time and propagation direction.

[0102] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By building a dynamic fault classification model, combined with multi-source data fusion and graph neural network technology, we achieve real-time fault classification, propagation path prediction, and risk assessment. The dynamic fault classification model dynamically adjusts classification criteria based on equipment operating status and environmental changes, improving classification accuracy. The graph neural network analyzes fault propagation paths, predicts the scope of fault impact, and calculates node risk scores based on risk assessment indicators, providing precise support for operation and maintenance decision-making. This solution significantly improves the real-time and accuracy of fault analysis, reduces operation and maintenance costs, and enhances network stability and reliability.

[0103] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A network log analysis and processing method based on AI, characterized in that: include: S1, collects log information of network devices through collection scripts that support multiple protocols; S2, performs preliminary classification and data supplement operations on the collected original logs; S3 sends the preliminarily processed log data to the AI big model for semantic analysis; S4, stores the analysis results of the AI model in the data think tank and performs historical comparison; S5: When a network device triggers a fault alarm, the data intelligence library is called to supplement the alarm content. S6, regularly upload equipment-related manuals provided by manufacturers to correct the analysis accuracy of the AI model.

2. The AI-based network log analysis and processing method according to claim 1, characterized in that: Said S3 specifically includes: S3a sends the preliminarily processed log data to the input port of the AI large model; S3b: The AI model performs deep semantic understanding of log content and performs basic classification of logs. In step 3c, the AI model performs secondary analysis on the classification results, correlating operation logs with fault logs to identify human factors or runtime status changes. S3d, the AI model provides treatment solution recommendations based on the analysis results.

3. The AI-based network log analysis and processing method according to claim 1, characterized in that: Said S4 specifically includes: S4a, stores the analysis results of the AI model in the data think tank according to the predetermined format; S4b, when storing new analysis results, first check whether there are analysis results of the same log in the think tank; S4c, if there are existing historical analysis results, compare the old and new analysis results to analyze the differences and fault causes; S4d, when the analysis results are inconsistent, the different analysis results are recorded and sent to the interface for manual confirmation for confirmation; S4e, manually confirm and correct inconsistent analysis results.

4. The AI-based network log analysis and processing method according to claim 1, wherein: Said S3 further includes: S31, dividing the timing pattern of device failure occurrence according to the timestamp of the log data based on the location of the device in the network; S32, performing fault-specific analysis on each event description in the time series pattern to identify the time distribution characteristics of the fault-specific event; S33, associate the upstream and downstream of the fault project and divide the fault propagation path; S34, constructs a fault impact range map based on the fault propagation path and timing pattern; S35, adds key influencing nodes to the AI big model for semantic analysis to predict the fault development trend of key influencing nodes.

5. The AI-based network log analysis and processing method according to claim 4, characterized in that: The S33 specifically includes: S33a, analyzing the causal and correlation relationships between different fault items; S33b, builds a fault propagation chain based on the upstream and downstream association relationships of the fault items; S33c, use a topology map to visualize the fault propagation path, and clearly mark the direction of fault propagation and the relationship between each fault item in the topology map.

6. The AI-based network log analysis and processing method according to claim 4, characterized in that: Said S34 comprises: S34a, combining the fault propagation path and timing pattern, analyzes the impact of the fault on various devices and business functions at different time points; S34b, using devices and business functions as nodes and fault propagation relationships as edges, constructs a fault impact range map; S34c, mark the importance, impact degree, and fault propagation direction and intensity information of each node in the graph.

7. The AI-based network log analysis and processing method according to claim 4, characterized in that: In the S33, it also includes: S331, using a machine learning algorithm to build a dynamic fault special classification model, and inputting the log data after upstream and downstream association of the dynamic fault special into the dynamic classification model; S332, the model classifies log records into different dynamic fault categories in real time based on the learned rules; S333, collecting network traffic data and device performance indicator data as multi-source data; S334, uses data fusion technology to correlate and integrate multi-source data; S335, analyzes the correlation between different dynamic fault special projects, the time series pattern of propagation, and the key node characteristics on the propagation path; S336, select the graph neural network to predict the fault propagation path after the dynamic fault special project. When a new fault occurs, the relevant information of the fault is input into the trained prediction model; among which, the relevant information of the fault is the fault special project, occurrence time, involved equipment, etc. The model should output the possible propagation path and propagation probability of the fault.

8. The AI-based network log analysis and processing method according to claim 7, characterized in that: S332 specifically includes: for each newly input log record, the model calculates the probability that the log belongs to each dynamic fault special item based on the classification rules learned in the training phase; when using a deep learning model, the scores of each category of the output layer are calculated through forward propagation, and the softmax function is used to convert these scores into a probability distribution: ;in, Represents log records Belong to category The probability of is the score of the i-th category in the model output layer, and K is the total number of categories.

9. The AI-based network log analysis and processing method according to claim 7, characterized in that: In S336, the relevant information of the fault is represented as follows: , G is the graph structure, X is the feature matrix of nodes and edges, W is the parameter of the model, and f is the prediction function of the graph neural network.

10. The AI-based network log analysis and processing method according to claim 7, characterized in that: The risk score in S337 specifically includes: determining risk assessment indicators, assigning weights to each indicator, and calculating the risk score of the node; the risk score formula is: , is the weight of the i-th indicator, is the score of the i-th indicator. The risk scores of key influencing nodes, fault special items, propagation paths and other information are integrated into a feature vector, and the feature vector is input into the AI big model for semantic analysis.

Images