Network log intelligent research and judgment and automatic response system and method based on large model fusion

Abstract

The application relates to the technical field of data processing, in particular to a network log intelligent research and judgment and automatic response system and method fusing a large model, the method comprising the following steps: acquiring a to-be-processed log; preprocessing the to-be-processed log to obtain a to-be-analyzed log; performing correlation analysis by using a global correlation knowledge graph in a preset analysis model to obtain a first analysis result; in the case that the first analysis result comprises an abnormal event, performing event accurate analysis by using an event diagnosis model in the preset analysis model to obtain a second analysis result; in the case that the first analysis result and the second analysis result both comprise abnormal events, determining an abnormal level of the abnormal events according to a preset event grading strategy; and generating an abnormal processing strategy according to the abnormal level of the abnormal events. The manual feature engineering link can be saved, and the analysis efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a system and method for intelligent analysis and automated response of network logs that integrates large models. Background Technology

[0002] As network scale expands and business complexity increases, network devices (routers, switches, firewalls, etc.), servers, and application systems generate massive amounts of network logs. These logs contain critical information such as device operating status, user operation behavior, and data transmission records, serving as the core basis for network security protection, troubleshooting, and operation and maintenance management.

[0003] Currently, traditional log analysis relies on rule engines, requiring manual pre-setting of a large number of rules, such as multiple failed login attempts = package breakdown. Faced with a log volume of 100,000 logs per second, the rule matching speed is slow and cannot cover unknown anomalies, resulting in a high frequency of missed and false judgments and low log analysis efficiency. Therefore, there is an urgent need for a network log intelligent judgment and automated response system and method that integrates large models. Summary of the Invention

[0004] This application provides a network log intelligent analysis and automated response system and method that integrates large models, which can save manual feature engineering and improve analysis efficiency.

[0005] On the one hand, embodiments of this application provide a method for intelligent analysis and automated response of network logs that integrates large models, the method including:

[0006] Get the logs to be processed;

[0007] The logs to be processed are preprocessed to obtain the logs to be analyzed;

[0008] The first analysis result is obtained by performing association analysis using the global association knowledge graph in the preset analysis model.

[0009] If the first analysis result includes the presence of abnormal events, the event diagnosis model in the preset analysis model is used to perform precise event analysis to obtain the second analysis result.

[0010] If both the first analysis result and the second analysis result contain abnormal events, the abnormality level of the abnormal event is determined according to the preset event classification strategy.

[0011] An exception handling strategy is generated based on the exception level of the abnormal event.

[0012] Optionally, the preprocessing of the log to be processed to obtain the log to be analyzed includes:

[0013] The log to be processed is semantically parsed using the semantic analysis network in the preset analysis model to obtain multiple key fields;

[0014] The multiple key fields are formatted in a unified manner to obtain the key format corresponding to each key field;

[0015] And / or, use the semantic analysis network in the preset analysis model to filter noise from the logs to be processed to obtain valid logs;

[0016] The key format and / or the valid logs are identified as the logs to be analyzed.

[0017] Optionally, after determining the key format and / or the valid log as the log to be analyzed, the process includes:

[0018] The log to be analyzed is compressed using a preset compression algorithm to obtain the compressed log to be analyzed.

[0019] Optionally, before performing association analysis using the global association knowledge graph in the preset analysis model to obtain the first analysis result, the method further includes:

[0020] Retrieve historical training logs;

[0021] Based on the historical training logs, an initial global relational knowledge graph is constructed;

[0022] Feature extraction is performed on the historical training logs to obtain multidimensional correlation features;

[0023] Multiple multidimensional association features are associated according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph.

[0024] Optionally, after associating multiple multidimensional association features according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph, the method further includes:

[0025] Retrieve historical exception logs and corresponding historical exception events;

[0026] The historical anomaly logs are input into the global relational knowledge graph to obtain predicted anomaly events;

[0027] Based on the predicted abnormal events and the historical abnormal events, the weights of the global association knowledge graph are adjusted until the predicted abnormal events and the historical abnormal events meet the training conditions.

[0028] Optionally, the event diagnosis model includes an abnormal propagation chain, and the step of performing precise event analysis using the event diagnosis model in the preset analysis model to obtain a second analysis result includes:

[0029] In the global association knowledge graph, the initial abnormal node corresponding to the abnormal event to be judged is determined;

[0030] In the event diagnosis model, a preset anomaly propagation chain is used to trace back to the anomaly source based on the initial anomaly node to obtain the anomaly cause corresponding to the anomaly event to be determined. The anomaly propagation chain is used to characterize the propagation link of the initial anomaly node in the global association knowledge graph.

[0031] Based on the stated cause of the anomaly and the anomaly event to be determined, the anomaly event is identified and identified as the second analysis result.

[0032] Optionally, generating an exception handling strategy based on the exception level of the abnormal event includes:

[0033] Obtain the event level and event type corresponding to the abnormal event;

[0034] Based on the event level and event type, an exception handling strategy is generated according to a preset processing strategy.

[0035] On the other hand, embodiments of this application provide a network log intelligent analysis and automated response system that integrates large models, the system comprising:

[0036] The acquisition module is used to acquire logs to be processed.

[0037] The preprocessing module is used to preprocess the logs to be processed to obtain the logs to be analyzed.

[0038] The association analysis module is used to perform association analysis using the global association knowledge graph in the preset analysis model to obtain the first analysis result.

[0039] The precision analysis module is used to perform precise event analysis using the event diagnosis model in the preset analysis model when the analysis results include the presence of abnormal events, and to obtain a second analysis result.

[0040] The determination module is used to determine the abnormality level of the abnormal event according to a preset event classification strategy when both the first analysis result and the second analysis result contain abnormal events.

[0041] The generation module is used to generate an exception handling strategy based on the exception level of the abnormal event.

[0042] Optionally, the system further includes:

[0043] The automated execution module includes an execution engine and a status monitoring unit. The execution engine supports multiple execution modes, and the status monitoring unit monitors the execution status of actions and automatically retryes or triggers manual intervention when execution fails.

[0044] Optionally, the system further includes:

[0045] The storage module is used to store logs to be analyzed, time-loop logs, and model parameters and version information;

[0046] The visualization interaction module includes a report generation unit and a manual intervention unit. The report generation unit is used to generate and export visualization reports, and the manual intervention unit is used to provide a manual review interface to support administrators in viewing abnormal events, modifying abnormal handling strategies, and manually executing handling actions.

[0047] In another aspect, embodiments of this application provide an electronic device, the device including: a processor and a memory storing computer program instructions;

[0048] When the processor executes the computer program instructions, it implements the network log intelligent analysis and automated response method of the fusion large model as described in the first aspect.

[0049] In another aspect, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the network log intelligent analysis and automated response method of the fusion large model as described in the first aspect.

[0050] The network log intelligent analysis and automated response system and method based on the integrated large model of this application can perform log processing analysis for the first time through the global association knowledge graph of the preset analysis model, automatically extract log features, and then, if there are abnormal events in the first analysis result, that is, if the log is abnormal at this time, the event diagnosis model in the preset analysis model is used to perform a second precise analysis to obtain a second analysis result. In the process of precise analysis, complex supply chains can be identified, improving the accuracy of event analysis. After determining that the log is an abnormal event, the abnormality level of the abnormal event can be determined according to the preset event classification strategy, so as to carry out targeted abnormality handling. The system realizes a closed loop of automatic collection, automatic analysis and abnormality handling, which not only reduces labor costs, but also reduces human subjectivity and improves the efficiency and accuracy of log processing. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a method for intelligent analysis and automated response of network logs that integrates large models, as provided in an embodiment of this application.

[0052] Figure 2 This is a schematic diagram of the structure of a network log intelligent analysis and automated response system that integrates a large model, as provided in an embodiment of this application.

[0053] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0056] To address the problems in existing technologies, this application provides a system and method for intelligent analysis and automated response of network logs that integrates a large-scale model. In this application, log processing and analysis are initially performed using a global relational knowledge graph of a preset analysis model, automatically extracting log features. Then, if an anomaly is detected in the first analysis result (i.e., the log is now abnormal), a second, more precise analysis is performed using an event diagnosis model within the preset analysis model to obtain a second analysis result. During this precise analysis, complex supply chains can be identified, improving the accuracy of event analysis. After determining that the log is an anomaly, the anomaly level can be determined according to a preset event grading strategy, allowing for targeted anomaly handling. The system achieves a closed loop of automatic data collection, automatic analysis, and anomaly handling, reducing not only manual costs but also the subjectivity of human intervention, thus improving the efficiency and accuracy of log processing.

[0057] The following section first introduces the intelligent analysis and automated response method for network logs of the fusion large model provided in the embodiments of this application.

[0058] Figure 1 This illustration shows a flowchart of a network log intelligent analysis and automated response method based on an embodiment of this application. Figure 1 As shown, the intelligent analysis and automated response method for network logs that integrates large models can include S101-S106:

[0059] S101, Get the logs to be processed.

[0060] In this embodiment of the application, in order to obtain all network logs within the system, distributed log collection nodes can be deployed in the system. Specifically, a collection agent based on Flink can be used for collection, which can cover all log sources of network devices such as routers and firewalls, servers such as web servers and database servers, and application systems such as OA systems and transaction systems.

[0061] Specifically, during the process of acquiring logs to be processed, two modes can be set on the acquisition node: real-time capture and timed capture. The interval for real-time capture can be configured to 1 second, while the interval for timed capture can be customized.

[0062] S102, preprocess the logs to be processed to obtain the logs to be analyzed.

[0063] In some embodiments, to ensure accurate analysis of the logs to be processed, preprocessing is required to facilitate accurate subsequent analysis. Preprocessing may include format standardization, noise filtering, and data compression.

[0064] S103, use the global association knowledge graph in the preset analysis model to perform association analysis and obtain the first analysis result.

[0065] In this embodiment of the application, a global association analysis of the full preprocessed logs can be performed across devices and types using a global association knowledge graph to initially locate suspected abnormal events. That is, by performing cross-device and cross-type semantic understanding of multi-source logs, it is convenient to explore whether the log is abnormal from multiple dimensions and improve accuracy.

[0066] S104, if the first analysis result includes the existence of abnormal events, the event diagnosis model in the preset analysis model is used to perform precise event analysis to obtain the second analysis result.

[0067] In this embodiment of the application, when the log may contain abnormal events by using a global association knowledge graph, an event diagnosis model can be used for further judgment. If the second analysis result obtained by the event diagnosis model also contains abnormal events, it can be determined that the log to be analyzed is abnormal. The abnormality level of the abnormal event can be determined according to the preset event grading strategy, so as to determine the degree of harm of the log to the system according to different abnormality levels, which facilitates the generation of different processing strategies and optimizes resource utilization.

[0068] S105, if both the first and second analysis results contain abnormal events, determine the abnormality level of the abnormal events according to the preset event grading strategy.

[0069] S106, Generate an exception handling strategy based on the exception level of the exception event.

[0070] In this embodiment, log processing analysis can be performed for the first time using a global association knowledge graph of a preset analysis model to automatically extract log features. Then, if there are abnormal events in the first analysis result, i.e., the log is abnormal at this time, the event diagnosis model in the preset analysis model is used to perform a second precise analysis to obtain a second analysis result. During the precise analysis process, complex supply chains can be identified, improving the accuracy of event judgment. After determining that the log is an abnormal event, the abnormality level of the abnormal event can be determined according to the preset event grading strategy, so as to carry out targeted abnormality handling. The system realizes a closed loop of automatic collection, automatic judgment and abnormality handling, which not only reduces labor costs, but also reduces human subjectivity and improves the efficiency and accuracy of log processing.

[0071] In some embodiments, S102 may include:

[0072] The semantic analysis network in the preset analysis model is used to perform semantic parsing on the log to be processed, and multiple key fields are obtained.

[0073] The multiple key fields are formatted in a unified way to obtain the key format corresponding to each key field;

[0074] And / or, use the semantic analysis network in the preset analysis model to filter noise from the logs to be processed to obtain valid logs;

[0075] Identify the key format and / or valid logs as the logs to be analyzed.

[0076] In this embodiment, during the process of standardizing the format of the logs to be processed, the semantic analysis model in the preset analysis model can be used to perform semantic parsing on the logs to be processed to obtain multiple key fields. The key fields may include timestamp, device ID, event type and IP address. Then, they are uniformly formatted to obtain the key format corresponding to each key field. The key format may be Parquet format.

[0077] In this embodiment, while standardizing the format or converting it into a critical format, the contextual understanding capability of the preset analysis model can be used to identify and filter noise. The noise can be duplicate logs, test logs, or logs with missing fields, so as to cause information redundancy and interfere with the subsequent analysis of the logs.

[0078] Specifically, the pre-defined analysis model used for the preprocessing described above can be the lightweight semantic understanding model BERT-Tiny. Format standardization and noise filtering can reuse the model's semantic parsing capabilities and reduce the cost of cross-module model calls.

[0079] In other embodiments, to reduce storage costs, after identifying key format and / or valid logs as the logs to be analyzed, the following steps are included:

[0080] The log to be analyzed is compressed using a preset compression algorithm to obtain the compressed log.

[0081] In this embodiment, the Snappy compression algorithm can be used to compress the preprocessed logs, reducing storage costs.

[0082] In other embodiments, after the above preprocessing of the logs to be processed, in order to more clearly analyze the logs to be analyzed, the method may further include the following before S103:

[0083] Retrieve historical training logs;

[0084] Based on historical training logs, an initial global relational knowledge graph is constructed.

[0085] Feature extraction is performed on historical training logs to obtain multidimensional correlation features;

[0086] Multiple multidimensional association features are associated according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph.

[0087] Specifically, in order to make more accurate judgments on the logs to be analyzed, it is necessary to train the global relational knowledge graph in the preset analysis model so that the preset analysis model can analyze the logs to be analyzed more accurately.

[0088] The specific training process is as follows: historical training logs can be used as training samples. The historical training logs can include data entities and the historical logs uploaded by each entity and their corresponding event types. Entities can be users, IP addresses, servers, application terminals, etc., and event types can include login, access, and execution.

[0089] Then, multi-dimensional association features are constructed based on historical training logs. Specifically, a pre-defined graph database is used as the underlying storage, with entities as nodes and events or interactions between entities as edges. The edges can be accompanied by attribute information, excluding timestamps, source logs, and confidence levels, and can form relationships such as user-login time-host-IP address. As an example, the relationship edges between entities include, but are not limited to, access relationships, membership relationships, dependency relationships, and geographical location relationships. In the process of constructing a global association knowledge graph, each relationship edge can be assigned a weight attribute, which can include interaction frequency, time density, and business importance.

[0090] In this embodiment, the multidimensional key features may include topological features, behavioral sequence features, and statistical features. Topological features may include degree centrality features, betweenness centrality features, and community detection features. Specifically, degree centrality can be used to identify densely connected key nodes, betweenness centrality features are used to discover bridge nodes in the network, and community detection features can be used to automatically cluster groups of entities with similar behavioral patterns. Behavioral sequence features can be constructed by extracting cross-device behavioral sequence patterns of entities, calculating the behavioral transition probability matrix, and establishing a temporal dependency model. These behavioral sequence features can be extracted through the model. Statistical features can be obtained by statistically analyzing the activity frequency within a time window, calculating the entropy of behavioral distribution, and extracting outlier detection features. All of the above features can be implemented using relevant technologies, which will not be elaborated further here.

[0091] After obtaining the multidimensional association features, the multiple multidimensional association features are then associated according to the preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph. Specifically, in this embodiment, the preset association rules may include lateral movement detection rules, privilege escalation pattern recognition, and data leakage pattern matching. The lateral movement detection rules are used to detect abnormal jumps between different devices for the same account, the privilege escalation pattern is used to identify abnormal sequences from normal access to privileged operations, and the data leakage pattern matching can be used for behavioral combinations of immediate external transmission after reading a large amount of data.

[0092] In this embodiment, nodes and edges can be associated using the aforementioned preset association rules to achieve global association analysis across devices and types.

[0093] The trained global relational knowledge graph can identify multi-source logs such as abnormal IP access to the database, firewall blocking, and account logins from different locations, thus initially locating suspected abnormal events. This helps narrow down the scope of anomalies from massive log volumes, enabling faster identification of abnormal logs.

[0094] In other embodiments, to enable the global association knowledge graph in the preset analysis model to more accurately identify associations, after associating multiple multidimensional association features according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph, the method further includes:

[0095] Retrieve historical exception logs and corresponding historical exception events;

[0096] Historical anomaly logs are input into a global relational knowledge graph to obtain predicted anomaly events;

[0097] Based on predicted and historical anomalous events, the weights of the global relational knowledge graph are adjusted until the predicted and historical anomalous events meet the training conditions.

[0098] Specifically, the connections between entities and relational edges can be modified by changing the weight values, so that the global relational knowledge graph can be used to analyze logs more accurately.

[0099] In some other embodiments, to ensure the accuracy of log analysis, the event diagnosis model may include an anomaly propagation chain, and S104 may include:

[0100] In the global relational knowledge graph, identify the initial abnormal node corresponding to the abnormal event to be judged;

[0101] In the event diagnosis model, a pre-set anomaly propagation chain is used to trace back to the source of the anomaly based on the initial anomaly node to obtain the anomaly cause corresponding to the anomaly event to be determined.

[0102] Based on the cause of the anomaly and the anomaly event to be determined, the anomaly event is identified and designated as the second analysis result.

[0103] In this embodiment, the log may contain abnormal events, which are identified by the global association knowledge graph. Therefore, in order to accurately determine whether there are abnormal events, an event diagnosis model can be used for further identification. The event diagnosis model can be an attack identification sub-model or a fault diagnosis sub-model. The training of the event diagnosis model can be based on incremental learning of historical labeled log samples, which can include normal logs and abnormal logs.

[0104] In this embodiment, the event diagnosis model can distinguish attack characteristics and identify the specific type of attack. For example, attack characteristics can include false triggering of interception and malicious attacks, and specific attack types can include ransomware attacks and brute-force attacks.

[0105] In other embodiments, when the global relational knowledge graph identifies an abnormal event, a preset abnormal propagation chain can be used to trace back to the source of the abnormality based on the initial abnormal node to obtain the cause of the abnormality corresponding to the abnormal event to be determined. The abnormal propagation chain is used to characterize the propagation path of the initial abnormal node in the global relational knowledge graph. In this embodiment, the global relational knowledge graph can be used to start from the initial abnormal node and perform a breadth-first search along the relation edges to evaluate the possible propagation path and range of the abnormality, calculate the risk level of the affected entity, and then trace back to the origin node of the abnormal behavior according to the preset abnormal propagation chain to identify the key turning point in the abnormal propagation chain and locate the entity most likely to be abnormal in order to determine the cause of the abnormality. Then, the abnormal cause and the abnormal event to be determined are used to determine the abnormal event so as to determine the corresponding abnormality handling strategy.

[0106] In some embodiments, a scoring algorithm can be deployed in the preset analysis model to fuse and score multi-dimensional features, and a weighted scoring algorithm can be used to calculate the overall association confidence, identify the credibility of the cause of the anomaly, reduce false alarms, and improve the accuracy of anomaly log analysis. It is worth noting that the weighted summation can be based on the above-mentioned multi-dimensional features, and any techniques disclosed in the relevant technologies can be used, without any limitation here.

[0107] In other embodiments, in S105, events can be classified according to four levels: "severe-high-medium-low". The specific classification strategy can adopt a dual mechanism of "automatic model learning + manual customization". That is, the preset analysis model can automatically learn the classification logic by combining historical handling results. For example, based on historical cases, "attack events corresponding to data transmission logs" can be classified as severe and "suspected violations corresponding to a single device login failure" can be classified as low-risk. In order to ensure the adaptability of classification, manual customization according to industry scenarios can be supported. For example, in the financial industry, "suspected violations corresponding to abnormal transaction amount logs" can be classified as severe.

[0108] In this embodiment, if the first analysis result of the global association knowledge graph is inconsistent with the second analysis result of the event diagnosis model, a verification mechanism can be triggered. This mechanism involves cross-verification by combining the handling results of similar historical events with business compliance requirements (such as whether it affects the operation of core businesses). If the verification still cannot reach a unified conclusion, a manual review process is triggered. The administrator will combine the actual situation of the scenario (such as whether the IP is a legitimate IP of the partner and whether the access behavior has been reported by the business) to finally confirm the event type and level, ensuring that the classification result accurately matches the generation logic of the subsequent execution strategy (for example, if it is determined to be a "normal access event", there is no need to generate a handling strategy; if it is determined to be a "confirmed attack event", it can be handled according to the abnormal level determined by the system).

[0109] In other embodiments, different exception handling strategies can be set for different exception levels to optimize processing resources. Specifically, S106 may include:

[0110] Get the event level and event type corresponding to the abnormal event;

[0111] Based on the event level and event type, an exception handling strategy is generated according to the preset handling strategy.

[0112] In this embodiment, the anomaly level can include severe events, high-risk events, medium-risk events, and low-risk events. Different anomaly levels determine different anomaly handling strategies, as detailed below:

[0113] For severe incidents (such as data leakage): generate a policy of "blocking the source IP + isolating the host involved + backing up relevant logs + notifying the administrator";

[0114] High-risk events (such as brute-force attacks): Generate a strategy of "blocking the source IP + temporarily locking the account + sending an alert SMS";

[0115] For medium-risk events (such as high device CPU usage): generate a strategy of "restarting device services + monitoring device load + logging".

[0116] Low-risk events (such as a single login failure): Generate a strategy of "logging + continuous monitoring";

[0117] In this embodiment, in order to accurately handle abnormal events, the abnormal handling strategy can also be verified, that is, the generated abnormal handling strategy is verified for both compliance and feasibility, such as: whether it complies with the enterprise security system, whether it will affect the normal operation of core business, and whether there are resource conflicts.

[0118] Specifically, while the anomaly handling strategies generated by the preset analysis model are based on a preset "event level - handling logic" framework, the specific strategy parameters need to be dynamically generated in conjunction with real-time log details. These strategy parameters may include the duration of IP blocking, the scope of isolated hosts, and the start / stop of related services. Therefore, there may be scenarios where verification fails. For example, in a "critical data leakage incident" in the financial industry, the preset analysis model generates a strategy of "blocking IP 192.168.1.100 + isolating the affected host" according to the preset logic. However, it detects that "IP 192.168.1.100 is the backup communication IP of the core trading system, and blocking it will cause the interruption of trading data synchronization throughout the day," or "the affected host carries unbacked-up customer account data, and direct isolation may cause the risk of data loss." In this case, the strategy verification is deemed to have failed. The system will automatically feed back the verification result to the preset analysis model, which will then regenerate an adapted strategy. If the preset analysis model still fails to pass the verification after three regenerations, a manual review process is triggered, whereby the administrator adjusts the strategy details based on business priorities, where the priorities can be manually set.

[0119] In other embodiments, a network log intelligent analysis and automated response system integrating large models is also provided, the system may include:

[0120] Log collection module: Specifically, it may include a distributed collection agent and a transmission control unit;

[0121] Distributed data collection agent: Deployed on various log source devices, supports operating systems such as Linux, Windows, and Unix, and has log capture and format recognition functions;

[0122] Transmission control unit: responsible for controlling the transmission mode (real-time / timed) and encryption method (SSL / TLS) of the collected data, monitoring the transmission status, and automatically reconnecting when the transmission is interrupted.

[0123] The log preprocessing module may include a format parsing unit, a noise filtering unit, and a data compression unit;

[0124] Format parsing unit: Built-in lightweight large model (such as BERT-Tiny) to achieve automatic parsing and standardization of multi-format logs;

[0125] Noise filtering unit: Based on the contextual understanding capabilities of a large model, it filters invalid logs and supports custom filtering rules;

[0126] Data compression unit: Uses the Snappy compression algorithm to compress the standardized logs.

[0127] The model assessment module may include a model management unit, a correlation analysis unit, and an event classification unit;

[0128] Model Management Unit: Responsible for the deployment, updating, and incremental learning of the main model and sub-models, and supports model version management and rollback;

[0129] The correlation analysis unit calls the main model to perform multi-source log correlation, calls the sub-model to perform specific scenario analysis, and outputs preliminary analysis results.

[0130] Event rating unit: Automatically rates events based on preliminary assessment results and historical data, and supports manually defined rating rules.

[0131] The response decision module may include a strategy generation unit and a strategy verification unit;

[0132] Strategy generation unit: Based on event level and type, it calls the large model to generate handling strategies and supports custom strategy templates;

[0133] Strategy verification unit: Verifies the compliance and business impact of the strategy. If the verification fails, it triggers a second generation or manual review.

[0134] An automated execution module may include an execution engine and a status monitoring unit;

[0135] Execution Engine: Supports execution methods such as API calls, script execution, and command sending, and has built-in execution templates for common devices / systems (such as Huawei firewall API templates and Windows server script templates).

[0136] Status monitoring unit: Monitors the execution status of the handling action in real time, and automatically retryes or triggers manual intervention when the execution fails.

[0137] The results feedback module may include a log recording unit and a model optimization unit.

[0138] Log recording unit: Records event loop information and stores it in the results database;

[0139] Model optimization unit: Regularly analyzes closed-loop logs to optimize the judgment rules and handling strategies of large models.

[0140] The storage module may include a log storage center, a results database, and a model database;

[0141] Log storage center: HDFS distributed storage is used to store pre-processed structured logs;

[0142] Results database: A MySQL relational database is used to store the closed-loop log of events;

[0143] Model Database: A Redis cache database is used to store large model parameters and version information, improving the speed of model access.

[0144] The visual interaction module may include a report generation unit and a manual intervention unit;

[0145] Report generation unit: Generates visual charts such as abnormal event trend charts, handling success rate reports, and model accuracy reports, and supports export (Excel / PDF format).

[0146] Manual intervention unit: Provides a manual review interface, allowing administrators to view abnormal events, modify handling strategies, and manually execute handling actions. It also supports log query, filtering, and export functions.

[0147] Based on the network log intelligent analysis and automated response method of the fused large model provided in the above embodiments, this application also provides a specific implementation of the network log intelligent analysis and automated response system of the fused large model. Please refer to the following embodiments. First, see... Figure 2 The network log intelligent analysis and automated response system 200 with integrated large model provided in this application embodiment may include:

[0148] Module 201 is used to acquire logs to be processed.

[0149] Preprocessing module 202 is used to preprocess the logs to be processed to obtain the logs to be analyzed;

[0150] The association analysis module 203 is used to perform association analysis using the global association knowledge graph in the preset analysis model to obtain the first analysis result;

[0151] The precision analysis module 204 is used to perform precise event analysis using the event diagnosis model in the preset analysis model when the analysis results include the presence of abnormal events, and to obtain a second analysis result.

[0152] The determination module 205 is used to determine the abnormality level of the abnormal event according to a preset event classification strategy when both the first analysis result and the second analysis result contain abnormal events.

[0153] The generation module 206 is used to generate an exception handling strategy based on the exception level of the exception event.

[0154] As an alternative implementation, the preprocessing module 202 can also be specifically used for:

[0155] The semantic analysis network in the preset analysis model is used to perform semantic parsing on the log to be processed, and multiple key fields are obtained.

[0156] The multiple key fields are formatted in a unified way to obtain the key format corresponding to each key field;

[0157] And / or, use the semantic analysis network in the preset analysis model to filter noise from the logs to be processed to obtain valid logs;

[0158] Identify the key format and / or valid logs as the logs to be analyzed.

[0159] As an alternative implementation, the preprocessing module 202 can also be specifically used for:

[0160] The log to be analyzed is compressed using a preset compression algorithm to obtain the compressed log.

[0161] As an alternative implementation, the association analysis module 203 can also be specifically used for:

[0162] Retrieve historical training logs;

[0163] Based on historical training logs, an initial global relational knowledge graph is constructed.

[0164] Feature extraction is performed on historical training logs to obtain multidimensional correlation features;

[0165] Multiple multidimensional association features are associated according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph.

[0166] As an alternative implementation, the association analysis module 203 can also be specifically used for:

[0167] Retrieve historical exception logs and corresponding historical exception events;

[0168] Historical anomaly logs are input into a global relational knowledge graph to obtain predicted anomaly events;

[0169] Based on predicted and historical anomalous events, the weights of the global relational knowledge graph are adjusted until the predicted and historical anomalous events meet the training conditions.

[0170] As an alternative implementation, the precise analysis module 204 can also be specifically used for:

[0171] In the global relational knowledge graph, identify the initial abnormal node corresponding to the abnormal event to be judged;

[0172] In the event diagnosis model, a pre-set anomaly propagation chain is used to trace back to the anomaly source based on the initial anomaly node to obtain the anomaly cause corresponding to the anomaly event to be determined. The anomaly propagation chain is used to characterize the propagation link of the initial anomaly node in the global association knowledge graph.

[0173] Based on the cause of the anomaly and the anomaly event to be determined, the anomaly event is identified and designated as the second analysis result.

[0174] As an alternative implementation, the generation module 206 can also be specifically used for:

[0175] Get the event level and event type corresponding to the abnormal event;

[0176] Based on the event level and event type, an exception handling strategy is generated according to the preset handling strategy.

[0177] As an optional implementation, the system further includes:

[0178] The automated execution module includes an execution engine and a status monitoring unit. The execution engine supports multiple execution modes, while the status monitoring unit monitors the execution status of actions and automatically retryes or triggers manual intervention when execution fails.

[0179] As an optional implementation, the system further includes:

[0180] The storage module is used to store logs to be analyzed, time-loop logs, and model parameters and version information;

[0181] The visualization interaction module includes a report generation unit and a manual intervention unit. The report generation unit is used to generate and export visualization reports, while the manual intervention unit provides a manual review interface to support administrators in viewing abnormal events, modifying abnormal handling strategies, and manually executing actions.

[0182] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0183] An electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0184] Specifically, the processor 301 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0185] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 302 may include removable or non-removable (or fixed) media, or memory 302 may be non-volatile solid-state memory. Memory 302 may be internal or external to the integrated gateway disaster recovery device.

[0186] In one instance, memory 302 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0187] Memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described in the network log intelligent analysis and automated response method according to the first aspect of this disclosure.

[0188] The processor 301 reads and executes computer program instructions stored in the memory 302 to achieve... Figure 1 The embodiment shown is a method for intelligent analysis and automated response of network logs that integrates a large model.

[0189] In one example, the electronic device may also include a communication interface 303 and a bus 304. For example, Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 304 and complete communication with each other.

[0190] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0191] Bus 304 includes hardware, software, or both, that couples components of an electronic device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 304 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0192] This electronic device can execute the network log intelligent analysis and automated response method of the fusion large model in the embodiments of this application, thereby achieving a combination of Figures 1-2 The paper describes a method and system for intelligent analysis and automated response of network logs based on a fusion large model.

[0193] Furthermore, in conjunction with the network log intelligent analysis and automated response method of the fusion large model in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the network log intelligent analysis and automated response methods of the fusion large model in the above embodiments.

[0194] In an optional embodiment, in conjunction with the network log intelligent analysis and automated response method of the fusion large model in the above embodiments, this application embodiment can provide a computer program product to implement it. The instructions in the computer program product are executed by the processor of the electronic device, enabling the electronic device to implement any of the network log intelligent analysis and automated response methods of the fusion large model in the above embodiments.

[0195] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0196] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0197] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0198] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor may be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions. The above descriptions are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and such modifications or substitutions should be covered within the scope of protection of this application.

Claims

1. A method for intelligent analysis and automated response of network logs integrating a large model, characterized in that, include: Get the logs to be processed; The logs to be processed are preprocessed to obtain the logs to be analyzed; The first analysis result is obtained by performing association analysis using the global association knowledge graph in the preset analysis model. If the first analysis result includes the presence of abnormal events, the event diagnosis model in the preset analysis model is used to perform precise event analysis to obtain the second analysis result. If both the first analysis result and the second analysis result contain abnormal events, the abnormality level of the abnormal event is determined according to the preset event classification strategy. An exception handling strategy is generated based on the exception level of the abnormal event. The event diagnosis model includes an abnormal propagation chain. The second analysis result, obtained by using the event diagnosis model within the preset analysis model for precise event analysis, includes: In the global relational knowledge graph, the initial abnormal node corresponding to the abnormal event to be judged is determined; In the event diagnosis model, a preset anomaly propagation chain is used to trace back to the anomaly source based on the initial anomaly node to obtain the anomaly cause corresponding to the anomaly event to be determined. The anomaly propagation chain is used to characterize the propagation link of the initial anomaly node in the global association knowledge graph. Based on the stated cause of the anomaly and the anomaly event to be determined, the anomaly event is identified and identified as the second analysis result.

2. The method according to claim 1, characterized in that, The preprocessing of the logs to be processed to obtain the logs to be analyzed includes: The log to be processed is semantically parsed using the semantic analysis network in the preset analysis model to obtain multiple key fields; The multiple key fields are formatted in a unified manner to obtain the key format corresponding to each key field; And / or, use the semantic analysis network in the preset analysis model to filter noise from the logs to be processed to obtain valid logs; The key format and / or the valid logs are identified as the logs to be analyzed.

3. The method according to claim 2, characterized in that, After determining the key format and / or the valid log as the log to be analyzed, the process includes: The log to be analyzed is compressed using a preset compression algorithm to obtain the compressed log to be analyzed.

4. The method according to any one of claims 1-3, characterized in that, Before performing association analysis using the global association knowledge graph in the preset analysis model to obtain the first analysis result, the method further includes: Retrieve historical training logs; Based on the historical training logs, an initial global relational knowledge graph is constructed; Feature extraction is performed on the historical training logs to obtain multidimensional correlation features; Multiple multidimensional association features are associated according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph.

5. The method according to claim 4, characterized in that, After associating multiple multidimensional association features according to preset association rules to improve the initial global association knowledge graph and obtain the global association knowledge graph, the method further includes: Retrieve historical exception logs and corresponding historical exception events; The historical anomaly logs are input into the global relational knowledge graph to obtain predicted anomaly events; Based on the predicted abnormal events and the historical abnormal events, the weights of the global association knowledge graph are adjusted until the predicted abnormal events and the historical abnormal events meet the training conditions.

6. The method according to claim 4, characterized in that, The step of generating an exception handling strategy based on the exception level of the abnormal event includes: Obtain the event level and event type corresponding to the abnormal event; Based on the event level and event type, an exception handling strategy is generated according to a preset processing strategy.

7. A network log intelligent analysis and automated response system integrating a large model, characterized in that, The system includes: The acquisition module is used to acquire logs to be processed. The preprocessing module is used to preprocess the logs to be processed to obtain the logs to be analyzed. The association analysis module is used to perform association analysis using the global association knowledge graph in the preset analysis model to obtain the first analysis result. The precision analysis module is used to perform precise event analysis using the event diagnosis model in the preset analysis model when the analysis results include the presence of abnormal events, and to obtain a second analysis result. The determination module is used to determine the abnormality level of the abnormal event according to a preset event classification strategy when both the first analysis result and the second analysis result contain abnormal events. The generation module is used to generate an exception handling strategy based on the exception level of the abnormal event. The event diagnosis model includes an anomaly propagation chain, and the precise analysis module is specifically used for: In the global relational knowledge graph, the initial abnormal node corresponding to the abnormal event to be judged is determined; In the event diagnosis model, a preset anomaly propagation chain is used to trace back to the anomaly source based on the initial anomaly node to obtain the anomaly cause corresponding to the anomaly event to be determined. The anomaly propagation chain is used to characterize the propagation link of the initial anomaly node in the global association knowledge graph. Based on the stated cause of the anomaly and the anomaly event to be determined, the anomaly event is identified and identified as the second analysis result.

8. The system according to claim 7, characterized in that, The system also includes: The automated execution module includes an execution engine and a status monitoring unit. The execution engine supports multiple execution modes, and the status monitoring unit monitors the execution status of actions and automatically retryes or triggers manual intervention when execution fails.

9. The system according to claim 7 or 8, characterized in that, The system also includes: The storage module is used to store logs to be analyzed, time-loop logs, and model parameters and version information; The visualization interaction module includes a report generation unit and a manual intervention unit. The report generation unit is used to generate and export visualization reports, and the manual intervention unit is used to provide a manual review interface to support administrators in viewing abnormal events, modifying abnormal handling strategies, and manually executing handling actions.

Images