Network traffic log analysis method, electronic equipment, storage medium and program product

Abstract

The invention provides a network traffic log analysis method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a scene analysis request for a network traffic log generated in a target scene; and inputting a scene field in the scene analysis request into a field analysis engine and an artificial intelligence model, so that the field analysis engine analyzes the network traffic log according to the scene field, and predicts a log analysis report of the field analysis engine according to the scene field through the artificial intelligence model, and obtaining a prediction result corresponding to the scene analysis request. In the implementation process of the scheme, the to-be-processed fields are respectively input into the field analysis engine and the artificial intelligence model, and compared with the direct analysis speed of the field analysis engine, the artificial intelligence model can quickly predict and obtain the prediction result corresponding to the scene analysis request, so that the scene analysis efficiency is improved. Therefore, the waiting time of the network flow log analysis is effectively reduced, and the real-time performance of the network flow log analysis is improved.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence and computer network traffic analysis, and in particular, to a network traffic log analysis method, electronic device, storage medium and program product. Background Art

[0002] At present, in the analysis scenario of massive network traffic logs, these logs are queried from the log database that can reach hundreds of millions per day. The traditional analysis method is very time-consuming due to the huge amount of data, long query database and high resource usage. It is usually difficult for users to see the analysis results in time. Therefore, the real-time performance of current network traffic log analysis is poor. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a network traffic log analysis method, electronic device, storage medium and program product, which are used to improve the problem of poor real-time performance of network traffic log analysis.

[0004] The embodiment of the present application provides a network traffic log analysis method, including: obtaining a scenario analysis request for a network traffic log generated in a target scenario, the scenario analysis request including: scenario fields; inputting the scenario fields into a field analysis engine and an artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scenario fields, and predicting the log analysis report of the field analysis engine according to the scenario fields through the artificial intelligence model, to obtain the prediction result corresponding to the scenario analysis request. In the implementation process of the above scheme, by inputting the fields to be processed into the field analysis engine and the artificial intelligence model respectively, so that the artificial intelligence model predicts the analysis result of the field analysis engine, compared with the speed of direct analysis by the field analysis engine, the artificial intelligence model can quickly predict the prediction result corresponding to the scenario analysis request, thereby effectively reducing the waiting time of the network traffic log analysis and improving the real-time performance of the network traffic log analysis.

[0005] Optionally, in an embodiment of the present application, the scene field includes: a time field and multiple independent fields; analyzing the network traffic log according to the scene field includes: splitting the target scene according to the time field and multiple independent fields to obtain multiple sub-scenes, and the sub-scene is composed of a time field and at least one independent field; for each sub-scene in the multiple sub-scenes, using the fields in the sub-scene to analyze the network traffic log to obtain multiple analysis results; aggregating the multiple analysis results to obtain a log analysis report. In the implementation process of the above scheme, each sub-scene can be analyzed independently, which allows the analysis tasks of different sub-scenes to be executed in parallel, greatly shortening the total analysis time, and by splitting the target scene, reducing the complexity of a single analysis task, reducing the demand for computing resources, and improving the processing speed.

[0006] Optionally, in an embodiment of the present application, the network traffic log is analyzed using the fields in the sub-scenario, including: slicing the historical time in the network traffic log into multiple time segments according to the time field in the sub-scenario; for each time segment in the multiple time segments, frequency analysis, trend analysis and / or anomaly analysis are performed on the network traffic log according to the time segment and at least one independent field in the sub-scenario. In the implementation process of the above scheme, by slicing the historical time into multiple time segments, analysis can be performed on a finer time scale to capture more subtle changes and patterns. This fine-grained analysis helps to discover sudden behaviors or abnormal situations in a short period of time, thereby greatly reducing the query time and memory resources occupied by the original data. Furthermore, by combining time segments and independent fields (such as IP addresses, protocol types, etc.), a comprehensive analysis of network activities within a specific time period can be performed, providing multi-dimensional insights, thereby significantly improving the accuracy of network traffic log analysis.

[0007] Optionally, in an embodiment of the present application, multiple analysis results are aggregated, including: aggregating multiple analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm. In the implementation of the above scheme, by combining multiple algorithms to aggregate the analysis results of different sub-scenarios, a more comprehensive and accurate comprehensive perspective can be provided. A single algorithm may have its limitations, and the combination of multiple algorithms can make up for their respective shortcomings and improve the overall analysis accuracy.

[0008] Optionally, in an embodiment of the present application, before inputting the scenario fields into the field analysis engine and the artificial intelligence model respectively, it also includes: obtaining a sample traffic log and a sample field, the sample traffic log includes: a real traffic log generated under the target scenario, and / or a simulated traffic log generated under a simulated target scenario, the sample field is the same as the scenario field; preprocessing the sample traffic log to obtain a preprocessed sample traffic log; using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field to obtain a sample analysis result; using the sample field as training data and the sample analysis result as a training label to train the artificial intelligence model. In the implementation process of the above scheme, by simulating the simulated traffic log generated under the target scenario as part of the sample traffic log, and using the real traffic log generated under the target scenario as part of the sample traffic log, it can be ensured that the training data is closer to the actual environment, thereby improving the generalization ability and prediction accuracy of the model.

[0009] Optionally, in an embodiment of the present application, before using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field, it also includes: using real traffic logs to test and tune the field analysis engine. In the implementation process of the above scheme, by using real traffic logs for testing and tuning, the performance of the field analysis engine in the actual environment can be verified to ensure that its assumptions and predictions are consistent with the actual situation. The real traffic log contains a variety of actual network activities. By testing and tuning these data, the false positives and false negatives of the field analysis engine can be reduced, and the analysis accuracy of the field analysis engine can be improved.

[0010] Optionally, in an embodiment of the present application, the artificial intelligence model includes: a long short-term memory network LSTM model and / or an extreme gradient boosting XGBoost tree model. In the implementation process of the above scheme, the LSTM model is good at processing and predicting time series data, can capture long-term dependencies, and is very suitable for analyzing time series features in network traffic logs, while the XGBoost tree model can accurately perform classification and regression tasks, and is suitable for identifying abnormal behaviors and patterns in network traffic.

[0011] An embodiment of the present application also provides a network traffic log analysis device, including: an analysis request acquisition module, used to obtain a scenario analysis request for a network traffic log generated in a target scenario, the scenario analysis request including: a scenario field; a scenario field analysis module, used to input the scenario field into a field analysis engine and an artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scenario field, and predicts the log analysis report of the field analysis engine according to the scenario field through the artificial intelligence model, to obtain a prediction result corresponding to the scenario analysis request.

[0012] Optionally, in an embodiment of the present application, the scene field includes: a time field and multiple independent fields; the scene field analysis module includes: a target scene splitting sub-module, used to split the target scene according to the time field and multiple independent fields to obtain multiple sub-scenes, and the sub-scenes are composed of a time field and at least one independent field; an analysis result acquisition sub-module, used to analyze the network traffic log for each sub-scene in the multiple sub-scenes using the fields in the sub-scene to obtain multiple analysis results; an analysis result aggregation sub-module, used to aggregate multiple analysis results to obtain a log analysis report.

[0013] Optionally, in an embodiment of the present application, an analysis result obtaining submodule includes: a historical time slicing unit, used to slice the historical time in the network traffic log into multiple time segments according to the time field in the sub-scenario; a traffic log analysis unit, used to perform frequency analysis, trend analysis and / or anomaly analysis on the network traffic log for each time segment in the multiple time segments according to the time segment and at least one independent field in the sub-scenario.

[0014] Optionally, in an embodiment of the present application, the analysis result aggregation submodule includes: an analysis result aggregation unit, which is used to aggregate multiple analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

[0015] Optionally, in an embodiment of the present application, the network traffic log analysis device also includes: a sample data acquisition module, used to obtain sample traffic logs and sample fields, the sample traffic logs include: real traffic logs generated under the target scenario, and / or simulated traffic logs generated under simulated target scenarios, the sample fields are the same as the scenario fields; a traffic log preprocessing module, used to preprocess the sample traffic logs to obtain preprocessed sample traffic logs; a traffic log analysis module, used to use a field analysis engine to analyze the preprocessed sample traffic logs according to the sample fields to obtain sample analysis results; an intelligent model training module, used to train the artificial intelligence model using the sample fields as training data and the sample analysis results as training labels.

[0016] Optionally, in an embodiment of the present application, the network traffic log analysis device further includes: an engine testing and tuning module, which is used to test and tune the field analysis engine using real traffic logs.

[0017] Optionally, in an embodiment of the present application, the artificial intelligence model includes: a long short-term memory network LSTM model and / or an extreme gradient boosting XGBoost tree model.

[0018] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0019] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0020] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a network traffic log analysis method provided by an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of the process of training the artificial intelligence model provided by an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the structure of a network traffic log analysis device provided in an embodiment of the present application is shown;

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0028] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0029] It should be noted that the network traffic log analysis method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant, or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer, and a UNIX server.

[0030] The following is an example of an application scenario in which the network traffic log analysis method is applicable: In a scenario where the amount of network traffic log data is very large and TBs of network traffic logs generated every day need to be analyzed, the network traffic log analysis method can be used to predict the analysis results of the network traffic log, thereby effectively reducing the waiting time for the network traffic log analysis and improving the real-time performance of the network traffic log analysis.

[0031] See also Figure 1 The flowchart of the network traffic log analysis method provided by the embodiment of the present application is shown; the embodiment of the present application provides a network traffic log analysis method, including:

[0032] Step S110: Obtain a scenario analysis request for the network traffic log generated in the target scenario, the scenario analysis request including: a scenario field.

[0033] The target scenario refers to the scenario selected on the terminal device, which can be a scenario selected from the two categories of scenarios, namely, preset scenarios and specified scenarios, including scenario fields. The available scenarios include, but are not limited to: event trend scenarios, protocol trend scenarios, attack trend scenarios, protocol abnormal access scenarios, HyperText Transfer Protocol (HTTP) website distribution scenarios, user access network scenarios, etc. Among them, the preset scenario is a scenario that is prefabricated and solidified in the security software product by the security software manufacturer, while the specified scenario is a custom scenario generated by the scenario field selected by the user on the terminal device. The above-mentioned security software products include, for example, network audit products or intrusion detection system (IDS) products.

[0034] Network traffic logs are logs generated in the target scenario. These network traffic logs can be obtained by collecting network traffic from security software products in a bypass monitoring environment or a serial environment. The collected network traffic logs may include audit logs and / or alarm logs. The above-mentioned scenario analysis requests for these network traffic logs may include: analysis requests for preset scenarios and analysis requests for specified scenarios.

[0035] Step S120: Input the scenario fields into the field analysis engine and the artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scenario fields, and predicts the log analysis report of the field analysis engine according to the scenario fields through the artificial intelligence model to obtain the prediction results corresponding to the scenario analysis request.

[0036] As the generation and growth of Internet communication data continues to accelerate, the network traffic logs that need to be analyzed are also growing dramatically. Therefore, the above-mentioned field analysis engine needs more time to analyze the network traffic logs, and users need to wait patiently for the analysis results. In this case, the artificial intelligence model in the above-mentioned network traffic log analysis method can be used to quickly obtain the prediction results of the log analysis, so that users can quickly know the prediction results they are concerned about, because the prediction results contain the analysis trend information of the network traffic log.

[0037] It is understandable that the order in which the scene fields are input into the field analysis engine and the artificial intelligence model respectively can be not restricted. The following are some possible implementation methods: In the first implementation method, the scene field is input into the field analysis engine and the artificial intelligence model at the same time. For target scenes with a very large number of targets to be analyzed, the prediction results of the artificial intelligence model are usually obtained first. At this time, the prediction results can be returned to the terminal device first, and then the log analysis report of the field analysis engine is returned. Correspondingly, for target scenes with a very small number of targets to be analyzed, the log analysis report of the field analysis engine is usually obtained first. At this time, the log analysis report of the field analysis engine can be returned to the terminal device first, and then the prediction results of the artificial intelligence model are returned. The scene field is input into the field analysis engine and the artificial intelligence model at the same time. These two processes can be executed in parallel instead of serially, which can greatly shorten the total processing time. In the second implementation method, the scene field is first input into the artificial intelligence model, and then the scene field is input into the field analysis engine. The prediction results of the artificial intelligence model are usually obtained first. At this time, the prediction results can be returned to the terminal device first, and then the log analysis report of the field analysis engine is returned. The third implementation method is to first input the scene field into the field analysis engine, and then input the scene field into the artificial intelligence model. In this case, the first result is returned to the terminal device.

[0038] Optionally, after the artificial intelligence model predicts the prediction result of the log analysis report, the prediction result can also be sent to the user's terminal device so that the user can obtain analysis trend information based on the prediction result. Since the artificial intelligence model can provide millisecond-level analysis response compared to the field analysis engine, the artificial intelligence model predicts and obtains the prediction result more quickly, and therefore can meet the user's timeliness requirements for viewing the prediction results of the analysis trend. Furthermore, after obtaining the log analysis report of the field analysis engine, a more accurate log analysis report is sent to the user's terminal device to meet the user's accuracy requirements for the log analysis report.

[0039] After the terminal device receives the log analysis report from the field analysis engine, the log analysis report can be rendered according to the target scenario. It can be rendered into graphics such as trend charts, heat maps, proportion charts, or comparison charts. The analysis data in the log analysis report can also be rendered into display tables and conclusion summaries. The trend information in the log analysis report can also be rendered into trend charts. The trend charts here can include trend comparison charts of historical data and future data.

[0040] During the implementation of the above solution, the fields to be processed are respectively input into the field analysis engine and the artificial intelligence model, so that the artificial intelligence model predicts the analysis results of the field analysis engine. Compared with the speed of direct analysis by the field analysis engine, the artificial intelligence model can quickly predict the prediction results corresponding to the scene analysis request, thereby effectively reducing the waiting time of network traffic log analysis and improving the real-time performance of network traffic log analysis.

[0041] An optional implementation of the above step S110, for example: the user selects the network traffic log that needs to be analyzed in the target scenario on the terminal device, and then, after clicking the button or right-click menu for performing scenario analysis on the network traffic log generated in the target scenario, a scenario analysis request is generated based on the scenario field required for the target scenario and the identifier of the network traffic log, and the scenario analysis request is sent to the server. Then, the server receives the scenario analysis request sent by the terminal device, and parses the scenario field and the identifier of the network traffic log required for the target scenario from the scenario analysis request. Among them, the above-mentioned server can be a server installed in the user's local area network, or it can be a cloud server on the Internet, which can be set according to the specific situation.

[0042] As an optional implementation of the above step S120, the above scenario field may include: a time field and multiple independent fields; the implementation of the above field analysis engine analyzing the network traffic log according to the scenario field may include:

[0043] Step S121: split the target scene according to the time field and multiple independent fields to obtain multiple sub-scenes, where the sub-scenes are composed of the time field and at least one independent field.

[0044] The implementation method of the above step S121 is, for example: assuming that the target scenario is "the protocol access of the source IP from 12:00 to 13:00 every day", it can be seen that the target scenario contains a time field (i.e., 12:00-13:00 every day), an independent field of a source IP, and an independent field of a protocol. In this case, the field analysis engine can split the target scenario into two sub-scenarios, including: the first sub-scenario is "time + source IP", and the second sub-scenario is "time + protocol". Optionally, the specific splitting method can be customized by the user according to the needs of the scene field combination splitting method, which enables the system to adapt to different business needs and technical environments, for example: parse the splitting identifier from the scene analysis request, and find the splitting strategy corresponding to the splitting identifier, and then combine and analyze and split the time field and multiple independent fields in the target scenario according to the splitting strategy to obtain multiple sub-scenarios. Through the combined analysis of multiple independent fields and time fields, the system can provide a rich perspective to help users fully understand the situation of network traffic. Users can freely combine time fields and independent fields according to needs to form different sub-scenarios to meet specific analysis needs. This flexibility enables the system to quickly adapt to new business requirements and technical environments.

[0045] Step S122: for each sub-scenario in the multiple sub-scenarios, the network traffic log is analyzed using the fields in the sub-scenario to obtain multiple analysis results.

[0046] Step S123: Aggregate multiple analysis results to obtain a log analysis report.

[0047] It is understandable that there are many analysis methods and aggregation methods, such as frequency analysis, trend analysis and / or anomaly analysis, etc. Therefore, the implementation methods of step S122 and step S123 will be described in detail below.

[0048] As an optional implementation of the above step S122, the above implementation of using the fields in the sub-scenario to analyze the network traffic log may include:

[0049] Step S122a: Slice the historical time in the network traffic log into multiple time segments according to the time field in the sub-scenario.

[0050] For example, the implementation method of the above step S122a is: assuming that the time field in the sub-scenario is 12:00-13:00 every day, the historical time in the network traffic log can be first sliced ​​into daily time segments, and then the daily time segments are divided and cut into hourly time segments, and finally the log data corresponding to the hourly time segment of 12:00-13:00 every day is obtained from the network traffic log, so as to reduce the memory resources and computing resources required for analysis, and also reduce the memory resources, communication resources and computing resources required for data acquisition. Of course, time can also be divided into other time segments according to specific needs, such as dimensional features such as week or timestamp.

[0051] Step S122b: For each time segment in the multiple time segments, perform frequency analysis, trend analysis, cluster analysis and / or anomaly analysis on the network traffic log according to the time segment and at least one independent field in the sub-scenario.

[0052] For example, the implementation method of the above step S122b is as follows: taking 12:00-13:00 today as an example, after obtaining the first part of the network traffic log corresponding to the time segment of 12:00-13:00 today, the "time + source IP" sub-scenario can be used to perform frequency analysis, trend analysis, cluster analysis and / or anomaly analysis on the first part of the network traffic log to obtain the analysis result of the first sub-scenario, and the "time + protocol" sub-scenario can also be used to perform frequency analysis, trend analysis, cluster analysis and / or anomaly analysis on the first part of the network traffic log to obtain the analysis result of the second sub-scenario. By performing trend analysis on historical data of different time segments, the system can dynamically adjust the threshold of anomaly detection according to actual data, thereby improving the accuracy and sensitivity of anomaly detection.

[0053] As an optional implementation of the above step S123, the above implementation of aggregating multiple analysis results may include:

[0054] Step S123a: Aggregate the multiple analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

[0055] The implementation method of the above step S123 is, for example: aggregating multiple analysis results with time as the pivot. Specifically, data merging algorithm, statistical analysis algorithm, anomaly detection algorithm, clustering algorithm, association rule algorithm and / or multidimensional data analysis algorithm, etc. can be used to aggregate the analysis results of the first sub-scenario with the analysis results of the second sub-scenario to obtain a log analysis report. The combined application of multiple algorithms such as the above data merging algorithm, statistical analysis algorithm, anomaly detection algorithm, clustering algorithm, association rule algorithm and / or multidimensional data analysis algorithm can analyze data from different angles and provide richer information. For example, statistical analysis can reveal the overall trend, the association rule algorithm can discover the implicit relationship between different fields, and the clustering algorithm can identify similar behavior patterns. Further, the above multidimensional data analysis algorithm can process high-dimensional data sets and help identify complex patterns and hidden relationships, which is particularly important for understanding subtle changes and potential threats in network traffic. The cross-validation mechanism of the above multiple algorithms can effectively reduce the false alarm rate. For example, the anomaly detection algorithm can identify potential abnormal behaviors, and the clustering algorithm can help confirm whether these behaviors belong to normal patterns, thereby improving the accuracy of judgment.

[0056] Optionally, after obtaining the log analysis report of the field analysis engine analyzing the network traffic log and the prediction result of the artificial intelligence model predicting the log analysis report, if the user finds that there is a large gap between the log analysis report and the prediction result, the network traffic log and the log analysis report can be used as training data for the artificial intelligence model to train the artificial intelligence model. Specifically, it can include: obtaining the real traffic log generated in the target scenario and the log analysis report corresponding to the real traffic log, first preprocessing the real traffic log to obtain the preprocessed traffic log. Use the field analysis engine to analyze the preprocessed traffic log according to the scene field to obtain the log analysis result, and finally, use the scene field as training data and the log analysis result as training label to train the artificial intelligence model. Among them, training the artificial intelligence model includes: using the artificial intelligence model to predict the log analysis report of the field analysis engine according to the scene field, obtain the model prediction result, and calculate the loss value between the model prediction result and the log analysis result, and then update the model parameters of the artificial intelligence model according to the loss value until the model parameters of the artificial intelligence model tend to converge, thereby obtaining the retrained artificial intelligence model.

[0057] See also Figure 2 The flowchart of training an artificial intelligence model provided by an embodiment of the present application is shown; as an optional implementation of the above-mentioned network traffic log analysis method, before the scene fields are respectively input into the field analysis engine and the artificial intelligence model, the artificial intelligence model can also be trained, and the implementation may include:

[0058] Step S210: Obtain sample traffic logs and sample fields. The sample traffic logs include: real traffic logs generated under the target scenario, and / or simulated traffic logs generated under the simulated target scenario. The sample fields are the same as the scenario fields.

[0059] It is understandable that the above-mentioned sample traffic logs can be real traffic logs generated in the target scenario as reported by users, or simulated traffic logs collected in a simulated target scenario. By continuously learning from historical sample traffic log data, the artificial intelligence model can identify complex patterns and implicit relationships in the sample traffic logs, thereby more accurately predicting future trends and abnormal behaviors. The traffic logs of the above-mentioned simulated target scenarios can cover extreme situations or special events that may be difficult to capture in reality, enriching the diversity of training data and enabling the model to better cope with various complex network environments. By using the real traffic logs generated in the target scenario as part of the sample traffic logs, it can ensure that the training data is closer to the actual environment, thereby improving the generalization ability and prediction accuracy of the model.

[0060] Step S220: pre-process the sample traffic log to obtain the pre-processed sample traffic log.

[0061] In the process of preprocessing sample traffic logs, the key fields to focus on may include: log time, source IP, destination IP, destination port, application protocol type, attack type, attack result, and risk level, etc. Users can generate simulated traffic logs according to specific needs, flexibly configure key fields and different scenario parameters, so that the system can quickly adapt to new business needs and technical environments.

[0062] The implementation method of the above step S220 is, for example: the sample traffic log is preprocessed by data preprocessing software, such as data cleaning, format conversion, removal of missing fields or missing value filling, etc., to obtain the preprocessed sample traffic log, to ensure that the data input to the analysis engine is of high quality and good consistency, and to reduce the influence of noise and outliers. After obtaining the preprocessed sample traffic log and sample field, the preprocessed sample traffic log and sample field can also be split according to a preset ratio (for example, 8:2) to obtain a training set and a test set, and the training set can be subjected to the following training process, and the test set can be used to perform performance testing on the trained artificial intelligence model.

[0063] Step S230: Use the field analysis engine to analyze the preprocessed sample traffic log according to the sample field to obtain a sample analysis result.

[0064] An example of an implementation of the above-mentioned step S230 is as follows: the above-mentioned sample field may include: a time field and multiple independent fields; the target scene is split according to the time field and multiple independent fields by the field analysis engine to obtain multiple sub-scenes, and the sub-scenes are composed of a time field and at least one independent field. For each of the multiple sub-scenes, the sample traffic log is analyzed using the fields in the sub-scene to obtain multiple analysis results. The multiple analysis results are aggregated to obtain a sample analysis result. It can be understood that the implementation of each of the above-mentioned sub-steps is similar or identical to the implementation of steps S121 to S123, and therefore will not be repeated. The preprocessing and analysis tasks of the sample traffic log can be executed in parallel on multiple computing nodes, making full use of distributed computing resources and accelerating the entire training process.

[0065] Step S240: Train the artificial intelligence model using the sample fields as training data and the sample analysis results as training labels.

[0066] An implementation example of the above step S240 is: using an artificial intelligence model to predict the log analysis report of the field analysis engine according to the sample field, obtaining the model prediction result, and calculating the loss value between the model prediction result and the sample analysis result, and then updating the model parameters of the artificial intelligence model according to the loss value until the model parameters of the artificial intelligence model tend to converge, thereby obtaining a trained artificial intelligence model. After obtaining the trained artificial intelligence model, the trained artificial intelligence model can also be exported. The staff can also use the trained artificial intelligence model for performance testing. If the performance of the artificial intelligence model exceeds the preset performance index, the artificial intelligence model can be imported into the network audit product or intrusion detection system (IDS) product in the user's local area network.

[0067] As an optional implementation of the above network traffic log analysis method, before using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field, the field analysis engine can also be tested and tuned, and then the analysis result of the field analysis engine can be used to train the model. The implementation of testing and tuning the field analysis engine can include:

[0068] Step S231: Use real traffic logs to test and tune the field analysis engine.

[0069] The implementation method of the above step S231 is, for example: using real traffic logs to test and tune the field analysis engine to obtain a tuned field analysis engine. After obtaining the tuned field analysis engine, the tuned field analysis engine can also be exported so that the staff can import the tuned field analysis engine into the network audit product or intrusion detection system (IDS) product in the user's local area network.

[0070] Optionally, after obtaining the above-mentioned tuned field analysis engine and trained artificial intelligence model, the tuned field analysis engine and the trained artificial intelligence model can also be added to the system upgrade package, so that users can use the system upgrade package to upgrade security software products such as network audit products or intrusion detection systems (IDS) products in the local area network, and optimize the performance of security software products. For example, the gap between the log analysis report of the field analysis engine and the prediction results of the artificial intelligence model can be reduced, thereby enhancing the accuracy of the security software product.

[0071] As an optional implementation of the above-mentioned network traffic log analysis method, the artificial intelligence model includes: a long short-term memory network (Long Short Term Memory, LSTM) model, a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, BiLSTM) model and / or an extreme gradient boosting XGBoost tree model, wherein LSTM is a special recursive neural network (RNN) that is good at processing and predicting time series data, and XGBoost is a decision tree-based gradient boosting framework with efficient computing power and powerful feature selection capabilities.

[0072] Optionally, in the process of using an artificial intelligence model to predict the log analysis report of the field analysis engine according to the scenario field, the above-mentioned artificial intelligence model can be selected from multiple artificial intelligence models according to the target scenario to select the artificial intelligence model with the best performance for prediction. For example, in the scenario of analyzing time series network traffic logs, an LSTM model or a BiLSTM model can be selected for prediction. Since the LSTM model or the BiLSTM model is good at processing and predicting time series data and can capture long-term dependencies, it is very suitable for analyzing time series features in network traffic logs, such as traffic trends, periodicity, and anomaly detection. For another example: for the scenario of analyzing the topology of network traffic logs collected under the network topology, the extreme gradient boosting XGBoost tree model can be selected for prediction. The XGBoost tree model performs well in processing structured data, can accurately perform classification and regression tasks, and is suitable for identifying abnormal behaviors and patterns in network traffic.

[0073] See also Figure 3 The structure diagram of the network traffic log analysis device provided in the embodiment of the present application is shown; the embodiment of the present application provides a network traffic log analysis device 300, including:

[0074] The analysis request acquisition module 310 is used to obtain a scenario analysis request for the network traffic log generated in the target scenario. The scenario analysis request includes: a scenario field.

[0075] The scenario field analysis module 320 is used to input the scenario fields into the field analysis engine and the artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scenario fields, and predicts the log analysis report of the field analysis engine according to the scenario fields through the artificial intelligence model to obtain the prediction results corresponding to the scenario analysis request.

[0076] As an optional implementation manner of the above device, the scene field includes: a time field and multiple independent fields; the scene field analysis module includes:

[0077] The target scene splitting submodule is used to split the target scene according to the time field and multiple independent fields to obtain multiple sub-scenes, and the sub-scene is composed of the time field and at least one independent field.

[0078] The analysis result obtaining submodule is used to analyze the network traffic log for each sub-scenario in the multiple sub-scenario using the fields in the sub-scenario to obtain multiple analysis results.

[0079] The analysis result aggregation submodule is used to aggregate multiple analysis results to obtain a log analysis report.

[0080] As an optional implementation of the above device, the analysis result obtaining submodule includes:

[0081] The historical time slicing unit is used to slice the historical time in the network traffic log into multiple time segments according to the time field in the sub-scenario.

[0082] The traffic log analysis unit is used to perform frequency analysis, trend analysis and / or anomaly analysis on the network traffic log for each time segment in the multiple time segments according to the time segment and at least one independent field in the sub-scenario.

[0083] As an optional implementation of the above device, the analysis result aggregation submodule includes:

[0084] The analysis result aggregation unit is used to aggregate multiple analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

[0085] As an optional implementation of the above device, the network traffic log analysis device further includes:

[0086] The sample data acquisition module is used to obtain sample traffic logs and sample fields. The sample traffic logs include: real traffic logs generated under the target scenario, and / or simulated traffic logs generated under the simulated target scenario. The sample fields are the same as the scenario fields.

[0087] The traffic log preprocessing module is used to preprocess the sample traffic log to obtain the preprocessed sample traffic log.

[0088] The traffic log analysis module is used to use the field analysis engine to analyze the preprocessed sample traffic log according to the sample field to obtain the sample analysis result.

[0089] The intelligent model training module is used to train the artificial intelligence model using sample fields as training data and sample analysis results as training labels.

[0090] As an optional implementation of the above device, the network traffic log analysis device further includes:

[0091] The engine testing and tuning module is used to test and tune the field analysis engine using real traffic logs.

[0092] As an optional implementation of the above-mentioned device, the artificial intelligence model includes: a long short-term memory network LSTM model and / or an extreme gradient boosting XGBoost tree model.

[0093] It should be understood that the device corresponds to the above-mentioned network traffic log analysis method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0094] See also Figure 4 An electronic device 400 provided in an embodiment of the present application includes: a processor 410 and a memory 420, wherein the memory 420 stores machine-readable instructions executable by the processor 410, and when the machine-readable instructions are executed by the processor 410, the above method is executed.

[0095] The embodiment of the present application also provides a computer-readable storage medium 430, on which a computer program is stored, and the computer program is executed by the processor 410 to execute the above method. The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0096] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0097] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0098] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0099] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0100] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A network traffic log analysis method, characterized in that: include: Obtaining a scenario analysis request for a network traffic log generated in a target scenario, the scenario analysis request comprising: a scenario field; The scenario fields are respectively input into the field analysis engine and the artificial intelligence model, so that the field analysis engine analyzes the network traffic log according to the scenario fields, and the artificial intelligence model predicts the log analysis report of the field analysis engine according to the scenario fields to obtain the prediction result corresponding to the scenario analysis request.

2. The method according to claim 1, characterized in that The scenario field includes: a time field and a plurality of independent fields; and analyzing the network traffic log according to the scenario field includes: Splitting the target scene according to the time field and the plurality of independent fields to obtain a plurality of sub-scenes, wherein the sub-scenes are composed of the time field and at least one of the independent fields; For each sub-scenario in the multiple sub-scenarios, analyzing the network traffic log using the fields in the sub-scenario to obtain multiple analysis results; The multiple analysis results are aggregated to obtain the log analysis report.

3. The method according to claim 2, characterized in that The analyzing the network traffic log using the fields in the sub-scenario includes: Slice the historical time in the network traffic log into multiple time segments according to the time field in the sub-scenario; For each time segment of the multiple time segments, frequency analysis, trend analysis and / or anomaly analysis is performed on the network traffic log according to the time segment and at least one of the independent fields in the sub-scenario.

4. The method according to claim 2, characterized in that: The aggregating the multiple analysis results includes: The multiple analysis results are aggregated using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

5. The method according to claim 1, characterized in that Before inputting the scenario fields into the field analysis engine and the artificial intelligence model respectively, the method further includes: Obtaining a sample traffic log and a sample field, wherein the sample traffic log includes: a real traffic log generated under the target scenario, and / or a simulated traffic log generated under the target scenario, wherein the sample field is the same as the scenario field; Preprocessing the sample traffic log to obtain a preprocessed sample traffic log; Using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field to obtain a sample analysis result; The artificial intelligence model is trained using the sample fields as training data and the sample analysis results as training labels.

6. The method according to claim 5, characterized in that Before using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field, the method further includes: The field analysis engine is tested and tuned using the real traffic log.

7. The method according to any one of claims 1 to 6, characterized in that: The artificial intelligence model includes: a long short-term memory network LSTM model and / or an extreme gradient boosting XGBoost tree model.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images