Network traffic log analysis method, electronic device, storage medium, and program product

Abstract

The application provides a network traffic log analysis method, an electronic device, a storage medium and a program product. The method comprises: obtaining a scene analysis request for a network traffic log generated in a target scene; inputting a scene field in the scene analysis request into a field analysis engine and an artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scene field, and the artificial intelligence model predicts a log analysis report of the field analysis engine according to the scene field, to obtain a prediction result corresponding to the scene analysis request. In the implementation process of the above scheme, by inputting the to-be-processed field into the field analysis engine and the artificial intelligence model respectively, compared with the speed of direct analysis by the field analysis engine, the artificial intelligence model can quickly predict and obtain the prediction result corresponding to the scene analysis request, thereby effectively reducing the waiting time of network traffic log analysis and improving the real-time performance of network traffic log analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and computer network traffic analysis, in particular to a network traffic log analysis method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Currently, in the analysis scene of massive network traffic logs, querying these logs from a log database with tens of millions of logs per day, the traditional analysis method is time-consuming and resource-consuming due to the large amount of data, long database query time and high resource occupation, which leads to a very time-consuming network traffic log analysis process, and users usually cannot see the analysis results in time. Therefore, the real-time performance of the current network traffic log analysis is poor. SUMMARY

[0003] The embodiments of the present application aim to provide a network traffic log analysis method, an electronic device, a storage medium and a program product, to improve the problem of poor real-time performance of network traffic log analysis.

[0004] The embodiments of the present application provide a network traffic log analysis method, comprising: obtaining a scene analysis request for network traffic logs generated under a target scene, the scene analysis request comprising a scene field; inputting the scene field into a field analysis engine and an artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic logs according to the scene field, and the artificial intelligence model predicts the log analysis report of the field analysis engine according to the scene field, to obtain a prediction result corresponding to the scene analysis request. In the implementation process of the above scheme, by inputting the to-be-processed field into the field analysis engine and the artificial intelligence model respectively, the artificial intelligence model can quickly predict the analysis result of the field analysis engine, compared with the speed of direct analysis by the field analysis engine, so as to effectively reduce the waiting time of network traffic log analysis and improve the real-time performance of network traffic log analysis.

[0005] Optionally, in the embodiments of the present application, the scene field comprises a time field and a plurality of independent fields; and the network traffic log is analyzed according to the scene field, which comprises: splitting the target scene according to the time field and the plurality of independent fields to obtain a plurality of sub-scenes, the sub-scenes being composed of the time field and at least one independent field; for each of the plurality of sub-scenes, the network traffic log is analyzed using the fields in the sub-scene to obtain a plurality of analysis results; and the plurality of analysis results are aggregated to obtain a log analysis report. In the implementation process of the above scheme, each sub-scene can be independently analyzed, which enables the analysis tasks of different sub-scenes to be executed in parallel, greatly shortening the total analysis time, reduces the complexity of a single analysis task by splitting the target scene, reduces the demand for computing resources, and improves the processing speed.

[0006] Optionally, in the embodiments of the present application, the network traffic log is analyzed using the fields in the sub-scene, which comprises: dividing the historical time in the network traffic log into a plurality of time segments according to the time field in the sub-scene; and for each of the plurality of time segments, the network traffic log is subjected to frequency analysis, trend analysis and / or anomaly analysis according to the time segment and at least one independent field in the sub-scene. In the implementation process of the above scheme, by dividing the historical time into a plurality of time segments, analysis can be performed on a finer time scale to capture more subtle changes and patterns. Such fine-grained analysis helps to discover burst behaviors or abnormal situations in a short time, thereby greatly reducing the query time of the original data and the occupied memory resources. Further, in combination with the time segment and the independent field (such as IP address, protocol type, etc.), the network activities in a specific time period can be comprehensively analyzed to provide multi-dimensional insights, thereby significantly improving the accuracy of network traffic log analysis.

[0007] Optionally, in the embodiments of the present application, the plurality of analysis results are aggregated, which comprises: using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multi-dimensional data analysis algorithm to aggregate the plurality of analysis results. In the implementation process of the above scheme, by combining a plurality of algorithms to aggregate the analysis results of different sub-scenes, a more comprehensive and accurate comprehensive perspective can be provided. A single algorithm may have its limitations, and the combination of a plurality of algorithms can make up for the respective shortcomings and improve the overall analysis accuracy.

[0008] Optionally, in the embodiments of the present application, before the scene field is input into the field analysis engine and the artificial intelligence model respectively, the method further includes: obtaining a sample traffic log and a sample field, the sample traffic log including: a real traffic log generated under a target scene, and / or a simulated traffic log generated under a simulated target scene, the sample field being the same as the scene field; preprocessing the sample traffic log to obtain a preprocessed sample traffic log; using the field analysis engine to analyze the preprocessed sample traffic log according to the sample field to obtain a sample analysis result; and using the sample field as training data and using the sample analysis result as a training label to train the artificial intelligence model. In the implementation process of the above scheme, by using the simulated traffic log generated under the simulated target scene as part of the sample traffic log and using the real traffic log generated under the target scene as part of the sample traffic log, it can be ensured that the training data is closer to the actual environment, thereby improving the generalization ability and prediction accuracy of the model.

[0009] Optionally, in the embodiments of the present application, before the preprocessed sample traffic log is analyzed according to the sample field using the field analysis engine, the method further includes: testing and optimizing the field analysis engine using a real traffic log. In the implementation process of the above scheme, by testing and optimizing using the real traffic log, it can be verified that the performance of the field analysis engine in the actual environment is consistent with the actual situation, and the real traffic log contains various actual network activities. By testing and optimizing these data, the false positives and false negatives of the field analysis engine can be reduced, and the analysis accuracy of the field analysis engine is improved.

[0010] Optionally, in the embodiments of the present application, the artificial intelligence model includes: a long short-term memory (LSTM) model and / or an extreme gradient boosting (XGBoost) tree model. In the implementation process of the above scheme, the LSTM model is good at processing and predicting time series data and can capture long-term dependencies, which is very suitable for analyzing time series features in network traffic logs, and the XGBoost tree model can accurately perform classification and regression tasks and is suitable for identifying abnormal behaviors and patterns in network traffic.

[0011] The embodiments of the present application also provide a network traffic log analysis device, which includes: an analysis request obtaining module, configured to obtain a scene analysis request for a network traffic log generated under a target scene, the scene analysis request including a scene field; and a scene field analysis module, configured to input the scene field into a field analysis engine and an artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scene field, and the artificial intelligence model predicts a log analysis report of the field analysis engine according to the scene field to obtain a prediction result corresponding to the scene analysis request.

[0012] Optionally, in the embodiment of the present application, the scene field comprises a time field and a plurality of independent fields; and the scene field analysis module comprises a target scene splitting submodule, configured to split the target scene according to the time field and the plurality of independent fields to obtain a plurality of sub-scenes, wherein each sub-scene is composed of the time field and at least one independent field; an analysis result obtaining submodule, configured to, for each sub-scene of the plurality of sub-scenes, analyze the network traffic log using the fields in the sub-scene to obtain a plurality of analysis results; and an analysis result aggregating submodule, configured to aggregate the plurality of analysis results to obtain a log analysis report.

[0013] Optionally, in the embodiment of the present application, the analysis result obtaining submodule comprises a historical time slicing unit, configured to slice historical time in the network traffic log into a plurality of time segments according to the time field in the sub-scene; and a traffic log analysis unit, configured to, for each time segment of the plurality of time segments, perform frequency analysis, trend analysis and / or anomaly analysis on the network traffic log according to the time segment and at least one independent field in the sub-scene.

[0014] Optionally, in the embodiment of the present application, the analysis result aggregating submodule comprises an analysis result aggregating unit, configured to aggregate the plurality of analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

[0015] Optionally, in the embodiment of the present application, the network traffic log analysis apparatus further comprises a sample data obtaining module, configured to obtain sample traffic logs and sample fields, wherein the sample traffic logs comprise real traffic logs generated under the target scene and / or simulated traffic logs generated under the target scene, and the sample fields are the same as the scene fields; a traffic log preprocessing module, configured to preprocess the sample traffic logs to obtain preprocessed sample traffic logs; a traffic log analysis module, configured to analyze the preprocessed sample traffic logs according to the sample fields using the field analysis engine to obtain sample analysis results; and an intelligent model training module, configured to train the artificial intelligence model by taking the sample fields as training data and taking the sample analysis results as training labels.

[0016] Optionally, in the embodiment of the present application, the network traffic log analysis apparatus further comprises an engine test and optimization module, configured to test and optimize the training of the field analysis engine using real traffic logs.

[0017] Optionally, in the embodiment of the present application, the artificial intelligence model comprises a long short-term memory (LSTM) model and / or an extreme gradient boosting (XGBoost) tree model.

[0018] The embodiment of the present application further provides an electronic device, comprising a processor and a memory, the memory stores machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method described above.

[0019] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the method described above.

[0020] The embodiment of the present application further provides a computer program product, comprising a computer program or computer instructions, and the computer program or computer instructions are executed by the processor to perform the method described above. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 The flowchart of the network traffic log analysis method provided by the embodiment of the present application is shown;

[0023] Figure 2 The flowchart of training the artificial intelligence model provided by the embodiment of the present application is shown;

[0024] Figure 3 The structural diagram of the network traffic log analysis device provided by the embodiment of the present application is shown;

[0025] Figure 4 The structural diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. It should be understood that the accompanying drawings in the embodiments of the present application are only for the purpose of illustration and description, and are not used to limit the scope of protection of the embodiments of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the embodiments of the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the embodiments of the present application.

[0027] In addition, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but only represents selected embodiments of the present application.

[0028] It can be understood that "first", "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that "first", "second" and the like do not limit the number and execution order, and "first", "second" and the like do not necessarily mean different. In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship. The term "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more groups (including two groups).

[0029] It should be noted that the network traffic log analysis method provided by the embodiments of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or a server having the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. The server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, including a mainframe, a minicomputer and a UNIX server.

[0030] The following introduces an example of an application scenario to which the network traffic log analysis method is applicable: in a scenario in which the amount of data of network traffic logs is very large and TBs of network traffic logs are generated every day, the network traffic log analysis method can be used to predict the analysis result of the network traffic logs, thereby effectively reducing the waiting time for network traffic log analysis and improving the real-time performance of network traffic log analysis.

[0031] See Figure 1 A flowchart of the network traffic log analysis method provided by the embodiments of the application is shown. The embodiments of the application provide a network traffic log analysis method, which comprises the following steps:

[0032] In step S110, a scene analysis request for network traffic logs generated in a target scene is acquired, and the scene analysis request comprises a scene field.

[0033] The target scene refers to a selected scene on a terminal device, which can be a scene including a scene field selected from two categories of preset scenes and specified scenes. The selectable scenes include but are not limited to the following: an event trend scene, a protocol trend scene, an attack trend scene, a protocol abnormal access scene, a HyperText Transfer Protocol (HTTP) website distribution scene, a user access network situation scene, and the like. The preset scenes are scenes pre-prepared and fixed in a security software product by a security software manufacturer, and the specified scenes are custom scenes generated by selecting a scene field on a terminal device. The security software product mentioned above is, for example, a network audit product or an Intrusion Detection System (IDS) product.

[0034] The network traffic logs are logs generated in the target scene. These network traffic logs can be obtained by a security software product in a bypass monitoring environment or a series environment, and the obtained network traffic logs can include audit logs and / or alarm logs. The scene analysis request for the network traffic logs can include an analysis request for a preset scene and an analysis request for a specified scene.

[0035] In step S120, the scene field is input to a field analysis engine and an artificial intelligence model, so that the field analysis engine analyzes the network traffic logs according to the scene field, and the artificial intelligence model predicts the log analysis report of the field analysis engine according to the scene field, to obtain a prediction result corresponding to the scene analysis request.

[0036] With the generation and growth of Internet communication data accelerating continuously, network traffic logs for analysis also grow dramatically, so the field analysis engine needs more time to analyze the network traffic logs, and the user needs to wait patiently for the analysis results. In this case, the artificial intelligence model in the above network traffic log analysis method can be used to quickly obtain the prediction results of log analysis, so that the user can quickly obtain the prediction results of interest, because the prediction results contain analysis trend information of the network traffic logs.

[0037] It can be understood that the order of inputting the scene field into the field analysis engine and the artificial intelligence model can not be limited, and some possible implementation modes are introduced as follows: in a first implementation mode, the scene field is input into the field analysis engine and the artificial intelligence model at the same time. For a target scene with a very large number of logs to be analyzed, the prediction result of the artificial intelligence model is usually obtained first, so the prediction result can be returned to the terminal device first, and then the log analysis report of the field analysis engine is returned. Correspondingly, for a target scene with a very small number of logs to be analyzed, the log analysis report of the field analysis engine is usually obtained first, so the log analysis report of the field analysis engine can be returned to the terminal device first, and then the prediction result of the artificial intelligence model is returned. The scene field is input into the field analysis engine and the artificial intelligence model at the same time, and the two processes can be executed in parallel instead of in series, so that the total processing time can be greatly shortened. In a second implementation mode, the scene field is input into the artificial intelligence model first, and then the scene field is input into the field analysis engine. The prediction result of the artificial intelligence model is usually obtained first, so the prediction result can be returned to the terminal device first, and then the log analysis report of the field analysis engine is returned. In a third implementation mode, the scene field is input into the field analysis engine first, and then the scene field is input into the artificial intelligence model. In this case, the result obtained first is returned to the terminal device.

[0038] Optionally, after the prediction result of the log analysis report by the artificial intelligence model, the prediction result can also be sent to the terminal device of the user, so that the user can obtain the analysis trend information according to the prediction result. Because the artificial intelligence model can provide millisecond-level analysis response compared with the field analysis engine, the artificial intelligence model can more quickly predict the prediction result, so as to meet the timeliness requirement of the user to view the prediction result of the analysis trend. Further, after obtaining the log analysis report of the field analysis engine, the log analysis report with higher accuracy is sent to the terminal device of the user, so as to meet the accuracy requirement of the user for the log analysis report.

[0039] After the terminal device receives the log analysis report of the field analysis engine, the log analysis report can be rendered according to a target scenario, which can be rendered into a trend chart, a heat map, a proportion chart, or a contrast chart, or the analysis data in the log analysis report can be rendered into a display table and a conclusion summary, or the trend information in the log analysis report can be rendered into a trend chart. The trend chart can include a trend comparison chart of historical data and future data.

[0040] In the implementation process of the above scheme, the field to be processed is input into the field analysis engine and the artificial intelligence model respectively, so that the artificial intelligence model predicts the analysis result of the field analysis engine. Compared with the speed of direct analysis by the field analysis engine, the artificial intelligence model can quickly predict the prediction result corresponding to the scene analysis request, thereby effectively reducing the waiting time of network traffic log analysis and improving the real-time performance of network traffic log analysis.

[0041] An optional implementation of the above step S110 is, for example: the user selects the network traffic log that needs to be analyzed in the target scenario on the terminal device, and then clicks the button or right-click menu for analyzing the network traffic log generated in the target scenario, generates a scene analysis request according to the scene field required by the target scenario and the identifier of the network traffic log, and sends the scene analysis request to the server. Then, the server receives the scene analysis request sent by the terminal device, and parses the scene field required by the target scenario and the identifier of the network traffic log from the scene analysis request. The above-mentioned server can be a server installed in a user local area network, or a cloud server in the Internet, which can be set according to specific conditions.

[0042] As an optional implementation of the above step S120, the scene field can include a time field and a plurality of independent fields; and the implementation of the field analysis engine analyzing the network traffic log according to the scene field can include:

[0043] Step S121: splitting the target scenario according to the time field and the plurality of independent fields to obtain a plurality of sub-scenes, each sub-scene being composed of the time field and at least one independent field.

[0044] An embodiment of the above step S121 is as follows: assuming that the target scenario is "protocol access of source IP from 12:00 to 13:00 every day", it can be seen that the target scenario contains a time field (i.e., from 12:00 to 13:00 every day), an independent field of a source IP, and an independent field of a protocol. In this case, the field analysis engine can split the target scenario into two sub-scenarios, including: a first sub-scenario of "time + source IP", and a second sub-scenario of "time + protocol". Alternatively, the user can customize the splitting manner of the scene field combination according to the needs, which enables the system to adapt to different business requirements and technical environments. For example: the system parses the splitting identifier from the scene analysis request and finds the splitting strategy corresponding to the splitting identifier, and then combines and splits the time field and the multiple independent fields in the target scenario according to the splitting strategy to obtain multiple sub-scenarios. Through the combination analysis of the multiple independent fields and the time field, the system can provide rich perspectives to help the user comprehensively understand the network traffic. The user can freely combine the time field and the independent field to form different sub-scenarios to meet specific analysis needs. This flexibility enables the system to quickly adapt to new business requirements and technical environments.

[0045] Step S122: for each of the multiple sub-scenarios, using the fields in the sub-scenario to analyze the network traffic log to obtain multiple analysis results.

[0046] Step S123: aggregating the multiple analysis results to obtain a log analysis report.

[0047] It can be understood that since there are many analysis methods and aggregation methods, such as frequency analysis, trend analysis, and / or anomaly analysis, etc., embodiments of the two steps S122 and S123 will be described in detail below.

[0048] As an optional embodiment of the above step S122, the above embodiment of using the fields in the sub-scenario to analyze the network traffic log can include:

[0049] Step S122a: according to the time field in the sub-scenario, the historical time in the network traffic log is divided into multiple time segments.

[0050] An embodiment of the above step S122a is, for example: assuming that the time field in the sub-scene is 12:00-13:00 every day, the historical time in the network traffic log can be first divided into time segments of each day, and then each time segment of each day is divided into time segments of each hour, and finally the log data corresponding to the time segment of 12:00-13:00 every day is obtained from the network traffic log, so as to reduce the required memory resources and computing resources for analysis, and also reduce the required memory resources, communication resources and computing resources for data acquisition. Of course, the time can also be divided into other time segments according to specific needs, such as week or timestamp dimension characteristics.

[0051] Step S122b: for each time segment in the plurality of time segments, performing frequency analysis, trend analysis, clustering analysis and / or anomaly analysis on the network traffic log according to at least one independent field in the time segment and the sub-scene.

[0052] An embodiment of the above step S122b is, for example: taking 12:00-13:00 today as an example, after obtaining the first part of the network traffic log corresponding to the time segment of 12:00-13:00 today, the first part of the network traffic log can be analyzed using the sub-scene of “time+source IP” to obtain the analysis result of the first sub-scene, and the first part of the network traffic log can also be analyzed using the sub-scene of “time+protocol” to obtain the analysis result of the second sub-scene. By performing trend analysis on historical data of different time segments, the system can dynamically adjust the threshold of anomaly detection according to actual data, thereby improving the accuracy and sensitivity of anomaly detection.

[0053] As an optional embodiment of the above step S123, the above embodiment of aggregating the plurality of analysis results can include:

[0054] Step S123a: aggregating the plurality of analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multi-dimensional data analysis algorithm.

[0055] The implementation of the above step S123 is, for example: time as the hub of a plurality of analysis results are aggregated, specifically, can use data merging algorithm, statistical analysis algorithm, anomaly detection algorithm, clustering algorithm, association rule algorithm and / or multidimensional data analysis algorithm, etc. The combination of the above data merging algorithm, statistical analysis algorithm, anomaly detection algorithm, clustering algorithm, association rule algorithm and / or multidimensional data analysis algorithm, etc. The application of multiple algorithms can analyze data from different angles and provide more information, for example, statistical analysis can reveal overall trends, association rule algorithm can find the implicit relationship between different fields, and clustering algorithm can identify similar behavior patterns. Further, the multidimensional data analysis algorithm described above can process high-dimensional data sets to help identify complex patterns and hidden relationships, which is particularly important for understanding subtle changes and potential threats in network traffic. The cross-validation mechanism of the above-mentioned multiple algorithms can effectively reduce the false alarm rate, for example, the anomaly detection algorithm can identify potential abnormal behavior, and the clustering algorithm can help confirm whether these behaviors belong to normal patterns, thereby improving the accuracy of judgment.

[0056] Optionally, after obtaining the log analysis report of the network traffic log analysis by the field analysis engine and the prediction result of the log analysis report by the artificial intelligence model, if it is found that the gap between the log analysis report and the prediction result is very large, the network traffic log and the log analysis report can be used as training data for the artificial intelligence model to train the artificial intelligence model. Specifically, it can include: obtaining real traffic log generated under the target scene and log analysis report corresponding to the real traffic log, preprocessing the real traffic log to obtain preprocessed traffic log. Analyze the preprocessed traffic log according to the scene field using the field analysis engine to obtain the log analysis result, and finally use the scene field as the training data and the log analysis result as the training label to train the artificial intelligence model. Wherein, training the artificial intelligence model includes: using the artificial intelligence model to predict the log analysis report of the field analysis engine according to the scene field, obtaining the model prediction result, and calculating the loss value between the model prediction result and the log analysis result, then updating the model parameters of the artificial intelligence model according to the loss value, until the model parameters of the artificial intelligence model converge, thereby obtaining the artificial intelligence model after retraining.

[0057] Please refer to Figure 2 The flowchart of training the artificial intelligence model provided by the embodiments of the application is shown; as an optional implementation of the above network traffic log analysis method, the artificial intelligence model can also be trained before the scene field is input into the field analysis engine and the artificial intelligence model. This implementation can include:

[0058] Step S210: Obtain a sample traffic log and a sample field, the sample traffic log including: a real traffic log generated under a target scenario, and / or a simulated traffic log generated under a simulated target scenario, the sample field being the same as the scenario field.

[0059] It can be understood that the above-mentioned sample traffic log can be a real traffic log generated under a target scenario fed back by a user, or a simulated traffic log collected under a simulated target scenario. The artificial intelligence model can recognize complex patterns and implicit relationships in the sample traffic log by continuously learning from historical sample traffic log data, so as to more accurately predict future trends and abnormal behaviors. The above-mentioned simulated traffic log under a target scenario can cover extreme situations or special events that can be difficult to capture in reality, enriching the diversity of training data, so that the model can better cope with various complex network environments. By using a real traffic log generated under a target scenario as part of the sample traffic log, it can be ensured that the training data is closer to the actual environment, thereby improving the generalization ability and prediction accuracy of the model.

[0060] Step S220: Preprocess the sample traffic log to obtain a preprocessed sample traffic log.

[0061] In the preprocessing process of the sample traffic log, the fields of interest can include: log time, source IP, destination IP, destination port, application protocol type, attack type, attack result, and risk level, etc. The user can generate a simulated traffic log according to specific needs, flexibly configure the fields of interest and different scenario parameters, so that the system can quickly adapt to new business needs and technical environments.

[0062] The implementation of the above-mentioned step S220 is, for example: through a data preprocessing software, the sample traffic log is preprocessed, such as data cleaning, format conversion, removal of missing fields or missing value filling, etc., to obtain a preprocessed sample traffic log, so as to ensure that the data input into the analysis engine has high quality and good consistency, and reduce the influence of noise and outliers. After obtaining the preprocessed sample traffic log and the sample field, the preprocessed sample traffic log and the sample field can be segmented according to a preset ratio (for example, 8:2) to obtain a training set and a test set. The training set can be subjected to the following training process, and the test set can be used to test the performance of the trained artificial intelligence model.

[0063] Step S230: Analyze the preprocessed sample traffic log according to the sample field using a field analysis engine to obtain a sample analysis result.

[0064] The implementation of the above step S230 may be, for example: the above-mentioned sample field may include a time field and a plurality of independent fields; the target scene is split according to the time field and the plurality of independent fields by the field analysis engine, and a plurality of sub-scenes are obtained, wherein each sub-scene is composed of the time field and at least one independent field. For each of the plurality of sub-scenes, the sample traffic log is analyzed using the fields in the sub-scene to obtain a plurality of analysis results. The plurality of analysis results are aggregated to obtain a sample analysis result. It can be understood that the implementation of each sub-step is similar or identical to the implementation of steps S121 to S123, and thus will not be described again. The preprocessing and analysis tasks of the sample traffic log can be performed in parallel on a plurality of computing nodes, fully utilizing distributed computing resources to accelerate the entire training process.

[0065] Step S240: training the artificial intelligence model using the sample field as the training data and the sample analysis result as the training label.

[0066] The implementation of the above step S240 may be, for example: using the artificial intelligence model to predict the log analysis report of the field analysis engine according to the sample field, obtaining a model prediction result, and calculating a loss value between the model prediction result and the sample analysis result, then updating the model parameters of the artificial intelligence model according to the loss value until the model parameters of the artificial intelligence model converge, thereby obtaining the trained artificial intelligence model. After obtaining the trained artificial intelligence model, the trained artificial intelligence model can also be exported. The staff can also use the trained artificial intelligence model to perform performance testing, and if the performance of the artificial intelligence model exceeds the preset performance indicators, the artificial intelligence model can be imported into the network audit product or the intrusion detection system (IDS) product in the user LAN.

[0067] As an optional implementation of the above network traffic log analysis method, before the preprocessed sample traffic log is analyzed by the field analysis engine according to the sample field, the field analysis engine can also be tested and optimized, and then the analysis result of the field analysis engine is used to train the model. The implementation of testing and optimizing the field analysis engine may include:

[0068] Step S231: testing and optimizing the field analysis engine using the real traffic log.

[0069] The implementation of the above step S231 may be, for example: testing and optimizing the field analysis engine using the real traffic log to obtain an optimized field analysis engine. After obtaining the optimized field analysis engine, the optimized field analysis engine can also be exported so that the staff can import the optimized field analysis engine into the network audit product or the intrusion detection system (IDS) product in the user LAN.

[0070] Optionally, after obtaining the above-mentioned optimized field analysis engine and the trained artificial intelligence model, the optimized field analysis engine and the trained artificial intelligence model can also be added to a system upgrade package, so that the user uses the system upgrade package to upgrade the network audit product or the intrusion detection system (IDS) product and the like in the local area network, and optimizes the performance of the security software product, for example, can reduce the gap between the log analysis report of the field analysis engine and the prediction result of the artificial intelligence model, thereby enhancing the accuracy of the security software product.

[0071] As an optional implementation of the above-mentioned network traffic log analysis method, the artificial intelligence model includes: a long short-term memory (LSTM) model, a bidirectional long short-term memory (BiLSTM) model, and / or an extreme gradient boosting XGBoost tree model, wherein the LSTM is a special recurrent neural network (RNN) that is good at processing and predicting time series data, and the XGBoost is a gradient boosting framework based on decision tree, which has high computing power and strong feature selection ability.

[0072] Optionally, in the process of using the artificial intelligence model to predict the log analysis report of the field analysis engine according to the scene field, the above-mentioned artificial intelligence model can be the performance-optimal artificial intelligence model selected from multiple artificial intelligence models according to the target scene for prediction. For example, for the scene of analyzing the time series network traffic log, the LSTM model or the BiLSTM model can be selected for prediction, because the LSTM model or the BiLSTM model is good at processing and predicting time series data, and can capture long-time dependence, which is very suitable for analyzing the time series features in the network traffic log, such as traffic trend, periodicity and anomaly detection. For example, for the scene of analyzing the network topology collected from the network topology, the extreme gradient boosting XGBoost tree model can be selected for prediction. The XGBoost tree model performs well in processing structured data, and can accurately perform classification and regression tasks, and is suitable for identifying abnormal behaviors and patterns in network traffic.

[0073] Please refer to Figure 3 The structure schematic diagram of the network traffic log analysis device provided by the embodiment of the application is shown; the embodiment of the application provides a network traffic log analysis device 300, which comprises:

[0074] The analysis request acquisition module 310 is configured to acquire a scene analysis request for network traffic log generated under a target scene, wherein the scene analysis request comprises a scene field.

[0075] The scene field analysis module 320 is configured to input the scene field into the field analysis engine and the artificial intelligence model respectively, so that the field analysis engine analyzes the network traffic log according to the scene field, and the artificial intelligence model predicts the log analysis report of the field analysis engine according to the scene field, to obtain a prediction result corresponding to the scene analysis request.

[0076] As an optional implementation of the above device, the scene field includes a time field and a plurality of independent fields; and the scene field analysis module includes:

[0077] The target scene splitting sub-module is configured to split the target scene according to the time field and the plurality of independent fields, to obtain a plurality of sub-scenes, each of which is composed of the time field and at least one independent field.

[0078] The analysis result obtaining sub-module is configured to, for each of the plurality of sub-scenes, analyze the network traffic log using the fields in the sub-scene, to obtain a plurality of analysis results.

[0079] The analysis result aggregating sub-module is configured to aggregate the plurality of analysis results, to obtain a log analysis report.

[0080] As an optional implementation of the above device, the analysis result obtaining sub-module includes:

[0081] The historical time slicing unit is configured to slice historical time in the network traffic log into a plurality of time segments according to the time field in the sub-scene.

[0082] The traffic log analysis unit is configured to, for each of the plurality of time segments, perform frequency analysis, trend analysis and / or anomaly analysis on the network traffic log according to the time segment and at least one independent field in the sub-scene.

[0083] As an optional implementation of the above device, the analysis result aggregating sub-module includes:

[0084] The analysis result aggregating unit is configured to aggregate the plurality of analysis results using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multidimensional data analysis algorithm.

[0085] As an optional implementation of the above device, the network traffic log analysis device further includes:

[0086] The sample data obtaining module is configured to obtain sample traffic logs and sample fields, the sample traffic logs including real traffic logs generated under the target scene and / or simulation traffic logs generated under the target scene, and the sample fields being the same as the scene fields.

[0087] The traffic log preprocessing module is configured to preprocess the sample traffic log to obtain a preprocessed sample traffic log.

[0088] The traffic log analysis module is configured to analyze the preprocessed sample traffic log according to the sample fields using the field analysis engine to obtain a sample analysis result.

[0089] The intelligent model training module is configured to train an artificial intelligence model using the sample fields as training data and using the sample analysis result as a training label.

[0090] As an optional implementation of the above device, the network traffic log analysis device further comprises:

[0091] The engine test and optimization module is configured to test and optimize the field analysis engine using the real traffic log.

[0092] As an optional implementation of the above device, the artificial intelligence model comprises a long short-term memory (LSTM) model and / or an extreme gradient boosting (XGBoost) tree model.

[0093] It should be understood that the device corresponds to the above network traffic log analysis method embodiments, and can perform each step involved in the above method embodiments. The specific functions of the device can be referred to the description above, and the detailed description is appropriately omitted here. The device comprises at least one software function module which can be stored in the memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0094] Please refer to Figure 4 The electronic device 400 provided by the embodiment of the present application comprises a processor 410 and a memory 420. The memory 420 stores machine readable instructions executable by the processor 410. When the machine readable instructions are executed by the processor 410, the method described above is performed.

[0095] The embodiment of the present application further provides a computer readable storage medium 430, and the computer readable storage medium 430 stores a computer program. The computer program is run by the processor 410 and executes the method described above. The computer readable storage medium 430 can be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0096] The embodiment of the present application further provides a computer program product, comprising: a computer program or computer instruction, which is run by a processor to execute the method described above.

[0097] It should be noted that each embodiment in the specification adopts a progressive manner for description, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment.

[0098] In several embodiments provided by the embodiment of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other ways. The apparatus embodiments described above are only schematic, and the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can be performed in a different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved.

[0099] In addition, each function module in each embodiment in the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0100] The above description is only an optional implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. A network traffic log analysis method characterized by, The method comprises the following steps: obtaining a scene analysis request for network traffic logs generated under a target scene, wherein the scene analysis request comprises a scene field; inputting the scene field into a field analysis engine and an artificial intelligence model respectively, so as to make the field analysis engine analyze the network traffic logs according to the scene field, and make the artificial intelligence model predict a log analysis report of the field analysis engine according to the scene field, and obtain a prediction result corresponding to the scene analysis request; wherein the scene field comprises a time field and a plurality of independent fields; the analysis of the network traffic logs according to the scene field comprises: splitting the target scene according to the time field and the plurality of independent fields to obtain a plurality of sub-scenes, wherein each sub-scene is composed of the time field and at least one independent field; for each sub-scene in the plurality of sub-scenes, using the fields in the sub-scene to analyze the network traffic logs to obtain a plurality of analysis results; and aggregating the plurality of analysis results to obtain the log analysis report.

2. The method of claim 1, wherein, The analysis of the network traffic logs using the fields in the sub-scene comprises: dividing historical time in the network traffic logs into a plurality of time segments according to the time field in the sub-scene; for each time segment in the plurality of time segments, performing frequency analysis, trend analysis and / or anomaly analysis on the network traffic logs according to at least one independent field in the time segment and the sub-scene.

3. The method of claim 1, wherein, The aggregation of the plurality of analysis results comprises: using a data merging algorithm, a statistical analysis algorithm, an anomaly detection algorithm, a clustering algorithm, an association rule algorithm and / or a multi-dimensional data analysis algorithm to aggregate the plurality of analysis results.

4. The method of claim 1, wherein, Before the scene field is inputted into the field analysis engine and the artificial intelligence model, the method further comprises the following steps: obtaining sample traffic logs and sample fields, wherein the sample traffic logs comprise real traffic logs generated under the target scene and / or simulated traffic logs simulated under the target scene, and the sample fields are the same as the scene fields; preprocessing the sample traffic logs to obtain preprocessed sample traffic logs; using the field analysis engine to analyze the preprocessed sample traffic logs according to the sample fields to obtain sample analysis results; training the artificial intelligence model by taking the sample fields as training data and taking the sample analysis results as training labels.

5. The method of claim 4, wherein, Before the preprocessed sample traffic logs are analyzed according to the sample fields by using the field analysis engine, the method further comprises the following step: testing and optimizing the field analysis engine by using the real traffic logs.

6. The method according to any one of claims 1 to 5, characterized in that, The artificial intelligence model comprises a long short-term memory (LSTM) model and / or an extreme gradient boosting (XGBoost) tree model.

7. An electronic device, comprising: The method comprises the following steps: a processor and a memory, wherein the memory stores machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by the processor to execute the method in any one of claims 1 to 6.

9. A computer program product, characterised in that, comprises: a computer program or computer instructions, and the computer program or the computer instructions are run by the processor to execute the method in any one of claims 1 to 6.

Images