Weblog data analysis method and device, computer equipment and storage medium

Abstract

The invention relates to a weblog data analysis method and device, computer equipment and a storage medium. The method comprises the following steps: collecting weblog data of target network equipment; according to the log analysis rule, analyzing the weblog data to generate a log analysis result; the log analysis rule is generated according to a network environment modeling result in a target time period; the network environment modeling result is constructed based on sensing data of a network environment where the target network equipment is located in a historical time period; the historical time period is before the target time period; and outputting network security alarm information of the target network equipment according to a log analysis result. By adopting the method, the log analysis rule can be generated in combination with the dynamic modeling result of the network environment under the condition of carrying out dynamic modeling on the network environment, and the network log data can be accurately analyzed by utilizing the log analysis rule, so that log analysis is more suitable for complex and changeable network environments and network security requirements, and the user experience is improved. And the accuracy of log analysis is improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing security technology, and in particular to a network log data parsing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of cloud computing and the increasing complexity of cybersecurity threats, anomaly detection and parsing in log data have become key technical challenges. In cloud computing environments, log data, as a key source of information recording system activity, contains rich operational details and behavioral patterns. However, its massive size and complex structure make extracting valuable information from it extremely challenging. Accurate and reliable log parsing can help identify potential security risks.

[0003] Traditional technologies mainly use pre-set static rules to parse logs. However, traditional technologies cannot adapt to complex and changing network environments and network security requirements, which is not conducive to improving the accuracy of log parsing. Summary of the Invention

[0004] Based on this, it is necessary to provide a network log data parsing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of log parsing in order to address the above technical problems.

[0005] In a first aspect, the present application provides a network log data parsing method, comprising:

[0006] Collect network log data from target network devices;

[0007] Parsing the network log data according to log parsing rules to generate log parsing results; wherein the log parsing rules are generated based on network environment modeling results within a target time period; the network environment modeling results are constructed based on perception data of the network environment in which the target network device is located within a historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information, and device information;

[0008] According to the log analysis result, network security alarm information of the target network device is output.

[0009] In one embodiment, the method further comprises:

[0010] Obtaining traffic prediction information of the network environment in which the target network device is located during the target time period;

[0011] Determine network environment information within the historical time period based on network traffic information, user behavior information, and device information of the network environment in which the target network device is located within the historical time period;

[0012] Inputting the traffic prediction information and the network environment information into a pre-trained network environment modeling model to obtain a network environment modeling result within the target time period;

[0013] The difference between the network environment modeling result within the target time period and the network environment information within the historical time period meets a preset condition.

[0014] In one embodiment, after the step of obtaining the network environment modeling result within the target time period, the method further includes:

[0015] Clustering log events corresponding to the network log data according to the network log data and the network environment modeling result within the target time period to obtain log event clusters;

[0016] A log parsing rule for parsing the network log data of the target network device is generated based on the log event cluster.

[0017] In one embodiment, generating log parsing rules for parsing network log data of the target network device based on the log event clusters includes:

[0018] For each of the log event clusters, generating an initial parsing rule according to the log events in the log event cluster;

[0019] Obtaining a weight of the initial parsing rule and an applicability probability of the initial parsing rule under a network environment modeling result within the target time period, and determining evaluation information of the initial parsing rule based on the weight and the applicability probability;

[0020] The log parsing rule is filtered out from the initial parsing rules according to the evaluation information.

[0021] In one embodiment, after the step of collecting network log data of the target network device, the method further includes:

[0022] Determining information entropy based on the frequency of occurrence of network log data of the target network device;

[0023] Noise data is filtered from the network log data of the target network device according to the information entropy and a preset information entropy threshold.

[0024] In one embodiment, the method further comprises:

[0025] Obtaining a network security identification result of the target network device, and optimizing the log parsing rule based on the network security identification result and the network security alarm information;

[0026] and / or

[0027] Obtain user feedback information regarding the log parsing rule, and optimize the log parsing rule based on the feedback information.

[0028] In a second aspect, the present application further provides a network log data parsing device, comprising:

[0029] A collection module, used to collect network log data of target network devices;

[0030] a parsing module, configured to parse the network log data according to log parsing rules to generate log parsing results; wherein the log parsing rules are generated based on network environment modeling results within a target time period; the network environment modeling results are constructed based on perception data of the network environment in which the target network device is located within a historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information, and device information;

[0031] The alarm module is used to output network security alarm information of the target network device according to the log analysis result.

[0032] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0034] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0035] The network log data parsing method, apparatus, computer device, computer-readable storage medium, and computer program product described above collect network log data from a target network device, parse the network log data according to log parsing rules, and generate log parsing results; wherein the log parsing rules are generated based on the network environment modeling results within a target time period; the network environment modeling results are constructed based on the perception data of the network environment in which the target network device is located within a historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information, and device information, thereby utilizing the network traffic, user behavior, and device information of the network environment in which the target network device is located within the historical time period. Perceive data, build the network environment modeling results within the target time period, and generate log parsing rules based on the network environment modeling results within the target time period, and then parse the network log data. According to the log parsing results, output the network security alarm information of the target network device, and thus analyze the network security alarm of the target network device based on the log parsing results obtained by parsing the network log data using the log parsing rules. In the case of dynamic modeling of the network environment, log parsing rules can be generated in combination with the dynamic modeling results of the network environment, and the log parsing rules can be used to accurately parse the network log data, so that log parsing can be more adaptable to complex and changeable network environments and network security requirements, thereby improving the accuracy of log parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A diagram illustrating an application environment of a network log data parsing method according to an embodiment;

[0038] Figure 2 A schematic diagram of a flow chart of a network log data parsing method according to an embodiment;

[0039] Figure 3 A flowchart of an intelligent log parsing method based on dynamic environment modeling in one embodiment;

[0040] Figure 4 is a structural block diagram of a network log data parsing device in one embodiment;

[0041] Figure 5 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The network log data parsing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104, or placed on the cloud or other network servers. Terminal 102 collects network log data from target network devices. Terminal 102 parses the network log data according to log parsing rules to generate log parsing results. The log parsing rules are generated based on the network environment modeling results within a target time period. The network environment modeling results are constructed based on the perception data of the network environment in which the target network device is located within a historical time period. The historical time period is before the target time period. The perception data includes network traffic information, user behavior information, and device information. Based on the log parsing results, terminal 102 outputs network security alarm information for the target network device. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0044] In an exemplary embodiment, Figure 2 As shown, a network log data parsing method is provided, which is described by taking the method applied to a terminal as an example, and includes the following steps S202 to S206. Among them:

[0045] Step S202: collecting network log data of the target network device.

[0046] The target network device may include but is not limited to a server, a user terminal, etc.

[0047] Network log data refers to recorded information generated by servers, applications, or network devices during network activity. Network log data can represent network requests, responses, error messages, user behavior, and more. In practice, network log data may include, but is not limited to, access time, IP address, request content, and status code.

[0048] As an example, with the rapid development of cloud computing and the increasing complexity of network security threats, anomaly detection and parsing of log data have become key technical challenges. Traditional log parsing methods mostly rely on static rule sets or single-dimensional environmental perception, making them difficult to adapt to the ever-changing network environment and its potential threats. To accurately parse network log data, terminals can obtain network log data from target network devices such as servers and user terminals for subsequent parsing.

[0049] Step S204: parse the network log data according to the log parsing rules to generate a log parsing result.

[0050] Among them, log parsing rules can be generated based on the network environment modeling results within the target time period. The network environment modeling results can be constructed based on the perception data of the network environment in which the target network device is located within the historical time period. The perception data includes network traffic information, user behavior information and device information (such as device status). The historical time period is before the target time period.

[0051] In practical applications, user behavior information can be determined by recording user behavior patterns (such as login frequency, access paths, and abnormal operations). For example, a terminal can pre-build a user behavior model, and the terminal inputs user behavior records into the user behavior model to obtain user behavior information. In specific implementations, the user behavior model can be expressed as:

[0052] .

[0053] in, Can refer to user behavior The weight of B can refer to the user behavior record.

[0054] The log parsing results may refer to the information obtained by parsing network log data using log parsing rules. In practical applications, the log parsing results may include but are not limited to log event classification, abnormal behavior, system performance, and security threats.

[0055] As an example, the terminal can obtain log parsing rules and parse the network log data according to the log parsing rules, analyze the log event classification, abnormal behavior, system performance and security threats of the network log data, and generate log parsing results. In actual applications, the log parsing rules can be generated by the network environment modeling results of the target network device within the target time period, and the network environment modeling results of the target network device within the target time period can be constructed by the network traffic information, user behavior information and device information and other perception data of the network environment in which the target network device is located in the historical time period (the historical time period is before the target time period).

[0056] Step S206: Output network security alarm information of the target network device according to the log analysis result.

[0057] Among them, network security alarm information can refer to information that characterizes the security risks existing in the target network equipment. In actual applications, security risks can include security threat events, which can include abnormal behavior, security vulnerabilities, malicious activities, data leakage risks and system configuration problems.

[0058] As an example, the terminal can analyze whether there is a security risk in the target network device and the specific security risk content when there is a security risk based on the log analysis results. Whether there is a security risk and the content of the security risk can be used as a network security alarm information. The terminal can output the network security alarm information and display the network security alarm information to the user in the form of text, images, voice, etc.

[0059] In actual applications, the terminal can analyze whether the log event meets the alarm triggering conditions of each alarm based on the log parsing results. For example, the terminal can use the Bayesian network model to perform risk assessment and determine whether the log event meets the alarm triggering conditions:

[0060] .

[0061] Here, A may refer to a log event in a log parsing result of network log data, and B may refer to a log feature of the network log data.

[0062] The terminal can also determine the alarm level based on the severity and frequency of the log event. The calculation expression of the alarm level L can be expressed as:

[0063] .

[0064] Among them, a j It can refer to the weight of the jth type of alarm, a j It can be preset and can be modified by the user. j It can refer to the frequency of log events, S j It can refer to the risk score of the jth type of alarm, S j The default values can be set and can be modified by the user. By comprehensively considering the weight, frequency and risk score of the alarm, the accurate classification and effective management of network security alarm information are ensured. Network security alarm information can be sent to security administrators through the alarm notification system and recorded in the system log.

[0065] In the above-mentioned network log data parsing method, network log data of a target network device is collected, and the network log data is parsed according to log parsing rules to generate log parsing results. The log parsing rules are generated based on the network environment modeling results within a target time period. The network environment modeling results are constructed based on the perception data of the network environment in which the target network device is located within a historical time period. The historical time period is before the target time period. The perception data includes network traffic information, user behavior information, and device information. Thus, the network environment modeling results within the target time period are constructed using the perception data such as network traffic, user behavior, and device information of the network environment in which the target network device is located within the historical time period. Based on the network environment modeling results within the target time period, log parsing rules are generated to parse the network log data. Based on the log parsing results, network security alarm information of the target network device is output. Based on the log parsing results obtained by parsing the network log data using the log parsing rules, network security alarms of the target network device are analyzed. In the case of dynamic modeling of the network environment, log parsing rules can be generated in combination with the dynamic modeling results of the network environment, and the network log data can be accurately parsed using the log parsing rules, thereby making log parsing more adaptable to complex and changing network environments and network security requirements, thereby improving the accuracy of log parsing.

[0066] In an exemplary embodiment, the above method also includes: obtaining traffic prediction information of the network environment in which the target network device is located in a target time period; determining the network environment information in the historical time period based on the network traffic information, user behavior information and device information of the network environment in which the target network device is located in the historical time period; inputting the traffic prediction information and the network environment information into a pre-trained network environment modeling model to obtain the network environment modeling result in the target time period; wherein the difference between the network environment modeling result in the target time period and the network environment information in the historical time period meets the preset conditions.

[0067] Among them, the traffic prediction information can be obtained by a traffic prediction model. The traffic prediction model can refer to a model used to combine the network traffic in a certain time period to determine the predicted value of the network traffic in a period of time after the time period. In actual applications, the traffic prediction model can be obtained by training the network traffic in the sample time period.

[0068] Among them, traffic prediction information can refer to information output by the traffic prediction model based on the network traffic information of the network environment where the target network device is located in the historical time period, which characterizes the predicted value of the network traffic of the network environment where the target network device is located in the target time period.

[0069] The network environment information within the historical time period may refer to information used to characterize the network environment, which is determined based on network traffic information, user behavior information, and device information of the network environment where the target network device is located within the historical time period.

[0070] Among them, the network environment modeling model can refer to a model used to combine the network environment information within a certain time period and the network traffic within the prediction time period to determine the predicted value of the network environment within the prediction time period. In actual applications, the output result of the network environment modeling model can be used as the network environment modeling result, and the network environment modeling model can be trained by the network environment information within the sample time period.

[0071] Among them, the network environment modeling result within the target time period may refer to the information output by the network environment modeling model in combination with the traffic prediction information of the target time period and the network environment information within the historical time period, which is used to characterize the network environment within the target time period. In actual applications, the difference between the network environment modeling result within the target time period and the network environment information within the historical time period meets the preset conditions.

[0072] As an example, the terminal can input the network traffic information of the network environment in which the target network device is located during a historical time period into a pre-trained traffic prediction model. The traffic prediction model can combine the network traffic information during the historical time period to determine the traffic prediction information for the target time period. The terminal can then analyze the network environment during the historical time period based on the network traffic information, user behavior information, and device information of the network environment in which the target network device is located during the historical time period, and determine the network environment information during the historical time period. The terminal can then input the traffic prediction information for the target time period and the network environment information during the historical time period into a pre-trained network environment modeling model. The network environment modeling model can combine the traffic prediction information for the target time period and the network environment information during the historical time period to analyze the network environment during the target time period and determine the network environment modeling result for the target time period.

[0073] In practical applications, the training process of the traffic prediction model may include: obtaining network traffic information within a first sample time period and network traffic information within a second sample time period following the first sample time period, inputting the network traffic information within the first sample time period into the traffic prediction model to be trained, obtaining traffic prediction information within the second sample time period, and training the traffic prediction model to be trained based on the network traffic information within the second sample time period and the traffic prediction information within the second sample time period. In a specific implementation, the traffic prediction model can be expressed as:

[0074] .

[0075] Among them, X tIt can refer to the traffic prediction information in the second sample time period, X t-i It can refer to the network traffic information in the first sample time period, i can refer to the time difference between the first sample time period and the second sample time period, and p can refer to the preset maximum value of the time difference between the first sample time period and the second sample time period. can refer to the autoregressive coefficient, It can refer to the moving average coefficient, which is used to measure the impact of the random error at the jth time point in the past on the current value. Q can refer to the order of the moving average term, that is, the coefficient of the random error term at the past q time points included in the model. can be a random error term at time t−j, It can refer to the white noise term.

[0076] In the specific implementation, the terminal can analyze the network traffic information of the network environment where the target network device is located in a historical time period and the traffic prediction information output by the traffic prediction model, and monitor network traffic changes. At the same time, the terminal can also detect abnormal traffic peaks and potential malicious activities based on the network traffic information of the network environment where the target network device is located in a historical time period and / or the traffic prediction information output by the traffic prediction model, and perform traffic anomaly detection in combination with quantile regression.

[0077] In practical applications, the training process of the network environment modeling model may include: obtaining network environment information within a first sample time period, network environment information within a second sample time period after the first sample time period, and traffic prediction information within the second sample time period, inputting the network environment information within the first sample time period and the traffic prediction information within the second sample time period into the network environment modeling model to be trained, obtaining the network environment modeling results within the second sample time period, and training the network environment modeling model to be trained based on the network environment modeling results within the second sample time period and the network environment information within the second sample time period. In a specific implementation, the network environment modeling model can adapt to ridge regression analysis and introduce regularization terms to avoid overfitting. The network environment modeling model can be expressed as:

[0078] .

[0079] in, It can refer to the weight vector of the network environment modeling results, It can refer to the target weight vector, Y can refer to the network environment information, Y can be used as the target variable, X can refer to the network environment modeling result determined by the traffic prediction information, X can be used as the independent variable, It can refer to a regularization parameter. In combination with the expression of the network environment modeling result, it can be understood that the difference between the network environment modeling result in the target time period and the network environment information in the historical time period can meet the preset condition.

[0080] The terminal can use the Bayesian inference method to predict the network status based on the model (such as the network environment modeling results). and the actual observed network status (such as network environment information) Y, dynamically adjust the weight vector , ensuring the real-time accuracy of the network environment modeling model:

[0081] .

[0082] in, It can refer to the posterior probability, can refer to the likelihood function, It can refer to the prior distribution, that is, the posterior probability is proportional to the product between the likelihood function and the prior distribution.

[0083] In this embodiment, traffic prediction information of the network environment in which the target network device is located in a target time period is obtained; network environment information in the historical time period is determined based on network traffic information, user behavior information and device information of the network environment in which the target network device is located in the historical time period; the traffic prediction information and network environment information are input into a pre-trained network environment modeling model to obtain a network environment modeling result in the target time period; wherein the difference between the network environment modeling result in the target time period and the network environment information in the historical time period meets a preset condition, and the network environment in the target time period can be accurately analyzed in combination with traffic prediction and network environment modeling, so as to generate log parsing rules in combination with the network environment in the target time period, and realize log parsing in combination with a dynamic network environment, thereby improving the accuracy of log parsing.

[0084] In some embodiments, after the step of obtaining the network environment modeling results within the target time period, the above method also includes: clustering the log events corresponding to the network log data based on the network log data and the network environment modeling results within the target time period to obtain log event clustering clusters; based on the log event clustering clusters, generating log parsing rules for parsing the network log data of the target network device.

[0085] The log event corresponding to the network log data may refer to a specific, particular piece of information extracted from the network log data. The log event may represent a specific action or state change, such as a login attempt, file access, etc.

[0086] The log event cluster may refer to a result obtained by clustering log events. In practical applications, each log event cluster may include at least one log event.

[0087] As an example, after obtaining the network environment modeling results for a target time period, the terminal can analyze the types / patterns of log events in the network log data under the network environment represented by the network environment modeling results for the target time period based on the network log data and the network environment modeling results for the target time period, and cluster the log events corresponding to the network log data to obtain log event clusters. The terminal can then analyze the impact of the log event type / pattern and information such as specific fields in the log on the log parsing method based on the log events in each log event cluster, and generate log parsing rules for parsing the network log data of the target network device.

[0088] In practical applications, the terminal can analyze network log data and the network environment modeling results within the target time period, use clustering algorithms (such as K-means clustering algorithm) to identify the types / patterns of log events in the network log data, and generate log event clusters C. In order to ensure the accuracy of log event clustering, the terminal can also analyze the clustering accuracy J (C) of the log event clusters. For example, for each log event cluster, the terminal can analyze the log event x i The cluster center (such as the centroid) of the cluster of the log event ) and sum them to obtain the accuracy of each log event cluster. The accuracy of each log event cluster is then summed to obtain the cluster accuracy J(C), where k can refer to the type of log event cluster. If the cluster accuracy J(C) meets the preset accuracy requirement, the terminal can determine that the log event cluster is sufficiently accurate. In a specific implementation, the calculation expression for cluster accuracy J(C) can be expressed as:

[0089] .

[0090] Among them, n k It can refer to the number of log events in the k-th log event cluster, where K is the number of log event clusters. It can refer to the centroid / cluster center of the k-th type of log event cluster.

[0091] In this embodiment, log events corresponding to the network log data are clustered according to the network log data and the network environment modeling results within the target time period to obtain log event clustering clusters; based on the log event clustering clusters, log parsing rules for parsing the network log data of the target network device are generated. This can be combined with the network environment within the target time period to analyze the categories of log events in the network log data, and accurately generate log parsing rules based on log events of different categories, thereby realizing the generation of log parsing rules in combination with a dynamic network environment, so as to accurately parse the logs and improve the accuracy of log parsing.

[0092] In some embodiments, log parsing rules for parsing network log data of a target network device are generated based on log event clusters, including: for each log event cluster, generating an initial parsing rule based on the log events in the log event cluster; obtaining the weight of the initial parsing rule and the applicability probability of the initial parsing rule under the network environment modeling results within the target time period, and determining evaluation information of the initial parsing rule based on the weight and the applicability probability; and screening out log parsing rules from the initial parsing rules based on the evaluation information.

[0093] The initial parsing rule may refer to a log parsing rule initially generated based on log events in a log event cluster and used to parse network log data.

[0094] Among them, the weight of the initial parsing rule can refer to information that characterizes the importance of the initial parsing rule in the set of initial parsing rules. In actual applications, the weight of the initial parsing rule can be generated in real time by the terminal or set by the user. It can be understood that the weight of the initial parsing rule can support modification by the user.

[0095] Among them, the applicability probability may refer to information that characterizes the applicability of the initial parsing rule in the network environment represented by the network environment modeling result within the target time period. In actual applications, the more applicable the initial parsing rule is to the network environment represented by the network environment modeling result within the target time period, the higher the applicability probability is; the less applicable the initial parsing rule is to the network environment represented by the network environment modeling result within the target time period, the lower the applicability probability is. The applicability probability may be generated in real time by the terminal or set by the user. It is understandable that the applicability probability may be modified by the user.

[0096] The evaluation information may refer to information representing the accuracy of the initial parsing rule. In practical applications, the evaluation information may be determined by a preset calculation expression.

[0097] As an example, for each log event cluster, the terminal can analyze the type / pattern of the log event and the impact of information such as specific fields in the log on the log parsing method based on the log events in the log event cluster, and generate initial parsing rules. The initial parsing rules can form an initial parsing rule set. The initial parsing rule set R can be expressed as:

[0098] .

[0099] The terminal can also obtain the weight of each initial parsing rule in the initial parsing rule set and the applicability probability of the initial parsing rule under the network environment modeling results within the target time period. The terminal can then determine the evaluation information of the initial parsing rule based on the weight and applicability probability, and filter out log parsing rules from the initial parsing rules based on the evaluation information.

[0100] In practical applications, filtering out log parsing rules from the initial parsing rules can be achieved by maximizing the objective function. The log parsing rules filtered out from the initial parsing rules can be used as an optimal rule set R * , the objective function can be expressed as:

[0101] .

[0102] in, Can refer to the initial parsing rules The weight of Can refer to the initial parsing rules The applicability probability of the network environment modeled within the target time period. The objective function can be used to generate a set of optimal rules based on the current network environment, improving the accuracy of log parsing rules by optimizing weights and applicability probabilities.

[0103] In the specific implementation, the terminal can also use genetic algorithms to update the rule set composed of log parsing rules, and optimize the objective function through natural selection and cross-mutation, for example: obtaining preset selection factor information , select the first rule from the rule set consisting of log parsing rules And the second rule is different from the first rule To update the rule set, the process of genetic algorithm updating the rule set can be expressed as:

[0104] .

[0105] in, It can refer to the updated log parsing rules.

[0106] In this embodiment, for each log event cluster, an initial parsing rule is generated according to the log events in the log event cluster; the weight of the initial parsing rule and the applicability probability of the initial parsing rule under the network environment modeling result within the target time period are obtained, and the evaluation information of the initial parsing rule is determined according to the weight and the applicability probability; the log parsing rule is screened out from the initial parsing rule according to the evaluation information, and the initial parsing rule can be first generated based on the log event, and then the log parsing rule can be accurately screened out from the initial parsing rule in combination with the weight and the applicability probability of the initial parsing rule, thereby improving the accuracy of the log parsing rule and further improving the accuracy of the log parsing.

[0107] In some embodiments, after the step of collecting network log data of the target network device, the above method also includes: determining information entropy based on the frequency of occurrence of the network log data of the target network device; filtering noise data from the network log data of the target network device based on the information entropy and a preset information entropy threshold.

[0108] The occurrence frequency may refer to information representing the number of times network log data appears within a specific time period.

[0109] The information entropy threshold may refer to information used to distinguish whether network log data is noise data.

[0110] The noise data may refer to data containing errors or anomalies (such as data that deviates from expected values) in network log data.

[0111] As an example, after obtaining the network log data of the target network device, the terminal can perform denoising on the network log data. For example, the terminal can determine the probability of occurrence of the network log data (or log features) based on the frequency of occurrence of the network log data of the target network device (or the log features of the network log data of the target network device). The terminal then determines the information entropy H based on the occurrence probability. The terminal can then filter out the noise data from the network log data of the target network device based on the size relationship / difference between the information entropy and a preset information entropy threshold. The information entropy H(X) of network log data X can be expressed as:

[0112] .

[0113] Among them, P (x i ) can refer to the log feature x i The probability of occurrence, the information entropy threshold can be expressed as H threshold If H(X)>H threshold , the terminal can determine that the network log data X is noise data, and the terminal can filter the noise data from the network log data.

[0114] In this embodiment, information entropy is determined based on the frequency of occurrence of network log data of the target network device; noise data is filtered from the network log data of the target network device based on the information entropy and a preset information entropy threshold. This can analyze the frequency of occurrence of network log data, accurately screen out and filter noise data from the network log data, and avoid interference of noise data in the generation of log parsing rules and log parsing, thereby improving the accuracy of log parsing.

[0115] In some embodiments, the above method also includes: obtaining the network security identification results of the target network device, and optimizing the log parsing rules based on the network security identification results and network security alarm information; and / or obtaining user feedback information on the log parsing rules, and optimizing the log parsing rules based on the feedback information.

[0116] The network security identification result may refer to information obtained after performing network security identification on a target network device. In practical applications, the network security identification result may represent the actual network security (such as security risks, etc.) of the target network device.

[0117] The feedback information may refer to information indicating the user's need to modify the log parsing rules.

[0118] As an example, the terminal can obtain the network security identification results of the target network device, and based on the network security identification results and network security alarm information, analyze whether there are false positives in the network security alarm information, and optimize the log parsing rules based on the false positives. For example, the terminal can use the confusion matrix to evaluate the performance of the log parsing rules and calculate the precision P and recall R:

[0119]

[0120] .

[0121] Among them, TP can refer to true positive (that is, the network security identification result indicates that there is no security risk, and the network security alarm information indicates that there is no security risk), FP can refer to false positive (that is, the network security identification result indicates that there is a security risk, and the network security alarm information indicates that there is no security risk), and FN can refer to false negative (that is, the network security identification result indicates that there is no security risk, and the network security alarm information indicates that there is a security risk).

[0122] The terminal can also obtain user feedback on log parsing rules to optimize them. For example, the terminal can analyze the feedback to determine the required changes to the log parsing rules and modify the log parsing rules that match the required changes based on the required changes, thereby optimizing the log parsing rules.

[0123] In this embodiment, by obtaining the network security identification results of the target network device and optimizing the log parsing rules based on the network security identification results and network security alarm information; and / or obtaining user feedback information on the log parsing rules and optimizing the log parsing rules based on the feedback information, the log parsing rules can be optimized in a timely manner, and the adaptability of the log parsing rules to network security requirements can be improved, thereby improving the accuracy of log parsing.

[0124] In some embodiments, as Figure 3The figure shows a flow chart of an intelligent log parsing method based on dynamic environment modeling. The terminal can collect log data from various network devices, servers, user terminals, and applications through the data collection module. The terminal can use sensors and network probes through the dynamic environment perception module to perceive dynamic changes in the network environment, including device status, network traffic, and user behavior, in real time, achieving dynamic environment perception. The terminal can use the environment modeling module to build an environment model of the current network based on the data from the dynamic environment perception module, thereby achieving environmental modeling and predicting potential security risks. The terminal can use the intelligent rule generation module to automatically generate and update log parsing rules based on the network environment modeling model, achieving intelligent rule generation. The terminal can use the intelligently generated log parsing rules through the log parsing module to perform real-time parsing of collected log data, obtaining log parsing results, achieving log parsing. The terminal can use the alarm identification module to identify potential security incidents based on the log parsing results and generate alarm information, achieving alarm identification. The terminal can use the rule optimization module to continuously optimize the parsing rules based on a feedback mechanism to ensure high accuracy and adaptability of the system, achieving rule optimization.

[0125] In actual applications, during the rule optimization process, the terminal can use user modification requirements for log parsing rules and / or false alarm information as feedback information, and combine the feedback information to learn emerging log patterns and generate new parsing rules. The learning process can be carried out through the Bayesian optimization model:

[0126] .

[0127] in, Can refer to the parameters of the parsing rules, It can refer to observed log data, Can refer to a given parameter The data likelihood is, This can refer to a prior distribution. Bayesian optimization methods dynamically adjust parsing rules by updating the posterior distribution of parameters to adapt to new data patterns and environmental changes. The optimized rules can be applied to the log parsing process, ensuring that the parsing rules can continuously adapt to changes in the network environment, improving the accuracy and flexibility of log parsing.

[0128] In this embodiment, by introducing a dynamic programming algorithm to generate an optimal log parsing rule set, the priority and dependency of different rules in the log parsing process can be effectively considered, thereby generating an optimal rule set more efficiently and ensuring the flexibility and parsing capability of the system; more emphasis is placed on real-time perception and modeling of dynamic environmental changes, and multi-dimensional environmental data (device status, network traffic, user behavior, etc.) is combined to build an accurate environmental model. Dynamic environmental perception and modeling significantly enhance the intelligence and adaptability of log parsing, and can cope with various complex and changing network environments; the intelligent solution for log parsing combined with multi-dimensional environmental data (such as device status, user behavior, network traffic, etc.) enhances the flexibility and adaptability of the system in different scenarios. Especially in complex network environments or unstable data sets containing noise, it is easier to perform accurate log anomaly detection; through machine learning and complex mathematical models, log parsing rules can be automatically generated and updated in real time, and the accuracy of the system can be continuously improved through optimization algorithms. The automatic generation and real-time adjustment of rules reduce manual intervention, making log parsing more intelligent and adaptive. Rule optimization based on machine learning algorithms greatly reduces the false alarm rate and improves parsing accuracy; through complex mathematical models and optimization algorithms, the accuracy of log parsing is significantly improved, especially in reducing the false alarm rate. The use of more complex mathematical methods and optimization algorithms in the log parsing and alarm identification process ensures higher accuracy and fewer false alarms, and is suitable for a wider range of network environments and application scenarios.

[0129] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0130] Based on the same inventive concept, embodiments of the present application also provide a network log data parsing device for implementing the network log data parsing method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more network log data parsing device embodiments provided below can be found in the limitations of the network log data parsing method described above and will not be repeated here.

[0131] In an exemplary embodiment, Figure 4 As shown, a network log data analysis device is provided, including: a collection module 402, a parsing module 404 and an alarm module 406, wherein:

[0132] The collection module 402 is used to collect network log data of the target network device.

[0133] The parsing module 404 is used to parse the network log data according to the log parsing rules and generate a log parsing result; wherein the log parsing rules are generated based on the network environment modeling results within the target time period; the network environment modeling results are constructed based on the perception data of the network environment in which the target network device is located within the historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information and device information.

[0134] The alarm module 406 is configured to output network security alarm information of the target network device according to the log analysis result.

[0135] In one of the exemplary embodiments, the device also includes a modeling module, which is specifically used to obtain traffic prediction information of the network environment in which the target network device is located in the target time period; determine the network environment information in the historical time period based on the network traffic information, user behavior information and device information of the network environment in which the target network device is located in the historical time period; input the traffic prediction information and the network environment information into a pre-trained network environment modeling model to obtain a network environment modeling result in the target time period; wherein the difference between the network environment modeling result in the target time period and the network environment information in the historical time period meets a preset condition.

[0136] In one of the exemplary embodiments, the device also includes a rule generation module, which is specifically used to cluster the log events corresponding to the network log data based on the network log data and the network environment modeling results within the target time period to obtain log event clusters; and generate log parsing rules for parsing the network log data of the target network device based on the log event clusters.

[0137] In one of the exemplary embodiments, the rule generation module is specifically used to generate an initial parsing rule for each of the log event clusters based on the log events in the log event clusters; obtain the weight of the initial parsing rule and the applicability probability of the initial parsing rule under the network environment modeling results within the target time period, and determine the evaluation information of the initial parsing rule based on the weight and the applicability probability; and filter out the log parsing rule from the initial parsing rules based on the evaluation information.

[0138] In one of the exemplary embodiments, the device also includes a filtering module, which is specifically used to determine information entropy based on the frequency of occurrence of the network log data of the target network device; and filter noise data from the network log data of the target network device based on the information entropy and a preset information entropy threshold.

[0139] In one of the exemplary embodiments, the device also includes an optimization module, which is specifically used to obtain the network security identification result of the target network device, and optimize the log parsing rules based on the network security identification result and the network security alarm information; and / or obtain user feedback information on the log parsing rules, and optimize the log parsing rules based on the feedback information.

[0140] Each module in the network log data parsing device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0141] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for parsing network log data. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0142] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0143] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0145] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0147] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0148] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0149] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A network log data parsing method, characterized in that: The method comprises: Collect network log data from target network devices; Parsing the network log data according to log parsing rules to generate log parsing results; wherein the log parsing rules are generated based on network environment modeling results within a target time period; the network environment modeling results are constructed based on perception data of the network environment in which the target network device is located within a historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information, and device information; According to the log analysis result, network security alarm information of the target network device is output.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining traffic prediction information of the network environment in which the target network device is located during the target time period; Determine network environment information within the historical time period based on network traffic information, user behavior information, and device information of the network environment in which the target network device is located within the historical time period; Inputting the traffic prediction information and the network environment information into a pre-trained network environment modeling model to obtain a network environment modeling result within the target time period; The difference between the network environment modeling result within the target time period and the network environment information within the historical time period meets a preset condition.

3. The method according to claim 2, characterized in that After the step of obtaining the network environment modeling result within the target time period, the method further includes: Clustering log events corresponding to the network log data according to the network log data and the network environment modeling result within the target time period to obtain log event clusters; A log parsing rule for parsing the network log data of the target network device is generated based on the log event cluster.

4. The method according to claim 3, characterized in that Generating log parsing rules for parsing network log data of the target network device based on the log event clusters includes: For each of the log event clusters, generating an initial parsing rule according to the log events in the log event cluster; Obtaining a weight of the initial parsing rule and an applicability probability of the initial parsing rule under a network environment modeling result within the target time period, and determining evaluation information of the initial parsing rule based on the weight and the applicability probability; The log parsing rule is filtered out from the initial parsing rules according to the evaluation information.

5. The method according to claim 1, wherein After the step of collecting network log data of the target network device, the method further includes: Determining information entropy based on the frequency of occurrence of network log data of the target network device; Noise data is filtered from the network log data of the target network device according to the information entropy and a preset information entropy threshold.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a network security identification result of the target network device, and optimizing the log parsing rule based on the network security identification result and the network security alarm information; and / or Obtain user feedback information regarding the log parsing rule, and optimize the log parsing rule based on the feedback information.

7. A network log data analysis device, characterized in that: The device comprises: A collection module, used to collect network log data of target network devices; a parsing module, configured to parse the network log data according to log parsing rules to generate log parsing results; wherein the log parsing rules are generated based on network environment modeling results within a target time period; the network environment modeling results are constructed based on perception data of the network environment in which the target network device is located within a historical time period; the historical time period is before the target time period; the perception data includes network traffic information, user behavior information, and device information; The alarm module is used to output network security alarm information of the target network device according to the log analysis result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images