A method, device and system for network security situation awareness

Abstract

The present application provides a network security situation awareness method, device and system. By using the log representation vectors of multiple network security monitoring logs, risk-free network security monitoring logs are determined from the multiple network security monitoring logs, and then they are determined as network security monitoring training logs to calibrate and optimize the network security situation awareness neural network. By automatically determining risk-free network security monitoring logs from multiple network security monitoring logs based on the log representation vectors, the efficiency of sample selection is improved, human resources are saved, and at the same time, the objectivity and accuracy of the samples are ensured, the error of the obtained risk-free network security monitoring logs is reduced, which helps to improve the training quality of the network security situation awareness neural network, and the calibrated and converged network security situation awareness neural network has a higher risk perception accuracy.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a network security situation awareness method, device, and system. Background Art

[0002] In the modern information technology environment, network security is of crucial importance. Enterprises, organizations, and even the entire society highly rely on network systems for data transmission, information storage, and business operations. Network security situation awareness technology aims to comprehensively and real-time monitor and evaluate the network security status, so as to timely discover potential security threats and take effective countermeasures. This is of irreplaceable significance for protecting sensitive information, ensuring the continuity of network services, and maintaining the stability of the network environment. When constructing a neural network for training in traditional network security situation awareness technology, the selection of risk-free network security monitoring log samples often relies on manual operation or simple rule screening. Manual sample selection requires a large amount of human resources and is easily affected by subjective factors. For example, different operators may select different risk-free samples according to their own experience and judgment, making it difficult to ensure the consistency and objectivity of the samples.

[0003] In the traditional method, due to the lack of an accurate risk-free sample selection method, the obtained risk-free network security monitoring logs may have large errors. When these error samples participate in the neural network training, they will affect the learning effect of the neural network on normal network behavior patterns, and further reduce the training quality of the network security situation awareness neural network. The neural network may not be able to accurately distinguish normal and abnormal network behaviors, resulting in a low risk perception accuracy in actual applications. Summary of the Invention

[0004] In view of this, this application provides a network security situation awareness method, device, and system.

[0005] The technical solution of this application is realized as follows:

[0006] On the one hand, the present application provides a method for network security situation awareness. The method includes: obtaining a plurality of network security monitoring logs, each of the network security monitoring logs including a network monitoring event. The plurality of network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs. The network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks; based on the log representation vectors of the respective network security monitoring logs in the plurality of network security monitoring logs, one or more target network security monitoring logs are determined from the plurality of network security monitoring logs, and are determined as network security monitoring training logs for calibrating a network security situation awareness neural network, where the target network security monitoring logs are the risk-free network security monitoring logs determined by the log representation vectors; the network security situation awareness neural network is calibrated and optimized through the network security monitoring training logs to obtain a calibrated and converged network security situation awareness neural network, and the calibrated and converged network security situation awareness neural network is used to determine whether a network security monitoring log to be analyzed is the risk-free network security monitoring log or the risky network security monitoring log.

[0007] On the other hand, the present application provides a network security situation awareness device, including: a log acquisition module, configured to obtain a plurality of network security monitoring logs, each of the network security monitoring logs including a network monitoring event. The plurality of network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs. The network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks; a sample determination module, configured to determine one or more target network security monitoring logs from the plurality of network security monitoring logs based on the log representation vectors of the respective network security monitoring logs in the plurality of network security monitoring logs, and determine them as network security monitoring training logs for calibrating a network security situation awareness neural network, where the target network security monitoring logs are the risk-free network security monitoring logs determined by the log representation vectors; a network optimization module, configured to calibrate and optimize the network security situation awareness neural network through the network security monitoring training logs to obtain a calibrated and converged network security situation awareness neural network, and the calibrated and converged network security situation awareness neural network is used to determine whether a network security monitoring log to be analyzed is the risk-free network security monitoring log or the risky network security monitoring log.

[0008] In a third aspect, the present application provides a network security situation awareness system, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, the steps in the above-described method are implemented.

[0009] Advantages of the present application: The present application uses the log representation vectors of multiple network security monitoring logs to determine risk-free network security monitoring logs among the multiple network security monitoring logs, and then determines them as network security monitoring training logs to calibrate and optimize the network security situation awareness neural network. By automatically determining risk-free network security monitoring logs among multiple network security monitoring logs based on the log representation vectors, the efficiency of sample selection is improved, human resources are saved, and at the same time, the objectivity and accuracy of the samples are ensured, the error of the obtained risk-free network security monitoring logs is reduced, which helps to improve the training quality of the network security situation awareness neural network, and the calibrated and converged network security situation awareness neural network has a higher perception accuracy for risks.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the technical solution of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification. These drawings show embodiments in accordance with the present application and, together with the specification, are used to explain the technical solution of the present application.

[0012] Figure 1 It is a schematic flowchart of the implementation of a network security situation awareness method provided by an embodiment of the present application.

[0013] Figure 2 It is a schematic diagram of the composition structure of a network security situation awareness device provided by an embodiment of the present application.

[0014] Figure 3 It is a schematic diagram of the hardware entity of a network security situation awareness system provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0016] An embodiment of the present application provides a network security situation awareness method, which can be executed by a processor of a network security situation awareness system. Among them, the network security situation awareness system may refer to devices with data processing capabilities such as servers, laptop computers, tablet computers, desktop computers, etc.

[0017] Figure 1 It is a schematic diagram of the implementation process of a network security situation awareness method provided by an embodiment of the present application. As Figure 1 shown, the method includes:

[0018] Step S100: Obtain multiple network security monitoring logs. Each network security monitoring log includes network monitoring events. The multiple network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs. The network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks.

[0019] In step S100, the network security monitoring log is a set of relevant information recorded when monitoring network activities. These information reflect various events and related states in the network.

[0020] The sources of network security monitoring logs are diverse. For example, it can be a firewall log. A firewall is a network security system located between an internal network and an external network. It monitors and filters the information flow in and out of the internal network according to pre-set security policies. Firewall logs will record information such as the source IP address, destination IP address, port number, connection time, protocol type, etc. of a network connection, as well as events such as whether a connection is allowed or rejected. For example, a firewall may record an event that at a certain moment, an attempt is made to connect to server 10.0.0.1 in the internal network through port 80 of the TCP protocol from an external IP address 192.168.1.100, and this event will be recorded in the firewall log.

[0021] In addition to firewall logs, they can also be logs collected by intrusion detection systems (IDS), intrusion prevention systems (IPS), antivirus software, network traffic analysis tools, etc. An intrusion detection system is designed to detect intrusion behaviors or attempted intrusion behaviors, and its logs will contain information such as the characteristics of suspected intrusion behaviors, the occurrence time, the source, etc. For example, if the IDS detects an abnormal network traffic pattern, which may be the traffic characteristics of a known attack method, such as a SYN Flood attack, it will record relevant network traffic information, attack characteristics, the target being attacked, etc. in the logs. An intrusion prevention system can not only detect intrusion behaviors but also actively defend against intrusion behaviors. Based on recording content similar to that of the intrusion detection system, its logs may also contain relevant information about the implementation of defense measures, such as blocking malicious access from a certain IP address and what defense strategies were adopted.

[0022] Antivirus software logs mainly record events related to virus detection and handling. For example, when the antivirus software scans and detects that a certain file is infected with a virus, it will record information such as the path of the file, the name of the virus, and the discovery time. Network traffic analysis tools mainly focus on various situations of network traffic, and their logs may contain information such as the size of network traffic, the flow direction, and the proportion of traffic generated by different application programs. For example, it records that a video stream application occupied 50% of the total network traffic during a certain period.

[0023] In these network security monitoring logs, each log contains network monitoring events. Here, a network monitoring event refers to an activity or behavior with specific significance detected during network operation. These network monitoring events can be divided into two types: the network monitoring events in risk-free network security monitoring logs do not pose security risks, and the network monitoring events in risky network security monitoring logs pose security risks.

[0024] For the network monitoring events in risk-free network security monitoring logs, such as normal internal network communication records in firewall logs, for example, when an internal employee normally accesses the company's internal office system during working hours, the connection from a certain internal IP address to the IP address of the office system server uses legal ports and protocols. This kind of connection is a normal business requirement and does not pose a security risk. Such an event belongs to a risk-free network security monitoring event, and the log containing this event is a risk-free network security monitoring log.

[0025] However, the network monitoring events in risky network security monitoring logs may involve various security threats. Taking the IDS log as an example, if a frequent port scanning behavior from an external unknown IP address is detected, this behavior may be that an attacker is looking for system vulnerabilities for further intrusion. This is a network monitoring event with security risks, and the log containing this event is a risky network security monitoring log.

[0026] There are various technical means for the network security situation awareness system to obtain these network security monitoring logs. If the logs are generated by local network devices, such as firewall logs, the network security situation awareness system can obtain the logs by establishing a connection with the firewall device and using methods such as the Simple Network Management Protocol (SNMP) or direct file reading (if the firewall supports storing the logs in file form and allows reading). For devices such as intrusion detection systems and intrusion prevention systems, the logs can also be obtained according to the interfaces provided by the devices themselves (such as API interfaces or specific log export functions). If it is antivirus software in the network, the network security situation awareness system can obtain the relevant logs through communication with the host where the antivirus software is located and according to the log sharing mechanism set by the antivirus software. The log acquisition of network traffic analysis tools can also be carried out based on the log acquisition interfaces or data sharing protocols provided by themselves. For example, some network traffic analysis tools support sending log data in a specific format (such as JSON or XML format) to a specified server or storage location, and the network security situation awareness system can obtain the logs from this specified location.

[0027] For logs stored in file form, the network security situation awareness system can obtain the log content through file reading operations. If log acquisition involves network transmission, it can involve relevant operations of the network protocol stack. Taking SNMP as an example, as the management end, the network security situation awareness system needs to follow the operation process specified by the SNMP protocol, such as sending a Get-Request message to the managed device (such as a firewall). The managed device returns a response message (Get-Response) containing log information according to the request, and the network security situation awareness system then parses the obtained message to obtain the log content. In the whole process, operations such as the encapsulation, sending, receiving, and parsing of network data packets are involved, and these operations all follow the corresponding network protocol standards to ensure that the log information can be accurately transmitted from the log source to the network security situation awareness system.

[0028] Step S200: Based on the log characterization vectors of multiple network security monitoring logs, determine one or more target network security monitoring logs from the multiple network security monitoring logs and determine them as network security monitoring training logs for calibrating the network security situation awareness neural network, where the target network security monitoring logs are risk-free network security monitoring logs determined through the log characterization vectors.

[0029] In step S200, based on the log representation vectors of multiple network security monitoring logs, the network security situation awareness system determines one or more target network security monitoring logs from the multiple network security monitoring logs and determines them as network security monitoring training logs for calibrating the network security situation awareness neural network.

[0030] First, the log representation vector is a mathematical representation of the network security monitoring log, which can reflect the characteristics of the log in a digital and structured way. For example, assume that a network security monitoring log contains information such as source IP address, destination IP address, port number, protocol type, and event type. The network security situation awareness system can quantify this information. Map the source IP address to a specific numerical range, and do the same for the destination IP address. The port number can be directly used as a numerical value, and the protocol type can be represented by digital encoding (such as TCP is 1, UDP is 2, etc.), and a similar encoding is done for the event type. Then, through a specific algorithm, these quantified values are combined into a vector, and this vector is the log representation vector.

[0031] The network security situation awareness system determines the target network security monitoring logs by analyzing these log representation vectors. The target network security monitoring logs are risk-free network security monitoring logs. The reason for choosing risk-free network security monitoring logs as the target network security monitoring logs for calibrating the network security situation awareness neural network is that these risk-free logs can provide sample characteristics in the normal network state for the neural network, which is conducive to the neural network learning to distinguish normal and abnormal network behaviors.

[0032] The network security situation awareness system can use various methods to determine the target network security monitoring logs based on the log representation vectors. One possible method is to use the vector space model (VSM) technology. In the vector space model, each log representation vector can be regarded as a point in the vector space. The network security situation awareness system can calculate the distance or similarity between each log representation vector. For example, for two log representation vectors and , the Euclidean distance formula can be used to calculate the distance between them. The smaller the distance, the more similar the two vectors are.

[0033] The network security situation awareness system can set a threshold. When the distance between the representation vector of a risk-free network security monitoring log and the representation vectors of other risk-free network security monitoring logs is within this threshold range, it is determined as the target network security monitoring log. For example, if the threshold is set to T, when calculating that the representation vector of the risk-free log is less than T for the distances to the representation vectors of other risk-free logs, can be determined as the target network security monitoring log.

[0034] Another possible technical means is to use clustering analysis. The network security situation awareness system can cluster all network security monitoring logs (including those with risks and without risks) according to their log feature vectors. For example, using the K-Means clustering algorithm, the log feature vectors are divided into K clusters. After clustering, the network security situation awareness system can identify the clusters containing risk-free network security monitoring logs, and then select representative risk-free network security monitoring logs from these clusters as the target network security monitoring logs. The selection basis can be factors such as the center position of the cluster or the distribution density of the vectors within the cluster.

[0035] After determining the target network security monitoring logs, the network security situation awareness system determines them as the network security monitoring training logs for calibrating the network security situation awareness neural network. The network security situation awareness neural network is a neural network model specifically used for analyzing network security conditions. It requires a large amount of sample data for training to improve the awareness ability of network security situations. By using these determined risk-free network security monitoring logs as training logs, the neural network can learn the patterns of normal network behaviors, so as to better identify risky network behaviors in subsequent analyses. For example, if during the training process, the neural network often encounters normal internal network communication patterns (contents in risk-free network security monitoring logs), then when encountering network monitoring events with large differences from this pattern, it can more accurately judge as possible risky situations.

[0036] Step S300: Calibrate and optimize the network security situation awareness neural network through the network security monitoring training logs to obtain a calibrated and convergent network security situation awareness neural network, which is used to determine whether the network security monitoring log to be analyzed is a risk-free network security monitoring log or a risky network security monitoring log.

[0037] In step S300, the network security situation awareness system calibrates and optimizes the network security situation awareness neural network through the network security monitoring training logs to obtain a calibrated and convergent network security situation awareness neural network, which is used to determine whether the network security monitoring log to be analyzed is a risk-free network security monitoring log or a risky network security monitoring log.

[0038] The network security situation awareness neural network is a complex neural network structure specifically constructed for analyzing network security situations. It consists of multiple components, and each component plays a unique role in the calibration process of network security monitoring training logs. The encoder component is a feature extractor. For example, assume that the network security monitoring training logs contain various information such as the source IP address, destination IP address, port number, protocol type, and event occurrence time of network connections. After the network security situation awareness system loads the network security monitoring training logs into the encoder component, the encoder will convert these original log information into a representation form that is more suitable for subsequent network processing, that is, obtain the first log representation vector of the network security monitoring training logs. This process can be analogized to converting the information of complex objects in the real world into digital feature vectors that computers can better understand and process. From a technical implementation perspective, the encoder can adopt a multi-layer neural network structure, and each layer performs a non-linear transformation on the input data. For example, in a simple fully connected neural network layer, if the input layer has n neurons (corresponding to n features in the log), and the hidden layer has m neurons, then for the input vector , the output of the hidden layer can be calculated by the formula , where are the weights, is the bias, and f is the activation function (such as the ReLU function ).

[0039] Next, the first log representation vector is loaded into the space adaptation component. The main role of the space adaptation component is to map the first log representation vector into the representation space of the first network transmission scenario to obtain the second log representation vector. Different network transmission scenarios may have different feature spaces. For example, there are differences between the network transmission scenarios of enterprise internal networks and Internet service provider (ISP) networks. Assume that in the enterprise internal network scenario, the source IP address range in the network security monitoring logs is relatively fixed and concentrated in a specific internal network segment, while in the ISP network scenario, the source IP address range is wider and more dispersed. The space adaptation component can convert the representation vector of the network security monitoring training logs obtained from the enterprise internal network into a representation space suitable for a wider network transmission scenario (such as a scenario that includes the interaction between enterprise networks and external networks). This mapping process may involve a combination of linear transformation and non-linear transformation. For example, an affine transformation y = Ax + b (where A is the transformation matrix, x is the input vector, and b is the translation vector) combined with some non-linear activation functions may be used to achieve it.

[0040] Then, the second log characterization vector is loaded into the discriminator component. The discriminator component outputs a corresponding first risk data sequence based on the input second log characterization vector, and each data item in this sequence corresponds to a risk coefficient. For example, if the discriminator analyzes the second log characterization vector corresponding to a certain network security monitoring training log and obtains the first risk data sequence as , where represents the risk coefficient under a certain specific dimension or analysis perspective. These risk coefficients reflect the risk assessment results of the network security monitoring training log under the current neural network model. The discriminator component may determine these risk coefficients based on some predefined rules or patterns learned during the training process. For example, if the source IP address in a certain network security monitoring training log is marked as coming from a high-risk external network area and at the same time involves connection attempts to non-standard ports, then the discriminator can assign a higher risk coefficient to the relevant risk data item.

[0041] During the calibration process of the network security situation awareness system, various parameters in the network security situation awareness neural network need to be continuously adjusted so that the network security monitoring training log can obtain more accurate risk assessment results after being processed by the neural network. For example, the network security situation awareness system can adopt the backpropagation algorithm to adjust parameters such as weights and biases in the neural network. The backpropagation algorithm calculates the gradient based on the loss function, and the loss function is used to measure the difference between the output of the neural network (such as the risk data sequence) and the expected output (such as the ideal risk coefficient corresponding to the known risk-free or risky label). Assume the loss function is the mean squared error (MSE) function , where is the predicted output (risk coefficient) of the neural network, is the expected output, and n is the number of data points. The network security situation awareness system calculates the gradient of the loss function with respect to the neural network parameters (such as the weight w) , and then updates the parameters in the direction of the gradient descent , where is the learning rate.

[0042] By continuously inputting network security monitoring training logs into the network security situation awareness neural network, the network security situation awareness system calibrates and optimizes the network security situation awareness neural network according to the above mechanism until the neural network reaches a state of calibration convergence. Calibration convergence means that after multiple iterations of optimization of the parameters of the neural network, its output results (risk data sequences) have reached a relatively stable and accurate state on the given training data (network security monitoring training logs). At this time, the obtained network security situation awareness neural network can effectively perform risk assessment on new network security monitoring logs to be analyzed and accurately determine whether they are risk-free network security monitoring logs or risky network security monitoring logs.

[0043] In one implementation, step S200, based on the log representation vectors of multiple network security monitoring logs, determining one or more target network security monitoring logs from the multiple network security monitoring logs and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network may include:

[0044] Step S210: Based on the log representation vectors of multiple network security monitoring logs, grouping the multiple network security monitoring logs, and determining one or more target network security monitoring logs from the multiple network security monitoring logs based on the grouping result and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network.

[0045] In step S210, the network security situation awareness system groups the multiple network security monitoring logs based on the log representation vectors of the multiple network security monitoring logs, determines one or more target network security monitoring logs from the multiple network security monitoring logs based on the grouping result, and determines them as network security monitoring training logs for calibrating the network security situation awareness neural network.

[0046] The log representation vector is a quantitative representation form of the content features of the network security monitoring log. For example, a network security monitoring log records information such as source IP address, destination IP address, port number, protocol type, and event type. The network security situation awareness system can assign specific values or codes to these different types of information and then combine them into a vector. For example, the source IP address can be mapped to a value according to its address range, and the destination IP address is the same. The port number is directly used as a value, and the protocol type (such as TCP is 1, UDP is 2, etc.) and event type (such as normal access is 1, suspicious access is 2, etc.) are also encoded accordingly, and finally form a log representation vector similar to the following.

[0047] The network security situation awareness system performs grouping operations based on these log feature vectors. Grouping is a means of aggregating similar network security monitoring logs. A possible technical means is to use a clustering algorithm, such as a distance-based clustering algorithm. The network security situation awareness system calculates the distances between each log feature vector to determine which network security monitoring logs corresponding to the log feature vectors are similar. Here, the Euclidean distance formula can be used to calculate the distance between vectors. For two log feature vectors and , their Euclidean distance .

[0048] Suppose the network security situation awareness system has a set of network security monitoring logs. The feature vector of log A is , the feature vector of log B is , and the feature vector of log C is . By calculating the Euclidean distance, it is found that the distance between log A and log B is smaller, while the distance from log C is larger. This means that log A and log B are more similar in characteristics and may belong to the same type of network behavior, while log C belongs to another type.

[0049] The network security situation awareness system groups according to the set distance threshold. For example, if the set distance threshold is T = 5, when the distance between two log feature vectors is less than T, the corresponding network security monitoring logs will be grouped together. In this way, log A and log B may be grouped together, and log C will be in a separate group.

[0050] Based on such grouping results, the network security situation awareness system determines the target network security monitoring logs. The target network security monitoring logs are risk-free network security monitoring logs. After grouping is completed, the network security situation awareness system can identify those groups that mainly contain risk-free network security monitoring logs from each group. For example, in a certain group, most of the network security monitoring logs record normal internal network communications, such as employees accessing the company's internal office system during normal working hours. In this case, the network security monitoring logs in this group are probably risk-free. The network security situation awareness system determines the network security monitoring logs in these risk-free groups as the target network security monitoring logs and determines them as the network security monitoring training logs for calibrating the network security situation awareness neural network. This is because these risk-free network security monitoring logs can provide sample features in the normal network state for the neural network. The network security situation awareness neural network requires a large number of risk-free samples during the training process to learn the patterns of normal network behavior, so as to accurately distinguish normal and abnormal network behaviors.

[0051] Another grouping technique can be the Density-Based Spatial Clustering of Applications with Noise (DBSCAN). In this algorithm, the network security situation awareness system determines clusters based on the data point density around the log representation vectors. If the density of data points (log representation vectors) in a region exceeds a certain threshold, the network security monitoring logs corresponding to these points are grouped into one group. This algorithm does not require specifying the number of clusters in advance, but automatically determines the grouping according to the data distribution.

[0052] For example, for the representation vectors of a group of network security monitoring logs, if there are multiple representation vectors clustered together in a certain spatial region, that is, the density of this region is high, then the network security monitoring logs corresponding to these representation vectors will be grouped into one group. The network security situation awareness system groups all network security monitoring logs in this way, and then filters out the groups containing risk-free network security monitoring logs from the grouping results, and further determines the target network security monitoring logs.

[0053] In one implementation, step S210, based on the log representation vectors of multiple network security monitoring logs, group the multiple network security monitoring logs, determine one or more target network security monitoring logs from the multiple network security monitoring logs based on the grouping results, and determine them as the network security monitoring training logs for calibrating the network security situation awareness neural network, which may specifically include:

[0054] Step S211: For each network security monitoring log among the multiple network security monitoring logs, determine the first similarity measure between the log representation vector of the network security monitoring log and the log representation vectors of the remaining network security monitoring logs among the multiple network security monitoring logs, and obtain multiple first similarity measures corresponding to the network security monitoring log;

[0055] Step S212: Based on x first similarity measures among the multiple first similarity measures corresponding to the network security monitoring log, determine the mean similarity measure corresponding to the network security monitoring log, where the x first similarity measures are the largest x first similarity measures among the multiple first similarity measures corresponding to the network security monitoring log, and x≥1;

[0056] Step S213: Based on the mean similarity measures corresponding to the multiple network security monitoring logs respectively, determine one or more target network security monitoring logs from the multiple network security monitoring logs, and determine them as the network security monitoring training logs for calibrating the network security situation awareness neural network.

[0057] In steps S211 - S213 of the specific implementation of step S210, the network security situation awareness system performs a series of operations to determine target network security monitoring logs, which will be used as network security monitoring training logs for calibrating the network security situation awareness neural network.

[0058] In step S211, for each network security monitoring log among multiple network security monitoring logs, the network security situation awareness system determines a first similarity measure between the log representation vector of this network security monitoring log and the log representation vectors of the remaining network security monitoring logs among the multiple network security monitoring logs, thereby obtaining multiple first similarity measures corresponding to this network security monitoring log.

[0059] As mentioned above, the log representation vector is a quantitative representation form of the content features of the network security monitoring log. For example, a network security monitoring log records information such as source IP address, destination IP address, port number, protocol type, event occurrence time, and event type. The network security situation awareness system can assign specific numerical values or codes to these different types of information and then combine them into a vector. Suppose the value range of the source IP address is 0 - 255, and it can be normalized to a value between 0 - 1; the same goes for the destination IP address; the port number is directly used as a numerical value; the protocol type can be represented by specific numbers, such as 1 for TCP and 2 for UDP; the event occurrence time can be converted into a time difference value relative to a certain reference time; if the event type is normal access, it is set to 1, and if it is abnormal access, it is set to 2, etc. In this way, a network security monitoring log can be represented as a log representation vector, for example, V=(0.1, 0.2, 8080, 1, 10, 1).

[0060] The similarity measure is an index used to measure the similarity degree between two vectors. The network security situation awareness system calculates the similarity measure between log representation vectors to judge the similarity between network security monitoring logs. A feasible similarity measure method is cosine similarity. For two log representation vectors and , the cosine similarity formula between them is . For example, there is the representation vector of network security monitoring log A and the representation vector of log B. The similarity measure value between them can be calculated through the cosine similarity formula.

[0061] For each network security monitoring log, the network security situation awareness system performs such similarity measurement calculations one by one with the rest of the network security monitoring logs. For example, assume there are 10 network security monitoring logs. For log 1, the network security situation awareness system will calculate its similarity measurements with logs 2, 3, …, 10, thus obtaining 9 first similarity measurement values.

[0062] In step S212, based on x first similarity measurements among the multiple first similarity measurements corresponding to the network security monitoring log, the network security situation awareness system determines the similarity measurement mean corresponding to the network security monitoring log. Here, the x first similarity measurements are the largest x first similarity measurements among the multiple first similarity measurements corresponding to the network security monitoring log, and x ≥ 1.

[0063] The similarity measurement mean is the result obtained by averaging some of the similarity measurement values. Assume that for a certain network security monitoring log, the network security situation awareness system calculates the first similarity measurement values between it and the other 9 network security monitoring logs as 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 respectively. If x = 3, then the network security situation awareness system will select the largest 3 similarity measurement values, namely 0.7, 0.8, 0.9, and then calculate their mean (0.7 + 0.8 + 0.9) / 3 = 0.8. This 0.8 is the similarity measurement mean corresponding to this network security monitoring log. The purpose of doing this is to pay more attention to the situations of other logs that are most similar to this log to a certain extent, and highlight the influence of those logs with characteristics relatively similar to the current log on the similarity evaluation of the current log.

[0064] In step S213, based on the similarity measurement means corresponding to each of the multiple network security monitoring logs, the network security situation awareness system determines one or more target network security monitoring logs among the multiple network security monitoring logs, and determines them as the network security monitoring training logs for calibrating the network security situation awareness neural network.

[0065] The network security situation awareness system will select the target network security monitoring logs according to the similarity measurement means of all network security monitoring logs. For example, assume there are 10 network security monitoring logs, and their similarity measurement means are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 respectively.

[0066] One possible implementation for determining the target network security monitoring logs is to set a threshold. If the mean of the similarity metrics is greater than this threshold, the corresponding network security monitoring logs are determined as the target network security monitoring logs. Suppose the threshold is set to 0.6, then the network security monitoring logs with similarity metric means of 0.7, 0.8, 0.9, and 1.0 will be determined as the target network security monitoring logs. These target network security monitoring logs will be used as network security monitoring training logs to calibrate the network security situation awareness neural network.

[0067] Another implementation is to sort the network security monitoring logs according to the magnitude of the similarity metric means, and then select a certain number of network security monitoring logs with the top rankings as the target network security monitoring logs. For example, after sorting these 10 network security monitoring logs in descending order of the similarity metric means, the first 5 logs are selected as the target network security monitoring logs. This is because these logs perform well in terms of the similarity metric means and may have the characteristics of more typical risk-free network security monitoring logs. Since risk-free network security monitoring logs often have a certain similarity pattern in a normal network environment, the logs selected in this way are more likely to be risk-free and thus suitable as network security monitoring training logs to calibrate the network security situation awareness neural network, enabling the neural network to better learn the characteristics of the normal network state so as to accurately distinguish risk-free and risky network security monitoring logs in subsequent analysis.

[0068] Through the operations of steps S211 - S213, the network security situation awareness system can screen out suitable target network security monitoring logs from numerous network security monitoring logs, and these logs will help improve the neural network's accurate perception ability of the network security situation during the calibration process of the network security situation awareness neural network.

[0069] In one implementation, step S213, based on the similarity metric means corresponding to each of the multiple network security monitoring logs, determines one or more target network security monitoring logs from the multiple network security monitoring logs and determines them as the network security monitoring training logs for calibrating the network security situation awareness neural network. Specifically, it may include:

[0070] Step S2131: Sequentially arrange the multiple network security monitoring logs in the order of decreasing similarity metric means to obtain a set of network security monitoring logs;

[0071] Step S2132: Determine the first k network security monitoring logs in the set of network security monitoring logs as the target network security monitoring logs;

[0072] Step S2133: Take the target network security monitoring log and determine it as the network security monitoring training log for calibrating the network security situation awareness neural network.

[0073] In steps S2131 - S2133 of the specific implementation manner of step S213, the network security situation awareness system performs a series of operations to determine the target network security monitoring log and determines it as the network security monitoring training log for calibrating the network security situation awareness neural network.

[0074] In step S2131, the network security situation awareness system arranges multiple network security monitoring logs in sequence according to the decreasing mean value of similarity metrics to obtain a set of network security monitoring logs.

[0075] As mentioned before, the mean value of similarity metrics is a comprehensive index calculated based on the similarity metrics between network security monitoring logs and other logs. For example, for the network security monitoring log , the network security situation awareness system calculates its similarity metric with other logs (such as using cosine similarity metric). Suppose the similarity metric with log is 0.8, the similarity metric with log is 0.6, the similarity metric with log is 0.7, etc. If following the method of step S212, take the largest x similarity metrics among them (assuming x = 2, that is, take 0.8 and 0.7), and then calculate their mean value (0.8 + 0.7) / 2 = 0.75. This 0.75 is the mean value of the similarity metric of log L_1.

[0076] Arrange in decreasing order according to the mean value of similarity metrics: The network security situation awareness system will obtain the mean values of similarity metrics of all network security monitoring logs and then arrange these logs in descending order. Suppose there are 5 network security monitoring logs , and their mean values of similarity metrics are 0.8, 0.6, 0.7, 0.5, 0.9 respectively. The network security situation awareness system arranges these logs in the order of decreasing mean value of similarity metrics as , thus obtaining an ordered set of network security monitoring logs. The purpose of doing this is to arrange the logs that perform better in terms of similarity metrics (that is, are more similar to other logs in a certain feature) in the front for subsequent selection of the target network security monitoring log. Technically speaking, the network security situation awareness system can use a sorting algorithm, such as the bubble sort algorithm. For the array of mean values of similarity metrics of n network security monitoring logs , the basic idea of bubble sort is to compare adjacent elements and swap them if the order is incorrect. The specific steps are as follows:

[0077] The first round of comparison: Starting from the first element and comparing with , if < , then swap their positions; then compare with , and so on, until comparing with . After such a round of comparison, the largest element will "float" to the end of the array.

[0078] The second round of comparison: Repeat the above process, but this time only compare up to and , because the largest element is already at the end, and in this way the second-largest element will be ranked in the second-to-last position.

[0079] Repeat the above process until the entire array is arranged in descending order of the mean of the similarity metric.

[0080] In step S2132, the network security situation awareness system determines the first k network security monitoring logs in the network security monitoring log set as the target network security monitoring logs. Here, k is a preset value, which determines how many network security monitoring logs are selected as the target logs. For example, in the network security monitoring log set arranged in descending order of the mean of the similarity metric as mentioned above, if k = 3, then the network security situation awareness system will determine these 3 network security monitoring logs as the target network security monitoring logs. The reason for selecting the first k logs is that these logs perform well in terms of the mean of the similarity metric, and they may have the characteristics of more typical risk-free network security monitoring logs. Since risk-free network security monitoring logs often have a certain similarity pattern in a normal network environment, these logs ranked in the front are more likely to be risk-free. This selection method helps to ensure that the selected target network security monitoring logs can provide effective risk-free samples for the calibration of the network security situation awareness neural network. Technically speaking, the network security situation awareness system only needs to select the first k elements from the already sorted network security monitoring log set according to the set k value. For example, in programming implementation, if the network security monitoring log set is stored in the form of an array, the network security situation awareness system can obtain the first k elements through index operations.

[0081] In step S2133, the network security situation awareness system determines the target network security monitoring logs as network security monitoring training logs for calibrating the network security situation awareness neural network. These network security monitoring logs determined as targets will be used as network security monitoring training logs to calibrate the network security situation awareness neural network. The network security situation awareness neural network is a complex model that requires a large amount of sample data for training to improve the ability to perceive network security situations. For example, assume that the network security situation awareness neural network contains multiple neuron layers. During the training process, the information in these network security monitoring training logs (such as the information after quantization of the source IP address, destination IP address, port number, protocol type, etc. in the logs) will be used as input data, and the neural network will adjust its internal parameters such as weights and biases based on these input data. Taking a simple neural network layer as an example, for the input network security monitoring training log data , after the calculation of the neuron (where w_i is the weight, b is the bias, and f is the activation function), the neural network will adjust the values of and b according to the difference between the actual output and the expected output. By using these target network security monitoring logs as training logs, the neural network can learn the patterns of normal network behaviors, so as to better identify risky network behaviors in subsequent analyses.

[0082] In one implementation, before step S200, which determines one or more target network security monitoring logs from multiple network security monitoring logs based on the respective log representation vectors of the multiple network security monitoring logs and determines them as network security monitoring training logs for calibrating the network security situation awareness neural network, it may further include:

[0083] Step S201: For each network security monitoring log in the multiple network security monitoring logs, align the network monitoring events in the network security monitoring log with the semantics and formats of the network monitoring events in the reference network security monitoring log to obtain a unified network security monitoring log for the network security monitoring log. The reference network security monitoring log is used to standardize the text formats and language spaces of network monitoring events in different network security monitoring logs; among them, the network security monitoring training logs are determined from the unified network security monitoring logs of the multiple network security monitoring logs based on the respective log representation vectors of the unified network security monitoring logs of the multiple network security monitoring logs.

[0084] In step S201 of the embodiment before step S200, for each network security monitoring log among multiple network security monitoring logs, the network security situation awareness system aligns the network monitoring events in the network security monitoring log with the semantics and format of the network monitoring events in the reference network security monitoring log to obtain a unified network security monitoring log for the network security monitoring log, where the reference network security monitoring log is used to standardize the text format and language space of the network monitoring events in different network security monitoring logs.

[0085] A network monitoring event is specific information about network activities recorded in a network security monitoring log. For example, a network monitoring event in a firewall log may include information such as the source IP address, destination IP address, port number, the time when the connection is established or disconnected, the protocol type (such as TCP, UDP), and whether the connection is allowed; a network monitoring event in an intrusion detection system log may include information related to suspected intrusion behaviors, such as abnormal traffic patterns, suspicious user operations, etc.

[0086] The reference network security monitoring log is a log template with a standard format and semantic definition. Its role is to unify the format and semantics of network monitoring events in network security monitoring logs from different sources. For example, assume there are logs from different firewall devices (such as firewalls produced by different manufacturers). Although the network monitoring events recorded in these firewall logs are generally the same, there may be differences in format and semantic expression. The log of one firewall may record the source IP address as "192.168.1.10", while another firewall may record it as "IP-SRC:192.168.1.10"; in terms of semantics, for the representation of a connection being allowed, one may use "ALLOW", and the other may use "PASS". The reference network security monitoring log will define a standard format, such as uniformly recording the source IP address in the form of "source IP: [specific IP address]", and using specific vocabulary to represent allowed or denied for the connection status.

[0087] The process of the network security situation awareness system for semantic and format alignment is relatively complex. Technically speaking, the network security situation awareness system needs to first parse the network security monitoring log to identify the various components of the network monitoring events in it. For example, for a log containing network monitoring events in text form, the network security situation awareness system can use regular expressions to identify specific patterns. Assume that to identify an IP address, the regular expression can be "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", and in this way, relevant information such as the IP address can be extracted from the log text.

[0088] When performing format alignment, the network security situation awareness system reorganizes each part of the extracted network monitoring events according to the format definition of the reference network security monitoring log. For example, if the reference network security monitoring log requires the time format to be recorded as "year-month-day hour:minute:second", and the time in the original network security monitoring log is recorded in timestamp form (such as 1612345678, representing the number of seconds since a certain starting time), the network security situation awareness system needs to convert the timestamp into the required date and time format. This may involve some time conversion functions or algorithms. Assuming the starting time is January 1, 1970, 00:00:00, through the formula date = new Date(timestamp * 1000) (in JavaScript, here timestamp is the timestamp, and multiplying by 1000 is because the time unit in JavaScript is milliseconds, while the unit of the timestamp is usually seconds), the timestamp can be converted into a date object, and then output in the format of "year-month-day hour:minute:second".

[0089] For semantic alignment, the network security situation awareness system establishes the correspondence between the semantics in the original network security monitoring log and the semantics in the reference network security monitoring log. For example, if "SUCCESS" is used in the original log to indicate a successful connection, and "CONNECTED" is used in the reference log to represent the same semantics, the network security situation awareness system needs to replace "SUCCESS" with "CONNECTED". This may require establishing a semantic mapping table that lists the correspondence between the original semantics and the target semantics. After parsing the network monitoring events in the original network security monitoring log, the network security situation awareness system performs semantic conversion by querying this semantic mapping table.

[0090] Taking firewall logs and intrusion detection system logs as examples for further illustration. Firewall logs may record detailed information about network connections, such as "Source IP: 192.168.1.10, Destination IP: 10.0.0.1, Port: 80, Protocol: TCP, Status: ALLOW, Time: 1612345678", while intrusion detection system logs may record "IP-SRC: 192.168.1.10, IP-DST: 10.0.0.1, PORT: 80, PROTO: TCP, EVENT: Normal". The standard format defined for benchmark network security monitoring logs may be "Source IP: [IP address], Destination IP: [IP address], Port: [port number], Protocol: [protocol type], Connection Status: [status], Time: [date and time]". The network security situation awareness system first parses the firewall logs and intrusion detection system logs to extract relevant information. For firewall logs, they can be directly adjusted according to the format of the benchmark logs; for intrusion detection system logs, "IP-SRC" needs to be converted to "Source IP", "IP-DST" to "Destination IP", "PROTO" to "Protocol", "EVENT: Normal" to "Connection Status: ALLOW" (assuming such a corresponding relationship is defined in the semantic mapping table), and the timestamp is converted to the date and time format, so as to obtain a unified network security monitoring log that is consistent with the format and semantics of the benchmark network security monitoring log.

[0091] By executing step S201, the network security situation awareness system can convert network security monitoring logs from different sources, with different formats and semantics, into a unified format, which facilitates subsequent operations such as determining target network security monitoring logs based on these logs, and ensures that network security situation awareness-related processing is carried out under a unified standard.

[0092] In one implementation, in step S201, aligning the semantics and formats of the network monitoring events in the network security monitoring logs with those of the network monitoring events in the benchmark network security monitoring logs to obtain a unified network security monitoring log for the network security monitoring logs may include:

[0093] Step S2011: Identify the event boundaries of the network security monitoring logs, and extract the set of event trigger data items of the network security monitoring logs. Each event trigger data item in the set of event trigger data items of the network security monitoring logs corresponds to a first data item position;

[0094] Step S2012: Construct a first correspondence between the first data item position and the second data item position, where the second data item position represents the data item position corresponding to the event trigger data item in the benchmark network security monitoring log;

[0095] Step S2013: Based on the first correspondence and the first data item position conversion array, obtain the third data item position corresponding to the first data item position in the reference network security monitoring log;

[0096] Step S2014: Obtain the data item position difference between the third data item position and the second data item position;

[0097] Step S2015: Optimize the first data item position conversion array along the direction of reducing the data item position difference;

[0098] Step S2016: If the first data item position conversion array no longer changes, process the network security monitoring log through the first data item position conversion array to obtain the unified network security monitoring log of the network security monitoring log.

[0099] In step S2011, the network security situation awareness system performs event boundary recognition on the network security monitoring log, extracts the event trigger data item set of the network security monitoring log, and each event trigger data item in the event trigger data item set of the network security monitoring log corresponds to a first data item position.

[0100] Event boundary recognition is the process of determining the start and end positions of each network monitoring event in the network security monitoring log. The network security monitoring log may contain multiple network monitoring events, and the presentation of these events in the log may be continuous text or data arranged in a certain format. For example, a network security monitoring log may be a text file that contains multiple event records related to network connections. The network security situation awareness system needs to determine where each event record starts and ends. A possible technical means is to identify based on specific delimiters or predefined patterns. For example, in some network security monitoring logs, each event record ends with a line break, and the network security situation awareness system can determine the event boundary by identifying the line break.

[0101] After identifying the event boundary, the network security situation awareness system extracts the key information in each event, and these key information constitute the set of event-triggered data items. For example, for a network connection event, the event-triggered data items may include the source IP address, destination IP address, port number, protocol type, etc. Taking a simple firewall log as an example, a record in the log is "2023-05-10 10:00:00,192.168.1.10,10.0.0.1,80,TCP,ALLOW". Here, "2023-05-10 10:00:00" (event occurrence time), "192.168.1.10" (source IP address), "10.0.0.1" (destination IP address), "80" (port number), "TCP" (protocol type), "ALLOW" (connection status), etc. are the event-triggered data items.

[0102] The position of each event-triggered data item in the log is the first data item position. Continuing with the above firewall log as an example, if in the order from left to right, "2023-05-10 10:00:00" is regarded as the 1st data item, its first data item position is 1; "192.168.1.10" is the 2nd data item, and the first data item position is 2, and so on. Technically speaking, the network security situation awareness system can determine the first data item position through simple index counting. For example, when storing event-triggered data items in an array structure, the index of the array can be used as the first data item position.

[0103] In step S2012, the network security situation awareness system constructs the first correspondence relationship between the first data item position and the second data item position, and the second data item position represents the data item position corresponding to the event-triggered data item in the reference network security monitoring log.

[0104] The reference network security monitoring log is a standard log template, which defines the standard format and semantics of network monitoring events. For example, the reference network security monitoring log may stipulate that the standard format of a network connection event is "event occurrence time, source IP address, destination IP address, port number, protocol type, connection status".

[0105] In the reference network security monitoring log, each event-triggered data item also has its corresponding position. For example, in the above format of the reference network security monitoring log, the second data item position of "event occurrence time" is 1, the second data item position of "source IP address" is 2, and so on.

[0106] The first corresponding relationship in the construction of the network security situation awareness system is to associate the event trigger data item positions (the first data item positions) in the network security monitoring logs with the event trigger data item positions (the second data item positions) in the reference network security monitoring logs. For example, for the event trigger data items in the firewall logs mentioned above and the reference network security monitoring logs, if "2023-05-10 10:00:00" in the firewall logs (the first data item position is 1) corresponds to "event occurrence time" in the reference network security monitoring logs (the second data item position is 1), then a 1-1 corresponding relationship is established; "192.168.1.10" in the firewall logs (the first data item position is 2) corresponds to "source IP address" in the reference network security monitoring logs (the second data item position is 2), establishing a 2-2 corresponding relationship, and so on. The network security situation awareness system can store this corresponding relationship through a mapping table or an array. For example, create a two-dimensional array to represent this first corresponding relationship.

[0107] In step S2013, the network security situation awareness system obtains the third data item position corresponding to the first data item position in the reference network security monitoring logs based on the first corresponding relationship and the first data item position conversion array.

[0108] The first data item position conversion array is a tool used to assist in converting the data item positions in the network security monitoring logs to the data item positions in the reference network security monitoring logs. Suppose there are n event trigger data items in the network security monitoring logs and m event trigger data items in the reference network security monitoring logs. The first data item position conversion array may be an array of size n (or m, depending on the specific conversion logic). Each element in the array represents the conversion value from the data item position in the network security monitoring logs to the data item position in the reference network security monitoring logs. For example, if the first data item in the network security monitoring logs corresponds to the second data item in the reference network security monitoring logs, then the first element in the first data item position conversion array may be 2.

[0109] The third data item position is calculated through the first corresponding relationship and the first data item position conversion array, and is the corresponding position of the data item in the network security monitoring logs in the reference network security monitoring logs. For example, assume the first data item position conversion array is [2, 3, 4, 5, 6, 7]. For the second data item (the first data item position is 2) in the network security monitoring logs, by looking up the first data item position conversion array, the corresponding conversion value is 3, and this 3 is the third data item position, indicating the position of this data item in the reference network security monitoring logs.

[0110] In step S2014, the network security situation awareness system obtains the data item position difference between the third data item position and the second data item position.

[0111] Suppose that in the previous step, the third data item position is obtained as 3 and the second data item position is 2 (based on the standard definition in the reference network security monitoring log). Then the data item position difference is 3 - 2 = 1. This data item position difference reflects the offset between the data item position in the network security monitoring log and the standard data item position in the reference network security monitoring log. The calculation of such an offset helps the network security situation awareness system understand the degree of difference in format between the current network security monitoring log and the reference network security monitoring log, providing a basis for subsequent optimization operations.

[0112] In step S2015, the network security situation awareness system optimizes the first data item position conversion array in the direction of reducing the data item position difference.

[0113] The purpose of optimizing the first data item position conversion array is to make the data item position in the network security monitoring log closer to the standard data item position in the reference network security monitoring log. For example, if the data item position difference is 1 (as calculated above) and the current first data item position conversion array is [2, 3, 4, 5, 6, 7], the network security situation awareness system can adjust the elements in this array. For instance, it can adjust the second element (corresponding to the previous calculation of the third data item position) from 3 to 2 to reduce the data item position difference. A possible optimization algorithm is based on the idea of gradient descent (if the data item position difference is regarded as a loss function), gradually adjusting the elements in the first data item position conversion array so that the data item position difference gradually decreases. Although the data structure here is different from traditional numerical optimization, the basic idea is similar, that is, adjusting the parameters (here are the elements in the first data item position conversion array) in the direction of optimizing the objective function (here is reducing the data item position difference).

[0114] In step S2016, if the first data item position conversion array no longer changes, the network security monitoring log is processed through the first data item position conversion array to obtain a unified network security monitoring log of the network security monitoring log.

[0115] When the first data item position conversion array no longer changes after multiple optimizations, it indicates that a relatively stable state has been reached. At this time, this array can accurately convert the data item positions in the network security monitoring log to positions that conform to the benchmark network security monitoring log. For example, if the optimized first data item position conversion array is [1, 2, 3, 4, 5, 6], the network security situation awareness system can rearrange the event trigger data items in the network security monitoring log according to this array. Suppose the event trigger data items in the original network security monitoring log are A, B, C, D, E, F (in the order of the original first data item positions). Through the conversion of the first data item position conversion array, these data items will be arranged in a new order. For example, if the element in the conversion array corresponding to the original first data item A is 1, then A remains in the first position of the new unified network security monitoring log; if the element in the conversion array corresponding to the original second data item B is 2, then B is placed in the second position of the new unified network security monitoring log, and so on. Finally, a unified network security monitoring log with the same format and semantics as the benchmark network security monitoring log is obtained.

[0116] Through the operations in steps S2011 - S2016, the network security situation awareness system can effectively align the semantics and formats of the network monitoring events in the network security monitoring log with those in the benchmark network security monitoring log, providing a standardized data basis for subsequent network security situation awareness related operations based on the unified network security monitoring log. This series of operations is of great significance when dealing with network security monitoring logs from different sources with different formats and semantics, helping to improve the accuracy and efficiency of network security situation awareness.

[0117] In one implementation, the network security situation awareness neural network includes an encoder component, a spatial adaptation component, and a discriminator component; based on this, step S300, tuning and optimizing the network security situation awareness neural network through the network security monitoring training log to obtain a tuned and converged network security situation awareness neural network may include:

[0118] Step S310: Load the network security monitoring training log into the encoder component, and obtain the first log feature vector of the network security monitoring training log based on the encoder component;

[0119] Step S320: Load the first log feature vector into the spatial adaptation component, and obtain the second log feature vector based on the spatial adaptation component. The spatial adaptation component is used to map the first log feature vector to the feature space of the first network transmission scenario;

[0120] Step S330: Load the second log representation vector into the discriminator component, and based on the discriminator component, output the first risk data sequence corresponding to the second log representation vector. Each data item in the first risk data sequence corresponds to a risk coefficient;

[0121] Step S340: Load the second log representation vector fused with enhancement information into the discriminator component, and based on the discriminator component, output the second risk data sequence corresponding to the second log representation vector fused with enhancement information. Each data item in the second risk data sequence corresponds to a risk coefficient;

[0122] Step S350: Optimize the parameters of the spatial adaptation component and the discriminator component along the direction that minimizes the risk coefficients of the first risk data sequence and along the direction that maximizes the risk coefficients of the second risk data sequence, to obtain a calibrated and converged network security situation awareness neural network.

[0123] In step S310, the network security situation awareness system loads the network security monitoring training log into the encoder component, and based on the encoder component, obtains the first log representation vector of the network security monitoring training log.

[0124] The encoder component is a part of the network security situation awareness neural network, and its function is similar to a data feature extractor. It accepts the input data (here it is the network security monitoring training log) and converts it into a representation form that is more suitable for subsequent network processing, that is, the first log representation vector. For example, assume that the network security monitoring training log contains information such as source IP address, destination IP address, port number, protocol type, and event type. For the source IP address, the network security situation awareness system can convert it into a specific range of values. For example, convert the four bytes of an IPv4 address into values between 0 - 255, and then through a certain mapping function (such as a normalization function), map it to a more appropriate value range, such as 0 - 1. Similar processing is also performed on information such as the destination IP address and port number. The protocol type can be represented by digital encoding, such as TCP is 1, UDP is 2, etc., and the event type is also encoded accordingly. Assume that after processing, the source IP address is converted to 0.1, the destination IP address is 0.2, the port number is 8080 (either directly as a value or after some normalization processing), the protocol type is 1, and the event type is 1. Then these values combined form a first log representation vector, for example, V=(0.1,0.2,8080,1,1).

[0125] The encoder can adopt a multi-layer neural network structure, and each layer performs a non-linear transformation on the input data. For example, in a simple fully connected neural network layer, if the input layer has n neurons (corresponding to n features in the log), and the hidden layer has m neurons, then for the input vector , the output of the hidden layer can be calculated through the formula , where are weights, \(b_i\) is the bias, and \(f\) is the activation function (such as the ReLU function ). Through such a multi-layer neural network structure, the network security monitoring training log is gradually converted into the first log representation vector.

[0126] In step S320, the network security situation awareness system loads the first log representation vector into the space adaptation component, and obtains the second log representation vector based on the space adaptation component. The space adaptation component is used to map the first log representation vector into the representation space of the first network transmission scenario.

[0127] The space adaptation component plays a role in converting different feature spaces in the network security situation awareness neural network. Different network transmission scenarios may have different feature spaces. For example, in the enterprise internal network scenario, the source IP address range in the network security monitoring log is relatively fixed and concentrated in a specific internal network segment, while in the Internet service provider (ISP) network scenario, the source IP address range is wider and more dispersed. The first log representation vector may be constructed based on a certain initial feature space, and the space adaptation component can convert it into the representation space applicable to the first network transmission scenario to obtain the second log representation vector. Suppose the first log representation vector , and the representation space of the first network transmission scenario has specific structural and feature distribution requirements. The space adaptation component may achieve this mapping through a linear transformation matrix \(A\) and a translation vector \(b\), that is , where \(V_2\) is the second log representation vector.

[0128] The first network transmission scenario is a specific network environment with its unique network topology, network traffic pattern, and security requirements, etc. For example, in an enterprise's office network, the communication between internal devices follows specific security policies, and the network traffic is mainly concentrated in the interaction between office-related applications, such as between the mail client and the mail server, between the file sharing server and the office terminal, etc. The network transmission characteristics in this scenario are very different from other scenarios (such as the public wireless network scenario), so it is necessary to map the log representation vector into the representation space suitable for this scenario so that the network security situation awareness neural network can analyze the network security status more accurately.

[0129] In step S330, the network security situation awareness system loads the second log representation vector into the discriminator component, and outputs the first risk data sequence corresponding to the second log representation vector based on the discriminator component. Each data item in the first risk data sequence corresponds to a risk coefficient.

[0130] The discriminator component in the network security situation awareness neural network is responsible for performing risk assessment on the input log representation vector (here it is the second log representation vector) and outputting a sequence of risk data. For example, the discriminator component may judge the risk level of the input vector based on a pre-trained model or patterns learned during the current tuning process. Suppose the second log representation vector , there are a series of computing units and parameters inside the discriminator component. It can perform calculations based on each element in the vector and the internal weight parameters. For example, through the formula (where \(r_i\) is the \(i\)-th risk coefficient in the first risk data sequence, is the weight inside the discriminator, is the bias, and \(g\) is the activation function of the discriminator, which may be the sigmoid function etc.) to calculate each risk coefficient.

[0131] The first risk data sequence: This is the result sequence obtained by the discriminator component after performing risk assessment on the second log representation vector. For example, if the calculated risk coefficient sequence by the discriminator is ([[]] ), each numerical value (risk coefficient) in this sequence represents the risk level under a certain specific dimension or analysis perspective. For example, 0.1 may represent the risk coefficient under the security assessment dimension of the source IP address, and 0.3 may represent the risk coefficient under the risk assessment dimension of port number usage, etc.

[0132] In step S340, the network security situation awareness system loads the second log representation vector with fused enhanced information into the discriminator component, and based on the discriminator component, outputs the second risk data sequence corresponding to the second log representation vector with fused enhanced information. Each data item in the second risk data sequence corresponds to a risk coefficient.

[0133] The second log representation vector with fused enhanced information is fused with some additional information on the basis of the second log representation vector. For example, these additional information may come from external threat intelligence sources or other relevant analysis results of the network environment. Suppose the second log representation vector , the external threat intelligence source provides high-risk tip information about certain IP addresses. If the source IP address in the current network security monitoring log is related to these high-risk IP addresses, then this threat intelligence information can be fused into the second log representation vector in a certain way. For example, if the threat intelligence indicates that the probability of a certain IP address being a malicious attack source is 0.8, then this probability value can be fused into \(V\) as a new element or in a weighted manner to obtain the second log representation vector with fused enhanced information (where is the new element after fusion, for example).

[0134] The second risk data sequence is similar to the first risk data sequence, which is the result sequence obtained after the discriminator component performs risk assessment on the second log characterization vector of the fusion-enhanced information. Due to the fusion of enhanced information, this risk data sequence may be different from the first risk data sequence. For example, when the enhanced information is not fused, for a certain network security monitoring log, a certain risk coefficient in the first risk data sequence is 0.2, while after fusing the information of the high-risk IP address involved in the log from external threat intelligence, the risk coefficient in the corresponding second risk data sequence may become 0.5, indicating that after considering the additional information, the risk level is re-evaluated as higher.

[0135] In step S350, the network security situation awareness system optimizes the parameters of the space adaptation component and the discriminator component along the direction of minimizing the risk coefficient of the first risk data sequence and along the direction of maximizing the risk coefficient of the second risk data sequence, and obtains a calibrated and converged network security situation awareness neural network.

[0136] In a neural network, parameters (such as weights and biases, etc.) determine the output result of the network. The network security situation awareness system needs to adjust these parameters so that the output of the network (here is the risk data sequence) meets the expected goal. For example, parameters such as the linear transformation matrix A and the translation vector b in the space adaptation component and the weights and biases in the discriminator component need to be optimized.

[0137] The network security situation awareness system optimizes along the direction of minimizing the risk coefficient of the first risk data sequence and maximizing the risk coefficient of the second risk data sequence. From a mathematical perspective, this can be regarded as a multi-objective optimization problem. Assume that the objective function is a function of the risk coefficient of the first risk data sequence, and the objective function is a function of the risk coefficient of the second risk data sequence. For example, (where is the i-th risk coefficient in the first risk data sequence), is the i-th risk coefficient in the second risk data sequence), and the goal here is to minimize and maximize . The network security situation awareness system can adopt a gradient-based optimization algorithm, such as the Stochastic Gradient Descent (SGD) algorithm. For the parameters \(\theta\) of the space adaptation component and the discriminator component (including various weights and biases mentioned above), calculate the gradients and , and then update according to a certain update rule (where and to update the parameters with the learning rate).

[0138] As the parameters are continuously optimized in the above direction, the network security situation awareness neural network will gradually reach a calibrated convergence state. Calibrated convergence means that the output of the network (risk data sequence) has reached a relatively stable and accurate state on the given network security monitoring training logs. For example, the risk coefficients in the risk data sequence no longer change significantly, or after multiple iterative optimizations, the risk assessment results of the network for the network security monitoring training logs are very close to the actual risk situation. At this time, the obtained network security situation awareness neural network can effectively perform risk assessment on the new network security monitoring logs to be analyzed, and accurately determine whether it is a risk-free network security monitoring log or a risky network security monitoring log.

[0139] In one implementation, the method provided by the embodiments of the present application further includes:

[0140] Step S400: Obtain the network security monitoring logs to be analyzed;

[0141] Step S500: Align the semantics and formats of the network monitoring events in the network security monitoring logs to be analyzed with those of the network monitoring events in the reference network security monitoring logs to obtain the unified network security monitoring logs of the network security monitoring logs to be analyzed. The reference network security monitoring logs are used to standardize the text formats and language spaces of the network monitoring events in different network security monitoring logs;

[0142] Step S600: Load the unified network security monitoring logs of the network security monitoring logs to be analyzed into the calibrated and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring logs to be analyzed. The risk identification result indicates that the network security monitoring logs to be analyzed are risk-free network security monitoring logs or risky network security monitoring logs.

[0143] In step S400, the network security situation awareness system obtains the network security monitoring logs to be analyzed, which are the network security monitoring logs that need to be risk-assessed. Their sources are similar to the network security monitoring logs obtained in step S100 and can be logs from firewalls, intrusion detection systems, intrusion prevention systems, antivirus software, network traffic analysis tools, etc. For example, in the network environment of an enterprise, the firewall continuously records information related to network connections, and these information form firewall logs, and some or all of these firewall logs may be used as the network security monitoring logs to be analyzed. Suppose new devices are connected to the enterprise's network or abnormal network traffic patterns are detected, and the relevant firewall logs will be used as the network security monitoring logs to be analyzed to determine whether there are security risks in these network activities.

[0144] The technical means for the network security situation awareness system to obtain the network security monitoring logs to be analyzed are similar to those for obtaining the training logs. If it is obtained from a local device, for example, obtaining logs from a local firewall device, the network security situation awareness system can obtain them through file reading operations (if the logs are stored in file form) or through specific interfaces provided by the device (such as API interfaces or network management protocol interfaces). Taking the Simple Network Management Protocol (SNMP) as an example, the network security situation awareness system can send an SNMP-Get request to the firewall device as the management end, and the firewall device, as the agent end, responds to the request and returns an SNMP-Response message containing the network security monitoring log information to be analyzed. The network security situation awareness system then parses the message to obtain the log content. If it is obtained from a remote device or cloud service, it may involve network communication protocols (such as HTTP, HTTPS, etc.), and the log data is obtained by sending requests and receiving responses.

[0145] In step S500, the network security situation awareness system aligns the semantics and formats of the network monitoring events in the network security monitoring logs to be analyzed with those in the reference network security monitoring logs, and obtains the unified network security monitoring logs of the network security monitoring logs to be analyzed. The reference network security monitoring logs are used to standardize the text formats and language spaces of the network monitoring events in different network security monitoring logs.

[0146] The semantic and format alignment is similar to the operation in step S201. The purpose is to convert the network security monitoring logs to be analyzed into logs with the same semantics and formats as the reference network security monitoring logs, so as to perform risk identification under a unified standard subsequently. For example, assume that the network security monitoring logs to be analyzed come from an intrusion detection system, and the format of the recorded network monitoring events is "IP_SRC:192.168.1.10,IP_DST:10.0.0.1,PORT:80,PROTO:TCP,EVENT:Potential_Attack", while the format of the reference network security monitoring logs is "Source IP: [IP address], Destination IP: [IP address], Port: [port number], Protocol: [protocol type], Event: [event description]". The network security situation awareness system needs to convert "IP_SRC" to "Source IP", "IP_DST" to "Destination IP", "PROTO" to "Protocol", and "EVENT:Potential_Attack" to an event description that conforms to the semantics of the reference logs (such as "Event: Suspected attack").

[0147] The network security monitoring log to be analyzed after semantic and format alignment becomes the unified network security monitoring log. This makes the network security monitoring logs to be analyzed from different sources consistent in format and semantics, facilitating the processing of them by the network security situation awareness neural network in subsequent steps. Technically speaking, the network security situation awareness system can adopt a method similar to that in step S201. For example, semantic alignment can be processed by establishing a semantic mapping table. For format alignment, each data item in the original log can be rearranged and adjusted according to the format definition of the reference log.

[0148] In step S600, the network security situation awareness system loads the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring log to be analyzed. The risk identification result indicates that the network security monitoring log to be analyzed is a risk-free network security monitoring log or a risky network security monitoring log.

[0149] The calibrated and converged network security situation awareness neural network is a neural network obtained after calibration and optimization in step S300. It has learned the feature patterns in the network security monitoring training logs and can perform risk assessment on the input unified network security monitoring log. For example, this neural network has learned the patterns of normal network behaviors through a large number of risk-free network security monitoring training logs, such as normal network connection patterns (including combinations of legal source IP addresses, destination IP addresses, port numbers, and protocol types, etc.), and has learned the characteristics of abnormal network behaviors through risky network security monitoring training logs.

[0150] The risk identification result is the judgment result obtained by the network security situation awareness system based on the output of the neural network regarding whether there is a risk in the network security monitoring log to be analyzed. There are two possible results: a risk-free network security monitoring log or a risky network security monitoring log. For example, if the network security monitoring log to be analyzed is a record of normal communication between internal network devices, it may be determined to be a risk-free network security monitoring log after being processed by the neural network; if it is a record of frequent port scans from an external unknown IP address, it may be determined to be a risky network security monitoring log.

[0151] In one implementation, after the network security situation awareness system loads the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network, the neural network will output the risk data sequence corresponding to the network security monitoring log to be analyzed. Each data item in the risk data sequence corresponding to the network security monitoring log to be analyzed corresponds to a risk coefficient. For example, assume the risk data sequence is , where r_1 may correspond to the risk coefficient related to the source IP address, r_2 corresponds to the risk coefficient related to the destination IP address, and so on. The network security situation awareness system determines the risk coefficient of the network security monitoring log to be analyzed based on the risk data sequence corresponding to the network security monitoring log to be analyzed. A possible determination method is to perform a weighted sum of the risk coefficients in the risk data sequence. Assuming the weight vector is , then the risk coefficient of the network security monitoring log to be analyzed . If R is greater than the first set coefficient value (this set coefficient value can be obtained by analyzing a large number of known risky and risk-free network security monitoring logs during the training process, for example, determining a threshold that can effectively distinguish risks and non-risks through statistical analysis), it is determined that the risk identification result of the network security monitoring log to be analyzed is a risky network security monitoring log; if R is less than the first set coefficient value, it is determined as a risk-free network security monitoring log.

[0152] In another embodiment, after the network security situation awareness system loads the unified network security monitoring logs of the network security monitoring logs to be analyzed into the calibrated and converged network security situation awareness neural network, the neural network outputs the log representation vectors of the respective log local data of the network security monitoring logs to be analyzed. For example, assume that the unified network security monitoring logs of the network security monitoring logs to be analyzed contain multiple parts, such as the source IP address part, destination IP address part, port number part, etc. of the network connection. The log representation vector corresponding to each part is the log representation vector of the log local data. The network security situation awareness system obtains the second similarity measure between the log representation vector of each log local data in the multiple log local data of the network security monitoring logs to be analyzed and the log representation vectors of the corresponding log local data in each network security monitoring training log. For example, for the log representation vector of the source IP address part in the network security monitoring logs to be analyzed, the network security situation awareness system calculates the similarity measure between it and the log representation vectors of the source IP address part in all network security monitoring training logs (which can be calculated using methods such as cosine similarity). Then, the network security situation awareness system determines the maximum value among the multiple second similarity measures as the risk coefficient of the log local data. For example, for the source IP address part, if the maximum similarity measure with the source IP address part in a certain network security monitoring training log is 0.8, then the risk coefficient of the source IP address part is 0.8. The network security situation awareness system determines the risk coefficient of the network security monitoring logs to be analyzed based on the risk coefficients of the multiple log local data, such as by using methods such as weighted average. Finally, according to rules similar to those above, that is, if the risk coefficient of the network security monitoring logs to be analyzed is greater than the first set coefficient value, it is determined as a risky network security monitoring log; if it is less than the first set coefficient value, it is determined as a non-risky network security monitoring log.

[0153] Through the operations in steps S400 - S600, the network security situation awareness system can effectively identify the risks of the network security monitoring logs to be analyzed. This process relies on the previously obtained network security monitoring logs, the calibrated network security situation awareness neural network, and the unified log format and semantic processing, thereby providing an important decision-making basis for network security management and helping network administrators timely discover and respond to security risks in the network.

[0154] In one embodiment, in step S600, loading the unified network security monitoring logs of the network security monitoring logs to be analyzed into the calibrated and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring logs to be analyzed may include:

[0155] Step S610: Load the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network, and output the risk data sequence corresponding to the network security monitoring log to be analyzed based on the calibrated and converged network security situation awareness neural network. Each data item in the risk data sequence corresponding to the network security monitoring log to be analyzed corresponds to a risk coefficient;

[0156] Step S620: Determine the risk coefficient of the network security monitoring log to be analyzed based on the risk data sequence corresponding to the network security monitoring log to be analyzed;

[0157] Step S630: If the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, determine that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risky network security monitoring log;

[0158] Step S640: If the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, determine that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risk-free network security monitoring log.

[0159] In step S610, the network security situation awareness system loads the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network, and outputs the risk data sequence corresponding to the network security monitoring log to be analyzed based on the calibrated and converged network security situation awareness neural network. Each data item in the risk data sequence corresponding to the network security monitoring log to be analyzed corresponds to a risk coefficient.

[0160] The calibrated and converged network security situation awareness neural network is a neural network optimized through the previous step S300. During the calibration process, it learns through a large number of network security monitoring training logs (including risk-free and risky logs) and has been able to effectively evaluate the risks of the input network security monitoring logs. For example, the structure of this neural network may include multiple hidden layers, and the neurons in each hidden layer are connected to the neurons in the front and back layers through specific weights and biases. These weights and biases are continuously adjusted during the calibration process to adapt to the characteristics of different network security monitoring logs. Assume that this neural network adopts a feedforward neural network structure, and the number of neurons in the input layer corresponds to the number of characteristics of the unified network security monitoring log. For example, if the unified network security monitoring log includes 5 characteristics such as source IP address, destination IP address, port number, protocol type, and event type, the input layer may have 5 neurons.

[0161] The unified network security monitoring log is the network security monitoring log to be analyzed after semantic and format alignment in step S500. For example, the original network security monitoring log to be analyzed may come from different devices with different formats and semantics. After alignment with the benchmark network security monitoring log, the unified network security monitoring log has a standard format and semantics. For example, the source IP address in the original firewall log may be recorded in a specific format, and after alignment, it is presented in the unified format of "source IP: [specific IP address]" according to the requirements of the benchmark network security monitoring log.

[0162] When the network security situation awareness system loads the unified network security monitoring log into the neural network, the neural network will output a risk data sequence according to its internal calculation logic. Each data item in this risk data sequence corresponds to a risk coefficient, and these risk coefficients reflect the risk degree of the network security monitoring log to be analyzed in different dimensions. For example, assume the risk data sequence is , where may represent the risk coefficient related to the source IP address, represents the risk coefficient related to the destination IP address, represents the risk coefficient related to the port number, represents the risk coefficient related to the protocol type, represents the risk coefficient related to the event type. Taking the source IP address as an example, if the source IP address in the network security monitoring log to be analyzed comes from a known malicious IP address segment, then can be assigned a higher value, such as 0.8; if the source IP address is a trusted IP address within the enterprise, can be assigned a lower value, such as 0.1.

[0163] The calculation process of the neural network involves a large number of matrix operations. Assume the feature vector of the input unified network security monitoring log is , the weight matrix of the first layer of the neural network is , and the bias vector is (where m is the number of neurons in the first hidden layer), then the output of the first hidden layer can be calculated by the formula , where f is the activation function, such as the ReLU function . Through multiple such calculations, the final output risk data sequence is obtained.

[0164] In step S620, the network security situation awareness system determines the risk coefficient of the network security monitoring log to be analyzed based on the risk data sequence corresponding to the network security monitoring log to be analyzed.

[0165] The network security situation awareness system needs to comprehensively obtain a value from the risk data sequence that can represent the risk level of the entire network security monitoring log to be analyzed, namely the risk coefficient. A feasible method is weighted summation. Suppose the risk data sequence is , and the corresponding weight vector is , then the risk coefficient of the network security monitoring log to be analyzed. For example, if , then . The weights here can be determined according to the importance of different features in risk assessment. For example, in some network environments, the source IP address and event type may be more important for risk assessment, so their corresponding weights can be set relatively high.

[0166] The risk coefficient is a comprehensive indicator that reflects the overall risk level of the network security monitoring log to be analyzed. If the risk coefficient is relatively high, it means that high-risk factors act together in multiple dimensions (such as source IP address, destination IP address, port number, etc.); if the risk coefficient is relatively low, it indicates a relatively low overall risk.

[0167] In step S630, if the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, it is determined that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risky network security monitoring log.

[0168] The first set coefficient value is a predetermined threshold used to distinguish between risk-free and risky network security monitoring logs. Its determination is obtained by analyzing a large number of known risky and risk-free network security monitoring logs. For example, during the training process, the network security situation awareness system can use a part of representative network security monitoring logs (already clearly marked as risky or risk-free) to determine this threshold. Suppose 1000 network security monitoring logs are collected, of which 500 are risk-free and 500 are risky. After calculating the risk coefficients of these logs (according to the method in the previous steps), a value that can effectively distinguish these two types of logs is found through statistical analysis as the first set coefficient value. For example, after analysis, it is found that most of the logs with a risk coefficient greater than 0.5 are risky, so 0.5 can be set as the first set coefficient value.

[0169] When the risk coefficient of the network security monitoring log to be analyzed is greater than this first set coefficient value, it is determined to be a risky network security monitoring log. For example, if the risk coefficient R of the network security monitoring log to be analyzed is 0.6, and the first set coefficient value is 0.5, then the network security situation awareness system determines that this network security monitoring log to be analyzed is risky. This means that in the network monitoring events recorded in this log, there are some factors (such as suspicious source IP addresses, abnormal port number usage, or specific event types, etc.) that make the overall risk exceed the pre-set security range.

[0170] In step S640, if the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, it is determined that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risk-free network security monitoring log.

[0171] Similar to step S630, when the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, for example, R = 0.4, and the first set coefficient value is 0.5, the network security situation awareness system determines that this network security monitoring log to be analyzed is a risk-free network security monitoring log. This indicates that in the network monitoring events recorded in this log, the combined risk level of each factor (source IP address, destination IP address, port number, protocol type, event type, etc.) is within the acceptable security range, and these network monitoring events may be normal network activities, such as normal internal network communication within an enterprise, legal external network access, etc.

[0172] In another implementation, step S600, loading the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring log to be analyzed may include:

[0173] Step S600A: Loading the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network, and outputting the log representation vectors of the respective log local data of the network security monitoring log to be analyzed based on the calibrated and converged network security situation awareness neural network;

[0174] Step S600B: For each log local data among the multiple log local data of the network security monitoring log to be analyzed, obtaining the second similarity measure between the log representation vector of the log local data and the log representation vectors of the corresponding log local data in each network security monitoring training log;

[0175] Step S600C: Determining the maximum value among the multiple second similarity measures as the risk coefficient of the log local data;

[0176] Step S600D: Determine the risk coefficient of the network security monitoring log to be analyzed based on the risk coefficients of multiple log local data;

[0177] Step S600E: If the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, determine that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risky network security monitoring log;

[0178] Step S600F: If the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, determine that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risk-free network security monitoring log.

[0179] In step S600A, the network security situation awareness system loads the unified network security monitoring log of the network security monitoring log to be analyzed into the calibrated and converged network security situation awareness neural network, and outputs the log representation vectors of each of the multiple log local data of the network security monitoring log to be analyzed based on the calibrated and converged network security situation awareness neural network.

[0180] The calibrated and converged network security situation awareness neural network is a neural network optimized through the previous step S300 calibration. During the calibration process, it learns through a large number of network security monitoring training logs (including risk-free and risky logs), and the parameters of the neural network (such as weights and biases, etc.) have been adjusted to a relatively stable state, enabling it to effectively analyze the input network security monitoring logs. For example, this neural network may have a multi-layer structure, and each layer performs a specific transformation on the input data. Suppose it is a deep neural network, including an input layer, multiple hidden layers, and an output layer. During the training process, it learns the complex relationships between different network security monitoring log features, so as to be able to process new network security monitoring logs to be analyzed.

[0181] The unified network security monitoring log is the network security monitoring log to be analyzed that has been aligned in semantics and format in step S500. For example, the original network security monitoring log to be analyzed may come from different devices (such as firewalls, intrusion detection systems, etc.), and its format and semantics may vary. After alignment with the reference network security monitoring log, the unified network security monitoring log has a standard format and semantics. For example, for the record of the source IP address, regardless of the original log format, the unified network security monitoring log is recorded in the standard format, such as the form of "source IP: [specific IP address]".

[0182] Log local data refers to each part in the unified network security monitoring log. For example, for a unified network security monitoring log containing information such as source IP address, destination IP address, port number, protocol type, and event type, the source IP address part, destination IP address part, etc. can each be regarded as log local data. These log local data respectively contain information on different aspects of network monitoring events.

[0183] Log representation vector: For each log local data, the network security situation awareness system converts it into a log representation vector through a neural network. The log representation vector is a mathematical representation form of the characteristics of log local data. For example, for the log local data of the source IP address, the network security situation awareness system can convert it into a vector, which contains some characteristic information about this source IP address. Suppose the source IP address is classified and encoded according to its address segments, and then combined with some other relevant information (such as whether the network area to which this IP address belongs is a high-risk area, etc.), and finally forms a vector form, such as , this vector is the log representation vector of the log local data of the source IP address. Technically speaking, when the neural network processes log local data, it can adopt operations similar to those of a fully connected layer. Suppose the input of the log local data is , the weight matrix of the neural network is , and the bias vector is , then the output log representation vector can be calculated through the formula , where f is an activation function, such as the ReLU function .

[0184] In step S600B, the network security situation awareness system obtains the second similarity measure between the log representation vector of each log local data in multiple log local data of the network security monitoring log to be analyzed and the log representation vectors of the corresponding log local data in each network security monitoring training log. The similarity measure is an index used to measure the similarity degree between two vectors. In this step, it is to measure the similarity between the log representation vector of the log local data of the network security monitoring log to be analyzed and the log representation vectors of the corresponding log local data in the network security monitoring training log. For example, for the log representation vector of the source IP address in the network security monitoring log to be analyzed and the log representation vector of a certain source IP address in the network security monitoring training log, the cosine similarity can be used to calculate their second similarity measure. The cosine similarity formula is . In addition to the cosine similarity, other similarity measure methods such as the Euclidean distance can also be used. The Euclidean distance formula is , the smaller the Euclidean distance, the more similar it is, while the larger the cosine similarity value, the more similar it is.

[0185] The network security monitoring training logs are the logs used to train the network security situation awareness neural network in the previous steps (such as steps S200 and S300). They contain known risk-free and risky network security monitoring logs. Through the training of these logs, the neural network learns the characteristic patterns of normal and abnormal network behaviors. For example, in the source IP address part of the network security monitoring training logs, there are both IP addresses from normal internal networks and records from known malicious IP addresses. These different records help the neural network identify the characteristic features of different types of source IP addresses during the training process.

[0186] The network security situation awareness system calculates the similarity measure between each log local data (such as source IP address, destination IP address, etc.) of the network security monitoring log to be analyzed and the corresponding log local data in the network security monitoring training logs respectively. Suppose there are 5 log local data (source IP address, destination IP address, port number, protocol type, event type) in the network security monitoring log to be analyzed and 100 in the network security monitoring training logs. For the log representation vector of the source IP address, which is a log local data in the network security monitoring log to be analyzed, the network security situation awareness system needs to calculate the second similarity measure between it and the log representation vectors of the source IP addresses in 100 network security monitoring training logs, obtaining a total of 100 similarity measure values; similarly, similar calculations are performed for other log local data such as the destination IP address.

[0187] In step S600C, the network security situation awareness system determines the maximum value among multiple second similarity measures as the risk coefficient of the log local data.

[0188] By taking the maximum value among the similarity metrics of each log local data with the corresponding log local data in the network security monitoring training logs as the risk coefficient of this log local data. For example, for the log local data of the source IP address in the network security monitoring log to be analyzed, after the network security situation awareness system calculates the second similarity metric between its log feature vector and the log feature vectors of the source IP addresses in 100 network security monitoring training logs, assuming these similarity metric values are 0.1, 0.2, 0.3, …, 0.8 respectively, then the risk coefficient of this source IP address log local data is 0.8. The meaning of this risk coefficient is the maximum similarity degree between this log local data and a certain corresponding log local data in the network security monitoring training logs. If this maximum value is small, it indicates that this log local data has a large difference from the corresponding part in the known normal or abnormal training logs and may pose a risk; if this maximum value is large, it indicates that it is relatively similar to a certain part in the training logs, with relatively low risk (if it is similar to the risk-free training logs) or relatively high risk (if it is similar to the risky training logs).

[0189] In step S600D, the network security situation awareness system determines the risk coefficient of the network security monitoring log to be analyzed based on the risk coefficients of multiple log local data.

[0190] The network security situation awareness system comprehensively obtains the overall risk coefficient of the network security monitoring log to be analyzed from the risk coefficients of each log local data. A feasible method is weighted summation. Assume that the network security monitoring log to be analyzed has 5 log local data (source IP address, destination IP address, port number, protocol type, event type), and their risk coefficients are respectively , and the corresponding weight vector is , then the risk coefficient of the network security monitoring log to be analyzed. For example, if , then . The weights here can be determined according to the importance of different log local data in the overall risk assessment. For example, in some network environments, the source IP address and event type may be more important for risk assessment, so their corresponding weights can be set relatively high.

[0191] In step S600E, if the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, then determine that the risk identification result of the network security monitoring log to be analyzed is that the network security monitoring log to be analyzed is a risky network security monitoring log.

[0192] The first set coefficient value is a predetermined threshold used to distinguish between risk-free and risky cybersecurity monitoring logs. Its determination requires analyzing a large number of known risky and risk-free cybersecurity monitoring logs. For example, during the training process, the cybersecurity situational awareness system can use a representative portion of cybersecurity monitoring logs (already clearly marked as risky or risk-free) to determine this threshold. Suppose 1000 cybersecurity monitoring logs are collected, 500 of which are risk-free and 500 are risky. After calculating the risk coefficients of these logs (using the method in the previous steps), a value that can effectively distinguish between these two types of logs is found through statistical analysis as the first set coefficient value. For instance, after analysis, it is found that most logs with a risk coefficient greater than 0.5 are risky, then 0.5 can be set as the first set coefficient value.

[0193] When the risk coefficient of the cybersecurity monitoring log to be analyzed is greater than this first set coefficient value, for example, if the risk coefficient R of the cybersecurity monitoring log to be analyzed is 0.6 and the first set coefficient value is 0.5, then the cybersecurity situational awareness system determines that this cybersecurity monitoring log to be analyzed is risky. This means that in the network monitoring events recorded in this log, there are some factors (such as suspicious situations in the local data of the log, etc.) that cause the overall risk to exceed the pre-set security range.

[0194] In step S600F, if the risk coefficient of the cybersecurity monitoring log to be analyzed is less than the first set coefficient value, it is determined that the risk identification result of the cybersecurity monitoring log to be analyzed is that the cybersecurity monitoring log to be analyzed is a risk-free cybersecurity monitoring log.

[0195] Similar to step S600E, when the risk coefficient of the cybersecurity monitoring log to be analyzed is less than the first set coefficient value, for example, R = 0.4 and the first set coefficient value is 0.5, the cybersecurity situational awareness system determines that this cybersecurity monitoring log to be analyzed is a risk-free cybersecurity monitoring log. This indicates that in the network monitoring events recorded in this log, the combined risk level of the local data of each log is within the acceptable security range, and these network monitoring events may be normal network activities, such as normal internal network communication within an enterprise, legal external network access, etc.

[0196] Based on the foregoing embodiments, an embodiment of the present application provides a network security situation awareness device. Each unit included in the device, as well as each module included in each unit, may be implemented by a processor in a computer device; of course, it may also be implemented by specific logic circuits. During implementation, the processor may be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0197] Figure 2 FIG. is a schematic structural diagram of a network security situation awareness device provided by an embodiment of the present application. As Figure 2 shown, the network security situation awareness device 200 includes:

[0198] A log acquisition module 210, configured to acquire a plurality of network security monitoring logs. Each network security monitoring log includes a network monitoring event. The plurality of network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs. The network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks.

[0199] A sample determination module 220, configured to determine one or more target network security monitoring logs from the plurality of network security monitoring logs based on the log feature vectors of the plurality of network security monitoring logs, and determine them as network security monitoring training logs for calibrating the network security situation awareness neural network, where the target network security monitoring logs are risk-free network security monitoring logs determined through the log feature vectors.

[0200] A network optimization module 230, configured to calibrate and optimize the network security situation awareness neural network through the network security monitoring training logs to obtain a calibrated and converged network security situation awareness neural network, and the calibrated and converged network security situation awareness neural network is used to determine whether the network security monitoring log to be analyzed is a risk-free network security monitoring log or a risky network security monitoring log.

[0201] The description of the above device embodiments is similar to the description of the above method embodiments and has similar beneficial effects to the method embodiments. In some embodiments, the functions or modules included in the device provided by the embodiments of the present application may be used to execute the methods described in the above method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0202] Figure 3 The following is a schematic diagram of the hardware of a network security situation awareness system provided by an embodiment of the present application. As Figure 3 shown, the hardware entities of the network security situation awareness system 1000 include: a processor 1001 and a memory 1002. Among them, the memory 1002 stores a computer program that can run on the processor 1001, and when the processor 1001 executes the program, it implements the steps in the method of any of the above embodiments.

[0203] As mentioned above, it is only the implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Within the protection scope of the present application.

Claims

1. A network security situation awareness method, characterized in that: The method comprises: Acquire multiple network security monitoring logs, each of which includes a network monitoring event, and the multiple network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs. The network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks. Based on the log characterization vectors of each of the multiple network security monitoring logs, one or more target network security monitoring logs are determined from the multiple network security monitoring logs, and determined as network security monitoring training logs for calibrating the network security situation awareness neural network, wherein the target network security monitoring log is the risk-free network security monitoring log determined by the log characterization vector; The network security situation awareness neural network is tuned and optimized through the network security monitoring training log to obtain a converged network security situation awareness neural network, and the converged network security situation awareness neural network is used to determine whether the network security monitoring log to be analyzed is the risk-free network security monitoring log or the risky network security monitoring log; The method of determining one or more target network security monitoring logs from the multiple network security monitoring logs based on the log representation vectors of each of the multiple network security monitoring logs, and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network, includes: For each of the multiple network security monitoring logs, determine a first similarity measure between a log representation vector of the network security monitoring log and log representation vectors of other network security monitoring logs in the multiple network security monitoring logs, and obtain multiple first similarity measures corresponding to the network security monitoring log; Determine a mean of the similarity metrics corresponding to the network security monitoring log based on x first similarity metrics among the multiple first similarity metrics corresponding to the network security monitoring log, wherein the x first similarity metrics are the largest x first similarity metrics among the multiple first similarity metrics corresponding to the network security monitoring log, and x≥1; Based on the mean values ​​of the similarity metrics corresponding to the multiple network security monitoring logs, one or more target network security monitoring logs are determined from the multiple network security monitoring logs and determined as network security monitoring training logs for calibrating the network security situation awareness neural network.

2. The method according to claim 1, characterized in that The method of determining one or more target network security monitoring logs from the multiple network security monitoring logs based on the mean values ​​of the similarity metrics corresponding to each of the multiple network security monitoring logs, and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network, includes: Arranging the plurality of network security monitoring logs in order according to a decreasing manner of the similarity metric mean to obtain a network security monitoring log set; Determine the first k network security monitoring logs in the network security monitoring log set as the target network security monitoring logs; The target network security monitoring log is determined as a network security monitoring training log for calibrating the network security situation awareness neural network.

3. The method according to claim 1, characterized in that Before determining one or more target network security monitoring logs from the multiple network security monitoring logs based on the log representation vectors of each of the multiple network security monitoring logs and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network, the method further includes: For each of the multiple network security monitoring logs, the semantics and format of the network monitoring events in the network security monitoring log are aligned with those of the network monitoring events in the benchmark network security monitoring log to obtain a unified network security monitoring log of the network security monitoring logs, and the benchmark network security monitoring log is used to normalize the text format and language space of the network monitoring events in different network security monitoring logs; wherein the network security monitoring training log is determined in the unified network security monitoring log of the multiple network security monitoring logs based on the respective log representation vectors of the unified network security monitoring log of the multiple network security monitoring logs.

4. The method according to claim 3, characterized in that: The step of aligning the semantics and format of the network monitoring events in the network security monitoring log with those in the reference network security monitoring log to obtain a unified network security monitoring log of the network security monitoring log includes: Performing event boundary identification on the network security monitoring log, extracting a set of event trigger data items of the network security monitoring log, wherein each event trigger data item in the set of event trigger data items of the network security monitoring log corresponds to a first data item position; Constructing a first correspondence between the first data item position and the second data item position, wherein the second data item position represents a data item position corresponding to an event triggering data item in the reference network security monitoring log; Based on the first corresponding relationship and the first data item position conversion array, obtaining a third data item position corresponding to the first data item position in the reference network security monitoring log; Obtaining a data item position difference between the third data item position and the second data item position; Optimizing the first data item position conversion array in a direction of reducing the data item position difference; If the first data item position conversion array does not change any more, the network security monitoring log is processed by using the first data item position conversion array to obtain a unified network security monitoring log of the network security monitoring log.

5. The method according to claim 1, characterized in that The network security situation awareness neural network includes an encoder component, a spatial adaptation component and a discriminator component; The step of adjusting and optimizing the network security situation awareness neural network through the network security monitoring training log to obtain a network security situation awareness neural network that has been adjusted and converged includes: Loading the network security monitoring training log into the encoder component, and obtaining a first log representation vector of the network security monitoring training log based on the encoder component; Loading the first log representation vector into the space adaptation component, obtaining a second log representation vector based on the space adaptation component, wherein the space adaptation component is used to map the first log representation vector to a representation space of a first network transmission scenario; Loading the second log representation vector into the discriminator component, outputting a first risk data sequence corresponding to the second log representation vector based on the discriminator component, wherein each data item in the first risk data sequence corresponds to a risk coefficient; Loading the second log representation vector of the fused enhancement information to the discriminator component, outputting a second risk data sequence corresponding to the second log representation vector of the fused enhancement information based on the discriminator component, wherein each data item in the second risk data sequence corresponds to a risk coefficient; The parameters of the spatial adaptation component and the discriminator component are optimized along the direction that minimizes the risk coefficient of the first risk data sequence and along the direction that maximizes the risk coefficient of the second risk data sequence to obtain a convergently tuned network security situation awareness neural network.

6. The method according to claim 1, characterized in that The method further comprises: Obtain the network security monitoring log to be analyzed; Aligning the semantics and format of the network monitoring events in the network security monitoring log to be analyzed with those in the reference network security monitoring log to obtain a unified network security monitoring log of the network security monitoring log to be analyzed, wherein the reference network security monitoring log is used to normalize the text format and language space of the network monitoring events in different network security monitoring logs; The unified network security monitoring log of the network security monitoring log to be analyzed is loaded into the calibrated and converged network security situation awareness neural network to obtain a risk identification result of the network security monitoring log to be analyzed, wherein the risk identification result indicates that the network security monitoring log to be analyzed is the risk-free network security monitoring log or the risky network security monitoring log.

7. The method according to claim 6, characterized in that The step of loading the unified network security monitoring log of the network security monitoring log to be analyzed into the adjusted and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring log to be analyzed includes: Loading the unified network security monitoring log of the network security monitoring log to be analyzed into the adjusted and converged network security situation awareness neural network, outputting the risk data sequence corresponding to the network security monitoring log to be analyzed based on the adjusted and converged network security situation awareness neural network, wherein each data item in the risk data sequence corresponding to the network security monitoring log to be analyzed corresponds to a risk coefficient; Determining the risk coefficient of the network security monitoring log to be analyzed based on the risk data sequence corresponding to the network security monitoring log to be analyzed; If the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, determining the risk identification result of the network security monitoring log to be analyzed as the network security monitoring log to be analyzed is the risky network security monitoring log; If the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, determining the risk identification result of the network security monitoring log to be analyzed as the network security monitoring log to be analyzed is the risk-free network security monitoring log; Alternatively, the step of loading the unified network security monitoring log of the network security monitoring log to be analyzed into the adjusted and converged network security situation awareness neural network to obtain the risk identification result of the network security monitoring log to be analyzed includes: Loading the unified network security monitoring log of the network security monitoring log to be analyzed into the adjusted and converged network security situation awareness neural network, and outputting the log representation vectors of the multiple log local data of the network security monitoring log to be analyzed based on the adjusted and converged network security situation awareness neural network; For each of the multiple log local data of the network security monitoring log to be analyzed, obtaining a second similarity measure between a log representation vector of the log local data and a log representation vector of a corresponding log local data in each of the network security monitoring training logs; Determine a maximum value among a plurality of the second similarity measures as a risk coefficient of the log local data; Determining the risk coefficient of the network security monitoring log to be analyzed based on the risk coefficients of the plurality of local data of the logs; If the risk coefficient of the network security monitoring log to be analyzed is greater than the first set coefficient value, determining the risk identification result of the network security monitoring log to be analyzed as the network security monitoring log to be analyzed is the risky network security monitoring log; If the risk coefficient of the network security monitoring log to be analyzed is less than the first set coefficient value, the risk identification result of the network security monitoring log to be analyzed is determined as the network security monitoring log to be analyzed is the risk-free network security monitoring log.

8. A network security situation awareness device, characterized in that: include: A log acquisition module, used to acquire a plurality of network security monitoring logs, each of which includes a network monitoring event, wherein the plurality of network security monitoring logs include one or more risk-free network security monitoring logs and one or more risky network security monitoring logs, wherein the network monitoring events included in the risk-free network security monitoring logs do not have security risks, and the network monitoring events included in the risky network security monitoring logs have security risks; A sample determination module is used to determine one or more target network security monitoring logs from the multiple network security monitoring logs based on the log characterization vectors of each of the multiple network security monitoring logs, and determine them as network security monitoring training logs for calibrating the network security situation awareness neural network, wherein the target network security monitoring log is the risk-free network security monitoring log determined by the log characterization vector; the one or more target network security monitoring logs are determined from the multiple network security monitoring logs based on the log characterization vectors of each of the multiple network security monitoring logs, and determined as network security monitoring training logs for calibrating the network security situation awareness neural network, including: for each of the multiple network security monitoring logs, determining the date of the network security monitoring log The method comprises the following steps: determining a first similarity metric between a log characterization vector and the log characterization vectors of the remaining network security monitoring logs in the plurality of network security monitoring logs to obtain a plurality of first similarity metrics corresponding to the network security monitoring log; determining a mean value of the similarity metric corresponding to the network security monitoring log based on x first similarity metrics among the plurality of first similarity metrics corresponding to the network security monitoring log, wherein the x first similarity metrics are the largest x first similarity metrics among the plurality of first similarity metrics corresponding to the network security monitoring log, and x≥1; determining one or more target network security monitoring logs from the plurality of network security monitoring logs based on the mean values ​​of the similarity metrics corresponding to each of the plurality of network security monitoring logs, and determining them as network security monitoring training logs for calibrating the network security situation awareness neural network; The network optimization module is used to adjust and optimize the network security situation awareness neural network through the network security monitoring training log to obtain a network security situation awareness neural network that has been adjusted and converged. The network security situation awareness neural network that has been adjusted and converged is used to determine whether the network security monitoring log to be analyzed is the risk-free network security monitoring log or the risky network security monitoring log.

9. A network security situation awareness system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 7 are implemented.

Images