Abnormal flow monitoring methods, devices and storage media

Abstract

The present application provides an abnormal traffic monitoring method, device and storage medium, which embeds the log initial characterization information of the target traffic monitoring log to obtain a monitoring embedding mapping vector, embeds the reference log characterization information related to the target traffic monitoring log and extracted from the normal control traffic monitoring log to obtain a reference embedding mapping vector, performs one or more stages of joint reconstruction operations based on the monitoring embedding mapping vector and the reference embedding mapping vector to obtain a fused reconstruction vector, obtains log reconstruction characterization information for the target traffic monitoring log based on the fused reconstruction vector, and obtains monitoring information of the target traffic monitoring log based on the reconstruction error between the log reconstruction characterization information and the log initial characterization information. The present application can increase the accuracy of attack traffic identification based on the reconstruction vector obtained by the fused reconstruction vector.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an abnormal flow monitoring method, device and storage medium. Background Art

[0002] With the rapid development of information technology, the Internet plays an indispensable role in people's life, work and all areas of society.

[0003] Traditional rule-based abnormal traffic monitoring methods rely on pre-set rule sets. For example, abnormal traffic is identified by defining specific rules such as IP addresses, port numbers, and protocol types. However, this method has obvious limitations. On the one hand, the network environment is constantly changing, and new attack methods emerge in an endless stream. It is difficult for pre-set rules to cover all possible abnormal situations in a timely manner. On the other hand, attackers can use some means to disguise traffic to bypass rule-based detection, such as using IP spoofing technology or hiding threat behaviors on the basis of normal protocols, making it difficult for rule-based monitoring methods to accurately identify.

[0004] Statistical analysis methods detect anomalies by analyzing the statistical characteristics of network traffic, such as the mean, variance, and distribution of traffic. Although this method can detect abnormal traffic that differs greatly from normal traffic statistical characteristics to a certain extent, its accuracy is also challenged by complex network environments and diverse attack methods. For example, when network traffic fluctuates, the statistical characteristics of normal traffic may also change significantly. At this time, the statistical analysis method may mistakenly identify normal traffic as abnormal traffic, or conversely, fail to detect abnormal traffic disguised as normal traffic fluctuations.

[0005] Some early machine learning methods have been applied to abnormal traffic monitoring, such as decision trees and support vector machines. These methods have improved the ability of abnormal traffic monitoring to a certain extent, but there are still some problems. For example, the decision tree method is prone to overfitting, and its generalization ability may be insufficient for large-scale and complex network traffic data; the support vector machine has a high computational complexity when processing high-dimensional data, and has strict requirements for data labeling. If the data labeling is inaccurate, it will affect the accuracy of the monitoring results. Summary of the invention

[0006] The purpose of the present invention is to provide a method, device and storage medium for monitoring abnormal flow. This application is implemented as follows:

[0007] In a first aspect, the present application provides an abnormal traffic monitoring method, the method comprising: obtaining a target traffic monitoring log, and embedding and mapping the log initial representation information of the target traffic monitoring log to obtain a monitoring embedded mapping vector of the target traffic monitoring log; determining reference log representation information related to the target traffic monitoring log, and embedding and mapping the reference log representation information to obtain a reference embedded mapping vector; the reference log representation information is extracted from a normal control traffic monitoring log corresponding to the target traffic monitoring log; performing one or more stages of joint reconstruction operations based on the monitoring embedded mapping vector and the reference embedded mapping vector to obtain the A fused reconstruction vector is obtained by a joint reconstruction operation of one or more stages. In the joint reconstruction operation of each stage, a saliency vector is extracted for the target monitoring embedding mapping vector and the target reference embedding mapping vector input at that stage to obtain the reference embedding mapping vector obtained at that stage, and a saliency vector is extracted for the reference embedding mapping vector obtained at that stage and the target monitoring embedding mapping vector to obtain the monitoring embedding mapping vector obtained at that stage; log reconstruction representation information of the target flow monitoring log is obtained based on the fused reconstruction vector; monitoring information of the target flow monitoring log is obtained based on a reconstruction error between the log reconstruction representation information and the log initial representation information.

[0008] In a second aspect, the present application provides an abnormal flow monitoring device, comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method as described above is implemented.

[0009] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program runs on a processor, the processor executes the method as described above.

[0010] Beneficial effects of the present application: The abnormal traffic monitoring method, device and storage medium provided by the present application embed the initial log representation information of the target traffic monitoring log to obtain a monitoring embedded mapping vector, embed the reference log representation information related to the target traffic monitoring log and extracted from the normal control traffic monitoring log to obtain a reference embedded mapping vector, perform one or more stages of joint reconstruction operations based on the monitoring embedded mapping vector and the reference embedded mapping vector to obtain a fused reconstruction vector, in each stage of the joint reconstruction operation, extract the significance vector of the target monitoring embedded mapping vector and the target reference embedded mapping vector input at that stage to obtain the reference embedded mapping vector obtained at that stage, and extract the significance vector of the reference embedded mapping vector and the target monitoring embedded mapping vector obtained at that stage to obtain the monitoring embedded mapping vector obtained at that stage, obtain the log reconstruction representation information for the target traffic monitoring log based on the fused reconstruction vector, and obtain the monitoring information of the target traffic monitoring log based on the reconstruction error between the log reconstruction representation information and the log initial representation information. During the abnormal traffic monitoring process, the joint reconstruction operation of each stage uses the target monitoring embedding mapping vector and the target reference embedding mapping vector input at that stage to extract the saliency vector, and obtains the reference embedding mapping vector obtained at that stage, and extracts the saliency vector of the reference embedding mapping vector and the target monitoring embedding mapping vector obtained at that stage to obtain the monitoring embedding mapping vector obtained at that stage. The overall semantics of the normal control traffic monitoring log and the adaptive reference embedding mapping vector of each stage can be combined to generate a fused reconstruction vector, so that the feature representation quality of the fused reconstruction vector is improved, and the accuracy of attack traffic identification based on the reconstruction vector obtained by the fused reconstruction vector is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of an abnormal flow monitoring method provided in an embodiment of the present application.

[0012] Figure 2 It is a schematic diagram of the composition of an abnormal flow monitoring device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The execution subject of the abnormal flow monitoring method in the embodiment of the present application is an abnormal flow monitoring device, including but not limited to a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. Figure 1 As shown, the method includes:

[0014] Step S100: Acquire a target traffic monitoring log, and embed and map the initial characterization information of the target traffic monitoring log to obtain a monitoring embedding mapping vector of the target traffic monitoring log.

[0015] In step S100 of the embodiment of the present application, the target traffic monitoring log is the relevant data recorded by the abnormal traffic monitoring device during the network traffic monitoring process, such as firewall logs, intrusion detection system (IDS / IPS) records, Web server logs, etc. The firewall log records the network connection information through the firewall, including the source IP address, destination IP address, port number, protocol type, and connection time. Taking an enterprise network as an example, the firewall may record connection attempts from external IP addresses to access specific ports of the enterprise's internal servers, and these records constitute part of the firewall log. IDS / IPS records focus on detecting intrusion behaviors in the network, such as the characteristics of threatening attack traffic, and may contain some abnormal network activity patterns or feature identifiers of known attacks. Web server logs record access information related to Web services, such as user access requests, requested resources (such as web pages, pictures, etc.), access time, client information (such as browser type), and other content.

[0016] The initial characterization information of the log is the information that can represent the characteristics of the log obtained by preliminary analysis and extraction of the target traffic monitoring log. For firewall logs, the initial characterization information of the log includes, for example, the distribution characteristics of the source IP address (for example, whether it comes from a specific IP segment, such as a known threat IP source area or an IP range of a specific department within the enterprise), the nature of the destination IP address (is it an important internal server or an external service provider address), the frequency of use of the port number (frequent access to certain special ports is an abnormal manifestation) and the distribution of protocol types (such as the usage ratio of TCP and UDP protocols, etc.). For Web server logs, the initial characterization information of the log can be the access frequency of different pages, the trend of changes in the number of visits in different time periods, etc. Embedding mapping is a technical means of converting high-dimensional, discrete initial characterization information of the log into a low-dimensional, continuous vector representation. This operation helps the abnormal traffic monitoring device to better process and analyze data. Assuming that there are n features in the initial characterization information of the log (for example, for firewall logs, n is the total number of features such as source IP, destination IP, port number, protocol type, etc.), a simple linear embedding mapping method can be used. Let the original initial characterization information of the log be the vector , through a mapping matrix (where m < n, indicating the dimension of the low-dimensional space after mapping), the monitoring embedding mapping vector y = xW is obtained. This mapping can convert the original complex log initial representation information into a vector in a low-dimensional space. This vector retains the main features of the original information while reducing the complexity of the data, facilitating subsequent processing and analysis. Through such an embedding mapping operation, the abnormal traffic monitoring device can obtain the monitoring embedding mapping vector of the target traffic monitoring log. This vector will serve as important input data for subsequent steps, and will be used for joint reconstruction operations with the reference embedding mapping vector, etc., so as to further determine whether there is abnormal traffic in the target traffic monitoring log.

[0017] Step S200: Determine reference log representation information related to the target traffic monitoring log, and embed and map the reference log representation information to obtain a reference embedding mapping vector; the reference log representation information is extracted from a normal control traffic monitoring log corresponding to the target traffic monitoring log.

[0018] In step S200 of the embodiment of the present application, the abnormal traffic monitoring device determines reference log representation information related to the target traffic monitoring log, and then embeds and maps the reference log representation information to obtain a reference embedding mapping vector.

[0019] The reference log characterization information is extracted from the normal control traffic monitoring log corresponding to the target traffic monitoring log. The control traffic monitoring log here refers to the traffic log recorded under normal network operation conditions, such as firewall logs, IDS / IPS records or Web server logs collected during a period of time when the network environment is stable and not attacked. Taking the firewall log as an example, the normal control traffic log may show certain regularity. The information extracted from these normal control traffic monitoring logs that can represent their characteristics is the reference log characterization information. For example, for the normal control traffic log of the firewall, the reference log characterization information may include the average traffic rate of the specific port connection from the internal network to the external network under normal circumstances, the normal proportion of different internal department IP addresses accessing external services, etc. The next step is to determine the reference log characterization information related to the target traffic monitoring log. The abnormal traffic monitoring device screens and determines according to some key features of the target traffic monitoring log. For example, if the target traffic monitoring log is a firewall log within a specific time period, and there is a specific network application scenario within this time period (such as a large-scale data backup operation within the enterprise, which may affect the network traffic pattern), then the abnormal traffic monitoring device will search for normal traffic logs with similar network application scenarios or similar time periods (such as during previous data backup operations) in the reference traffic monitoring log, and extract appropriate reference log representation information from these logs. This process may involve comparing and matching some key indicators in the target traffic monitoring log (such as the range of traffic rates, the main IP address segments involved, etc.) with the reference traffic monitoring log.

[0020] Similar to the embedding mapping in step S100, the embedding mapping operation on the reference log representation information is intended to convert the reference log representation information into a form that is more convenient for the abnormal traffic monitoring device to process and analyze. Assume that the reference log representation information contains m features, which are represented as vectors , also through a mapping matrix (where k < m, indicating the dimension of the low-dimensional space after mapping), the reference embedding mapping vector b is obtained using the formula b = aP. This embedding mapping operation converts the high-dimensional, discrete reference log representation information into a low-dimensional, continuous reference embedding mapping vector.

[0021] Step S300: Perform one or more stages of joint reconstruction operations based on the monitoring embedded mapping vector and the reference embedded mapping vector to obtain a fused reconstruction vector obtained by the joint reconstruction operation of one or more stages. In the joint reconstruction operation of each stage, perform saliency vector extraction on the target monitoring embedded mapping vector and the target reference embedded mapping vector input at that stage to obtain the reference embedded mapping vector obtained at that stage, and perform saliency vector extraction on the reference embedded mapping vector and the target monitoring embedded mapping vector obtained at that stage to obtain the monitoring embedded mapping vector obtained at that stage.

[0022] In step S300 of the embodiment of the present application, the abnormal traffic monitoring device performs one or more stages of joint reconstruction operations based on the monitoring embedded mapping vector and the reference embedded mapping vector to obtain a fused reconstruction vector. The monitoring embedded mapping vector is a vector obtained by embedding and mapping the initial characterization information of the target traffic monitoring log in step S100. This vector contains the converted characteristic information of the target traffic monitoring log. The reference embedded mapping vector is a vector obtained by extracting the reference log characterization information from the normal control traffic monitoring log and embedding and mapping it in step S200. It represents the characteristic information of normal traffic.

[0023] The joint reconstruction operation performed by the abnormal traffic monitoring device is a complex process. Taking the firewall traffic monitoring in the network as an example, it is assumed that the monitoring embedded mapping vector contains the converted feature information such as the source IP address, destination IP address, port number and protocol type of the target traffic, and the reference embedded mapping vector contains the typical feature information of these elements under normal traffic. In the joint reconstruction operation, the abnormal traffic monitoring device attempts to integrate and adjust the features of the target traffic with the features of the normal traffic to find out possible abnormal situations.

[0024] The joint reconstruction operation is divided into one or more stages. In each stage, the abnormal traffic monitoring device must perform specific operations to process the input target monitoring embedding mapping vector and target reference embedding mapping vector. Assume that the target monitoring embedding mapping vector is , the target reference embedding mapping vector is A possible joint reconstruction technique is based on weighted summation and transformation of vectors.

[0025] For example, the abnormal flow monitoring device can set a weight vector , first perform a weighted operation on the target monitoring embedding mapping vector X to obtain , a similar weighted operation is performed on the target reference embedding mapping vector Y to obtain (Here we assume that the weight vector W is compatible with the dimensions of X and Y. Some preprocessing may be required to ensure that the dimensions match.) Then sum the weighted vectors , and then perform a linear transformation on Z, such as through a transformation matrix (where p and q are determined according to the dimension of vector Z), we get Z'=ZA, where Z' is the intermediate result vector obtained by the joint reconstruction operation at this stage.

[0026] As the stages progress, the intermediate result vectors of each stage will continuously integrate the characteristic information of the target traffic and normal traffic. For example, in network traffic monitoring, if the intermediate result vector obtained by the joint reconstruction operation in the first stage finds that the usage frequency of a certain port number in the target traffic is significantly different from that of the normal traffic, then in the joint reconstruction operation in the subsequent stage, the abnormal traffic monitoring device will adjust the weight vector or transformation matrix according to this difference in order to analyze the target traffic more accurately.

[0027] During the entire joint reconstruction operation, the abnormal traffic monitoring device continuously weighs and adjusts the characteristics of the target traffic and the characteristics of the normal traffic, and finally obtains a fused reconstruction vector. This fused reconstruction vector combines the information of the target traffic monitoring log and the normal control traffic monitoring log, and can more accurately reflect the actual situation of the target traffic, providing an important basis for the subsequent judgment of whether there is abnormal traffic in the target traffic monitoring log. Through multiple stages of joint reconstruction operations, the abnormal traffic monitoring device can gradually and deeply analyze the differences between the target traffic and the normal traffic, and improve the accuracy and reliability of abnormal traffic monitoring.

[0028] Step S400: obtaining log reconstruction representation information of the target traffic monitoring log based on the fused reconstruction vector.

[0029] In step S400 of the embodiment of the present application, the fused reconstruction vector is the result obtained by the abnormal traffic monitoring device in step S300 after one or more stages of joint reconstruction operations. This vector combines the monitoring embedded mapping vector of the target traffic monitoring log and the relevant information of the reference embedded mapping vector obtained from the normal control traffic monitoring log. For example, in a network traffic monitoring scenario, the fused reconstruction vector may contain the source IP address, destination IP address, port number, protocol type of the network connection, and the relationship between them, etc., after reconstruction.

[0030] The abnormal traffic monitoring device extracts log reconstruction representation information from this fused reconstruction vector. A possible technical means is through feature decomposition and selection. Assume that the fused reconstruction vector is , we can perform principal component analysis (PCA) on this vector. The basic principle of PCA is to project the original vector into a new coordinate system through linear transformation, so that the direction with the largest variance of the data in the new coordinate system is taken as the principal component.

[0031] Specifically, calculate the covariance matrix of the fusion reconstruction vector V , where is the mean vector of vector V. Then perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues and the corresponding eigenvectors . Sort the eigenvectors according to the magnitudes of the eigenvalues, and select the eigenvectors corresponding to the first k (k < n) eigenvalues to form a matrix . Then obtain a new vector through the formula , and this is the vector after feature selection, which contains the main information in the fusion reconstruction vector and can be used as part of the log reconstruction characterization information.

[0032] For another example, when processing firewall traffic logs, there may be multiple different value ranges for the source IP addresses in the fusion reconstruction vector. Through this method of eigenvalue decomposition and selection, those value ranges that are significant in differentiating normal and abnormal traffic can be highlighted as log reconstruction characterization information. If a certain source IP address segment rarely appears in normal traffic but is found to have a relatively high weight after analysis in the fusion reconstruction vector, then this source IP address segment may become an important element in the log reconstruction characterization information for subsequent determination of whether the traffic is abnormal.

[0033] Step S500: Obtain the monitoring information of the target traffic monitoring log based on the reconstruction error between the log reconstruction characterization information and the log initial characterization information.

[0034] The reconstruction error refers to the degree of difference between the log reconstruction characterization information and the log initial characterization information. There are various ways for the abnormal traffic monitoring device to calculate this difference. For example, the Euclidean distance formula can be used to calculate the distance between vectors. Assume that the log reconstruction characterization information is vector , and the log initial characterization information is vector , then the Euclidean distance , and this distance d can be used as a measure of the reconstruction error. Of course, other measurement methods can also be used, such as the Manhattan distance , etc.

[0035] The abnormal traffic monitoring device obtains the monitoring information of the target traffic monitoring log based on this reconstruction error. Taking the network firewall traffic monitoring as an example, if the reconstruction error is large, it indicates that the features of the target traffic monitoring log after processing deviate significantly from the original features. Under normal circumstances, if the target traffic is normal traffic, then after the processing of steps S100 to S400, the log reconstruction characterization information should be relatively similar to the log initial characterization information, and the reconstruction error is relatively small.

[0036] Assume that in an enterprise network environment, normal internal network traffic shows a specific source IP address range, destination IP address range, and port usage mode in the log initial characterization information. After processing, if these features in the log reconstruction characterization information have changed significantly compared with the original log initial characterization information, such as a large number of addresses from external unknown areas suddenly appear in the source IP address, the destination IP address points to sensitive servers within the enterprise, and the port usage mode does not conform to normal business logic, then the calculated reconstruction error will be larger. This larger reconstruction error indicates that there may be abnormal traffic in the target traffic monitoring log, and the abnormal traffic monitoring device can determine that the target traffic is abnormal traffic and mark it as an abnormal traffic monitoring result. On the contrary, if the reconstruction error is small and within an acceptable range, it means that the characteristics of the target traffic have not changed much before and after processing, which means that the target traffic is more likely to be normal traffic. For example, during normal office hours, the file sharing traffic between employees within the enterprise, after the entire abnormal traffic monitoring process, the difference between its log reconstruction characterization information and the log initial characterization information is small, and the abnormal traffic monitoring device will determine that this traffic is normal traffic. The abnormal traffic monitoring device can effectively monitor the target traffic monitoring log for abnormal traffic by accurately calculating and analyzing the reconstruction error, thereby ensuring the security and stability of the network. This monitoring method based on reconstruction error can comprehensively consider the relationship between the original characteristics of the target traffic and the processed characteristics, avoiding the limitations of single factor judgment and improving the accuracy and reliability of abnormal traffic monitoring.

[0037] In one embodiment, in step S300, extracting a saliency vector for the target monitoring embedding mapping vector and the target reference embedding mapping vector input at this stage to obtain the reference embedding mapping vector obtained at this stage may include:

[0038] Step S310: obtaining a first reference vector and a first content vector based on the target monitoring embedding mapping vector input at this stage, and obtaining a first request vector based on the target reference embedding mapping vector input at this stage;

[0039] Step S320: performing cross-saliency vector extraction on the first reference vector, the first content vector and the first request vector to obtain a reference embedding mapping vector obtained at this stage.

[0040] In step S300 of the embodiment of the present application, the target monitoring embedding mapping vector is a vector obtained after embedding and mapping the initial characterization information of the target flow monitoring log in step S100, which contains the rich feature information of the target flow monitoring log after conversion. The abnormal flow monitoring device further decomposes this vector to obtain a first reference vector (understood as a key) and a first content vector (understood as a value).

[0041] For example, suppose the target monitoring embedding mapping vector is The abnormal flow monitoring device can adopt a simple segmentation strategy to divide the vector X into two parts according to certain rules, one of which is used as the first reference vector , and the other part as the first content vector This division can be based on a certain logical relationship between the vector elements. For example, the first half of the elements are related to the source-related information of the network connection (such as the characteristic value of the source IP address after conversion, etc.) as the reference vector, and the second half of the elements are related to the destination-related information (such as the characteristic value of the destination IP address after conversion, etc.) as the content vector.

[0042] Meanwhile, for the target reference embedding mapping vector, it is the vector obtained after extracting the reference log representation information from the normal control flow monitoring log and embedding mapping in step S200. The abnormal flow monitoring device also processes this vector to obtain the first request vector. For example, the target reference embedding mapping vector is The abnormal traffic monitoring device can convert it into a first request vector by a mapping function f The mapping function f can be a linear transformation function, such as (in is the transformation coefficient), which is used to convert the information in the target reference embedding mapping vector into the first request vector according to specific requirements, so as to perform the subsequent cross-saliency vector extraction operation.

[0043] In step S320, in the scenario of network traffic monitoring, cross-significance vector extraction is an operation aimed at mining important correlation features between vectors.

[0044] Taking firewall traffic monitoring as an example, the first baseline vector may contain typical features related to the source IP address of the network connection under normal circumstances (the converted feature values), the first content vector contains features related to the destination IP address, and the first request vector contains feature request information extracted from the normal control traffic log for comparison with the current target traffic.

[0045] When the abnormal flow monitoring device extracts the cross-significance vector, it needs to comprehensively consider the relationship between the three vectors. A possible technical means is to measure the correlation between them based on the inner product operation of the vector. Assume that the first reference vector , the first content vector , the first request vector .

[0046] First, calculate the correlation score between the vectors. For the first reference vector K and the first request vector Q, their inner product can be calculated , this inner product value reflects a certain degree of association between them. Similarly, for the first content vector V and the first request vector Q, the inner product is calculated Then, the cross-significance vector is determined based on these correlation scores. For example, a threshold T can be set if , then select the elements with the strongest correlation with Q from the first content vector V and combine them with the elements with the strongest correlation with Q in the first reference vector K to obtain a new vector This new vector Z is the result of cross-saliency vector extraction, which will serve as an important component of the reference embedding mapping vector obtained in this stage.

[0047] Another possible technical approach is based on a matrix decomposition method. The first reference vector K, the first content vector V and the first request vector Q are combined into a matrix , and then perform singular value decomposition on the matrix M , where U, and V are the left singular vector matrix, singular value diagonal matrix, and right singular vector matrix, respectively. Select the part related to the significance from the decomposed matrix, for example, select the left singular vector and the right singular vector corresponding to the part with larger singular value and combine them to obtain the cross-significance vector. This matrix decomposition-based method can more comprehensively consider the complex relationship between the three vectors, dig out the significant features hidden in the vectors, and thus obtain a more accurate reference embedding mapping vector obtained at this stage, providing a more reliable basis for the subsequent joint reconstruction operation and the final abnormal traffic monitoring.

[0048] In one implementation, step S320, performing cross-saliency vector extraction on the first reference vector, the first content vector, and the first request vector to obtain the reference embedding mapping vector obtained at this stage, may include:

[0049] Step S321: extracting a cross-saliency vector based on the first reference vector, the first content vector and the first request vector according to a weighted focusing strategy to obtain a first cross-saliency vector;

[0050] Step S322: performing difference merging on the first cross-saliency vector and the first request vector to obtain a merged vector;

[0051] Step S323: performing one or more vector transformations on the merged vector to obtain a transformation vector;

[0052] Step S324: perform difference merging on the transformed vector and the merged vector to obtain the reference embedded mapping vector obtained at this stage.

[0053] In step S321, the first reference vector, the first content vector, and the first request vector respectively contain different feature information of the target traffic monitoring log and the normal control traffic monitoring log after processing. The weighted focus strategy (here taking the attention mechanism as an example) aims to assign different weights according to the importance of these vector elements, thereby highlighting key features for cross-significance vector extraction.

[0054] For example, assuming that the first reference vector , the first content vector , the first request vector When the abnormal traffic monitoring device adopts the attention mechanism, the attention score is first calculated. For each element in the reference vector K , calculate its relevance score with the request vector Q (here Representation vector and q), this score reflects the inner product of the element The importance of the request vector Q. Similarly, for each element in the content vector V , calculate its relevance score with the request vector Q .

[0055] Then, the reference vector K and the content vector V are weighted and summed according to these scores. For the reference vector K, the weighted vector is obtained ; For the content vector V, get the weighted vector Finally, the weighted reference vector K' and content vector V' are combined to obtain the first cross-significance vector The first cross-significant vector obtained by the attention mechanism can highlight the important features related to the request vector. For example, in firewall traffic monitoring, if the request vector Q contains the characteristic information of the port used under normal traffic, then the first cross-significant vector obtained by the attention mechanism The characteristic elements related to port usage in the reference vector K and the content vector V will be highlighted, thereby better reflecting the relationship between the target traffic and the normal traffic in this regard.

[0056] In step S322, the first cross-significance vector Contains important information related to target traffic and normal traffic characteristics after being processed by the weighted focus strategy. The first request vector It is a vector extracted from the normal control traffic monitoring log for comparison and extraction of significant features.

[0057] The abnormal traffic monitoring device performs a difference merging operation, for example, assuming that the first cross-significance vector , then merge the vectors The significance of this difference merging operation is to integrate the difference information between the first cross-significance vector and the first request vector. In network traffic monitoring, if the first cross-significance vector represents an adjusted relationship between the target traffic and the normal traffic in some features, and the first request vector Q is the original reference feature of the normal traffic, so the merged vector This can reflect the residual between this adjusted relationship and the original reference feature. For example, in Web server traffic monitoring, if the first cross-significance vector It reflects the characteristics of different page access frequencies in the current target traffic after adjustment relative to normal traffic. The first request vector Q is the page access frequency characteristic under normal traffic, so the merged vector The elements in represent the difference between the adjusted access frequency and the normal access frequency. These difference information helps to further explore the abnormal features in the target traffic.

[0058] In step S323, the vectors are merged The difference information between the first cross-significance vector and the first request vector is included, but further processing may be required to better mine its features. The abnormal traffic monitoring device performs a vector transformation operation on the merged vector.

[0059] One possible way to transform a vector is through linear transformation. For example, through a transformation matrix (where l is the merge vector , p is the dimension of the transformed vector), for the merged vector Transform and get the transformation vector This transformation matrix A can be pre-trained based on a large amount of normal traffic and abnormal traffic sample data. It can map the difference information in the merged vector to a space that is more conducive to analysis. For example, in the traffic monitoring of the intrusion detection system (IDS), if the merged vector The difference information in reflects the difference between the source IP address, destination IP address and port number of the network connection and the normal traffic. Then, after transformation by the appropriate transformation matrix A, the transformation vector These difference information can be expressed in a way that better reflects its relevance to abnormal traffic. In addition to linear transformation, nonlinear transformation can also be used, such as using an activation function (such as the ReLU function f(x)=max(0,x)) to perform nonlinear processing on the elements in the merged vector, and then performing a linear transformation, which can increase the flexibility of vector transformation and adaptability to complex traffic characteristics.

[0060] In step S324, the transformation vector is the vector obtained after vector transformation of the merged vector. It contains the features after processing the difference information of the merged vector. Contains the original difference information between the first cross-saliency vector and the first request vector.

[0061] The abnormal flow monitoring device performs the difference merging operation again. For example, assuming that the transformation vector , merge vector (Here we assume that p=l. If the dimensions are different, some preprocessing may be required to make the dimensions match.) The reference embedding mapping vector obtained at this stage This second difference merging operation is to further integrate and adjust the results obtained from the previous operation. In network traffic monitoring, this reference embedding mapping vector It will serve as an important input for subsequent operations (such as further interactive operations with the target monitoring embedded mapping vector, etc.). It integrates the feature information mined from a series of operations starting from the first reference vector, the first content vector, and the first request vector. This information can more accurately reflect the relationship between the target traffic and the normal traffic, thereby helping to better judge whether there is an abnormality in the target traffic during the joint reconstruction operation. For example, in firewall traffic monitoring, if the previous operation has mined the difference between the target traffic and the normal traffic in terms of source IP address, destination IP address, and port number, then this reference embedded mapping vector These difference characteristics will be expressed in a more comprehensive way, providing a more accurate reference for subsequent operations and improving the accuracy of abnormal traffic monitoring.

[0062] In one embodiment, in step S300, extracting a saliency vector from the reference embedding mapping vector and the target monitoring embedding mapping vector obtained in this stage to obtain the monitoring embedding mapping vector obtained in this stage may include:

[0063] Step S330: obtaining a second reference vector and a second content vector based on the reference embedding mapping vector obtained in this stage, and obtaining a second request vector based on the target monitoring embedding mapping vector input in this stage;

[0064] Step S340: performing cross-saliency vector extraction on the second reference vector, the second content vector and the second request vector to obtain the monitoring embedding mapping vector obtained in this stage.

[0065] In step S330, the abnormal traffic monitoring device obtains a second reference vector and a second content vector based on the reference embedded mapping vector obtained in this stage, and obtains a second request vector based on the target monitoring embedded mapping vector. The reference embedded mapping vector is the result of a series of operations, and it contains processed feature information related to normal traffic. For example, in network traffic monitoring, if the reference embedded mapping vector represents the relevant features of firewall traffic under normal circumstances, then the abnormal traffic monitoring device will decompose this vector. Assume that the reference embedded mapping vector is The abnormal traffic monitoring device can divide the element into the second reference vector according to certain rules, such as based on the traffic feature type represented by the element (such as the source IP address related feature part and the destination IP address related feature part). and the second content vector .

[0066] Meanwhile, the target monitoring embedding mapping vector is a vector obtained and converted from the target traffic monitoring log in step S100, which contains the characteristic information of the target traffic. The abnormal flow monitoring device also converts it into a second request vector through a mapping function g. The mapping function g can be a linear transformation function based on predefined rules, for example (in is a transformation coefficient), which is used to convert the information embedded in the target monitoring mapping vector into a second request vector according to specific requirements for subsequent operations.

[0067] Next, in step S340, the abnormal traffic monitoring device performs cross-significance vector extraction on the second reference vector, the second content vector, and the second request vector to obtain the monitoring embedding mapping vector obtained in this stage. In the network traffic scenario, this operation is to mine the significant feature differences between the target traffic and the normal traffic. For example, assuming that the second reference vector , the second content vector , the second request vector .

[0068] One implementation method is to extract cross-significant vectors by calculating the similarity between vectors. The abnormal traffic monitoring device can calculate the cosine similarity between the second reference vector K' and the second request vector Q' , and the cosine similarity between the second content vector V' and the second request vector Q' .

[0069] Based on these similarity values, the abnormal traffic monitoring device can determine which elements have more significant characteristics in the target traffic relative to the normal traffic. The value of is high, indicating that some elements in the second reference vector K' have a strong correlation with the second request vector Q' (i.e., some characteristics of the target flow). The abnormal flow monitoring device can filter out these elements with strong correlation according to the preset threshold T'. Assume , then select the part of elements with the strongest correlation with Q' from the second reference vector K', and select the part of elements with the strongest correlation with Q' from the second content vector V' (based on ), combine these elements to get a new vector , this new vector Z'' is the monitoring embedding mapping vector obtained at this stage by extracting the cross-saliency vector.

[0070] This cross-significance vector extraction process helps the abnormal traffic monitoring device to more accurately analyze the difference characteristics between the target traffic and the normal traffic in the joint reconstruction operation, so as to better judge whether the target traffic has abnormal conditions. For example, in the Web server traffic monitoring, if the access frequency of certain pages under normal traffic has a specific pattern (reflected in the reference embedded mapping vector), and the target traffic (the second request vector obtained by converting the target monitoring embedded mapping vector) has a significant difference in the access frequency of these pages in certain time periods from the normal pattern, then the monitoring embedded mapping vector Z'' obtained by the above steps can highlight these difference characteristics, providing an important basis for the subsequent judgment of whether the target traffic is abnormal based on the fusion reconstruction vector.

[0071] In one embodiment, step S340, performing cross-significance vector extraction on the second reference vector, the second content vector, and the second request vector to obtain the monitoring embedding mapping vector obtained in this stage, may include:

[0072] Step S341: extracting a cross-saliency vector based on the second reference vector, the second content vector and the second request vector according to a weighted focusing strategy to obtain a second cross-saliency vector;

[0073] Step S342: performing difference merging on the second cross-saliency vector and the second request vector to obtain the monitoring embedding mapping vector obtained in this stage.

[0074] In step S341, the abnormal traffic monitoring device extracts a cross-significance vector based on the second reference vector, the second content vector and the second request vector according to a weighted focusing strategy to obtain a second cross-significance vector.

[0075] Taking the firewall log in network traffic monitoring as an example, the second reference vector may contain information such as source IP address features related to normal traffic extracted from the reference embedding mapping vector. Assume that the second reference vector is ; The second content vector may contain normal traffic characteristics related to the destination IP address, set The second request vector is obtained based on the target monitoring embedded mapping vector, which may reflect certain characteristic requirements of the target traffic, such as the requirement to compare the source IP address and destination IP address of the target traffic with the normal traffic, and is set to .

[0076] The weighted focusing strategy can adopt the attention mechanism. The abnormal traffic monitoring device first calculates the attention weight. For each element in the second reference vector K , calculate its attention weight with the second request vector Q ,here Representation vector and the inner product of Q. Similarly, for each element in the second content vector V , calculate its attention weight with the second request vector Q .

[0077] Then, the second reference vector K and the second content vector V are weighted and summed according to these attention weights. The weighted second reference vector , the weighted second content vector . Finally, the weighted second reference vector K' and the second content vector V' are combined to obtain the second cross-saliency vector Z=(K',V'). For example, in an intrusion detection system (IDS), if the second request vector Q represents the need to compare the target traffic and normal traffic in terms of IP addresses under a certain attack mode, then the second cross-saliency vector Z obtained through this attention mechanism will highlight the difference between the source IP address and destination IP address features related to this attack mode in normal traffic and target traffic.

[0078] In step S342, the abnormal traffic monitoring device performs difference merging on the second cross-significance vector and the second request vector to obtain the monitoring embedding mapping vector obtained in this stage.

[0079] Assume that the second cross-significance vector , the second request vector (Here we assume that l=q. If the dimensions are different, preprocessing may be required to make them match.) The abnormal flow monitoring device calculates the difference The monitoring embedding mapping vector obtained in this stage is obtained.

[0080] Taking Web server traffic monitoring as an example, if the second cross-significance vector Z contains the characteristic difference in page access frequency between normal traffic and target traffic after weighting by the attention mechanism, and the second request vector Q is the original page access frequency-related characteristic vector of the target traffic, then the monitoring embedding mapping vector M obtained by difference merging can more intuitively reflect the quantitative result of this difference. This difference merging operation helps to further explore the special characteristics of the target traffic relative to the normal traffic, and provides a more accurate basis for the subsequent judgment of whether the target traffic is abnormal based on the fusion reconstruction vector. For example, if the access frequency of a certain type of page fluctuates within a certain range under normal traffic, and the access frequency of the page in the target traffic shows a large deviation from the normal fluctuation range in the monitoring embedding mapping vector through the above operation, this may imply that the target traffic is abnormal.

[0081] In one embodiment, the one or more stages include multiple stages; in step S300, performing a joint reconstruction operation of one or more stages according to the monitoring embedded mapping vector and the reference embedded mapping vector to obtain a fused reconstruction vector obtained by the joint reconstruction operation of the one or more stages may include:

[0082] Step S301: in a joint reconstruction operation of the first stage among the multiple stages, extracting a saliency vector from the monitoring embedding mapping vector and the reference embedding mapping vector inputted in the first stage to obtain a reference embedding mapping vector obtained in the first stage, and extracting a saliency vector from the reference embedding mapping vector and the monitoring embedding mapping vector obtained in the first stage to obtain a monitoring embedding mapping vector obtained in the first stage;

[0083] Step S302: In each stage starting from the second stage in the multiple stages, a joint reconstruction operation of this stage is performed based on the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the previous stage to obtain the reference embedded mapping vector and the monitoring embedded mapping vector obtained in this stage, until the joint reconstruction operation of the last stage in the multiple stages outputs the fused reconstruction vectors of multiple stages.

[0084] In step S301, for the joint reconstruction operation of the first stage among multiple stages, the abnormal traffic monitoring device performs significance vector extraction on the monitoring embedded mapping vector and the reference embedded mapping vector input in the first stage to obtain the reference embedded mapping vector obtained in the first stage, and performs significance vector extraction on the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the first stage to obtain the monitoring embedded mapping vector obtained in the first stage.

[0085] In the network traffic monitoring scenario, the monitoring embedding mapping vector is a vector obtained by embedding and mapping the target traffic monitoring log, which contains various characteristic information of the target traffic. For example, taking firewall traffic monitoring as an example, this vector may contain converted characteristic values ​​such as source IP address, destination IP address, port number, protocol type, etc. Suppose the monitoring embedding mapping vector is The reference embedding mapping vector is extracted from the normal control traffic monitoring log and embedded in the vector, which contains the characteristic information of normal traffic and is set as .

[0086] For the extraction of significant vectors, the abnormal traffic monitoring device can use a technical method based on distance measurement. For example, the Euclidean distance between vectors is calculated to determine the significance of the elements. For each element in X , calculate its difference with each element in Y The Euclidean distance Then, based on these distance values, the significance is determined, and the distance is selected to meet certain conditions (such as the distance is less than a certain threshold ) to obtain the reference embedding mapping vector obtained in the first stage .

[0087] Next, for the reference embedding mapping vector obtained in the first stage and monitor the embedding mapping vector X, and perform similar saliency vector extraction operations again. For example, calculate The degree of correlation between the elements in and the elements in X is set as the correlation function f(a,b) (for example, it can be a vector inner product, etc.). Each element in and every element in X ,calculate Based on these correlation values, select ) to obtain the monitoring embedding mapping vector obtained in the first stage Then, in step S302, in each stage starting from the second stage in the multiple stages, the abnormal traffic monitoring device performs a joint reconstruction operation of the stage based on the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the previous stage, and obtains the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the stage, until the joint reconstruction operation of the last stage in the multiple stages outputs the fused reconstruction vectors of the multiple stages.

[0088] Taking the second stage as an example, assuming that the reference embedding mapping vector obtained in the previous stage (first stage) is , the monitoring embedding mapping vector is The abnormal flow monitoring device can use the weighted summation technique to perform joint reconstruction operations.

[0089] Let the weight vector be (r is determined according to the specific vector dimension), for the reference embedding mapping vector Weighted , for monitoring embedding mapping vector Weighted The weighted vectors are then combined, for example, by addition to obtain a new intermediate vector (s is the dimension of the combined vector).

[0090] Then, the intermediate vector Z is processed to obtain the reference embedding mapping vector and the monitoring embedding mapping vector obtained in this stage. For example, Z can be mapped into two vectors by a mapping function g, which is set as and ,here and It is a mapping function defined according to specific traffic characteristic requirements.

[0091] For the subsequent stages, the abnormal flow monitoring device repeats this process and continuously performs joint reconstruction operations based on the results of the previous stage. and As input, similar weighting, combination, mapping and other operations are performed to obtain and , and so on. Until the last stage, after a series of joint reconstruction operations, a fused reconstruction vector is obtained. This fused reconstruction vector integrates the characteristic information of the target traffic and normal traffic at each stage, providing an important data basis for the subsequent acquisition of log reconstruction representation information of the target traffic monitoring log based on this vector and the final abnormal traffic monitoring.

[0092] Throughout the entire process, operations at each stage are adjusted and optimized based on the previous stage, gradually integrating the characteristics of target traffic and normal traffic to more accurately determine whether there are abnormalities in the target traffic.

[0093] In one implementation, step S400, obtaining log reconstruction representation information of a target traffic monitoring log based on the fused reconstruction vector, may include:

[0094] Step S410: performing cross-significance vector extraction on the monitoring embedding mapping vector and the reference embedding mapping vector included in the fused reconstruction vector to obtain log reconstruction representation information of the target traffic monitoring log.

[0095] The fused reconstruction vector is the result of the joint reconstruction operation of one or more stages of step S300, which integrates the information of the monitoring embedded mapping vector of the target traffic monitoring log and the reference embedded mapping vector (obtained from the normal control traffic monitoring log). For example, in the network traffic monitoring scenario, the monitoring embedded mapping vector part of the fused reconstruction vector may contain information of the source IP address, destination IP address, port number and other features of the target traffic after a series of processing, and the reference embedded mapping vector part contains the relevant information of these features under normal traffic.

[0096] Cross-significance vector extraction is an operation to mine key features in a vector. When the abnormal traffic monitoring device performs this operation, it first needs to determine how to measure the significance of the vector elements. Taking network firewall traffic as an example, assuming that the monitoring embedding mapping vector in the fusion reconstruction vector is , the reference embedding mapping vector is .

[0097] One possible technical approach is to determine the significance based on the variance of the vector elements. Calculate the variance of each element of the monitoring embedding mapping vector X (in is the mean of X), and similarly calculates the variance of each element of the reference embedding mapping vector Y Elements with larger variance are usually considered more significant because they represent features in the vector that have larger deviations from the mean.

[0098] The abnormal flow monitoring device selects elements with higher significance from the monitoring embedding mapping vector and the reference embedding mapping vector according to these significance determination methods. For example, a threshold T is set, and for the elements in X that satisfy Elements and Y satisfies Elements to combine.

[0099] Another technical approach can be based on the correlation between vectors. Calculate the correlation coefficient between the monitoring embedding mapping vector X and the reference embedding mapping vector Y, such as the Pearson correlation coefficient Then, the vector elements are weighted according to the correlation coefficient, and the elements with higher correlation are combined to form a new vector. This new vector is the result obtained by cross-significance vector extraction, which is part of the log reconstruction representation information of the target traffic monitoring log. This log reconstruction representation information can reflect the key characteristics of the target traffic after fusion and analysis with the normal traffic characteristics.

[0100] In one implementation, step S500, obtaining monitoring information of the target traffic monitoring log according to a reconstruction error between the log reconstructed representation information and the log initial representation information, may include:

[0101] Step S510: determining a reconstruction error between the log reconstruction representation information and the log initial representation information;

[0102] Step S520: converting the reconstruction error to obtain a conversion parameter;

[0103] Step S530: Determine the monitoring information of the target traffic monitoring log according to the conversion parameter.

[0104] In step S510, the log reconstruction characterization information is obtained in step S400 based on the fused reconstruction vector. It is a feature representation of the target traffic monitoring log after a series of complex processing (including embedding mapping, joint reconstruction operations, etc.). The log initial characterization information is the characterization information directly extracted from the target traffic monitoring log and obtained after preliminary processing in step S100, reflecting the characteristics of the target traffic monitoring log in its original state. The reconstruction error is the degree of difference between the two characterization information.

[0105] In the network traffic monitoring scenario, this difference can reflect the changes in the target traffic before and after processing. For example, in firewall traffic monitoring, the initial log characterization information may contain original features such as the source IP address range, the destination IP address range, and the port usage frequency. The log reconstruction characterization information is a feature representation obtained after comprehensively considering the target traffic and comparing it with the normal traffic. If the target traffic is normal traffic, the reconstruction error should be small in theory; if there is abnormal traffic, some features of the target traffic may change significantly, thereby increasing the reconstruction error.

[0106] There are many ways to calculate the reconstruction error. A common method is to calculate the Euclidean distance between vectors. Assume that the log reconstruction representation information is a vector , the log initial representation information is a vector , the Euclidean distance formula is , this distance d can be used as a measure of reconstruction error.

[0107] Another method is to calculate the Manhattan distance, the formula is In some cases, Manhattan distance can better reflect the absolute differences in feature values ​​in various dimensions. For example, in Web server traffic monitoring, if the elements in the log reconstruction representation information and the log initial representation information represent the number of visits to different pages, Manhattan distance can intuitively reflect the total amount of changes in the number of visits on each page.

[0108] In step S520, the directly obtained reconstruction error may not be convenient for directly using to judge whether the target flow is abnormal, and needs to be converted to obtain a more appropriate conversion parameter. This conversion parameter can be better combined with the subsequent judgment criteria to more accurately obtain the monitoring information of the target flow monitoring log.

[0109] For example, in some network environments, different types of traffic (such as internal office traffic, external access traffic, etc.) may have different tolerances for reconstruction errors. Through conversion, the representation of reconstruction errors can be adjusted according to the characteristics of different traffic types to make it more in line with actual anomaly judgment needs.

[0110] One possible conversion method is to use normalization. If the reconstruction error e (assuming it is calculated using the Euclidean distance) has a wide range of values, it can be converted to the [0,1] interval by normalization. For example, is the minimum reconstruction error obtained in normal traffic samples, is the maximum reconstruction error, then the normalized conversion parameter .

[0111] Another transformation method can be based on a logarithmic function. If the reconstruction error e>0, use Logarithmic transformation can compress large reconstruction error values, so that reconstruction errors of different sizes can more evenly affect the abnormal judgment results in subsequent judgments. For example, in the traffic monitoring of the intrusion detection system (IDS), if the original reconstruction error is large in the face of large-scale traffic, the conversion parameter after logarithmic transformation can more reasonably reflect the degree of abnormality.

[0112] In step S530, the conversion parameter combines the relationship between the log reconstruction characterization information and the log initial characterization information, and this parameter can be used to determine whether the target flow is abnormal. Generally speaking, if the conversion parameter exceeds a certain threshold, the target flow can be determined to be abnormal flow; if it is within the threshold range, it is considered to be normal flow.

[0113] In network traffic monitoring, this threshold can be determined based on a large amount of normal traffic and known abnormal traffic sample data analysis. For example, in an enterprise network environment, by analyzing historical traffic data (including traffic during different periods such as normal working days and holidays), a suitable threshold can be determined to distinguish normal traffic from abnormal traffic.

[0114] Assume that the conversion parameter obtained by step S520 is p, and the threshold is set to T. If p>T, the abnormal traffic monitoring device determines that the target traffic monitoring log is abnormal traffic, and the monitoring information is marked as abnormal. For example, in firewall traffic monitoring, if p=0.8 and the threshold T=0.6, it means that the reconstruction error of the target traffic is large, and there may be abnormal situations, such as a large number of accesses from external unknown areas to the source IP address or the destination IP address points to sensitive servers within the enterprise and the behavior does not conform to normal business logic. If p≤T, the target traffic monitoring log is determined to be normal traffic, and the monitoring information is marked as normal. For example, in Web server traffic monitoring, if p=0.3 and the threshold T=0.5, it means that the characteristics of the target traffic have not changed much before and after processing, and are consistent with the normal traffic pattern, such as normal user access traffic, such as normal web browsing by employees or normal data interaction of internal systems.

[0115] In one embodiment, the method may further include:

[0116] Step S501: extracting flow characterization information of one or more stages from the reconstruction error to obtain flow anomaly characterization information;

[0117] Step S530, determining the monitoring information of the target traffic monitoring log according to the conversion parameter, may include:

[0118] Step S531: Integrate the conversion parameter with the flow anomaly characterization information to obtain anomaly integration information;

[0119] Step S532: Determine the monitoring information of the target traffic monitoring log based on the abnormal integration information.

[0120] In step S501, the reconstruction error is the difference between the log reconstruction characterization information determined in step S510 and the log initial characterization information. This difference is not just a simple value, it contains the deviation of the target flow and the normal flow in various characteristics. The flow characterization information extraction is to deeply explore the flow characteristics represented by this deviation, so as to obtain the flow abnormality characterization information.

[0121] When the abnormal flow monitoring device extracts flow characterization information, a multi-stage processing method can be adopted. In the first stage, it is assumed that the reconstruction error is represented by vector Indicates (here It can be different aspects of the reconstruction error quantified in some way, such as Indicates the reconstruction error part related to the source IP address).

[0122] The abnormal traffic monitoring device can first decompose this vector. For example, according to the classification of network traffic characteristics, E is decomposed into sub-vectors related to different traffic characteristics. Assume that the sub-vector related to the source IP address is , the subvector related to the destination IP address is etc. This can be achieved by setting the index range of the elements, for example Contains the elements of E with indices 1 to m.

[0123] In the subsequent stage, the abnormal traffic monitoring device can further analyze these sub-vectors. For example, the abnormal flow monitoring device can calculate the statistical characteristics of the elements in the sub-vector, such as the mean ,variance Etc. These statistical features can help the abnormal traffic monitoring device to better understand the distribution of reconstruction errors related to source IP addresses.

[0124] Another technical method is based on cluster analysis. The abnormal flow monitoring device can cluster the elements in E according to their similarity. For example, using the K-Means clustering algorithm, the elements in E are divided into k clusters. For each cluster, the abnormal traffic monitoring device can calculate the cluster center , and statistics such as the distance from the element to the cluster center. Through cluster analysis, the abnormal traffic monitoring device can find the clustering pattern of reconstruction errors in different feature dimensions, thereby mining the traffic anomaly characterization information. For example, if the elements in a cluster are mainly related to the reconstruction error related to the port number, and the reconstruction error represented by the cluster center is large, this may imply that there are abnormal traffic characteristics in terms of the port number.

[0125] After one or more stages of such processing, the abnormal traffic monitoring device finally obtains traffic anomaly characterization information. This information is, for example, a description of various traffic feature anomalies, such as the abnormal change pattern of the source IP address, the abnormal access tendency of the destination IP address, the abnormal usage frequency of the port number, etc.

[0126] In step S531, the conversion parameter is obtained after converting the reconstruction error in step S520. It is a value that integrates the relationship between the log reconstruction characterization information and the log initial characterization information. The traffic anomaly characterization information is a description of the abnormal situation of the traffic characteristics obtained by in-depth mining of the reconstruction error in step S501. Integrating the two can obtain more comprehensive and accurate anomaly integration information, which can comprehensively consider the overall abnormality of the traffic and the specific abnormal characteristics.

[0127] In network traffic monitoring, such as web server traffic monitoring, conversion parameters may indicate the degree of deviation of the overall traffic from normal traffic, while traffic anomaly characterization information may indicate which pages have abnormal access frequencies, which user sources have abnormal access patterns, etc. Integrating the two can more accurately determine traffic anomalies.

[0128] A simple integration method is weighted summation. Assuming the conversion parameter is p, the traffic anomaly characterization information is represented by the vector Indicates (here It can indicate the abnormal degree of different traffic characteristics, such as The abnormal traffic monitoring device can set the weight vector , and then calculate the abnormal integration information (Here w is the weight of the conversion parameter p). The weight vector W can be set according to the importance of different traffic features in abnormality judgment. For example, in firewall traffic monitoring, if the abnormality of the source IP address is very critical to determine whether the traffic is attack traffic, then The value of can be set relatively large.

[0129] Another way of integration can be based on logical rule fusion. For example, if the conversion parameter p exceeds a certain threshold , and the abnormality of the source IP address in the traffic anomaly characterization information Exceeding a certain threshold , then the abnormal integration information I can be expressed as a specific abnormal level, such as "high-risk abnormality". If p is within a certain range, and If the traffic is also within the corresponding acceptable range, I indicates "normal" or "low risk abnormality", etc. This fusion based on logical rules can be flexibly defined according to actual network security requirements and traffic characteristics.

[0130] In step S532, the abnormal integration information integrates the overall abnormality degree of the traffic and the specific abnormal characteristic manifestations. Based on this information, the monitoring information of the target traffic monitoring log can be accurately determined, that is, whether the target traffic is normal traffic or abnormal traffic, and if it is abnormal traffic, its abnormality degree and type, etc.

[0131] In network traffic monitoring scenarios, such as traffic monitoring by an intrusion detection system (IDS), anomaly integration information can reflect whether the target traffic has signs of threat attacks, such as whether there are a large number of accesses from threat IP addresses (through traffic anomaly characterization information) and the degree of deviation of the overall traffic from normal traffic (through conversion parameters).

[0132] Assume that the abnormal integrated information I is a value obtained by weighted summation. The abnormal traffic monitoring device can set a threshold value T. If I>T, the target traffic monitoring log is determined to be abnormal traffic, and the monitoring information is marked as abnormal. For example, in the traffic monitoring of an enterprise network, if I=0.8 and the threshold value T=0.6, this indicates that there is an abnormality in the target traffic. The abnormal traffic monitoring device can further analyze the specific type of abnormality based on the traffic anomaly characterization information. For example, if the traffic anomaly characterization information shows that the source IP address has a large number of accesses from external unknown areas and the port number is used abnormally, then it can be determined as abnormal traffic that may be an attempt at external intrusion.

[0133] If I≤T, the target traffic monitoring log is determined to be normal traffic, and the monitoring information is marked as normal. For example, if I=0.3 and threshold T=0.5, it means that the characteristics of the target traffic are consistent with normal traffic in both overall and specific aspects, such as normal internal office traffic or legitimate external access traffic.

[0134] In one implementation, step S100, embedding and mapping the initial characterization information of the target traffic monitoring log to obtain the monitoring embedding mapping vector of the target traffic monitoring log, may include:

[0135] Step S110: extracting the initial characterization information of the target traffic monitoring log, and embedding the saliency characterization information of the initial characterization information of the log in one or more stages to obtain a monitoring embedding mapping vector of the target traffic monitoring log;

[0136] Step S200, determining reference log representation information related to the target traffic monitoring log, and embedding and mapping the reference log representation information to obtain a reference embedding mapping vector, may include:

[0137] Step S210: determining reference log representation information related to the target traffic monitoring log in the reference representation information set, wherein the reference representation information set includes one or more candidate reference log representation information extracted according to the reference traffic monitoring log;

[0138] Step S220: embedding the reference log representation information with saliency representation information at one or more stages to obtain a reference embedding mapping vector.

[0139] In the embodiment of the present application, the target traffic monitoring log contains rich network traffic information, and the first thing the abnormal traffic monitoring device needs to do is to extract the initial characterization information of the log. Taking the firewall log as an example, the log records information such as the source IP address, destination IP address, port number, protocol type, and connection timestamp. The abnormal traffic monitoring device will extract this information as the initial characterization information of the log. The abnormal traffic monitoring device embeds the extracted initial characterization information of the log into one or more stages of significant characterization information. The purpose of embedding significant characterization information is to convert the initial characterization information of the log into a form that is more suitable for subsequent processing and to highlight the important feature information. In the first stage, it is assumed that the initial characterization information of the log is a vector , the abnormal traffic monitoring device can adopt a weight-based embedding method. For example, for the elements related to the source IP address If the source IP address has a high importance in judging abnormal traffic according to historical data or predefined rules, then Give a higher weight The weighted elements become For the entire vector X, after such a weighted operation, we get .

[0140] In the subsequent stage, the abnormal traffic monitoring device can perform more complex embedding operations. For example, the autoencoder structure in the neural network is adopted. The autoencoder consists of an encoder and a decoder. The weighted vector X' is input into the encoder, and the encoder maps it to a low-dimensional latent space representation Z=f(X'), where f is the mapping function of the encoder. This low-dimensional latent space representation Z can capture the main features of the initial representation information of the log and remove some redundant information. Then, Z is input into the decoder to obtain the reconstructed vector X''=g(Z), where g is the mapping function of the decoder. By comparing X' and X'', the parameters of the autoencoder can be adjusted to minimize the reconstruction error. After one or more such stages, the monitoring embedding mapping vector of the target traffic monitoring log is finally obtained. This monitoring embedding mapping vector retains the key features of the initial representation information of the log while existing in a more compact and more conducive form for subsequent analysis.

[0141] In step S210, a reference characterization information set is constructed by the abnormal traffic monitoring device based on the control traffic monitoring log. The control traffic monitoring log is a log collected under normal network operation conditions, such as a firewall log or a Web server log in a time period when there is no attack and the network is running stably. The reference characterization information set contains one or more candidate reference log characterization information extracted based on these control traffic monitoring logs. The process of the abnormal traffic monitoring device determining the reference log characterization information related to the target traffic monitoring log is as follows. First, determine the reference characterization information set. Taking the control traffic log of the firewall as an example, the abnormal traffic monitoring device extracts characterization information from the firewall log of multiple normal operation periods to form a reference characterization information set. Assume that m groups of characterization information are extracted from logs of different time periods, and each group of characterization information contains relevant information such as the source IP address range, the destination IP address range, and the port number usage frequency. These groups of characterization information constitute a reference characterization information set. Then, the initial characterization information of the target traffic monitoring log is compared with each candidate reference log characterization information in the reference characterization information set. For example, the source IP address in the target traffic monitoring log is 192.168.1.100, the destination IP address is 10.0.0.5, the port number is 80, the protocol type is TCP, and the connection time is 2023-08-01 10:00:00. The abnormal traffic monitoring device compares this information with the corresponding information in each group of candidate reference log representation information in the reference representation information set. For the source IP address, it can be determined by judging whether the network segment to which it belongs is the same or whether it is within the normal access network segment range; for the destination IP address, it can be determined whether it is a normal internal server address or a trusted external service address; for the port number, it can be compared whether its usage frequency is within the normal range, etc. Finally, the candidate reference log representation information corresponding to the representation information comparison result indicating adaptation is used as the reference log representation information related to the target traffic monitoring log. For example, if there is a set of candidate reference log representation information in the reference representation information set, whose source IP address range includes 192.168.1.100, the destination IP address 10.0.0.5 is a normal internal server address, the usage frequency of port number 80 is within the normal range, and the protocol type and connection time also conform to the normal pattern, then this set of candidate reference log representation information is determined as the reference log representation information related to the target traffic monitoring log.

[0142] Step S220 is similar to the embedding of significant characterization information into the initial characterization information of the target flow monitoring log in step S110. The abnormal flow monitoring device also processes the reference log characterization information in one or more stages. In the first stage, assuming that the reference log characterization information is a vector , a weight-based embedding method can also be used. For example, if the destination IP address shows a high correlation in the comparison with the target traffic monitoring log, then the element y_i related to the destination IP address is given a higher weight , the weighted elements become , the entire vector is weighted to obtain .

[0143] In the subsequent stages, a structure similar to that of an autoencoder can also be used. The weighted vector Y' is input into the encoder to obtain a low-dimensional latent space representation Z'=f(Y'), and then the reconstructed vector Y''=g(Z') is obtained through the decoder. The reconstruction error is minimized by adjusting the parameters. After one or more such stages, a reference embedding mapping vector is finally obtained. This reference embedding mapping vector will serve as an important input for subsequent joint reconstruction operations, etc., and will be used together with the monitoring embedding mapping vector of the target traffic monitoring log for abnormal traffic monitoring.

[0144] In one implementation, step S210, determining reference log representation information related to the target traffic monitoring log in the reference representation information set, may include:

[0145] Step S211: determining a reference representation information set;

[0146] Step S212: performing a characterization information comparison on the initial log characterization information and each candidate reference log characterization information in the reference characterization information set to obtain a characterization information comparison result;

[0147] Step S213: taking the candidate reference log representation information corresponding to the representation information comparison result indicating adaptation as the reference log representation information related to the target traffic monitoring log.

[0148] In step S211, the reference characterization information set is constructed by the abnormal traffic monitoring device, which is an information set extracted from the normal control traffic monitoring log. The control traffic monitoring log is the traffic information recorded when the network is in normal operation, and this information covers the characteristics of various normal traffic. For example, in the monitoring of the network firewall, the control traffic monitoring log records the source IP address, destination IP address, port number, protocol type, and connection time of the network connection during the normal time period.

[0149] The control flow monitoring log of the Web server contains the access request information of users under normal circumstances, such as the access frequency of different pages, the access mode of different types of users (such as internal employees, external users), etc. The abnormal flow monitoring device analyzes and extracts these normal control flow monitoring logs to form a reference representation information set.

[0150] The abnormal traffic monitoring device can use technical means of data extraction and organization. Assume that the control traffic monitoring log of the firewall is stored in the form of a database table, and each record represents a network connection. The abnormal traffic monitoring device can write a query statement to extract relevant field information, such as "SELECT source_ip, destination_ip, port_number, protocol_type, connection_time FROM firewall_log WHERE status = 'normal'" (here it is assumed that "status = 'normal'" represents a normal connection record).

[0151] For the control flow monitoring log of the Web server, if it exists in the form of a log file, the abnormal flow monitoring device can use text processing technology to parse the log file. For example, for the common Apache Web server log format, its log record may contain a format such as "IP address - [access time] "Request method Request resource HTTP version" Status code Number of bytes sent". The abnormal flow monitoring device can write a parsing program to extract relevant information such as IP address, request resource, etc., and organize these information extracted from different normal control flow monitoring logs to form a reference characterization information set. Each element in this set is representative feature information extracted from normal traffic. In step S212, the initial log characterization information is extracted from the target flow monitoring log, which reflects the initial characteristics of the target flow. The candidate reference log characterization information in the reference characterization information set is extracted from the normal control flow monitoring log and represents the characteristics of normal traffic. By comparing the two, the abnormal flow monitoring device can find out the similarities and differences between the target flow and the normal flow, so as to determine which candidate reference log characterization information is more relevant to the target flow monitoring log.

[0152] In the example of firewall traffic monitoring, the source IP address in the initial characterization information of the log is, for example, the content that needs to be compared. Assuming that the source IP address in the target traffic monitoring log is 192.168.1.100, in the candidate reference log characterization information in the reference characterization information set, different normal traffic may have different source IP address ranges. The abnormal traffic monitoring device needs to determine whether 192.168.1.100 is within these normal source IP address ranges. The same comparison process is used for information such as the destination IP address, port number, and protocol type.

[0153] In terms of Web server traffic monitoring, if the initial characterization information of the log shows that the access frequency of a certain page suddenly increases, the abnormal traffic monitoring device will compare this access frequency with the access frequency of the page under normal circumstances in the reference characterization information set to see whether it exceeds the normal fluctuation range.

[0154] For information of numerical type, such as port number, the abnormal traffic monitoring device can adopt the range comparison method. Assuming that the normal range of port number in a candidate reference log representation information in the reference representation information set is 80-100, if the port number in the target traffic monitoring log is 85, then this port number is within the normal range. For the comparison of IP addresses, the IP address segment matching method can be adopted. If the target IP address belongs to a specific subnet segment, and this subnet segment is defined as a normal access source or destination address in the candidate reference log representation information in the reference representation information set, then it is considered to be a match in terms of IP address.

[0155] For non-numeric information, such as protocol type, an exact matching method can be used. If the protocol type in the target traffic monitoring log is "TCP", and the protocol type in the candidate reference log representation information in the reference representation information set is also "TCP", then there is a match in terms of protocol type. By comparing each feature in this way, the abnormal traffic monitoring device can obtain the representation information comparison result, which is a comprehensive description of the matching status of the target traffic monitoring log and each candidate reference log representation information in each feature.

[0156] In step S213, the characterization information comparison result indicates the degree of match between the target flow monitoring log and the candidate reference log characterization information. The abnormal flow monitoring device selects the candidate reference log characterization information corresponding to the characterization information comparison result with a high degree of match, that is, indicating adaptation, as the reference log characterization information related to the target flow monitoring log.

[0157] In the firewall traffic monitoring scenario, if a candidate reference log representation information matches the target traffic monitoring log in terms of source IP address, destination IP address, port number, protocol type, and connection time, or is within the normal fluctuation range, then this candidate reference log representation information is considered to be related to the target traffic monitoring log. For example, the source IP address in the target traffic monitoring log is an address in the enterprise's internal network, the destination IP address is the address of the enterprise's internal server, the port number is a normal business port, the protocol type is "TCP", and the connection time is also within normal office hours. If there is a candidate reference log representation information in the reference representation information set that matches the target traffic monitoring log in these aspects, then this candidate reference log representation information will be selected as the reference log representation information related to the target traffic monitoring log.

[0158] In this way, the abnormal traffic monitoring device can find the reference log representation information that is most relevant to the target traffic monitoring log. This information will play an important role in subsequent operations (such as embedded mapping, etc.), and help to perform abnormal traffic monitoring more accurately.

[0159] In the embodiment of the present application, the above method is performed by using a flow monitoring neural network. The embodiment of the present application also provides a training process of the flow monitoring neural network, which may specifically include the following steps:

[0160] Step S10: obtaining a regular training log, a control training log related to the regular training log, and an attack training log;

[0161] Step S20: extracting representation information from the regular training log, the control training log, and the attack training log respectively, to obtain regular log initial representation information of the regular training log, control log initial representation information of the control training log, and attack log initial representation information of the attack training log;

[0162] Step S30: using the traffic monitoring neural network to be calibrated, embedding and mapping the initial characterization information of the regular log, the initial characterization information of the control log, and the initial characterization information of the attack log, respectively, to obtain a regular embedding mapping vector of the regular training log, a reference embedding mapping vector of the control training log, and an attack embedding mapping vector of the attack training log;

[0163] Step S40: using the traffic monitoring neural network to be calibrated, performing one or more stages of joint reconstruction operations according to the conventional embedding mapping vector and the reference embedding mapping vector, obtaining a first fused reconstruction vector obtained by the one or more stages of joint reconstruction operations, and obtaining conventional log reconstruction representation information of the conventional training log based on the first fused reconstruction vector;

[0164] Step S50: using the traffic monitoring neural network to be calibrated, performing one or more stages of joint reconstruction operations according to the attack embedding mapping vector and the reference embedding mapping vector, obtaining a second fused reconstruction vector obtained by the one or more stages of joint reconstruction operations, and obtaining attack log reconstruction representation information of the attack training log based on the second fused reconstruction vector;

[0165] Step S60: iteratively adjust the traffic monitoring neural network to be adjusted according to the conventional log reconstruction characterization information, the attack log reconstruction characterization information and the conventional log initial characterization information, and stop when the network converges to obtain the traffic monitoring neural network.

[0166] Regular training logs are traffic monitoring logs collected by abnormal traffic monitoring devices from normal network operation environments. These logs contain various traffic information of the network under normal conditions, such as data transmission in the network when there is no attack or abnormal interference.

[0167] The control training log is related to the regular training log. It is also a log collected during normal network operation, but focuses on certain specific normal traffic patterns or network environments. For example, in an enterprise network, the control training log may specifically record the network traffic during specific business operations (such as the daily financial data backup period). The difference between it and the regular training log is that the control training log focuses more on the detailed information of certain specific normal network behavior patterns, while the regular training log is a broader traffic record under normal network operation. For example, the control training log may record in detail the traffic connection information from the specific IP address segment of the finance department to the specific port of the data storage server (such as a dedicated port for backup data transmission) during the financial data backup period, including more detailed information such as the frequency of the connection and the amount of data transmitted. This information helps to more accurately identify the characteristic patterns of normal traffic when training neural networks.

[0168] Attack training logs are network traffic logs that contain known attack behaviors. These logs are collected by simulating attacks or from actual network environments that have been attacked. For example, in a network security test environment, some network attack tools can be used to simulate common attack types, such as port scanning attacks and SQL injection attacks. Taking port scanning attacks as an example, the attack training log will record a series of port scanning behaviors from the attacker's IP address, including the target IP address of the scan (for example, the IP address of the server within the enterprise), the port range of the scan (such as ports from 1 to 1024), the frequency of the scan (for example, very fast continuous scans), and the protocol type (usually TCP, because port scanning is mainly based on TCP connections for detection). For SQL injection attacks, the attack training log will record request information containing threatening SQL statements. For example, if an attacker attempts to inject threatening SQL code through input points such as the login page or search box of a Web application, the log will record information such as the source IP address of the request, the target resource of the request (such as the URL of the login page), the request content containing the threatening SQL statement, and the time when the attack occurred.

[0169] In step S20, characterization information extraction is the process of extracting information that can represent traffic characteristics from the original training log. Its purpose is to convert complex log data into a form that can be processed by the neural network, that is, to simplify the data representation by extracting key features, while retaining information that is important for traffic classification (normal or abnormal). For firewall-related training logs (including regular, control and attack logs), the abnormal traffic monitoring device can adopt a rule-based feature extraction method. For example, for the source IP address feature, the network segment information to which it belongs can be extracted. If the source IP address is 192.168.1.100, its network segment can be extracted as 192.168.1.0 / 24. For port numbers, they can be classified into common ports (such as 80, 443, etc.) and uncommon ports, which can be achieved by setting a common port list and comparing them. For the training log of the Web server, the abnormal traffic monitoring device can extract features such as the type of page requested by the user (such as static page, dynamic page), the frequency of the request, and the geographical location information of the source IP address of the request (if there is a relevant IP-geolocation mapping database). For example, if the URL requested by the user contains the ".html" extension, it can be determined as a static page request; if it contains a specific dynamic script extension (such as ".php" or ".asp"), it is a dynamic page request. Through such a characterization information extraction process, the abnormal traffic monitoring device obtains regular log initial characterization information for regular training logs, obtains control log initial characterization information for control training logs, and obtains attack log initial characterization information for attack training logs.

[0170] In step S30, embedding mapping is the process of converting the high-dimensional, discrete log initial representation information into a low-dimensional, continuous vector representation. This operation helps the neural network to better process and analyze data because low-dimensional vectors are more efficient in calculation and can capture the potential semantic relationships in the data.

[0171] Assume that the initial representation information of the regular log is a vector containing multiple features , the traffic monitoring neural network can adopt a linear embedding mapping method. For example, through a weight matrix (where m < n), the vector X is mapped to a conventional embedding mapping vector Y = XW. A similar method is used to embed and map the initial characterization information of the control log and the initial characterization information of the attack log. For example, for the initial characterization information vector of the control log , through the weight matrix (q<p) obtains the reference embedding mapping vector B=AV for the training log; for the initial representation information vector of the attack log , through the weight matrix (s<r) The attack embedding mapping vector D=CU of the attack training log is obtained. This embedding mapping operation converts the initial representation information of different types of logs into a vector form suitable for subsequent processing by the neural network.

[0172] In step S40, the joint reconstruction operation aims to fuse and adjust the regular embedding mapping vector (representing the features of the regular training log) and the reference embedding mapping vector (representing the features of the specific normal traffic pattern) to generate a first fused reconstruction vector. This process allows the neural network to learn the feature representation of the regular training log after comparison and fusion with the specific normal traffic pattern, so as to better identify the feature pattern of normal traffic and build a model of normal traffic.

[0173] In each stage of the joint reconstruction operation, the abnormal flow monitoring device can use a method combining weighted summation and nonlinear transformation. Assume that in the first stage, the conventional embedding mapping vector is , the reference embedding mapping vector is First, the abnormal flow monitoring device sets the weight vector and , calculate the intermediate vector Then, Perform nonlinear transformations, such as through an activation function f (such as the ReLU function )get In the subsequent stage, the result vector obtained in the previous stage (such as ) performs similar weighted summation and nonlinear transformation operations with the new weight vector, and after multiple stages, the first fused reconstruction vector is obtained .

[0174] When obtaining the conventional log reconstruction representation information of the conventional training log based on the first fused reconstruction vector F_1, the abnormal traffic monitoring device can use a reverse mapping or decoding operation. For example, if a specific encoding function E is used in the embedding mapping process to encode the conventional log initial representation information X into a conventional embedding mapping vector Y (ie, Y=E(X)), then a decoding function D can be constructed so that the conventional log reconstruction representation information The regular log reconstruction representation information X' is the feature representation of the regular training log after a joint reconstruction operation with a specific normal traffic pattern.

[0175] Step S50 is similar to step S40, both of which perform a joint reconstruction operation, but here it is for attack training logs. The similarities are in the basic process and method of the operation, such as the combination of weighted summation of multiple stages and nonlinear transformation. The difference lies in the nature of the input vector. The attack embedding mapping vector here contains the characteristics of the attack traffic, while the conventional training log contains the characteristics of normal traffic. This comparison operation helps the neural network learn the difference characteristics between attack traffic and normal traffic when compared with a specific normal traffic pattern.

[0176] Assume that the attack embedding mapping vector is , the reference embedding mapping vector is In the first stage, the weight vector is set and , calculate the intermediate vector , and then perform a nonlinear transformation (f is the activation function). After multiple stages, the second fusion reconstruction vector is obtained .

[0177] Reconstructed vector based on the second fusion The process of obtaining the attack log reconstructed representation information is similar to that of the conventional training log. Assume that there is an encoding function E' that encodes the initial attack log representation information C into the attack embedding mapping vector D (i.e., D = E'(C)), and construct a decoding function D' so that the attack log reconstructed representation information C' = D'( ). The attack log reconstruction representation information C' is the feature representation of the attack training log after the joint reconstruction operation with the specific normal traffic pattern.

[0178] In step S60, iterative adjustment is to adjust the parameters of the traffic monitoring neural network by comparing the conventional log reconstruction characterization information, the attack log reconstruction characterization information and the conventional log initial characterization information, so that the network can better classify normal traffic and attack traffic. Its purpose is to minimize the reconstruction error of normal traffic (i.e., the difference between the conventional log reconstruction characterization information and the conventional log initial characterization information), while maximizing the recognition ability of attack traffic (i.e., the difference between the attack log reconstruction characterization information and the conventional log initial characterization information should be able to accurately reflect the attack characteristics).

[0179] First, the abnormal traffic monitoring device calculates the training reconstruction error between the conventional log reconstructed representation information X' and the conventional log initial representation information X. A common calculation method is to use the Euclidean distance formula , where n is the dimension of the information vector. Based on this training reconstruction error, the abnormal traffic monitoring device can define a training reconstruction cost value ,For example .

[0180] Next, the abnormal traffic monitoring device calculates the training recovery error between the attack log reconstruction characterization information C' and the regular log initial characterization information X. The Euclidean distance formula can also be used , and define a training recovery cost ,For example .

[0181] Then, the abnormal flow monitoring device iteratively adjusts the flow monitoring neural network according to the training reconstruction cost value L_r and the training restoration cost value L_a. For example, the gradient descent algorithm is used according to the loss function (in is a weight parameter that weighs the normal traffic reconstruction error and attack traffic identification ability) to update the weight of the neural network. In each iteration, the gradient of the loss function L with respect to the neural network weight is calculated , then according to the formula (in is the current weight, is the updated weight, is the learning rate) to update the weights.

[0182] This iterative process will continue until the network converges. The criterion for network convergence can be that the value of the loss function L changes less than a preset threshold in multiple consecutive iterations. , for example, if (where L_{t} ​​is the loss function value of the current iteration, is the loss function value of the previous iteration), the network is considered to have converged. When the network converges, the abnormal traffic monitoring device obtains a trained traffic monitoring neural network, which can be used to monitor abnormal traffic on actual network traffic.

[0183] In one embodiment, step S60, iteratively adjusting the traffic monitoring neural network to be adjusted according to the conventional log reconstruction characterization information, the attack log reconstruction characterization information and the conventional log initial characterization information, stopping when the network converges, and obtaining the traffic monitoring neural network may include:

[0184] Step S61: determining a training reconstruction cost value based on a training reconstruction error between the conventional log reconstruction representation information and the conventional log initial representation information;

[0185] Step S62: Determine a training restoration cost based on a training restoration error between the attack log reconstructed representation information and the regular log initial representation information;

[0186] Step S63: iteratively adjust the traffic monitoring neural network to be adjusted according to the training reconstruction cost value and the training restoration cost value, and stop when the network converges to obtain the traffic monitoring neural network.

[0187] In step S61, the conventional log reconstructed representation information is obtained in step S40 based on the first fused reconstruction vector, which is the feature representation of the conventional training log after a series of processing (including embedding mapping and joint reconstruction operations, etc.). The conventional log initial representation information is the representation information directly extracted from the conventional training log in step S20, reflecting the original features of the conventional training log.

[0188] The abnormal traffic monitoring device calculates the training reconstruction error between the two to measure the degree of deviation between the regular log features after the neural network processing and the original regular log features. For example, assuming that the regular log reconstruction representation information is vector , the initial characterization information of the conventional log is a vector A common method for calculating the training reconstruction error is the Euclidean distance, which is given by This distance value It represents a comprehensive measure of the difference in each feature dimension. For example, in network traffic monitoring, if a dimension in the initial representation information of the regular log represents a certain feature of the source IP address, and the corresponding dimension in the reconstructed representation information of the regular log represents the source IP address related features after the neural network processing, then the Euclidean distance reflects the overall degree of change of this feature before and after processing.

[0189] The training reconstruction cost is defined based on the training reconstruction error, which is an indicator used to measure the degree of loss of the neural network when processing regular training logs. The abnormal traffic monitoring device can use a simple square form to define the training reconstruction cost. For example, let the training reconstruction cost be ,but This definition makes the training reconstruction cost increase rapidly as the training reconstruction error increases, emphasizing the importance of accurately reconstructing the regular training log features. For example, if =0.5, then = 0.25. During the iterative tuning of the neural network, the goal is to minimize this training reconstruction cost to ensure that the neural network can accurately reconstruct the features of the regular training log.

[0190] In step S62, the attack log reconstructed characterization information is obtained in step S50 based on the second fused reconstruction vector, which is the feature representation of the attack training log after the joint reconstruction operation with the reference embedding mapping vector. The abnormal traffic monitoring device calculates the training restoration error between the attack log reconstructed characterization information and the regular log initial characterization information to measure the difference between the characteristics of the attack traffic after the neural network processing and the regular traffic characteristics. It is also assumed that the attack log reconstructed characterization information is vector , the initial characterization information of the conventional log is a vector The Euclidean distance formula is used to calculate the training restoration error. For example, in traffic monitoring of an intrusion detection system (IDS), if the initial representation information of the regular log represents the usage pattern of port numbers under normal traffic, and the reconstructed representation information of the attack log represents the port number-related features under processed attack traffic, then the training restoration error reflects the degree of deviation of the attack traffic from the normal traffic in terms of port number usage.

[0191] The training recovery cost is defined based on the training recovery error and is used to measure the neural network's ability to distinguish between attack traffic and regular traffic. The abnormal traffic monitoring device can define the training recovery cost ,For example Similar to the training reconstruction cost, this definition makes the training restoration cost increase as the training restoration error increases. For example, if =1.0, then =1.0. In the iterative adjustment process of the neural network, it is hoped that the training recovery cost value can be as large as possible within a reasonable range to improve the neural network's ability to identify attack traffic.

[0192] In step S63, the abnormal traffic monitoring device iteratively adjusts the traffic monitoring neural network according to the training reconstruction cost value L_r and the training restoration cost value L_a. The purpose is to minimize the training reconstruction cost value (to ensure accurate reconstruction of regular traffic) while maximizing the training restoration cost value (to improve the ability to identify attack traffic). During the iterative adjustment process, it is necessary to adjust the parameters of the neural network, such as the weights and biases of the neural network. This process is similar to finding an optimal solution in a multidimensional space, so that both the training reconstruction cost value and the training restoration cost value reach an ideal state. One implementation method is the gradient descent algorithm. First, define a loss function L, which is a combination of the training reconstruction cost value and the training restoration cost value. For example, suppose ,in is a weight parameter that weighs the normal traffic reconstruction error and attack traffic identification capability. =0.5, indicating that equal emphasis is placed on normal traffic reconstruction errors and attack traffic identification capabilities.

[0193] In each iteration, the abnormal flow monitoring device calculates the gradient of the loss function L with respect to the neural network weights For example, for a weight w in a neural network, we can calculate it by using the chain rule or other derivation rules. , and get the gradient vector Then, according to the formula Update the weights, where is the current weight, is the updated weight, is the learning rate. For example, if =0.3, =0.1, =0.01, then .

[0194] Network convergence means that as the number of iterations increases, the value of the loss function L no longer changes significantly. To determine whether the network has converged. For example, if ( is the loss function value of the current iteration, is the loss function value of the previous iteration), the network is considered to have converged. For example, =0.001. If the change of L is less than 0.001 in multiple consecutive iterations, the iterative adjustment process is stopped. At this time, the flow monitoring neural network obtained by the abnormal flow monitoring device is a trained network that can be used for actual abnormal flow monitoring.

[0195] In one embodiment, the method may further include:

[0196] Step S601: determining a first characterization information error between the reconstructed characterization information of a regular log and the initial characterization information of a regular log, and determining a second characterization information error between the reconstructed characterization information of an attack log and the initial characterization information of an attack log;

[0197] Step S602: using the flow monitoring neural network to be calibrated, extracting flow characterization information of one or more stages from the first characterization information error and the second characterization information error, respectively, to obtain first flow abnormality characterization information and second flow abnormality characterization information;

[0198] Step S603: determining an attack traffic monitoring cost value based on the attack traffic field priori mark corresponding to the attack training log, the first traffic anomaly characterization information, and the second traffic anomaly characterization information;

[0199] Step S63, iteratively adjusting the traffic monitoring neural network to be adjusted according to the training reconstruction cost value and the training restoration cost value, stopping when the network converges, and obtaining the traffic monitoring neural network, may include:

[0200] Step S631: Integrate the training reconstruction cost value, the training recovery cost value and the attack traffic monitoring cost value to obtain a target training cost value;

[0201] Step S632: iteratively adjust the flow monitoring neural network to be adjusted according to the target training cost value, stop when the network converges, and obtain the flow monitoring neural network.

[0202] In step S601, the task of the abnormal traffic monitoring device is to determine two characterization information errors. Specifically, the abnormal traffic monitoring device needs to calculate the first characterization information error between the conventional log reconstructed characterization information and the conventional log initial characterization information, and the second characterization information error between the attack log reconstructed characterization information and the attack log initial characterization information.

[0203] From a technical point of view, assuming that the initial representation information of the regular log is a vector , conventional log reconstruction represents information as a vector , the first characterization information error can be obtained by calculating a certain norm of the difference between the corresponding elements of vector A and vector B, such as the Euclidean norm, the formula is Similarly, for the attack log, assume that the initial characterization information of the attack log is a vector , the attack log reconstruction representation information is vector , the second characterization information error The purpose of this step is to quantify the difference between the reconstructed representation information and the initial representation information, and to provide basic data for subsequent steps.

[0204] In step S602, the abnormal flow monitoring device uses the flow monitoring neural network to be calibrated to perform one or more stages of flow characterization information extraction on the first characterization information error and the second characterization information error, respectively, to obtain the first flow abnormality characterization information and the second flow abnormality characterization information.

[0205] For example, when extracting traffic characterization information, the neural network may perform weighted summation, nonlinear transformation, and other operations on each element in the first characterization information error vector. Assume that the first characterization information error vector is , a neural network may have a set of weights , after weighted summation, we get Then, it is processed through a nonlinear activation function f(x) (such as the ReLU function f(x)=max(0,x)) to obtain an element in the first traffic anomaly characterization information. The processing of the entire vector may involve multiple such operation combinations, and for example, the processing result of a multi-layer neural network structure. The processing of the error of the second characterization information is similar, thereby obtaining the second traffic anomaly characterization information. This process aims to mine higher-level feature information related to traffic anomalies from the error information.

[0206] In step S603, the abnormal traffic monitoring device determines the attack traffic monitoring cost value based on the attack traffic field priori mark corresponding to the attack training log, the first traffic abnormality characterization information and the second traffic abnormality characterization information.

[0207] Assume that the attack traffic field is a priori marked as a binary vector , where f_i indicates whether a certain attack traffic field exists (for example, 1 indicates existence, and 0 indicates non-existence). The first traffic anomaly characterization information is a vector , the second traffic anomaly characterization information is vector The abnormal traffic monitoring device can determine the attack traffic monitoring cost by a weighted combination method. For example, first define a weight vector Used for attack traffic field priori marking, weight vector For the first traffic anomaly characterization information, the weight vector Used for the second traffic anomaly characterization information. Then the attack traffic monitoring cost value Here, the priori markers and traffic anomaly characterization information are weightedly combined to comprehensively reflect the monitoring value of attack traffic, which helps to more accurately evaluate the performance of neural networks in attack traffic monitoring.

[0208] In step S631, the abnormal traffic monitoring device integrates the training reconstruction cost value, the training recovery cost value and the attack traffic monitoring cost value to obtain the target training cost value.

[0209] Assume that the training reconstruction cost is , the training recovery cost is , the attack traffic monitoring cost is , the abnormal flow monitoring device can use a simple addition operation to integrate these three values, namely the target training cost value Of course, other more complex integration methods can also be used, such as weighted summation according to different weights. The purpose of this integration is to combine the values ​​reflecting different performance aspects of the neural network (conventional log reconstruction, attack log recovery, and attack traffic monitoring) to form a unified evaluation index for subsequent iterative adjustment of the neural network.

[0210] In step S632, the abnormal flow monitoring device iteratively adjusts the flow monitoring neural network to be adjusted according to the target training cost value, and stops when the network converges, thereby obtaining the flow monitoring neural network.

[0211] During the calibration process, the abnormal flow monitoring device uses an optimization algorithm (such as a gradient descent algorithm) to adjust the parameters of the neural network according to the target training cost. For example, for a certain weight parameter w in the neural network, its update formula can be ,in is the learning rate. In each iteration, the abnormal traffic monitoring device calculates the target training cost value based on the current input data (regular training log, control training log and attack training log), and then uses the above formula to update the parameters of the neural network. As the iteration proceeds, the target training cost value will gradually decrease. When the target training cost value no longer decreases significantly or reaches the preset convergence condition (such as the target training cost value is less than a certain threshold or the difference between multiple consecutive iterations is less than a certain threshold), the abnormal traffic monitoring device considers that the network has converged and stops the iterative adjustment process. The neural network obtained at this time is a trained traffic monitoring neural network, which can be used for subsequent abnormal traffic monitoring tasks.

[0212] The present application also provides an abnormal flow monitoring device, such as Figure 2 As shown, the abnormal flow monitoring device 100 includes: a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, such as through a bus 102. Optionally, the abnormal flow monitoring device 100 may also include a transceiver 104. It should be noted that in actual applications, the transceiver 104 is not limited to one, and the structure of the abnormal flow monitoring device 100 does not constitute a limitation on the embodiments of the present application.

[0213] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program runs on a processor, the processor can execute corresponding contents in the aforementioned method embodiment.

Claims

1. A method for monitoring abnormal flow, characterized in that: The method comprises: Acquire a target traffic monitoring log, and embed and map the log initial representation information of the target traffic monitoring log to obtain a monitoring embedding mapping vector of the target traffic monitoring log; Determine reference log representation information related to the target traffic monitoring log, and embed and map the reference log representation information to obtain a reference embedding mapping vector; the reference log representation information is extracted from a normal control traffic monitoring log corresponding to the target traffic monitoring log; One or more stages of joint reconstruction operations are performed according to the monitoring embedding mapping vector and the reference embedding mapping vector to obtain a fused reconstruction vector obtained by the joint reconstruction operations of the one or more stages, and in each stage of the joint reconstruction operation, saliency vector extraction is performed on the target monitoring embedding mapping vector and the target reference embedding mapping vector input at that stage to obtain the reference embedding mapping vector obtained at that stage, and saliency vector extraction is performed on the reference embedding mapping vector obtained at that stage and the target monitoring embedding mapping vector to obtain the monitoring embedding mapping vector obtained at that stage; Obtaining log reconstruction representation information of the target traffic monitoring log based on the fused reconstruction vector; Obtaining monitoring information of the target traffic monitoring log according to a reconstruction error between the log reconstruction representation information and the log initial representation information; The step of extracting a saliency vector from the target monitoring embedding mapping vector and the target reference embedding mapping vector input at this stage to obtain the reference embedding mapping vector obtained at this stage includes: Obtaining a first reference vector and a first content vector based on the target monitoring embedding mapping vector input at this stage, and obtaining a first request vector based on the target reference embedding mapping vector input at this stage; Performing cross-saliency vector extraction on the first reference vector, the first content vector, and the first request vector to obtain a reference embedding mapping vector obtained at this stage; The extracting of the saliency vector of the reference embedding mapping vector obtained in this stage and the target monitoring embedding mapping vector to obtain the monitoring embedding mapping vector obtained in this stage includes: Obtain a second reference vector and a second content vector based on the reference embedding mapping vector obtained in this stage, and obtain a second request vector based on the target monitoring embedding mapping vector input in this stage; Performing cross-significance vector extraction on the second reference vector, the second content vector, and the second request vector to obtain a monitoring embedding mapping vector obtained in this stage; The step of performing cross-significance vector extraction on the first reference vector, the first content vector, and the first request vector to obtain a reference embedding mapping vector obtained at this stage includes: Based on the first reference vector, the first content vector and the first request vector, extracting a cross-saliency vector according to a weighted focusing strategy to obtain a first cross-saliency vector; Performing difference merging on the first cross-significance vector and the first request vector to obtain a merged vector; Perform one or more vector transformations on the combined vector to obtain a transformation vector; Perform difference merging on the transformation vector and the merged vector to obtain a reference embedding mapping vector obtained in this stage; The step of extracting the cross-significance vector from the second reference vector, the second content vector, and the second request vector to obtain the monitoring embedding mapping vector obtained at this stage includes: Based on the second reference vector, the second content vector and the second request vector, extracting a cross-saliency vector according to a weighted focusing strategy to obtain a second cross-saliency vector; The second cross-saliency vector and the second request vector are difference-merged to obtain the monitoring embedding mapping vector obtained in this stage.

2. The method according to claim 1, characterized in that The one or more stages include multiple stages; performing a joint reconstruction operation of one or more stages according to the monitoring embedded mapping vector and the reference embedded mapping vector to obtain a fused reconstruction vector obtained by the joint reconstruction operation of the one or more stages includes: In a joint reconstruction operation of a first stage among the multiple stages, saliency vector extraction is performed on the monitoring embedding mapping vector and the reference embedding mapping vector inputted in the first stage to obtain a reference embedding mapping vector obtained in the first stage, and saliency vector extraction is performed on the reference embedding mapping vector and the monitoring embedding mapping vector obtained in the first stage to obtain a monitoring embedding mapping vector obtained in the first stage; In each stage starting from the second stage in the multiple stages, a joint reconstruction operation of the stage is performed according to the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the previous stage to obtain the reference embedded mapping vector and the monitoring embedded mapping vector obtained in the stage, until the joint reconstruction operation of the last stage in the multiple stages outputs a fused reconstruction vector of the multiple stages; The obtaining log reconstruction representation information of the target traffic monitoring log based on the fused reconstruction vector includes: Performing cross-significance vector extraction on the monitoring embedding mapping vector and the reference embedding mapping vector included in the fused reconstruction vector to obtain log reconstruction representation information of the target traffic monitoring log; The obtaining the monitoring information of the target traffic monitoring log according to the reconstruction error between the log reconstruction representation information and the log initial representation information includes: Determining a reconstruction error between the log reconstructed representation information and the log initial representation information; Converting the reconstruction error to obtain a conversion parameter; The monitoring information of the target traffic monitoring log is determined based on the conversion parameter.

3. The method according to claim 2, characterized in that The method further comprises: Extracting flow characterization information of one or more stages from the reconstruction error to obtain flow anomaly characterization information; The step of determining the monitoring information of the target flow monitoring log according to the conversion parameter includes: Integrate the conversion parameter with the flow anomaly characterization information to obtain anomaly integration information; The monitoring information of the target traffic monitoring log is determined based on the abnormal integration information.

4. The method according to claim 1, characterized in that The embedding and mapping of the log initial representation information of the target traffic monitoring log to obtain the monitoring embedding mapping vector of the target traffic monitoring log includes: Extracting the initial characterization information of the target traffic monitoring log, and embedding the initial characterization information of the log with significant characterization information of one or more stages to obtain a monitoring embedding mapping vector of the target traffic monitoring log; The determining of reference log representation information related to the target traffic monitoring log, and embedding and mapping the reference log representation information to obtain a reference embedding mapping vector, includes: Determining reference log representation information related to the target traffic monitoring log in a reference representation information set, wherein the reference representation information set includes one or more candidate reference log representation information extracted based on the reference traffic monitoring log; One or more stages of saliency representation information embedding are performed on the reference log representation information to obtain a reference embedding mapping vector.

5. The method according to claim 4, characterized in that The determining of the reference log representation information related to the target traffic monitoring log in the reference representation information set includes: determining a reference representation information set; Comparing the initial log representation information with each candidate reference log representation information in the reference representation information set to obtain a representation information comparison result; The candidate reference log representation information corresponding to the representation information comparison result indicating adaptation is used as the reference log representation information related to the target traffic monitoring log.

6. The method according to any one of claims 1 to 5, characterized in that: The method is performed using a flow monitoring neural network, which is trained through the following process: Obtaining a regular training log, a control training log related to the regular training log, and an attack training log; Extracting representation information from the regular training log, the control training log, and the attack training log respectively to obtain regular log initial representation information of the regular training log, control log initial representation information of the control training log, and attack log initial representation information of the attack training log; Using the traffic monitoring neural network to be calibrated, embedding and mapping the initial characterization information of the conventional log, the initial characterization information of the control log, and the initial characterization information of the attack log, respectively, to obtain a conventional embedding mapping vector of the conventional training log, a reference embedding mapping vector of the control training log, and an attack embedding mapping vector of the attack training log; Using the traffic monitoring neural network to be calibrated, performing one or more stages of joint reconstruction operations according to the conventional embedding mapping vector and the reference embedding mapping vector, obtaining a first fused reconstruction vector obtained by the one or more stages of joint reconstruction operations, and obtaining conventional log reconstruction representation information of the conventional training log based on the first fused reconstruction vector; Using the traffic monitoring neural network to be calibrated, performing one or more stages of joint reconstruction operations according to the attack embedding mapping vector and the reference embedding mapping vector, obtaining a second fused reconstruction vector obtained by the one or more stages of joint reconstruction operations, and obtaining attack log reconstruction representation information of the attack training log based on the second fused reconstruction vector; The flow monitoring neural network to be calibrated is iteratively calibrated according to the conventional log reconstruction characterization information, the attack log reconstruction characterization information and the conventional log initial characterization information, and the flow monitoring neural network is obtained by stopping when the network converges.

7. The method according to claim 6, characterized in that The iterative adjustment of the traffic monitoring neural network to be adjusted according to the conventional log reconstruction characterization information, the attack log reconstruction characterization information and the conventional log initial characterization information, and stopping when the network converges to obtain the traffic monitoring neural network, includes: Determining a training reconstruction cost value based on a training reconstruction error between the conventional log reconstructed representation information and the conventional log initial representation information; Determining a training restoration cost based on a training restoration error between the attack log reconstructed representation information and the regular log initial representation information; Iteratively adjusting the flow monitoring neural network to be adjusted according to the training reconstruction cost value and the training restoration cost value, stopping when the network converges, and obtaining the flow monitoring neural network; The method further comprises: Determine a first characterization information error between the conventional log reconstructed characterization information and the conventional log initial characterization information, and determine a second characterization information error between the attack log reconstructed characterization information and the attack log initial characterization information; Using the flow monitoring neural network to be calibrated, extracting flow characterization information of one or more stages from the first characterization information error and the second characterization information error, respectively, to obtain first flow abnormality characterization information and second flow abnormality characterization information; Determine an attack traffic monitoring cost based on the attack traffic field priori mark corresponding to the attack training log, the first traffic anomaly characterization information, and the second traffic anomaly characterization information; The iterative adjustment of the flow monitoring neural network to be adjusted according to the training reconstruction cost value and the training restoration cost value, stopping when the network converges, and obtaining the flow monitoring neural network, comprises: Integrating the training reconstruction cost value, the training recovery cost value, and the attack traffic monitoring cost value to obtain a target training cost value; The flow monitoring neural network to be calibrated is iteratively calibrated according to the target training cost value, and the process is stopped when the network converges to obtain the flow monitoring neural network.

8. An abnormal flow monitoring device, characterized in that: include: one or more processors; Memory; one or more computer programs; The one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program runs on a processor, the processor executes the method according to any one of claims 1 to 7.

Images