A log data compression and decompression method based on masked autoencoder

Abstract

The present invention belongs to the field of computer technology, and specifically relates to a log data compression and decompression method based on a masked autoencoder. The present invention includes: constructing an adjacency matrix, a node feature matrix, and a behavior weight matrix according to the subgraphs divided by the original origin graph; masking the node feature matrix, sending the adjacency matrix, the behavior weight matrix, and the masked node feature matrix to the encoder, and the encoder outputs a representation vector to obtain a reconstructed origin graph through a decoder; calculating the reconstruction error by comparing the reconstructed origin graph with the original origin graph, and iteratively updating the training parameters of the masked autoencoder model; compressing the log data using the encoder of the trained model, and decompressing it using the decoder of the trained model. The present invention can significantly reduce the bandwidth required for transmission by introducing a masked autoencoder model to compress the log data, while ensuring that the data can maintain a sufficient amount of information after compression and decompression for subsequent analysis and processing.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a log data compression and decompression method based on a masked autoencoder. Background Art

[0002] In the modern information technology environment, log data plays a vital role. It not only records the operating status of the system, but also provides valuable information for troubleshooting, performance optimization and security auditing. However, with the increase in system complexity and the expansion of operation scale, the amount of log data has also shown explosive growth. The transmission, storage and analysis of large-scale log data has brought a heavy burden to network and storage resources.

[0003] Especially in distributed systems, cloud computing environments or big data processing platforms, log data needs to be aggregated from various nodes to a central server for analysis. In this process, bandwidth resources often become a bottleneck restricting data transmission efficiency. Although traditional data compression technology can reduce the size of log data to a certain extent, its compression efficiency and decompression speed are often difficult to meet the needs of real-time monitoring and rapid response when faced with massive and frequently generated log data.

[0004] In recent years, the autoencoder technology in the field of deep learning has provided new ideas for data compression. In particular, the masked autoencoder combines the unsupervised learning characteristics of the autoencoder with the data enhancement capabilities of the masking mechanism. It can focus more on the key parts of the data while learning the data features, thereby improving the efficiency of compression and decompression.

[0005] In the field of log data analysis, although there are some methods based on machine learning and deep learning for tasks such as anomaly detection and prediction, applying deep learning technology to the compression and decompression of log data is a new research direction. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the bandwidth consumption of large-scale log data transmission and the compression efficiency and decompression speed of existing data compression technology are difficult to meet the needs of real-time monitoring and rapid response when dealing with real-time large-scale log data, and propose a log data compression and decompression method based on a masked autoencoder. By introducing a masked autoencoder model to compress log data, the bandwidth required for transmission can be greatly reduced, while ensuring that the data can maintain sufficient information after compression and decompression for subsequent analysis and processing, thereby achieving efficient and accurate log data compression and decompression.

[0007] In order to achieve the above object, the technical solution provided by the present invention is: a log data compression and decompression method based on a masked autoencoder, comprising:

[0008] Collect log data in real time, build the original origin graph based on the log data, divide the original origin graph into subgraphs of fixed size in fixed time windows, and build the adjacency matrix, node feature matrix and behavior weight matrix based on the subgraphs;

[0009] Construct a masked autoencoder model, perform mask processing on the node feature matrix, and send the adjacency matrix, behavior weight matrix, and masked node feature matrix to the encoder of the masked autoencoder model. The decoder of the masked autoencoder model receives the representation vector output by the encoder to obtain a reconstructed provenance graph.

[0010] Repeatedly reconstructing the original origin map into a reconstructed origin map, calculating the reconstruction error by comparing the reconstructed origin map with the original origin map, iteratively updating the training parameters of the masked autoencoder model, and completing the training of the masked autoencoder model when the reconstruction error reaches a preset value;

[0011] Input the adjacency matrix, node feature matrix, and behavior weight matrix of the log data to be compressed into the encoder of the trained masked autoencoder model, and output the representation vector, i.e., the compressed log data;

[0012] The compressed log data is input into the decoder of the trained masked autoencoder model to obtain the reconstructed provenance graph, which is stored in the form of a vector to complete the decompression of the log data.

[0013] Furthermore, constructing the adjacency matrix, the node feature matrix and the behavior weight matrix according to the subgraph includes:

[0014] Create a zero matrix of size N×N, where N is the number of nodes in the subgraph. i Point to Node j If there are directed edges, then the zero matrix i Line j The elements of the column are 1, otherwise they are 0. Traverse each directed edge in the subgraph to obtain the adjacency matrix;

[0015] According to the node attributes of each node in the preset subgraph, feature vectors are generated for all nodes in the subgraph, and a feature matrix of size N×F×D is created, where N is the number of nodes in the subgraph, F is the number of node attributes, and D is the highest dimension of all feature vectors. The feature vector corresponding to each node is filled into the feature matrix in the order of the nodes, and the tail of the feature vector with a dimension less than D is filled with 0 to obtain the node feature matrix;

[0016] Create a zero matrix of size N×N, where N is the number of nodes in the subgraph. Traverse each directed edge in the subgraph and read the corresponding weight value according to the importance of the operation relationship represented by the directed edge. If there is a directed edge from node i to node j in the subgraph, fill the corresponding weight value into the i-th row and j-th column of the zero matrix to obtain the behavior weight matrix.

[0017] Furthermore, the method generates a feature vector for all nodes in the subgraph according to the node attributes of each node in the preset subgraph, including: reading the node attributes of each node in the subgraph, using one-hot encoding to generate a type feature vector for the node attributes representing the type, and using a pre-trained BERT to generate a text feature vector for the node attributes of the text class.

[0018] Furthermore, the masked autoencoder model includes an encoder and a decoder, the encoder adopts a graph attention network, and the decoder adopts a Transformer network.

[0019] Furthermore, the masking of the node feature matrix includes: randomly selecting some nodes in the subgraph, and replacing feature vectors corresponding to the some nodes with mask marks MASK in the node feature matrix.

[0020] Compared with the prior art, the present invention has the following significant advantages: 1. It solves the bandwidth problem in the process of large-scale log data transmission, improves transmission efficiency, reduces the consumption of storage and network resources, and provides a more efficient and economical log data processing solution. 2. Compared with traditional compression technology, the compression technology of the present invention is more real-time, can better meet the real-time processing requirements of log data, has higher compression efficiency and lower compression and decompression performance overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a log data compression and decompression method based on a masked autoencoder of the present invention;

[0022] Figure 2 An example diagram of processing log data for the present invention;

[0023] Figure 3 A flowchart of the present invention using a masked autoencoder model to compress log data;

[0024] Figure 4 This is an example diagram of training a masked autoencoder model for the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] like Figure 1 As shown, the present invention provides a log data compression and decompression method based on a masked autoencoder, comprising the following steps:

[0027] (1) Processing log data: First, use the log collection tool to collect the log data provided by Windows ETW (Event Tracing for Windows) in real time. Represent objects such as processes, files, or socket entities in the log data as nodes, represent the operation relationship between nodes as edges, and add the timestamp to the edge attribute to indicate the time when the event occurred. Define the object that initiates the operation as the subject node, and the object being operated as the object node. The subject and object nodes are connected by a directed edge representing the operation relationship to form a triple. The original origin graph is composed of multiple triples and stored; divide the original origin graph into fixed-size subgraphs according to a fixed time window; perform a simple embedding on the subgraph to obtain three matrices representing the content of the graph: an adjacency matrix containing node structure information, a node feature matrix containing node feature information, and a behavior weight matrix containing edge information.

[0028] (1-1) The construction process of the original origin graph: Use log collection tools to collect log data in the system, represent objects such as processes, files or socket entities in the log data as nodes, represent the operation relationship between nodes as edges (such as read operation, write operation, download operation, delete operation, etc.), and add timestamps to the attributes of the edges to indicate the time when the event occurred. Define the object that initiates the operation as the subject node, and the object being operated as the object node. The subject and object nodes are connected by a directed edge representing the operation relationship to form a triple. The original origin graph is composed of multiple triples and stored;

[0029] As shown in Table 1, the following log data need to construct the original origin graph:

[0030] Table 1 Log data sample

[0031] Entity A Entity B operate Timestamp Feature information P1 F1 Read T1 {Path=D:\Document\file1,FileID=20973465} P2 F2 Write T2 {Path=E:\ file2,FileID=32989721} P2 F1 Write T3 {Path=D:\Document \file1,FileID=20973465} P2 F1 Delete T4 {Path =D:\Document,DIRECT DEL} P3 F2 Read T5 {Path=E:\file2,FileID=32989721} P3 F3 Read T6 {Path=C:\User\file3,FileID=75263168} P4 F3 Read T7 {Path=C:\User\file3,FileID=75263168} P4 F3 Write T8 {Path=C:\User\file3,FileID=75263168}

[0032] Each entity in the table has a unique identifier composed of numbers and letters to form a node (in actual applications, it can be UUID or other data that can be used to identify uniqueness). Entity A is the subject node, entity B is the object node, the operation relationship is extracted as a directed edge, and the timestamp information is added to the edge attribute. The subject node and the object node are connected with the directed edge to form a triple. The 8 log data in the table constitute a total of 8 triples, and the subject and object nodes have repeated parts. Figure 2 As shown in the figure, the processing process of the log data, P1, P2, P3, and P4 represent different processes and serve as subject nodes in the triplet. F1, F2, and F3 represent different files and serve as object nodes in the triplet. The nodes are connected by edges to complete the construction of the original provenance graph.

[0033] (1-2) Method of dividing subgraphs according to time windows: Determine the size of the time window based on specific business needs and the requirements for real-time data processing, and allocate data to the data shards corresponding to the corresponding time window based on the timestamp information in the log data. When the time window scrolls over time, the data will be synchronously sharded to obtain several subgraphs.

[0034] like Figure 3 As shown, the log data generated in real time will be segmented after the original origin graph is constructed. This method uses the time window to segment the graph.

[0035] (1-3) The process of representing a subgraph as a matrix using simple embedding:

[0036] (a) Construct an adjacency matrix to represent node structure information: The adjacency matrix represents the directed connections between nodes. i Point to Node j If there are directed edges, then the adjacency matrix i Line j The elements of the column are 1, otherwise they are 0. Create a N × N The zero matrix of N is the number of nodes in the subgraph. Traverse each directed edge in the graph in a depth-first search. If a node i Point to Node j , then the first i Line j The elements of the columns are set to 1, and an adjacency matrix is ​​obtained. Although depth-first search is used as a graph traversal method in this step, it should be understood that any graph traversal method that can achieve the same or similar functions can be used as an alternative.

[0037] (b) Construct a node feature matrix representing node feature information: First, preset the node attributes of each node in the subgraph (such as process number, file name, file type, file path, socket type, etc.). A node can have multiple node attributes. Secondly, select different feature value encoding methods for different node attributes. Use one-hot encoding to generate type feature vectors for node attributes such as file type and socket type. Use BERT (Bidirectional Encoder Representations from Transformer) that has been pre-trained with samples to generate text feature vectors for node attributes such as process number, file name, and file path. After obtaining the feature vector corresponding to the feature of each node, create a size of N × F × D The matrix of N is the number of nodes in the subgraph, F is the number of node attributes, D is the highest dimension in the node’s feature vector. Fill each feature vector corresponding to each node into the feature matrix in node order, and D The tail of the feature vector is padded with 0 to obtain the node feature matrix. Although one-hot and BERT are used for encoding in this step, it should be understood that any encoding method that can achieve the same or similar functions can be used as an alternative.

[0038] (c) Construct a behavior weight matrix to represent edge information: The behavior weight matrix represents the weights of directed edges, which reflect the strength or importance of behaviors between different nodes. Create a N × N The zero matrix of N is the number of nodes in the subgraph. Traverse each directed edge in the subgraph in a depth-first search manner, and preset the corresponding weight value according to the importance of the operation relationship represented by the edge (for example, the network transmission operation of the process and the reading operation of the file can be regarded as operations with a higher degree of importance). Finally, fill the weight value into the corresponding position in the matrix (that is, the first i Line j Columns), the behavior weight matrix is ​​obtained. Although depth-first search is used as the graph traversal method in this step, it should be understood that any graph traversal method that can achieve the same or similar functions can be used as an alternative.

[0039] like Figure 2The process of extracting the matrix content shown in the figure, after simple embedding, obtains the adjacency matrix, node feature matrix and behavior weight matrix from the original origin graph. In the adjacency matrix, the horizontal represents P1, P2, F1, F2 in turn, and the vertical represents P1, P2, F1, F2 in turn. Each node is reachable by itself, so it is represented as 1. There is a directed edge from P1 to F1, and there is a directed edge from P2 to F1 and F2, so the value is also 1. There is no directed edge between other nodes, so the value is 0. The value of the feature vector in the node feature matrix is ​​determined according to the feature information of each node in the original origin graph, including the type of process, process number, file type, file name and other additional information, etc., and the feature information representing the type (file type, socket type, etc.) is encoded using one-hot, and the text information (process number, file name, file path, etc.) is encoded using BERT to obtain the corresponding feature vectors, and finally the feature vectors are filled into the feature matrix. The feature vectors are represented by uppercase X and Y in the figure. In this embodiment, X and Y are used to distinguish the feature vectors corresponding to P and F. The values ​​in the behavior weight matrix are determined by the weight information related to the directed edges. The weight values ​​are determined by the importance of the operations represented by the edges. In the log data used in this embodiment, the Read operation, Write operation, and Delete operation are all operations with high importance, so the corresponding weight values ​​are relatively high. Figure 2 As shown in the figure, R represents the Read operation, W represents the Write operation, and D represents the Delete operation; a weight value of 0 means that there is no connection between nodes and there is no relationship; 1 represents the relationship between the node itself and is meaningless; 2 represents the Delete / Write operation; and 3 represents the Read operation.

[0040] (2) Training the masked autoencoder model: Mask the node feature matrices in the three matrices obtained in step (1), i.e., hide the feature information of these nodes; send the adjacency matrix, behavior weight matrix, and masked node feature matrix to the encoder; use GAT (Graph Attention Network) as the encoder, the encoder will convert the three matrices representing the graph content into corresponding embedding representations, i.e., representation vectors; the decoder uses Transformer as the network structure, receives the representation vector output by the encoder, and attempts to restore the original origin graph through reconstruction operations; the above-mentioned real-time collection of log data, construction of the original origin graph, sub-graph division, matrix conversion, masking, encoding, decoding, and reconstruction processes are repeated, and the reconstruction error is continuously reduced by optimizing the masked autoencoder model structure, adjusting the training strategy, and using regularization techniques until the reconstruction error reaches the preset value, completing the training of the masked autoencoder model. Although GAT is used as the encoder and Transformer is used as the decoder in this step, it should be understood that any encoding and decoding technology that can achieve the same or similar functions can be used as an alternative.

[0041] (2-1) A portion of the nodes in the original origin graph is randomly selected for masking. Specifically, in the matrix representation of the original origin graph, the feature vector related to the node is replaced by a mask mark MASK in the node feature matrix, so that all feature vectors of the node are invisible;

[0042] like Figure 4 As shown, in the node feature matrix, the feature content of the P2 node is replaced by a mask mark MASK, and the feature data of the original node is invisible.

[0043] (2-2) Training of the masked autoencoder model: self-supervised training, using GAT as the encoder, inputting the adjacency matrix, node feature matrix and behavior weight matrix representing the graph content into the encoder (where the node feature matrix has been masked). The encoder extracts the features of the visible part of the node feature matrix and predicts the features of the missing part based on the context of the masked part and the intrinsic structure of the data. It combines the adjacency matrix and the behavior weight matrix to output a representation vector containing the content of the original origin graph and passes the output representation vector to the decoder. The decoder is designed with a Transformer network structure and reconstructs the original origin graph using the received representation vector.

[0044] The reconstruction error is calculated by comparing the reconstructed origin map with the original origin map. The mean square error loss function is used as the loss function of the feature reconstruction part, and the cross entropy loss function is used as the loss function of the structure reconstruction part.

[0045] The training parameters of the masked autoencoder model are adjusted by optimizing the above two loss functions. As the training progresses, the representation vector output by the encoder will become more comprehensive and predictive, the reconstruction accuracy of the decoder will improve, and the masked autoencoder model will gradually have the ability to effectively extract the features of the embedded representation data of the original source image and the ability to reconstruct the data.

[0046] like Figure 4As shown in the figure, the three matrices obtained after the original origin graph is simply embedded are input into the encoder, and the encoder outputs a representation vector V, which contains the structural features and content feature information of all nodes. The structural features and content features are low-dimensional representations extracted from the data of the three matrices. Subsequently, the input representation vector V is input into the decoder, and the decoder reconstructs the content of the original origin graph, where R represents the Read operation, W represents the Write operation, and D represents the Delete operation. The reconstruction error is calculated by comparing the reconstructed origin graph with the original origin graph, and the mean square error loss function is used as the loss function of the feature reconstruction part. The feature reconstruction refers to the content in the reconstructed node feature matrix, and the cross entropy loss function is used as the loss function of the structural reconstruction part. The structural reconstruction refers to the connection relationship between the reconstructed nodes, that is, the adjacency matrix and the behavior weight matrix. By optimizing the above loss function, the training parameters of the masked autoencoder model are adjusted, and the reconstruction error is continuously reduced during training, and finally a masked autoencoder model with excellent capabilities is obtained.

[0047] (3) Log data compression: Using a log data processing method similar to that in step (1), the log data generated by the device in real time is converted into the original origin graph and divided into several subgraphs according to the time window. The subgraphs are represented as adjacency matrices, node feature matrices, and behavior weight matrices using simple embedding. The encoder of the masked autoencoder model trained in step (2) is used to receive the matrix representation of the subgraph and output the corresponding representation vector, i.e., the compressed log data.

[0048] (3-1) The encoder compresses the subgraph: the adjacency matrix, node feature matrix, and behavior weight matrix representing the subgraph content are input into the encoder. At this time, the encoder has the ability to extract the features of the nodes in the subgraph from the matrix content because it applies the trained masked autoencoder model. The encoder outputs the three input matrices as a representation vector to complete the compression of the graph content.

[0049] like Figure 3 As shown in the figure, after a simple embedding, the subgraph outputs three matrices: adjacency matrix, node feature matrix and behavior weight matrix. After these three matrices are input to the encoder, a representation vector V=[v1, v2, v3…v n ], where v n It represents the structural and content feature information of the nth node. The representation vector contains the content information of the three matrices. The representation vector is the compressed log data.

[0050] (4) Data transmission: The compressed log data will be transmitted to the log analysis device in real time via the network.

[0051] (5) On the log analysis device, apply the decoder of the masked autoencoder model trained in step (2), input the received compressed log data into the decoder, the decoder receives the compressed log data, restores the original origin graph from the representation vector through reconstruction operation, and stores it in the form of a vector, thereby completing the decompression of the log data.

[0052] (5-1) The process of the decoder reconstructing the features of the original origin graph: the representation vector output by the encoder is input into the decoder. At this time, the decoder has the ability to reconstruct the features of the original origin graph through the representation vector because it applies the trained masked autoencoder model. The decoder restores the content features and structural features of the nodes and edges in the original origin graph through content reconstruction and structural reconstruction of the input representation vector to obtain the reconstructed origin graph. The reconstructed origin graph exists in the form of feature values, and the feature values ​​of the reconstructed origin graph are stored in the form of vectors.

[0053] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A log data compression and decompression method based on a masked autoencoder, characterized in that: The log data compression and decompression method based on the masked autoencoder comprises: Collect log data in real time, build the original origin graph based on the log data, divide the original origin graph into subgraphs of fixed size in fixed time windows, and build the adjacency matrix, node feature matrix and behavior weight matrix based on the subgraphs; Construct a masked autoencoder model, perform mask processing on the node feature matrix, and send the adjacency matrix, behavior weight matrix, and masked node feature matrix to the encoder of the masked autoencoder model. The decoder of the masked autoencoder model receives the representation vector output by the encoder to obtain a reconstructed provenance graph. Repeatedly reconstructing the original origin map into a reconstructed origin map, calculating the reconstruction error by comparing the reconstructed origin map with the original origin map, iteratively updating the training parameters of the masked autoencoder model, and completing the training of the masked autoencoder model when the reconstruction error reaches a preset value; Input the adjacency matrix, node feature matrix, and behavior weight matrix of the log data to be compressed into the encoder of the trained masked autoencoder model, and output the representation vector, i.e., the compressed log data; The compressed log data is input into the decoder of the trained masked autoencoder model to obtain the reconstructed provenance graph, which is stored in the form of a vector to complete the decompression of the log data.

2. The log data compression and decompression method based on masked autoencoder according to claim 1, characterized in that: The step of constructing an adjacency matrix, a node feature matrix, and a behavior weight matrix according to the subgraph includes: Create a size of N × N The zero matrix of N is the number of nodes in the subgraph. If i Point to Node j If there are directed edges, then the zero matrix i Line j The elements of the column are 1, otherwise they are 0. Traverse each directed edge in the subgraph to obtain the adjacency matrix; According to the node attributes of each node in the preset subgraph, generate feature vectors for all nodes in the subgraph and create a N × F × D The characteristic matrix of N is the number of nodes in the subgraph, F is the number of node attributes, D The highest dimension among all feature vectors is filled into the feature matrix according to the node order. D The tail of the eigenvector is filled with 0 to obtain the node feature matrix; Create a size of N × N The zero matrix of N is the number of nodes in the subgraph. Traverse each directed edge in the subgraph and read the corresponding weight value according to the importance of the operation relationship represented by the directed edge. If the node in the subgraph i Point to Node j If there is a directed edge, fill the corresponding weight value into the zero matrix. i Line j Columns, get the behavior weight matrix.

3. The log data compression and decompression method based on masked autoencoder according to claim 2, characterized in that: The method generates a feature vector for all nodes in the subgraph according to the node attributes of each node in the preset subgraph, including: reading the node attributes of each node in the subgraph, using one-hot encoding to generate a type feature vector for the node attributes representing the type, and using a pre-trained BERT to generate a text feature vector for the node attributes of the text class.

4. The log data compression and decompression method based on masked autoencoder according to claim 1, characterized in that: The masked autoencoder model includes an encoder and a decoder, the encoder adopts a graph attention network, and the decoder adopts a Transformer network.

5. The log data compression and decompression method based on masked autoencoder according to claim 1, characterized in that: The masking process for the node feature matrix includes: randomly selecting some nodes in the subgraph, and replacing the feature vectors corresponding to the some nodes with mask marks MASK in the node feature matrix.

Images