Traceability log storage and query method based on deep embedding and single-layer RNN prediction

Abstract

The invention provides a traceability log storage and query method based on deep embedding and single-layer RNN prediction, and relates to the technical field of traceability graph compression, and the method comprises the steps: converting a system bottom layer audit log into a traceability graph, carrying out the coding mapping of path class and identification class attribute fields through deep learning embedding representation, and obtaining a traceability graph; coding prefixes are distributed for the path type attribute fields and the identification type attribute fields through self-adaptive dictionary coding, and numerical attribute fields are recorded through incremental coding; preprocessing and learning the traceability graph through a single-layer recurrent neural network (RNN) unit; performing local adjustment on the traceability graph during data updating based on incremental coding and adaptive mapping; and mapping the query condition into an attribute coding prefix, reconstructing an attribute sequence based on the attribute coding prefix, and performing correction through the adjustment table. For redundant information and structural similarity, a deep learning embedding representation technology and an adaptive dictionary coding strategy are adopted, and shared codes of similar attribute fields are effectively learned and extracted, so that redundant data storage is reduced.

Description

Source tracing log storage and querying method based on deep embedding and single-layer RNN prediction Technical Field

[0001] This application relates to the field of source graph compression technology, and more specifically, to a method for storing and querying source logs based on deep embedding and single-layer RNN prediction. Background Technology

[0002] Existing security analysis and attack attribution systems typically construct attribution graphs by continuously collecting log information such as system calls, file accesses, and network communications to support attack attribution and causal analysis. To ensure the integrity and immutability of attribution data, existing systems generally use an append-only approach for data storage, causing the attribution graph to grow continuously over time. Furthermore, cyberattacks against government and enterprise systems often exhibit multi-stage and long-term latency characteristics, requiring long-term storage of attribution data, further increasing the pressure on attribution graph storage systems in terms of space consumption and access efficiency.

[0003] To address the issue of excessive storage overhead in source tracing graphs, existing technologies primarily focus on improvements through data compression and storage optimization.

[0004] One type of method reduces storage space usage by eliminating redundant nodes (such as Chinese invention patent CN115277124A, an online system and server for searching and matching attack patterns based on system source graph), edges (such as Chinese invention patent CN118827222A, an APT attack detection method based on multi-dimensional edge optimization of the source graph), or duplicate attribute fields (such as Chinese invention patent CN105488043A, a data query method and system based on key-value data blocks). However, some of these methods involve lossy processing, which may affect data integrity and subsequent... Further analysis of the results; another type of method reduces storage space requirements by using encoding optimization (such as Chinese invention patent with publication number CN119229152A, a subgraph matching method and related equipment based on dense perception graph contrast learning, which optimizes the encoder and projection head) or general compression format (such as Chinese invention patent with publication number CN118568067A, a source graph compression method, device, electronic device and storage medium, which matches the part isomorphic to the template subgraph from the source graph to be compressed as the region to be compressed), but its query capability often depends on the underlying storage format, making it difficult to efficiently support complex graph structures and time-series queries.

[0005] In addition, some systems use relational databases (such as Chinese invention patent CN120429300A, "A Method and Apparatus for Image Tracing", whose relational database includes a global feature image node set and an edge set, extracts global features from the images contained in the image dataset in the basic database to obtain a global feature image node set; traverses all image nodes, establishes directed edges between image nodes with modification relationships and obtains an edge set) or graph databases (such as Neo4j, a commonly used graph database in this field) as the traceability graph storage carrier. Although this balances storage and query efficiency to a certain extent, it still has problems such as high storage costs and decreased query efficiency when facing large-scale, long-term accumulated traceability log data.

[0006] In recent years, some technical solutions have attempted to introduce deep learning models to model and optimize the storage of source graph data (such as Chinese invention patent CN115065556A, a method and system for detecting malicious behavior in logs based on graph contrastive learning). However, existing methods generally suffer from drawbacks such as large model size, high consumption of training and storage resources, and insufficient utilization of the semantics of node and edge structures.

[0007] In summary, existing source graph storage technologies still suffer from problems such as large storage space consumption, high data redundancy, insufficient expression of structural similarity, high model cost, and limited query efficiency in massive log scenarios, making it difficult to simultaneously meet the practical application requirements of long-term storage and efficient querying. Summary of the Invention

[0008] To address the aforementioned issues, this application employs a source tracing log storage and query method based on deep embedding and single-layer RNN prediction, comprising: S1 Encoding Compression: Converting the system's underlying audit logs into a source tracing graph containing nodes and edges. During this conversion, deep learning embedding representations are used to encode and map path-class and identifier-class attribute fields in the audit logs. In the encoding and mapping process, adaptive dictionary encoding is used to assign encoding prefixes to path-class and identifier-class attribute fields, and incremental encoding is used to record numerical attribute fields; S2 Encoding Learning: Preprocessing and learning the encoding vectors of nodes and edges in the source tracing graph using a single-layer recurrent neural network (RNN) unit; S3 Dataset Update: Locally adjusting the source tracing graph based on incremental encoding and adaptive mapping during data updates; S4 Query Output: Mapping query conditions to corresponding attribute encoding prefixes, reconstructing attribute sequences based on attribute encoding prefixes using a single-layer recurrent neural network (RNN) module, and correcting the prediction results through an adjustment table.

[0009] Optionally, the step S1 of encoding and mapping the path class and identifier class attribute fields in the audit log using deep learning embedding representation includes: generating a high-dimensional embedding vector for each attribute using deep learning embedding representation, and encoding and mapping the attribute fields using the semantic association and frequency of occurrence of the attributes.

[0010] Optionally, assigning encoding prefixes to path class and identifier class attribute fields through adaptive dictionary encoding in step S1 includes: dynamically adjusting the encoding based on frequency, thereby assigning shared encoding prefixes to attribute fields with similar spatial structure characteristics.

[0011] Optionally, in step S1, incremental encoding is used to record numerical attribute fields, including: recording the relative offset between adjacent events instead of storing complete numerical information each time.

[0012] Optionally, the data preprocessing in step S2 includes one-dimensional flattening of the encoded vectors of nodes and edges, followed by window partitioning according to a pre-defined delimiter. After this process, the embedding representation of the sequence can be formally expressed as the following expression: In the formula, e represents the encoding of each independent node or edge, vector T represents the total length of the sequence, and d represents the dimension of the feature.

[0013] Optionally, the learning in step S2 includes using an RNN to process the input sequence through temporal modeling and using a fully connected network to predict the fields, specifically as follows: In the formula, W represents the weight matrix, b represents the bias term, and ReLU represents the activation function.

[0014] Optionally, step S3, local adjustment, includes: mapping the newly added attribute to the existing coding system; if the attribute has a structure similar to an existing class or category, mapping the newly added attribute to the corresponding category and obtaining a new unique suffix; if there is no similar category, assigning the newly added attribute to a new category and generating a new prefix based on its characteristics.

[0015] Optionally, step S4, reconstructing the attribute sequence, includes: when a user initiates a query request for a specific node or event, the query conditions are first mapped to the corresponding attribute encoding prefix, and the prefix is ​​fed into a single-layer recurrent neural network (RNN) unit. The single-layer RNN unit predicts subsequent encoding values ​​field by field according to the sequence generation method based on the historical hidden state and the input query prefix, until a preset termination identifier is encountered, thus completing the reconstruction of the attribute sequence.

[0016] Optionally, step S4 corrects the prediction results using an adjustment table, which includes: maintaining a unique identifier for each node or edge and its corresponding true encoding value to construct an adjustment table; using the adjustment table to verify the encoding in the reconstructed attribute sequence; when the encoding in the reconstructed attribute sequence is inconsistent with the encoding recorded in the adjustment table, the true encoding value in the adjustment table is used first for replacement; if the encoding information for a certain position is not recorded in the adjustment table, the encoding result in the reconstructed attribute sequence is used by default as the reliable encoding for that position.

[0017] This application also provides a method for storing and querying source logs based on deep embedding and single-layer RNN prediction, including a source graph encoding and compression module based on spatial structure and node attributes, and a source data storage and query module based on recurrent neural network sequence prediction. The source graph encoding and compression module based on spatial structure and node attributes is used to convert the system's underlying audit logs into a source graph containing nodes and edges. In this conversion process, path class and identifier class attribute fields in the audit logs are encoded and mapped using deep learning embedding representation. During the encoding and mapping process, an adaptive dictionary encoding is used to assign encoding prefixes to path class and identifier class attribute fields, and incremental encoding is used to record numerical attribute fields. Based on incremental encoding and adaptive mapping, the source graph is locally adjusted when the data is updated. The source data storage and query module based on recurrent neural network sequence prediction is used to preprocess and learn the encoding vectors of nodes and edges in the source graph using a single-layer recurrent neural network (RNN) unit. The query conditions are mapped to the corresponding attribute encoding prefixes, and the single-layer recurrent neural network (RNN) module reconstructs the attribute sequence based on the attribute encoding prefixes. The prediction results are corrected by adjusting the table.

[0018] The beneficial effects of the source log storage and query method based on deep embedding and single-layer RNN prediction provided in this application are as follows: (1) Collect and parse the underlying audit logs, and convert the log events into a source graph structure containing node and edge attribute information in chronological order. For the large number of attribute fields contained in the nodes and edges of the source graph, this module performs structured processing on the attribute data and constructs an encoding mapping relationship based on the structural features and frequency of occurrence of the fields. For redundant information and structural similarity, this component adopts deep learning embedding representation technology and adaptive dictionary encoding strategy. These technologies can effectively learn and extract the shared encoding of similar attribute fields, thereby reducing redundant data storage and preserving semantic relevance. Deep learning embedding representation will learn high-dimensional representation according to data features, thereby assigning similar encoding representations to similar data points and improving the compression effect.

[0019] (2) This application constructs a lightweight sequence prediction model based on the encoded source graph data to model and predict the attribute sequences of nodes or edges. By predicting partial field information to replace complete explicit storage, this application can further reduce the storage scale while ensuring data recoverability. During the query process, the model generates corresponding prediction results based on the input sequence and combines them with auxiliary data structures used to correct prediction errors to perform consistency correction on the prediction results, thereby ensuring the accuracy and reliability of the query results. Through the synergistic effect of this prediction and correction mechanism, stable and efficient storage and query performance can be maintained in long-term operation and resource-constrained environments. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0021] Figure 1 is a schematic diagram of the architecture of the source log storage and query method based on deep embedding and single-layer RNN prediction provided in the embodiments of this application. Detailed Implementation

[0022] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0023] As shown in Figure 1, this application provides a method for storing and querying source logs based on deep embedding and single-layer RNN prediction, including: S1 Encoding Compression: Converting the system's underlying audit logs into a source graph containing nodes and edges. During this conversion, deep learning embedding representation is used to encode and map the path class and identifier class attribute fields in the audit logs. In the encoding and mapping process, adaptive dictionary encoding is used to assign encoding prefixes to the path class and identifier class attribute fields, and incremental encoding is used to record numerical attribute fields; S2 Encoding Learning: Preprocessing and learning the encoding vectors of nodes and edges in the source graph through a single-layer recurrent neural network (RNN) unit; S3 Dataset Update: Locally adjusting the source graph based on incremental encoding and adaptive mapping during data updates; S4 Query Output: Mapping query conditions to corresponding attribute encoding prefixes, reconstructing attribute sequences based on attribute encoding prefixes using a single-layer recurrent neural network (RNN) module, and correcting the prediction results through an adjustment table.

[0024] In the source graph, nodes are the specific objects that generate log events in the system. Figure 1 shows four core node types, including: node identifier v1, whose corresponding unique identifier (represented as hashid in the figure) is c721, type (represented as type in the figure) is process, attribute 1 (represented as property1 in the figure) is syscall, attribute 2 (represented as property2 in the figure) is cache2 I / O, the corresponding type encoding is 0 (represented as type:process:0 in the initial encoding table of node-edge attributes), and attribute 1 encoding is 0 (represented as property1:syscall:0 in the initial encoding table of node-edge attributes).

[0025] Node identifier v2 has a unique identifier of 18bd, type subject (operation subject), attribute 1 is mmap (memory mapping), attribute 2 is compiz (program name), the corresponding type encoding is 1 (represented as type:subject:1 in the initial encoding table of node-edge attributes), and attribute 1 is encoded as 1 (represented as property1: mmap:1 in the initial encoding table of node-edge attributes).

[0026] The node identifier is v3, and its corresponding unique identifier is 4a7b. Its type is file, its attribute 1 is ioctl (device control), its attribute 2 is ext4 (file system), and its corresponding type encoding is 2 (represented as type:file:2 in the initial encoding table of node-edge attributes). The attribute 1 encoding is 2 (represented as property1:ioctl:2 in the initial encoding table of node-edge attributes).

[0027] The node identifier is v4, and its corresponding unique identifier is d532. Its type is network, its attribute 1 is sendmsg, its attribute 2 is eth0 (network interface), its corresponding type encoding is 3 (represented as type:network:3 in the initial encoding table of node-edge attributes), and its attribute 1 encoding is 3 (represented as property1:sendmsg:3 in the initial encoding table of node-edge attributes).

[0028] Figure 1 also shows three temporal edges, containing association relationships and event information, including: the connection relationship of edge e1 (represented as edge in the figure) is v1 (process) - v2 (operation subject), the operation attribute is write (represented as property in the figure), and the timestamp (represented as time in the figure) is 10.0; the connection relationship of edge e2 is v3 (file) - v2 (operation subject), the operation attribute is open (open), and the timestamp is 20.5; the connection relationship of edge e3 is v2 (operation subject) - v4 (network), the operation attribute is read (read), and the timestamp is 30.0; the temporal event flow is formed by timestamps 10.0-20.5-30.0, representing the causal link of process writing to subject - file being opened by subject - subject reading and sending network data.

[0029] For the redundant path / home / log1 / .cache / mozilla / firefox, a shared encoding prefix is ​​assigned. The figure shows an example of adaptive dictionary encoding: the path segment " / " is assigned the encoding prefix 0, the path segment "firefox" is assigned the encoding prefix 1000, the path segment "home" is assigned the encoding prefix 1011, and the path segment "log1" is assigned the encoding prefix 1010.

[0030] The adjustment table in Figure 1 shows the adjustment entry v1:[5:1,6:8,29:0,40:1,...] for node v1, where 5:1 indicates that the actual code for position 5 is 1, the actual code for position 6 is 8, the actual code for position 29 is 0, and the actual code for position 40 is 1. The adjustment table also includes the adjustment entry for edge e1 (the specific corresponding actual code is not shown in Figure 1).

[0031] Step S1, which encodes and maps the path class and identifier class attribute fields in the audit log using deep learning embedding representation, includes: generating a high-dimensional embedding vector for each attribute using deep learning embedding representation, and encoding and mapping the attribute fields using the semantic association and frequency of occurrence of the attributes.

[0032] Path and identifier attributes are cleaned and standardized, and a dedicated vocabulary for each attribute is established. First, all path and identifier attributes are deduplicated, retaining only the first occurrence of each distinct field attribute. This ensures that each unique field attribute is encoded only once. After deduplication, a dedicated vocabulary is created to store the unique identifier for each attribute. This vocabulary will serve as the basis for subsequent training.

[0033] Define the embedding dimension by initializing a high-dimensional embedding matrix, where each attribute (such as a path or identifier) ​​is mapped to a high-dimensional vector. The embedding dimension is typically determined based on the complexity of the data and the requirements of the model, and may need to be adjusted to meet dataset compression needs.

[0034] During path attribute processing, the path strings " / home / log1 / firefox / 976DA8F" and " / home / log1 / mozilla / firefox / anmp96vk" are encoded as IDs 30 and 40, respectively. Since these two paths are very similar in their main structure, differing only in their end identifiers, they are assigned similar embedding vectors to reflect their semantic similarity. By using a forward slash (" / ") as a separator, the model can split the path string into different sub-paths, further improving the model's semantic understanding ability and achieving semantic training.

[0035] For attributes with low frequency, negative sampling is used for frequency-based training to avoid overfitting of the embedding vectors for these attributes. Negative sampling preserves the semantic features of high-frequency attributes and prevents low-frequency attributes from affecting the model's generalization ability. Negative sampling also helps maintain the semantic consistency of the entire embedding matrix.

[0036] After completing the above training, the final step is to encode and map each attribute to its corresponding high-dimensional embedding vector, thus performing attribute-high-dimensional vector encoding mapping. These vectors can effectively represent the semantic space information of each attribute and provide support for subsequent analysis or model training.

[0037] In step S1, assigning encoding prefixes to path class and identifier class attribute fields through adaptive dictionary encoding includes: dynamically adjusting the encoding based on frequency, thereby assigning shared encoding prefixes to attribute fields with similar spatial structure characteristics.

[0038] First, all path-class and identifier-class attribute fields are represented using high-dimensional embedding vectors generated by deep learning. These embedding vectors effectively capture the semantic relationships between attributes. Next, the cosine similarity between all attribute embedding vectors is calculated to measure the similarity of different attribute fields in the semantic space, and clustering based on embedding vector similarity is performed.

[0039] For each pair of attribute field embedding vectors, calculate their cosine similarity. Cosine similarity reflects the angular similarity between two attributes in the vector space; the higher the similarity, the more semantically similar the two attributes are. Sort all attribute field similarities and select the attributes with the highest similarity.

[0040] Attributes with high similarity are grouped into the same category. Here, a hierarchical clustering method is used to assign similar attribute fields to the same category, ensuring that the attribute fields in each category have similar spatial structure features.

[0041] A unique shared encoding prefix is ​​assigned to the attribute fields in each class. This prefix represents the common characteristics of the attributes in that class, effectively reducing storage space and improving query efficiency. The prefix for each class is determined based on the clustering results of the attributes, ensuring that attributes within a class share the same prefix.

[0042] For each attribute field within a class, a unique suffix is ​​assigned. The suffix is ​​numbered based on the uniqueness of the attribute within the class, ensuring that the code for each attribute is unique within the class. This suffix is ​​typically generated using a simple numbering system.

[0043] The final encoding of each attribute is formed by combining a shared prefix and a class-unique suffix. In this way, attribute fields with similar spatial structure characteristics will share the same prefix, while each attribute within the same class is distinguished by its suffix, ultimately forming an efficient and compact encoding.

[0044] Suppose the path attributes " / home / log1 / firefox / 976DA8F" and " / home / log1 / mozilla / firefox / anmp96vk" are clustered into the same class. Due to their similar structure, the model assigns them a shared prefix "01". Then, the path " / home / log1 / firefox / 976DA8F" is assigned the suffix "01", while the path " / home / log1 / mozilla / firefox / anmp96vk" is assigned the suffix "02". Therefore, their final encodings are "01_01" and "01_02" respectively.

[0045] In step S1, incremental encoding is used to record numerical attribute fields, including: recording the relative offset between adjacent events instead of storing complete numerical information each time.

[0046] In the source graph encoding and compression module based on spatial structure and node attributes, the system collects and parses the underlying audit logs, converting log events into a source graph structure containing node and edge attribute information in chronological order. For the numerous attribute fields contained in the nodes and edges of the source graph, this module performs structured processing on the attribute data and constructs encoding mapping relationships based on the structural features and frequency of occurrence of the fields. To address redundant information and structural similarity, this component employs deep learning embedding representation technology and an adaptive dictionary encoding strategy. These technologies can effectively learn and extract shared codes for similar attribute fields, thereby reducing redundant data storage and preserving semantic relevance. Deep learning embedding representation learns high-dimensional representations based on data features, thus assigning similar encoding representations to similar data points and improving compression performance. Deep learning embedding representation can learn the high-dimensional semantic space of attribute fields, mapping similar data points to similar vector spaces, thereby effectively improving encoding accuracy and compression performance.

[0047] Step S2, data preprocessing, includes one-dimensional flattening of the encoded vectors of nodes and edges, followed by window partitioning based on a pre-defined delimiter. After this process, the embedding representation of the sequence can be formally expressed as the following expression: In the formula, e represents the encoding of each independent node or edge, vector T represents the total length of the sequence, and d represents the dimension of the feature.

[0048] This design means that the model only needs to pay attention to all the field information of the current node or edge during the prediction process until it encounters the termination identifier, which marks the end of a complete query.

[0049] All encoded nodes and events are sequence data composed of 0s and 1s. During sequence prediction, time windows need to be segmented, but delimiters must be set between different nodes and events. This invention utilizes unique numbers not used in the encoding process as delimiters. This invention uses 2 as the delimiter identifier between nodes and 3 as the event identifier. This is for the purpose of dividing the current node identifier and attribute field training during sequence prediction training. Attribute fields are uniformly distinguished using 4, and a 6 is used as the terminator at the end.

[0050] Step S2 involves learning the input sequence using RNN through temporal modeling and predicting fields using a fully connected network, specifically as follows: In the formula, W represents the weight matrix, b represents the bias term, and ReLU represents the activation function.

[0051] Through the above processing, the RNN model can gradually complete the prediction task of the attribute sequence, thereby realizing the storage and query of the source data.

[0052] Training data consists of the true attribute sequence and the model's predicted sequence: for example, using the true sequence e1-e2-e3 as the label and e1-e2 as the input, predict e3 and compare it with the true e3. Mean squared error or cross-entropy loss is used to calculate the error between the predicted and true values.

[0053] Forward and backward propagation are performed, and the predicted values ​​are calculated according to the E-RNN-Flatten-Fully Connected-y algorithm, recording the hidden state at each time step. Starting from the last time step, the gradient of each parameter is calculated backward along the time axis. All parameters are updated, and the loss function is minimized until the model converges.

[0054] The query conditions are mapped to attribute encoding prefixes. The RNN reconstructs the attribute sequence field by field, and the encoding prefix is ​​used as the initial sequence input into the RNN. The model predicts subsequent attributes field by field at time steps until it encounters a termination identifier: First step: predict the operation type based on the prefix target node; Second step: predict the time increment based on the target node and operation type; Third step: predict the final reconstructed sequence of the source node based on the results of the first two steps.

[0055] Step S3, local adjustment, includes: mapping the newly added attribute to the existing coding system; if the attribute has a structure similar to an existing class or category, mapping the newly added attribute to the corresponding category and obtaining a new unique suffix; if there is no similar category, assigning the newly added attribute to a new category and generating a new prefix based on its characteristics.

[0056] This application provides a dynamic encoding update mechanism. When new or updated data arrives, there is no need to re-encode the entire dataset. Incremental encoding and adaptive mapping adjustment strategies can make local adjustments when data is updated, thus avoiding the cost of re-encoding existing data. This mechanism ensures efficient system operation during data updates and achieves a balance between storage overhead and computational performance.

[0057] In practical applications, data is constantly being updated, such as with the addition of paths and identifier fields. To effectively encode newly added attributes, an incremental update monitoring mechanism is first needed to identify the new data. This new data may be attributes that have never appeared before or changes to existing attributes. When new data arrives, the first step is to determine whether an incremental update to the existing encoding is necessary.

[0058] When a new attribute field is detected, incremental encoding is generated for these new attributes. Unlike batch encoding, incremental encoding primarily makes local adjustments to the new data, avoiding impact on the encoding structure of existing data. Specifically, the newly added attributes are mapped to the appropriate position in the existing encoding system. If an attribute has a structure similar to an existing class or category, it will be mapped to the corresponding category and receive a new unique suffix. If no similar category exists, it will be assigned to a new category, and a new prefix will be generated based on its characteristics.

[0059] During data updates, not only are new codes needed, but existing codes also need to be adjusted. An adaptive mapping adjustment strategy dynamically adjusts the mapping relationships of existing codes. Specific steps include: adjusting semantic mapping relationships: for encoded attributes, if semantic changes occur between attributes, the mapping between attributes and codes is adjusted based on the new data. The allocation of shared prefixes and suffixes is updated by recalculating the similarity between embedding vectors and attributes.

[0060] Dynamic frequency adjustment: For attributes with low frequency or those that no longer occur frequently, their encoding is automatically compressed or merged to ensure that high-frequency attributes still occupy a higher priority encoding position. Low-frequency attributes may be re-encoded or removed from certain categories to avoid wasting storage space.

[0061] Merging and splitting adjustments: When multiple attribute fields are detected to have highly similar features, a merging operation can be performed to create a new shared encoding prefix, reducing the total number of codes. Conversely, if the semantic differences between attributes increase, previously merged codes will be split to ensure the independence and accuracy of the codes.

[0062] The local adjustment component of the adaptive mapping adjustment strategy primarily optimizes already encoded attributes. Based on updated data features, the embedding vectors of these attributes are adjusted to more accurately position them in the semantic space. The updated embedding vectors influence subsequent attribute encoding mappings, ensuring flexibility in responding to attribute changes. For example, when the frequency of a path attribute suddenly increases, its embedding vector can be adjusted to more closely align with higher-frequency attribute fields, enhancing the semantic expressiveness of the encoding.

[0063] Step S4, reconstructing the attribute sequence, includes: when a user initiates a query request for a specific node or event, the query conditions are first mapped to the corresponding attribute encoding prefix, and the prefix is ​​fed into a single-layer recurrent neural network (RNN) unit. The single-layer RNN unit predicts subsequent encoding values ​​field by field according to the sequence generation method based on the historical hidden state and the input query prefix, until a preset termination identifier is encountered, thus completing the reconstruction of the attribute sequence.

[0064] During the query phase, by employing a single-layer RNN, this method not only avoids the complexity of multi-layer neural networks but also completes predictions more quickly, reducing computational resource consumption. Through this process, this application can dynamically recover the required data content during querying without explicitly storing complete node or edge attribute information, further reducing storage requirements.

[0065] Step S4 corrects the prediction results using an adjustment table, which includes: maintaining a unique identifier for each node or edge and its corresponding true encoding value to construct an adjustment table; using the adjustment table to verify the encoding in the reconstructed attribute sequence; when the encoding in the reconstructed attribute sequence is inconsistent with the encoding recorded in the adjustment table, the true encoding value in the adjustment table is used first for replacement; if the encoding information for a certain position is not recorded in the adjustment table, the encoding result in the reconstructed attribute sequence is used as the reliable encoding for that position by default.

[0066] Given the potential for prediction bias in sequence prediction models during complex storage tasks, this method introduces an adjustment table mechanism to accurately correct prediction results and ensure the consistency and reliability of source data. The adjustment table maintains a unique identifier for each node or edge in memory, along with its corresponding true encoding value. This mechanism corrects key fields during the prediction process. When the predicted encoding output by the model differs from the encoding recorded in the adjustment table, the true encoding value from the adjustment table is used first to replace it, ensuring the accuracy of the query results. If the adjustment table does not record the encoding information for a certain position, the model's prediction result is used as the reliable encoding for that position by default. This correction mechanism effectively avoids errors caused by model prediction bias and improves the accuracy of data recovery.

[0067] By combining prediction and correction, this method can significantly reduce data storage size without explicitly storing complete node or edge attribute information. Furthermore, the size of the adjustment table can be dynamically configured based on system resource availability and query accuracy requirements, flexibly balancing storage space usage, computational complexity, and query accuracy. In resource-constrained environments, the dynamic adjustment of the adjustment table enables the system to operate continuously and efficiently, making it particularly suitable for long-term source log data storage and querying tasks. The size of the adjustment table itself is determined by the prediction accuracy after converting the dataset to sequence modeling and the size of the data itself. If the dataset is too large, the adjustment table will also increase in size. However, the adjustment table can be compressed and its accuracy improved through multiple training iterations.

[0068] Ultimately, by combining the efficient predictive capabilities of a single-layer RNN with its adjustment table mechanism, this component significantly improves the efficiency of traceability data storage and retrieval while ensuring data consistency and recoverability. This solution provides reliable technical support for the management and analysis of large-scale traceability log data, and is particularly suitable for efficient and flexible long-term storage and retrieval needs.

[0069] This application also provides a source tracing log storage and query system based on deep embedding and single-layer RNN prediction, including a source tracing graph encoding and compression module based on spatial structure and node attributes, and a source tracing data storage and query module based on recurrent neural network sequence prediction. The source tracing graph encoding and compression module based on spatial structure and node attributes is used to convert the system's underlying audit logs into a source tracing graph containing nodes and edges. In this conversion process, path class and identifier class attribute fields in the audit logs are encoded and mapped using deep learning embedding representation. During the encoding and mapping process, an adaptive dictionary encoding is used to assign encoding prefixes to path class and identifier class attribute fields, and incremental encoding is used to record numerical attribute fields. Based on incremental encoding and adaptive mapping, the source tracing graph is locally adjusted when the data is updated. The source tracing data storage and query module based on recurrent neural network sequence prediction is used to preprocess and learn the encoding vectors of nodes and edges in the source tracing graph using a single-layer recurrent neural network (RNN) unit. The query conditions are mapped to the corresponding attribute encoding prefixes, and the single-layer recurrent neural network (RNN) module reconstructs the attribute sequence based on the attribute encoding prefixes. The prediction results are corrected by adjusting the table.

[0070] The source graph encoding and compression component based on spatial structure and node attributes is mainly used to solve the problems of high redundancy in node and edge attribute fields and difficulty in expressing the structural similarity of path-like strings in source graphs. While ensuring the semantic integrity of the source graph, this component significantly reduces storage overhead by efficiently compressing and storing the source graph data.

[0071] The core objective of the source data storage and query component based on recurrent neural network sequence prediction is to further reduce the explicit storage size of source graph data while improving query efficiency, all while ensuring the consistency and recoverability of source data. By employing an efficient single-layer recurrent neural network (RNN) model, this component can accurately model and predict attribute sequences field by field, effectively reducing storage requirements and ensuring accuracy and efficiency during data recovery.

[0072] This application presents a source tracing log storage and query method based on deep embedding and single-layer RNN prediction. This method is designed for storing and querying massive source tracing log data, aiming to reduce storage overhead and improve query efficiency while ensuring data integrity and consistency. It does not rely on manual screening or prior rules for log data; instead, it performs structured encoding and predictive storage processing on the data based on the structural features and attribute distribution patterns of nodes and edges in the source tracing graph. By combining attribute encoding mechanisms with sequence prediction models, this application effectively compresses highly redundant source tracing data and achieves a balance between space efficiency and computational efficiency during storage and querying.

[0073] In the source data storage and query module based on recurrent neural network sequence prediction, this application constructs a lightweight sequence prediction model based on the encoded source graph data to model and predict the attribute sequences of nodes or edges. By predicting partial field information instead of full explicit storage, this application can further reduce the storage scale while ensuring data recoverability. During the query process, the model generates corresponding prediction results based on the input sequence and combines them with auxiliary data structures to correct prediction errors, thereby ensuring the accuracy and reliability of the query results. Through the synergistic effect of this prediction and correction mechanism, stable and efficient storage and query performance can be maintained in long-term operation and resource-constrained environments.

[0074] To address the challenges of existing source tracing graph storage technologies in massive log scenarios, such as large storage space consumption, high data redundancy, insufficient structural similarity representation, high training and storage costs of deep models, and limited query efficiency, this application presents a source tracing log storage and query method based on deep embedding and single-layer RNN prediction. This method reduces storage overhead caused by redundant fields by structurally encoding and compressing the attribute information of nodes and edges in the source tracing log, while ensuring the integrity and consistency of the source data. Simultaneously, a lightweight sequence prediction mechanism is constructed to achieve efficient storage and fast querying of source data, thereby reducing the scale of model parameters and computational resource consumption. Furthermore, this application aims to introduce a prediction result correction mechanism to improve the reliability of storage and query results while ensuring query accuracy, making the method applicable to practical source tracing analysis applications that require long-term operation, limited resources, and high timeliness.

[0075] By achieving the aforementioned objectives, this application provides a feasible and efficient technical solution for the storage and retrieval of large-scale probabilities log data, while balancing storage efficiency, computational efficiency, and query accuracy. In experiments using the same open-source dataset (project name: Leonard; related conference: Usenix Security'23), this invention successfully compressed the original 18MB dataset to a storage size of 1.37MB. In contrast, the mainstream graph database PostgreSQL requires 41MB of storage space, a difference of 29 times. Compared to the current state-of-the-art deep compression storage method (Leonard's method in the paper "The Case for Learned Provenance GraphStorage Systems" by Usenix Security), its adjusted table size is 51282KB, far exceeding the 30031KB adjusted table size of this invention. Furthermore, its training time after four iterations is 241.6516 seconds. This invention achieves the same accuracy as the Leonard method after four iterations, with a training time of 209.4117 seconds.

[0076] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for storing and querying source logs based on deep embedding and single-layer RNN prediction, characterized in that, include: S1 Encoding Compression: The system's underlying audit logs are converted into a source graph containing nodes and edges. During this conversion, deep learning embedding representations are used to encode and map the path class and identifier class attribute fields in the audit logs. In the encoding and mapping process, adaptive dictionary encoding is used to assign encoding prefixes to the path class and identifier class attribute fields, and incremental encoding is used to record numerical attribute fields. S2 Encoding Learning: The encoding vectors of nodes and edges in the source graph are preprocessed and learned using a single-layer recurrent neural network (RNN) unit. S3 Dataset Update: Based on incremental encoding and adaptive mapping, the source graph is locally adjusted when the data is updated. S4 Query Output: The query conditions are mapped to the corresponding attribute encoding prefixes. The single-layer recurrent neural network (RNN) module reconstructs the attribute sequence based on the attribute encoding prefixes and corrects the prediction results by adjusting the table.

2. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: The step S1 of encoding and mapping the path class and identifier class attribute fields in the audit log using deep learning embedding representation includes: generating a high-dimensional embedding vector for each attribute using deep learning embedding representation, and encoding and mapping the attribute fields using the semantic association and frequency of occurrence of the attributes.

3. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 2, characterized in that: The step S1 of assigning encoding prefixes to path class and identifier class attribute fields through adaptive dictionary encoding includes: dynamically adjusting the encoding according to frequency, thereby assigning shared encoding prefixes to attribute fields with similar spatial structure characteristics.

4. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: The incremental encoding method used in step S1 to record numerical attribute fields includes: recording the relative offset between adjacent events instead of storing complete numerical information each time.

5. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: The data preprocessing in step S2 includes: flattening the encoded vectors of nodes and edges in one dimension, and then performing window partitioning according to a pre-defined delimiter. After this process, the embedding representation of the sequence can be formally expressed as the following expression: In the formula, e represents the encoding of each independent node or edge, vector T represents the total length of the sequence, and d represents the dimension of the feature.

6. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 5, characterized in that: The learning described in step S2 includes RNN processing the input sequence through temporal modeling and using a fully connected network to predict the field, specifically as follows: In the formula, W represents the weight matrix, b represents the bias term, and ReLU represents the activation function.

7. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: The local adjustment in step S3 includes: mapping the newly added attribute to the existing coding system; if the attribute has a structure similar to an existing class or category, mapping the newly added attribute to the corresponding category and obtaining a new unique suffix; if there is no similar category, assigning the newly added attribute to a new category and generating a new prefix based on its characteristics.

8. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: Step S4, reconstructing the attribute sequence, includes: when a user initiates a query request for a specific node or event, the query conditions are first mapped to the corresponding attribute encoding prefix, and the prefix is ​​fed into a single-layer recurrent neural network (RNN) unit. The single-layer RNN unit predicts subsequent encoding values ​​field by field according to the sequence generation method based on the historical hidden state and the input query prefix, until a preset termination identifier is encountered, thus completing the reconstruction of the attribute sequence.

9. The source tracing log storage and query method based on deep embedding and single-layer RNN prediction according to claim 1, characterized in that: Step S4, which involves correcting the prediction results using an adjustment table, includes: maintaining a unique identifier for each node or edge and its corresponding true encoding value to construct an adjustment table; using the adjustment table to verify the encoding in the reconstructed attribute sequence; when the encoding in the reconstructed attribute sequence is inconsistent with the encoding recorded in the adjustment table, the true encoding value in the adjustment table is used first for replacement; if the encoding information for a certain position is not recorded in the adjustment table, the encoding result in the reconstructed attribute sequence is used by default as the reliable encoding for that position.

10. A source tracing log storage and query system based on deep embedding and single-layer RNN prediction, characterized in that: The system includes a source graph encoding and compression module based on spatial structure and node attributes, and a source data storage and query module based on recurrent neural network sequence prediction. The source graph encoding and compression module based on spatial structure and node attributes converts the system's underlying audit logs into a source graph containing nodes and edges. During this conversion, deep learning embedding representations are used to encode and map path and identifier attribute fields in the audit logs. Adaptive dictionary encoding is used to assign encoding prefixes to path and identifier attribute fields during the encoding and mapping process, and incremental encoding is used to record numerical attribute fields. Based on incremental encoding and adaptive mapping, the source graph is locally adjusted during data updates. The source data storage and query module based on recurrent neural network sequence prediction preprocesses and learns the encoding vectors of nodes and edges in the source graph using a single-layer recurrent neural network (RNN) unit. The query conditions are mapped to the corresponding attribute encoding prefixes. The single-layer recurrent neural network (RNN) module reconstructs the attribute sequence based on the attribute encoding prefixes and corrects the prediction results by adjusting the table.

Images