A structured storage method of cyber security situation awareness log data

Abstract

The application relates to the field of network security and data storage technology, in particular to a network security situation awareness log data structured storage method; the method comprises the following steps: acquiring heterogeneous log data to be processed and determining a corresponding underlying storage system, extracting feature information, acquiring generated structured data table quantity information and table interconnection complexity information, inputting a preset evaluation model to generate a storage mode fragmentation index; when the index is lower than a danger threshold, first-level structured analysis is carried out to generate first structured data; when the index is greater than or equal to the danger threshold, second-level structured analysis is carried out to generate second structured data; the first structured data or the second structured data is subjected to cryptography signature and hash calculation to generate target solidified data, and the target solidified data is written into the underlying storage system; the application considers the query availability and the cryptography credibility, and reduces the table structure fragmentation risk.

Description

Technical Field

[0001] This invention relates to the fields of network security and data storage technology, specifically a structured storage method for network security situation awareness log data. Background Technology

[0002] Currently, log data storage mechanisms in network security situational awareness scenarios are typically implemented using fixed table structure parsing or unified wide table storage. When the accessed heterogeneous logs simultaneously contain alarm events, traffic characteristics, and variant format content reported by various security devices, existing methods struggle to balance fine-grained structured parsing, cross-table join query efficiency, and reliable verification after log data solidification. This can easily lead to increased table structure fragmentation, decreased query response performance, and difficulty in meeting the requirements of subsequent auditing and forensics regarding data integrity and source credibility. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a structured storage method for network security situation awareness log data. Specifically, the technical solution of this invention includes: Obtain the heterogeneous log data to be processed and determine the underlying storage system corresponding to the heterogeneous log data. The heterogeneous log data includes alarm events and traffic characteristics of network security devices, and the underlying storage system stores structured data tables. Extract the feature information of the heterogeneous log data. Obtain information on the number of structured data tables generated in the underlying storage system and the complexity of inter-table relationships; The quantity information and the complexity information of inter-table associations are input into the preset evaluation model for calculation, generating the current structured burden weight, and based on the structured burden weight, generating the storage mode fragmentation index. When the storage mode fragmentation index is less than the preset danger threshold, the heterogeneous log data is parsed in the first level of structured parsing based on the feature information to generate the first structured data. When the storage mode fragmentation index is greater than or equal to the danger threshold, the heterogeneous log data is subjected to a second-level structured parsing based on the feature information to generate second-level structured data. The second-level structured parsing extracts the key index fields in the heterogeneous log data and retains the original content fragments of the heterogeneous log data to reduce the complexity of inter-table associations. Perform cryptographic signature and hash calculation on the first or second structured data to generate target solidified data; write the target solidified data into the underlying storage system.

[0004] Preferably, the quantity information and inter-table association complexity information are input into a preset evaluation model for calculation to generate the current structured burden weight, including: multiplying the quantity information of the structured data table by a preset single-table maintenance penalty factor to obtain the basic quantity loss value; extracting the number of cross-table hops required for typical threat queries from the inter-table association complexity information, and multiplying the number of cross-table hops by a preset association loss coefficient to obtain the association complexity loss value; and adding the basic quantity loss value and the association complexity loss value to generate the current structured burden weight.

[0005] Preferably, the feature information is mapped into feature vectors with multiple business attribute dimensions. The similarity between the feature vectors and the structure vectors of each table in the preset template library in the corresponding dimensions is calculated, and the table structure with the highest similarity is selected as the corresponding micro data table structure. The heterogeneous log data is split and mapped according to the micro data table structure to generate the first structured data.

[0006] Preferably, the heterogeneous log data is subjected to a second-level structured parsing based on feature information to generate second structured data, including: mapping the heterogeneous log data to a preset generalized wide table structure, wherein the generalized wide table structure includes predefined stable common columns and extended field blocks for accommodating low-frequency extended content, generating semi-structured data blocks; extracting the key index fields of the semi-structured data blocks as key index fields of the heterogeneous log data, generating second structured data.

[0007] Preferably, a preset cryptographic algorithm is used to perform digital signature processing on the first structured data or the second structured data to generate signature data; a data digest is extracted from the first structured data or the second structured data; a hash operation is performed on the data digest to generate a hash value; and the signature data, hash value, and the first structured data or the second structured data are merged to generate target solidified data containing the signature data and hash value.

[0008] Preferably, after writing the target solidified data into the underlying storage system, the cross-table query latency of the underlying storage system is monitored; if the cross-table query latency is greater than a preset latency threshold, the value of the danger threshold is reduced; if the cross-table query latency is less than or equal to the latency threshold, the value of the danger threshold is kept unchanged.

[0009] Preferably, if the field splitting and mapping of heterogeneous log data according to the micro data table structure fails, the heterogeneous log data is determined to be mutated format data; the mutated format data is redirected to the second-level structured parsing to generate second structured data.

[0010] Preferably, the process involves: obtaining a data retrieval request containing a retrieval hash value; extracting the corresponding target solidified data from the underlying storage system based on the retrieval hash value; performing cryptographic verification on the signature data in the target solidified data; returning the target solidified data if the cryptographic verification passes; and sending a data tampering alarm message if the cryptographic verification fails.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. This application calculates the storage mode fragmentation index by combining the number of tables in the underlying storage system and the complexity of the associations. Based on this index, it dynamically selects either the first-level fine-grained parsing or the second-level degraded parsing with fewer fields. This mechanism can adaptively switch the parsing path according to the database pressure, effectively balancing the need for fine-grained parsing and the efficiency of cross-table join queries, and proactively avoids the problem of decreased query response performance caused by the increased fragmentation of the table structure. 2. This application targets structured data, extracts a digest and performs a hash operation, generates a digital signature using cryptographic algorithms, and integrates the signature, hash value, and original data into the target solidified data for storage. During retrieval, it supports cryptographic verification of the signature data. This mechanism adds proof of source credibility and content integrity to structured logs, fully meeting the business requirements of subsequent network security audits and judicial evidence collection for credible verification of solidified data. 3. This application uses vector similarity matching to pre-set a micro data table structure to suppress disordered table expansion; the second level maps logs to a generalized wide table to preserve the original text and extract key indexes; when the micro table field mapping fails, the system determines it as a mutated format and redirects it to the second level; this closed loop not only achieves high-precision feature extraction, but also includes abnormal logs through a degradation path, ensuring the system's retrieval survivability in a high-adversarial environment; 4. After the data is stored in the database, this application continuously monitors the cross-table query latency of the underlying storage system. When the latency exceeds a preset threshold, it actively lowers the danger threshold that triggers degradation resolution. This mechanism incorporates the query latency business indicator into the storage decision-making closed loop, realizes dynamic adaptive correction of the fragmentation threshold, avoids the system losing responsiveness under high pressure due to fixed static thresholds, and ensures the timeliness of alarms and analysis. Attached Figure Description

[0012] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart illustrating a method for structured storage of network security situation awareness log data, provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0014] A method for structured storage of network security situation awareness log data, the method comprising: Obtain the heterogeneous log data to be processed and determine the underlying storage system corresponding to the heterogeneous log data. The heterogeneous log data includes alarm events and traffic characteristics of network security devices, and the underlying storage system stores structured data tables. Extract the feature information of the heterogeneous log data. Obtain the quantity information and inter-table association complexity information of the structured data tables already generated in the underlying storage system; input the quantity information and inter-table association complexity information into the preset evaluation model for calculation to generate storage mode fragmentation index; When the storage mode fragmentation index is less than the preset danger threshold, the heterogeneous log data is parsed in the first level of structured parsing based on the feature information to generate the first structured data. When the storage mode fragmentation index is greater than or equal to the danger threshold, the heterogeneous log data is subjected to a second-level structured parsing based on the feature information to generate second-level structured data. The second-level structured parsing extracts the key index fields in the heterogeneous log data and retains the original content fragments of the heterogeneous log data to reduce the complexity of inter-table associations. Perform cryptographic signature and hash calculation on the first or second structured data to generate target solidified data; write the target solidified data into the underlying storage system; This embodiment provides a structured storage mechanism for network security situation awareness log data, such as... Figure 1 As shown; specifically, this mechanism is deployed in the national-level financial backbone network security operation center and operates on the heterogeneous log streams continuously reported by core routers, next-generation firewalls, intrusion detection devices, host audit probes, database audit devices, and honeypot nodes; In this scenario, the logs do not come from a single device, but simultaneously contain alarm events, session 5-tuples, traffic direction, byte count changes, protocol behavior fragments, process access traces, and some extended fields after being privately encoded by the device. Therefore, before entering the underlying storage, it is necessary to first determine the underlying storage system that it is compatible with, and then decide whether to use fine-grained parsing or downgraded parsing based on the current database structure load, so as to ensure the credibility of subsequent judicial evidence collection while avoiding the reduction of situational awareness query efficiency due to fragmented table structure. Furthermore, after obtaining the logs to be processed, the underlying storage system to which the batch of logs is to be written can be determined based on the data source identifier, log header format, collection channel type, and access tenant or region information. The underlying storage system here can be a distributed columnar library for high-frequency writing, a document-oriented library for wide-field semi-structured storage, or a hybrid storage pool combining the two. The reason for determining the storage location first is that if the same logs are sent to a library that focuses on fine-grained relational queries, the system pays more attention to the field splitting depth; if they are sent to a library that focuses on preserving the original context, the system pays more attention to write stability and subsequent verifiable evidence collection. Extract feature information; the feature information here is not limited to text keywords, but includes field appearance order, field completeness rate, timestamp reliability, device type label, protocol category, source and destination address pattern, whether there are a large number of unknown key names, whether there are garbled segments, whether there are abnormal length jumps, etc.; these features reflect whether the log format is stable, whether it has a standardized splitting basis, and whether continuing to create an independent table structure for it may introduce additional database structure burden; While extracting log features, the system reads the number of structured data tables and the complexity of inter-table relationships from the underlying storage system to be written to. The number of tables reflects whether the existing structure is already highly dispersed. The complexity of inter-table relationships reflects how many intermediate tables, indexes and connection conditions need to be crossed when splicing threat links later. For situational awareness systems, what truly affects alarm timeliness is not only the write speed of a single table, but more importantly, whether cross-table queries will result in large-scale scanning and multi-hop associations. To this end, these two types of information are input into a preset evaluation model to obtain a storage mode fragmentation index. This index is essentially a real-time representation of whether the database is still suitable for generating new fine structures. To facilitate understanding, a simplified analogy can be presented: Assume there are three types of security tables in the underlying storage, denoted as T1, T2, and T3. T1 and T2 are associated with host identifiers, and T2 and T3 are associated with session identifiers. Querying a lateral movement link requires traversing at least two associations. If new logs then trigger the generation of T4 and T5, and T4 is only applicable to variant format data below a preset threshold, subsequent queries may need to jump from T1 to T2, and then to T4 or T5, lengthening the association path and dispersing index hits. The evaluation model does not simply count the number of tables, but comprehensively considers the engineering fact that the more tables there are and the more interdependent they are, the greater the query latency. When the fragmentation index is below the danger threshold, it indicates that the system is still under control and it is appropriate to use the first level of structured parsing. At this time, the log can be extracted to meet the first preset extraction depth. For example, an intrusion detection log can be split into multiple fields such as device number, collection time, event type, source Internet Protocol address, destination Internet Protocol address, source port, destination port, protocol, threat level, signature number, attack stage, payload length, and associated hostname, so as to facilitate fine-grained retrieval and threat association in the future. When the fragmentation index reaches or exceeds the danger threshold, it indicates that the system has reached or exceeded the preset performance degradation threshold. At this time, the second-level structured parsing is adopted. This path does not pursue complete field restoration, but retains a small number of key indexes, such as time, source Internet Protocol address, destination Internet Protocol address, device type, event category, digest fingerprint, and pointers to the original semi-structured content, in order to reduce the write and association system burden caused by new table expansion and field surge. In other words, the system actively abandons some detail extraction depth at this stage to prioritize the availability and timeliness of threat queries. After completing the first or second structured data, the system performs cryptographic signing and hash calculation on it to form the target solidified data. The signature is used to prove the source of the data and the responsible party for writing it, and the hash is used to prove that the content has not been modified after being written. The solidified data is then written to the underlying storage system, so that the structured result can not only be called by the situational awareness engine, but also has the verifiability required for subsequent auditing and evidence collection. In the anomaly handling process, if the corresponding underlying storage system cannot be identified during log access, the batch of logs can be written to a buffer queue and mapped to a preset isolated storage partition by default according to the device source, and then archived after subsequent identification is completed; if key header fields are found to be completely missing when extracting feature information, at least the acquisition channel identifier, reception time, and original content summary can be retained to avoid logs being directly discarded due to format abnormalities; if the evaluation model is temporarily unavailable, a conservative strategy can be enabled, and second-level structured parsing can be used directly to ensure that the system does not become unstable due to continued expansion of the table structure; if the signature module times out within a preset time, the local pending signature task can be generated first and the corresponding hash value can be frozen, and then transferred to the formal retrieval area after the supplementary signature is completed to avoid confusion between unfixed data and fixed data; During the nighttime clearing window of the financial backbone network, a large number of border firewalls, database auditing devices and virtual private network access gateways simultaneously report logs. The system detects that a batch of logs from the intrusion detection cluster contain rule numbers, attack stages, session information and relatively stable field arrangements. Furthermore, the current database only has fewer topic tables than the preset table number threshold, and the cross-table query latency is still within a controllable range. Therefore, the logs are sent to the first-level structured parsing to form fine-grained records that can be directly used for advanced persistent threat link splicing. Foreign attackers launched a large number of high-concurrency log probes with large-scale format variations. A batch of logs with suspicious origins claimed to come from the probe, but their internal field names changed frequently, the timestamp format was inconsistent, and the number of extended key values ​​exceeded the preset limit. The system synchronously detected a significant increase in the number of newly generated micro-tables in the database and a trend toward more complex cross-table relationships. Therefore, the batch of logs was changed to use second-level structured parsing, extracting only the time, source address, destination address, event tag, and original fragment summary, and writing it into the wide table area after the signature and hash were solidified. In this way, even if the query system is under high pressure, it can still quickly locate suspicious behavior around key indexes, and will not exceed the preset five-minute alarm response time requirement due to the infinite expansion of the table structure. The purpose of this step is to establish a solidified path that can adaptively switch according to the pressure of the database structure in a heterogeneous log environment with strong adversarial, high concurrency and low latency, so that log storage no longer only pursues the parsing accuracy of a single batch, but also takes into account the long-term query availability and cryptographic trustworthiness, thereby achieving proactive management of fragmentation risk.

[0015] Furthermore, the quantity information and inter-table association complexity information are input into a preset evaluation model for calculation to generate the current structured burden weight, including: multiplying the quantity information of the structured data table by a preset single-table maintenance penalty factor to obtain the basic quantity loss value; extracting the number of cross-table hops required for typical threat queries from the inter-table association complexity information, and multiplying the number of cross-table hops by a preset association loss coefficient to obtain the association complexity loss value; and adding the basic quantity loss value and the association complexity loss value to generate the current structured burden weight.

[0016] This embodiment provides an evaluation mechanism for generating storage mode fragmentation indicators. Specifically, in the aforementioned financial backbone network scenario, determining whether to downgrade parsing solely based on whether the number of current tables exceeds a preset threshold is often insufficient to reflect the true risk. This is because some tables, although numerous, are independent of each other, and the cross-table query pressure is not high; while some tables, even if they do not exceed the first preset limit, can cause a rapid decline in query performance due to shared indexes, multi-hop connections, and redundant mappings. Therefore, in this embodiment, a structured burden weight is first generated, and then the fragmentation indicator is obtained by mapping this weight, making the evaluation results closer to the actual operational risks. Furthermore, the structured burden weight reflects the system overhead that the current underlying storage needs to incur to maintain the existing table structure. This system overhead can be understood as: how many additional table definitions, index relationships, join paths, and query routing branches the database needs to maintain as new logs continue to be stored in a fine-grained manner. The more tables there are, the more dispersed the structural boundaries become; the more complex the relationships between tables, the more difficult it is for the situational awareness engine to quickly hit key data when piecing together attack paths. After inputting these two types of information into the evaluation model, the result is not an abstract value for mathematical research, but an engineering weight used for storage scheduling. Specifically, the calculation logic of the evaluation model is as follows: multiply the quantity information of the structured data table by a preset single-table maintenance penalty factor to obtain the basic quantity loss value; extract the number of cross-table jumps required for typical threat queries from the inter-table association complexity information, and multiply it by a preset association loss coefficient to obtain the association complexity loss value; add the basic quantity loss value and the association complexity loss value to generate the current structured burden weight. This weighted, structured decomposition based on engineering factors ensures complete transparency and traceability in the calculation process; as a quantifiable concrete example, the structured burden weights are expressed through a formula. The calculation yields the current structured burden weights. From quantity information Compared with the preset single-table maintenance penalty factor The product of these factors, plus the number of cross-table hops required for a typical threat query extracted from the inter-table join complexity information. Correlation loss coefficient with preset The product of these factors constitutes the system's structure; through this formula, the system can transform the abstract storage structure pressure into specific engineering numerical indicators. As a simplified example: Suppose there are four tables in a certain underlying storage, used for host auditing, network sessions, intrusion events, and authentication, respectively. If a lateral movement path of an advanced persistent threat requires sequential association of host login failure records, abnormal sessions, lateral scans, and privilege escalation alerts, then at least three to four of the four tables are involved. When the number of such links exceeds a preset limit, the jump range of a single query increases, and even if the database write latency is lower than the preset standard, the retrieval end will experience response timeouts. At this point, a database with 4 tables and 3 hops in the main association path can be understood as already having a significant structural burden. If several more mini tables that only serve a few mutation logs are subsequently added, although the write volume of each table is small, the combined effect will further increase the weight of the structural burden. When generating fragmentation metrics based on this weight, a hierarchical mapping approach can be adopted. For example, when the structured burden is low, the fragmentation metric indicates that finer-grained splitting is possible; when the structured burden reaches a medium level, the metric indicates that new table structures need to be suppressed; when the structured burden is high, the metric indicates that query viability should be prioritized. Through this two-stage generation method, the system does not directly use the number of original tables as the sole criterion, but distinguishes between large numbers with high association complexity and large numbers with low association complexity. In the exception handling process, if the system cannot fully obtain the complexity of inter-table relationships in a short time, a temporary burden weight can be generated based on the number of tables, and the result can be marked as a conservative estimate. If the underlying storage is in the initial state of just starting up or the number of tables is less than the preset start threshold, resulting in a low weight, the system can still set a minimum safety boundary to prevent the unrestrained creation of too many temporary tables during the cold start phase. If there are legacy tables that are rarely queried recently, they can be removed from the active relationship statistics to avoid misleading the current fragmentation judgment. After the aforementioned financial backbone network reached a period of high attack incidence, the system discovered that in addition to the several types of topic tables originally used for boundary traffic, host processes, and identity authentication, a number of temporary micro-tables corresponding only to certain unidentified key values ​​appeared, exceeding the preset threshold. Although each of these tables only carried less than the preset capacity threshold of data, multiple tables still needed to be concatenated using user identifiers, session identifiers, and host addresses. Based on this, the evaluation model first formed a high structured burden weight, and then mapped it to a fragmentation index close to the danger zone. At this time, the scheduling module no longer encouraged the generation of new fragmented structures for newly arrived mutated logs, but instead prepared to switch to wide tables or semi-structured storage paths in advance. The purpose of this mechanism is to improve the assessment of database operation risk from simply counting the number of tables to a comprehensive evaluation of the number of tables and the complexity of relationships between tables. This will enable a more accurate characterization of fragmentation trends and prevent the system from becoming severely overloaded before the table structure is completely out of control.

[0017] Furthermore, the heterogeneous log data is subjected to first-level structured parsing based on feature information to generate first-structured data, including: mapping feature information into feature vectors with multiple business attribute dimensions; calculating the similarity between the feature vectors and the structure vectors of each table in the preset template library in the corresponding dimensions; selecting the table structure with the highest similarity as the corresponding micro data table structure; and splitting and mapping the fields of the heterogeneous log data according to the micro data table structure to generate first-structured data.

[0018] This embodiment provides a first-level structured parsing mechanism; specifically, in the aforementioned main scenario, when the underlying storage still has the ability to continue to support fine-grained table structures, the system does not dynamically create a brand new table for each new log, but instead prioritizes finding the closest table structure template in the preset template library and aggregating similar logs into the same type of micro data table structure; this can maintain field precision while suppressing disordered table expansion; Furthermore, the process of mapping log features into feature vectors can be understood as transforming the structural outline of the original log into a comparable format profile. This profile does not require reproducing specific business content, but rather emphasizes which type of structure the log resembles. For example, it may focus on whether it contains a fixed time field, whether there are source and destination address pairs, whether it has a port, whether it has a rule number, the range of the total number of fields, whether the key-value pair arrangement is stable, and whether it often has a device-specific prefix. Each table structure vector in the template library represents a stable storage skeleton, such as firewall session class, host process audit class, database access audit class, or intrusion detection alarm class. To achieve accurate and transparent comparison, the similarity calculation process is as follows: assign a preset business importance weight to each dimension of the feature vector; calculate the deviation of the feature vector from the corresponding table structure vector in each dimension; multiply the deviation of each dimension by the corresponding business importance weight and sum them to obtain the overall weighted distance; generate the final similarity score based on the inverse proportional mapping result of the overall weighted distance; this plaintext calculation rule avoids the risk of uninterpretability introduced by complex algorithms. As a quantifiable concrete example, the calculation process of the overall weighted distance and similarity score is implemented through the formula:

[0019] The total dimension of the feature vector is denoted as . By calculating the eigenvector at the th Values ​​in each dimension The corresponding table structure vector is in the th Values ​​in each dimension The absolute difference, multiplied by the business importance weight assigned to that dimension. Then, the weighted differences of all dimensions are summed to obtain the overall weighted distance. Finally, the reciprocal of the distance plus one is used to generate the final similarity score. The above formula clarifies the complete mapping logic from multidimensional feature differences to single similarity evaluation; As a simplified example: Suppose there are three templates in the current template library. A represents the firewall alarm template, with the fields outlined as time + source Internet Protocol address + destination Internet Protocol address + port + action + policy number; B represents the host audit template, with the fields outlined as time + host identifier + process name + user + operation result. C represents the intrusion detection template, with the field outline being time + source internet address + destination internet address + threat ID + threat level + session digest. A log entry to be processed contains time, source internet address, destination internet address, threat ID, threat level, and a small amount of payload fragments; its structural outline is clearly closer to C. Based on this, the system selects the micro-data table structure corresponding to C, then splits the fields according to this table structure and maps them to the corresponding columns to generate the first structured data. Using a micro-table structure instead of a uniform wide table is beneficial for maintaining high retrieval accuracy when the risk of fragmentation is still low. This is because when similar logs are mapped to similar structures, subsequent conditional filtering of fields such as threat ID, device action, account, and port will be more accurate, and the query optimizer will be more likely to hit the corresponding index. In the exception handling process, if the highest similarity is significantly lower than the preset acceptance threshold, it is not advisable to force the application of a certain template to avoid writing semantic errors in the fields. In this case, the current path can be skipped and the process can be handled by the subsequent degradation parsing path. If two templates have similarity, the template that has been bound to this device type more often in the past can be selected to improve stability. If the corresponding template in the template library has been marked as old by the system and rarely hit, its selection priority can be reduced to avoid historical templates interfering with the convergence of the current structure. During a nighttime attack and defense peak on the financial backbone network, the boundary intrusion detection cluster reported a batch of logs. The fields consistently included alarm time, attack source address, target address, threat signature number, risk level, and hit rule description. Since the current fragmentation index was still below the danger zone, the system mapped these features into a structural profile and compared it with intrusion detection alarm templates, virtual private network authentication templates, and database audit templates in the template library. Finally, the intrusion detection alarm template was selected. The system maps the source address in the original log to the attack source field, the target address to the victim target field, the threat signature number to the rule field, and the risk level to the alarm level field, generating the first structured data that can be directly retrieved by the situation engine. The purpose of this step is to reuse existing structures as much as possible through template matching, rather than constantly creating new structures, thereby achieving a balance between high-precision field extraction and controllable structure growth.

[0020] Furthermore, based on the feature information, a second-level structured parsing is performed on the heterogeneous log data to generate second-structured data. This includes mapping the heterogeneous log data to a preset generalized wide table structure, wherein the generalized wide table structure includes predefined stable common columns and extended field blocks for accommodating low-frequency extended content, generating semi-structured data blocks; extracting the key index fields of the semi-structured data blocks as key index fields of the heterogeneous log data, generating second-structured data.

[0021] This embodiment provides a second-level structured parsing mechanism. Specifically, in the aforementioned scenario, while relying solely on fine-grained template matching is beneficial for accurate queries, it can quickly overload the database when encountering a format-variant log storm. If the database continues to subdivide fields and maintain new templates, it will be overloaded by a large number of micro-structures that only adapt to a few abnormal logs. Therefore, this embodiment introduces a generalized wide table structure to converge complex or unstable logs into a limited storage skeleton, ensuring that the retrieval link is not interrupted due to excessive expansion of the table structure. Furthermore, the generalized wide table structure can be understood as a predefined set of stable common columns, such as reception time, event time, device type, source region, source Internet Protocol address, destination Internet Protocol address, event category, risk label, raw log fragment, digest fingerprint, extended field block, etc. Among them, the extended field block is used to accommodate content that cannot be stably classified or appears infrequently, forming a semi-structured data block. The key to doing this is not to break down all fields, but to ensure that the retrieval entry point most frequently called by the situation engine is always available. As a simplified example of deduction: Suppose that in addition to the time, source Internet Protocol address, and destination Internet Protocol address, a certain mutated log also contains multiple random keys, such as key 17, variable _x, flag _9, etc., and these keys may change immediately in the next batch of logs; if we still design an independent table structure for them, it will continuously generate independent fields with low reusability. When using this approach, the entire log entry can first be encapsulated into a semi-structured data block B1. Stable columns are then filled into the wide table, such as event time = t1, source internet protocol address = a1, destination internet protocol address = b1, and device type = d1. The remaining random key content is then uniformly entered into the extended field block. Key index fields, such as time, address, device type, event tag, and hash digest, are extracted from B1 to form the second structured data. The number of these key fields is less than the number of fields extracted in the first level of parsing, but it is sufficient to support high-priority retrieval and subsequent tracing. From an engineering perspective, this approach is equivalent to shifting from ensuring all fields are indexable to ensuring key fields are indexable and all original text is verifiable when database pressure increases. The former emphasizes parsing completeness, while the latter emphasizes system survivability. For a situational awareness system approaching the five-minute alarm boundary, the latter is more likely to prevent overall failure. In the anomaly handling process, if a log cannot be extracted even if the source and destination Internet Protocol addresses are not available, the receiving time, collection node identifier, device type, and original content summary can still be retained as a minimum index. If the log body is too large to be directly embedded into a wide table, it can be stored in object storage or segmented storage area, and only the external reference address and summary value can be retained in the wide table. If it is subsequently proven that a certain type of variant log has gradually stabilized, it can be migrated from this path back to the fine-grained template path to avoid occupying analysis capacity in a coarse-grained manner for a long time. Following the outbreak of format mutation attacks, a large number of forged probe logs flooded the financial backbone network security center. These logs shared common characteristics: they all contained coarse communication addresses and times, but other field names changed frequently and were mixed with device vendor-specific garbled characters. The system detected that the current fragmentation index had entered a dangerous range, so it stopped trying to generate detailed structures for each of these logs and instead mapped them uniformly to a pre-defined generalized wide table. The wide table stores event time, source address, destination address, device origin, event category, and original semi-structured blocks, while extracting source address, destination address, time window, and digest fingerprint as key indexes. In this way, the situational awareness engine can still quickly retrieve whether a certain source address frequently probes multiple targets within a certain time period, without causing the retrieval to stall due to the infinite expansion of the underlying table structure. The purpose of this step is to use a limited number of field indexes to handle unstable log formats under increased storage pressure and attack interference, thereby achieving a degraded storage effect that prioritizes query efficiency, preserves original evidence, and allows for controllable structural growth.

[0022] Further, cryptographic signing and hash calculation are performed on the first structured data or the second structured data to generate target solidified data, including: digitally signing the first structured data or the second structured data using a preset cryptographic algorithm to generate signature data; extracting a data digest from the first structured data or the second structured data; performing a hash operation on the data digest to generate a hash value; and fusing the signature data, the hash value, and the first structured data or the second structured data to generate target solidified data containing the signature data and the hash value.

[0023] This embodiment provides a log solidification mechanism. Specifically, in the aforementioned scenario, simply writing the structured results into the database is insufficient to meet the audit and evidence collection requirements of the financial backbone network. This is because attackers may attempt to tamper with the data already stored in the database after intrusion, or confuse the chain of responsibility for the event by forging intermediate processing results. Therefore, after the structured data is formed, the system further performs digital signature, digest extraction, and hash calculation, and integrates these results with the structured content to form the target solidified data. Furthermore, digital signatures emphasize who confirmed the writing of the data, binding structured records to trusted signatories to prevent issues where records exist but their origin is unknown. Digest extraction focuses on compressing data into a stable fingerprint, extracting a fixed representation from complete structured content that can be used for integrity verification. Performing a hash operation on the digest further generates a hash value that is easy to retrieve and compare. Finally, binding and storing the signature data, hash value, and original structured result together creates target fixed data that balances readability and verifiability. As a simplified example: Suppose a first piece of structured data is denoted as R1, which contains time, source Internet Protocol address, destination Internet Protocol address, threat level, and rule number; the system first signs R1 to obtain S1; then extracts digest D1 from R1; hashes D1 to obtain H1; and generates a target solidified data P1, which contains R1, S1, and H1; if another second piece of structured data obtained through a degradation path is denoted as R2, although it has fewer fields, it can still generate S2, D2, H2, and form P2 according to the same process; in this way, regardless of whether the data comes from a fine-grained path or a degradation path, it is ultimately stored in the database in a unified trusted solidified format; It should be noted that the structured results are obtained first, followed by signing and hashing, because the verification object required for evidence collection is not the original noise stream of the device, but the structured record that has actually been accepted by the system and is available for retrieval. By solidifying the final stored object, it can be ensured that subsequent queries, exports, and verifications point to the same definite content. During the exception handling process, if the signature service experiences a short-term increase in latency, the structured data can be locked in the security buffer and the digest and hash calculations can be completed. Once the signature is restored, the signature data can be supplemented. During this period, the relevant records can be marked as pending complete solidification and are prohibited from entering the judicial export channel. If an abnormal field encoding is found during the digest extraction process, the field encoding can be uniformly rectified before generating the digest to avoid different verification results due to encoding differences for the same semantic content. If a mismatch between the signature data and the structured content is found before writing, the data entry should be terminated immediately and the processing node alarm should be recorded. During the clearing period of the financial backbone network, the system receives two types of logs: one is host audit records formed through fine-grained parsing, and the other is format mutation detection records formed through downgraded parsing. The system generates signature data and hash values ​​containing complete fields for the former, and performs the same solidification process on the key indexes and semi-structured reference information for the latter. Ultimately, regardless of the source, what is stored in the database is not simply a queryable record, but target solidified data with a signature and hash value. When a security incident is subsequently traced back, the operations and maintenance personnel can directly verify whether the record has remained unchanged since it was entered into the database. The purpose of this mechanism is to add proof of source credibility and content integrity to structured logs, thereby achieving a consistent connection between subsequent situational analysis, audit documentation, and judicial evidence collection.

[0024] Furthermore, the method also includes: after writing the target solidified data into the underlying storage system, monitoring the cross-table query latency of the underlying storage system; if the cross-table query latency is greater than a preset latency threshold, reducing the value of the danger threshold; if the cross-table query latency is less than or equal to the latency threshold, keeping the value of the danger threshold unchanged.

[0025] This embodiment provides an adaptive adjustment mechanism for the danger threshold. Specifically, in the aforementioned scenario, if the danger threshold remains fixed for a long time, the system may exhibit a lag phenomenon: although the fragmentation index has not yet reached the originally set danger line, the cross-table query latency has increased significantly, indicating that the database's tolerance to structural expansion is decreasing. If the original threshold is still used for judgment at this time, the most critical intervention opportunity may be missed. Therefore, this embodiment continuously monitors the cross-table query latency after writing the target fixed data and dynamically lowers the danger threshold accordingly. Furthermore, cross-table query latency is a direct indicator of a situational awareness system's ability to trace a complete attack chain. No matter how well-designed the table structure or how complete the signature mechanism, if the cross-table query latency exceeds a preset threshold when querying a threat chain, the system will still fail to meet the preset alarm timeliness requirements in an adversarial environment. Therefore, using cross-table query latency as a reverse correction measure can make fragmented risk assessment more closely reflect the actual business consequences. Lowering the danger threshold does not mean simply tightening the standards, but rather triggering degraded storage earlier when the database begins to experience performance degradation, thus preventing structural pressure from continuing to rise. As a quantifiable and concrete example, when the current cross-table query latency is monitored... Exceeding the preset delay threshold At that time, through the formula

[0026] The danger threshold is adjusted to be lowered, thus based on the old value of the danger threshold before adjustment. Preset smoothing adjustment coefficient The new, reduced danger threshold value is calculated based on the relative proportion exceeding the delay threshold. By introducing a delay timeout ratio for dynamic adjustment, system oscillations caused by excessive threshold drop are avoided. As a simplified example: Suppose the system originally switched to second-level parsing only when the fragmentation index reached 0.7. However, recently, it was observed that typical three-table join queries changed from stable responses to near-critical waiting times. This indicates that although the database has not yet fully entered the high-fragmentation zone, the remaining safety margin is already very small. At this point, the system appropriately lowers the danger threshold from 0.7 to a more conservative level, such as close to the 0.6 range. In this way, newly arriving logs are more likely to be guided to the degraded parsing path, thereby slowing down the growth rate of newly added micro-tables. If subsequent query delays return to normal, the current threshold is kept unchanged to avoid threshold oscillation. From an engineering perspective, this is equivalent to establishing a feedback control based on query health: the slower the query, the closer the database performance is to the preset performance bottleneck threshold, and the earlier the system should trigger the degradation parsing mechanism; if the query latency does not exceed the preset threshold, it means that the existing structural expansion is still within the range supported by system resources, and the system does not need to reduce the parsing accuracy prematurely. In the anomaly handling process, if there are insufficient statistical samples for query latency, such as a lack of typical related queries during low business hours, the danger threshold can be temporarily left unadjusted to avoid misadjustment due to accidental samples. If the detected latency increase is caused by unstructured factors such as underlying network maintenance or disk failure, the threshold adjustment can be temporarily frozen, and the health diagnosis module can first rule out infrastructure anomalies. If lowering the danger threshold multiple times still cannot suppress the latency deterioration, a more conservative mode can be entered, and second-level structured parsing can be directly applied to more logs. After the format mutation attack lasted for several tens of minutes, although the Financial Backbone Network Security Center was still able to complete log writing, the situational awareness engine's response time began to approach the failure threshold when performing multi-hop queries involving abnormal login, host migration, and database access. Based on this, the system determined that the existing database's tolerance for newly added microstructures had decreased, so it proactively lowered the danger threshold, causing more subsequent logs to enter the wide table or semi-structured index path in advance. After a period of time, the rate of microtable additions decreased, the cross-table query pressure stabilized, and the system was able to continue to complete threat linkage alerts within five minutes. The purpose of this mechanism is to incorporate the final business metric of query latency into the storage decision-making closed loop, thereby achieving dynamic correction of the fragmentation threshold and preventing the system from losing responsiveness under high pressure due to a static threshold that remains unchanged for a long time.

[0027] Furthermore, the method also includes: if field splitting and mapping of heterogeneous log data according to the micro data table structure fails, the heterogeneous log data is determined to be mutated format data; the mutated format data is redirected to the second-level structured parsing to generate second structured data.

[0028] This embodiment provides a fallback mechanism for redirection after field splitting failure. Specifically, in the aforementioned scenario, even if the template matching result seems reasonable, field splitting and mapping may still fail. For example, the log header structure may be similar to the template, but the body may contain vendor-specific encoding, key name distortion, missing fields, reversed order, or abnormal escape characters, causing some columns to be unstable. If the database is forcibly entered according to the established template at this time, misaligned fields, semantic pollution, or even incorrect indexes will occur. Therefore, in this embodiment, when mapping fails, the log is determined to be data with a variant format and redirected to the second-level structured parsing. Furthermore, the so-called field splitting and mapping failure can manifest in several ways: First, key fields cannot be located, for example, the template requires a rule number, but the corresponding position in the original text has been replaced by an unknown code; second, multiple semantic conflicts occur at the same position, for example, the position that should be the destination port is filled with a string label; third, the number of parsed fields deviates significantly from the reasonable range that the template can handle; fourth, the dependencies between fields are not valid, for example, there is a destination Internet Protocol address but no source Internet Protocol address, or there is an operation result but no subject identifier; in these cases, to prevent erroneous structures from entering the fine-grained table, the system determines the log as a variant format and transfers it to the second-level path, which has a higher fault tolerance and is more stable in operation. As a simplified example: Suppose a log entry is determined to be close to template C after similarity comparison. Template C requires the field order to be: time—source Internet Protocol address—destination Internet Protocol address—threat number—threat level. However, during actual splitting, it is found that the original content has become: time—unknown encoded segment—source Internet Protocol address—threat level—extended fragment, with the destination Internet Protocol address missing and the threat number replaced by garbled characters. Even if it is forced to be written, it will result in an empty destination Internet Protocol address and an incorrect threat number, which will pollute subsequent queries. Therefore, the system marks it as a variant format data and no longer continues to write it to the table in a fine-grained manner. Instead, it is transferred to the second level of parsing, retaining only the verifiable time, source Internet Protocol address, risk label, and original fragment. This fault tolerance compensation mechanism solves the boundary problem of the previous implementation method: template matching only has preliminary structural similarity, while truly reliable data entry must be verified by field placement; only by adding a failure redirection mechanism can the template path be prevented from being contaminated by abnormal logs. In the exception handling process, if only non-critical field mapping fails, while critical index fields can be extracted stably, the first-level parsing can continue according to the business strategy, and the failed fields are written to the extended retention area; if the failure occurs on a critical field, the second-level parsing should be switched directly; if a large number of mapping failures occur continuously from the same device source, the priority of that source in the template path can be temporarily reduced to prevent repeated consumption of parsing resources. During a format mutation attack on the financial backbone network, a batch of logs disguised as intrusion detection alerts still closely resembled existing templates in terms of structural profiles, and were therefore initially sent to the template matching process. However, upon further analysis, the system discovered that the rule number field of these logs was frequently replaced with meaningless random strings, and the destination address was often missing. To avoid writing erroneous values ​​into the fine-grained alert table, the system uniformly classified these logs as mutated format data and redirected them to a generalized wide table path, retaining only the time, source address, event category, and original block summary. This prevented erroneous structures from polluting the main analysis table while preserving the most basic clues needed for subsequent review. The purpose of this mechanism is to add a reliable failure exit for fine-grained structured paths, thereby achieving a closed-loop parsing mechanism that is as fine as possible when possible and resorts to a safer degradation approach when finer is not possible.

[0029] Furthermore, the method also includes: obtaining a data retrieval request containing a retrieval hash value; extracting the corresponding target solidified data from the underlying storage system based on the retrieval hash value; performing cryptographic verification on the signature data in the target solidified data; returning the target solidified data if the cryptographic verification passes; and sending a data tampering alarm message if the cryptographic verification fails.

[0030] This embodiment provides a retrieval and verification mechanism for solidified data. Specifically, in the aforementioned scenario, after logs are structured, signed, and hashed, they still need to support rapid retrieval and authenticity verification. Especially after a major security incident occurs in the financial backbone network, the personnel handling the incident often do not perform a fuzzy search based on the original full text, but rather retrieve the corresponding record precisely based on the hash value obtained from the previous analysis of the data chain. Therefore, this embodiment ensures that the returned data is not only findable but also trustworthy through a combination of hash retrieval and signature verification. Furthermore, data retrieval requests can originate from situational awareness platforms, audit review platforms, or judicial export interfaces, and the requests include retrieval hash values. The system uses these hash values ​​to locate the corresponding target fixed data in the underlying storage. Since the hash value corresponds to a unique fingerprint of fixed content, complex condition concatenation can be bypassed to directly find the target record. After finding the record, it does not return it immediately but first performs cryptographic verification on the signature data. If the verification passes, it indicates that the currently returned content is consistent with the content at the time of signing and was indeed generated by an authorized node or authorized module. If the verification fails, it indicates that the data may have been tampered with or replaced during storage, migration, or retrieval, and the system should immediately send a data tampering alarm message. As a simplified example: Suppose an analyst obtains hash value H9 in a suspicious lateral movement link and initiates a search; the system finds a fusion package P9 in the underlying storage, which contains a structured record R9, a signature S9, and a hash H9; S9 is verified using the corresponding verification public key; if the verification is successful, P9 or R9 within it is returned according to the permissions; if the verification fails, it means that although the record location can be located through the hash index, the trust status of the record itself has been compromised, and an alarm is returned instead of directly delivering the content; From a security mechanism perspective, hash retrieval solves the problem of quickly finding a record, while signature verification solves the problem of whether the found record is trustworthy; the combination of the two can prevent the mistaken trust of contaminated data based solely on an index hit. In the exception handling process, if the retrieved hash value exists but the corresponding record has been archived and migrated, the system can first return the archive location information and continue to perform signature verification on the migrated copy; if there are duplicate copies with the same hash value in the underlying storage, the signatures should be verified one by one to determine which one is the trusted master copy; if the signature verification module is temporarily unavailable, the return operation can be frozen and the record can be prompted that it is in a pending verification state to prevent unverified data from entering the evidence chain. In the post-mortem analysis of a database unauthorized access incident on the financial backbone network, the personnel handling the incident obtained a set of suspicious record hash values ​​from the initial alarm chain to locate key audit logs. The system quickly extracted the target hardened data from the underlying storage based on the hash values ​​and verified the signature data within it. For records that passed verification, the system returned them to the forensic terminal to confirm the time, source address, and associated account of the unauthorized access. For records that failed verification, the system immediately triggered a data tampering alarm and included the relevant storage nodes in the scope of further investigation to prevent contaminated data from misleading the characterization of the incident. The purpose of this mechanism is to establish a closed loop for solidified logs, from accurate retrieval to complete verification, thereby ensuring the availability, reliability, and traceability of log data in a highly adversarial environment.

[0031] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for structured storage of network security situation awareness log data, characterized in that, The method includes: Obtain heterogeneous log data to be processed and determine the underlying storage system corresponding to the heterogeneous log data, wherein the heterogeneous log data includes alarm events and traffic characteristics of network security devices, and the underlying storage system stores structured data tables; extract the feature information of the heterogeneous log data; Obtain the quantity information and inter-table association complexity information of the structured data tables already generated in the underlying storage system; The quantity information and the complexity information of inter-table associations are input into a preset evaluation model for calculation to generate the current structured burden weight, and based on the structured burden weight, a storage mode fragmentation index is generated. When the storage mode fragmentation index is less than a preset danger threshold, the heterogeneous log data is subjected to first-level structured parsing based on the feature information to generate first structured data. When the storage mode fragmentation index is greater than or equal to the danger threshold, the heterogeneous log data is subjected to a second-level structured parsing based on the feature information to generate second structured data. The second-level structured parsing extracts key index fields from the heterogeneous log data and retains the original content fragments of the heterogeneous log data to reduce the complexity of the inter-table association. Cryptographic signature and hash calculation are performed on the first structured data or the second structured data to generate target solidified data; the target solidified data is written into the underlying storage system.

2. The method for structured storage of network security situation awareness log data according to claim 1, characterized in that, The step of inputting the quantity information and the inter-table association complexity information into a preset evaluation model for calculation to generate the current structured burden weight includes: multiplying the quantity information of the structured data table by a preset single-table maintenance penalty factor to obtain a basic quantity loss value; extracting the number of cross-table hops required for typical threat queries from the inter-table association complexity information, and multiplying the number of cross-table hops by a preset association loss coefficient to obtain an association complexity loss value; and adding the basic quantity loss value and the association complexity loss value to generate the current structured burden weight.

3. The method for structured storage of network security situation awareness log data according to claim 1, characterized in that, The step of performing a first-level structured parsing of the heterogeneous log data based on the feature information to generate first structured data includes: mapping the feature information into feature vectors with multiple business attribute dimensions; calculating the similarity between the feature vectors and the structure vectors of each table in a preset template library in the corresponding dimensions; selecting the table structure with the highest similarity as the corresponding micro data table structure; and performing field splitting and mapping on the heterogeneous log data according to the micro data table structure to generate the first structured data.

4. The method for structured storage of network security situation awareness log data according to claim 1, characterized in that, The step of performing second-level structured parsing of the heterogeneous log data based on the feature information to generate second structured data includes: mapping the heterogeneous log data to a preset generalized wide table structure, wherein the generalized wide table structure includes predefined stable common columns and extended field blocks for accommodating low-frequency extended content, generating semi-structured data blocks; extracting the key index fields of the semi-structured data blocks as key index fields of the heterogeneous log data, generating the second structured data.

5. A structured storage method for network security situation awareness log data according to claim 1, characterized in that, The step of performing cryptographic signing and hash calculation on the first structured data or the second structured data to generate target solidified data includes: performing digital signature processing on the first structured data or the second structured data using a preset cryptographic algorithm to generate signature data; Extract a data digest from the first structured data or the second structured data; perform a hash operation on the data digest to generate a hash value; merge the signature data, the hash value, and the first structured data or the second structured data to generate target solidified data containing the signature data and the hash value.

6. A structured storage method for network security situation awareness log data according to claim 1, characterized in that, The method further includes: after writing the target solidified data into the underlying storage system, monitoring the cross-table query latency time of the underlying storage system; if the cross-table query latency time is greater than a preset latency threshold, reducing the value of the danger threshold; if the cross-table query latency time is less than or equal to the latency threshold, keeping the value of the danger threshold unchanged.

7. A structured storage method for network security situation awareness log data according to claim 3, characterized in that, The method further includes: if the field splitting and mapping of the heterogeneous log data according to the micro data table structure fails, determining that the heterogeneous log data is mutated format data; redirecting the mutated format data to the second-level structured parsing to generate the second structured data.

8. A structured storage method for network security situation awareness log data according to claim 5, characterized in that, The method further includes: obtaining a data retrieval request containing a retrieval hash value; extracting the corresponding target solidified data from the underlying storage system based on the retrieval hash value; performing cryptographic verification on the signature data in the target solidified data; returning the target solidified data if the cryptographic verification passes; and sending a data tampering alarm message if the cryptographic verification fails.

Images