Data access method and device of real-time auditing system, equipment and storage medium

Abstract

The invention discloses a data access method and device of a real-time auditing system, equipment and a storage medium, and relates to the field of computers, and the method comprises the following steps: obtaining to-be-stored auditing data of a target real-time auditing system, and determining a target hash value corresponding to the to-be-stored auditing data by using a target hash algorithm to determine a target fragment index corresponding to the to-be-stored auditing data; determining a first target hash fragment based on the target fragment index, and storing the to-be-stored audit data to the first target hash fragment of the target data structure based on a corresponding target storage rule; determining a second target hash fragment from all the hash fragments based on the audit data access request, and accessing the second target hash fragment based on a corresponding target access rule to obtain corresponding target audit data; the target data structure is a Map and Vector mixed storage structure constructed based on a storage rule list. According to the method and the device, quick access to the data stored in the data structure in a high-concurrency environment is realized.

Description

Technical Field

[0001] This invention relates to the field of computers, and in particular to data access methods, apparatus, devices, and storage media for real-time auditing systems. Background Technology

[0002] In modern high-concurrency systems, the efficiency of data structure access directly impacts overall system performance. Traditional data access methods face the following challenges: When implementing concurrency control mechanisms, if a global lock is used, all threads share a single lock, easily leading to severe serialization bottlenecks; while a read-write lock mechanism allows multiple read operations to run concurrently, write operations still require exclusive access. Furthermore, common data structures used to store data are often single data structures like Vector or Map, which cannot simultaneously meet the requirements of sequential access and deduplication.

[0003] In summary, how to achieve fast access to data stored in a data structure under high concurrency and meet various business needs is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data access method, apparatus, device, and storage medium for a real-time auditing system, enabling fast access to data stored in a data structure under high concurrency environments. The specific solution is as follows:

[0005] In a first aspect, this application discloses a data access method for a real-time auditing system, including:

[0006] The system retrieves audit data to be stored from memory, uses a target hash algorithm to determine the target hash value corresponding to the audit data to be stored, and determines the target shard index corresponding to the audit data to be stored based on the target hash value.

[0007] Based on the target shard index, the first target hash shard is determined from all available hash shards corresponding to the current business scenario in the target real-time audit system, and the target storage rule corresponding to the first target hash shard is determined based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule;

[0008] Obtain an audit data access request, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access request, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request;

[0009] The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid storage structure of Map and Vector built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure.

[0010] Optionally, before determining the first target hash shard from all available hash shards corresponding to the current business scenario based on the target shard index, the method further includes:

[0011] Based on business configuration data, target multi-dimensional features are extracted from audit log data to determine the current business scenario; the target multi-dimensional features include policy ID, rule ID, source IP, destination IP, account information, and protocol type.

[0012] Optionally, saving the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule includes:

[0013] If the first target hash shard is a first type of target hash shard, then the audit data to be stored is sequentially saved to the target storage queue based on the timestamp data corresponding to the audit data to be stored, and the audit data to be stored is saved to the first target hash shard of the target data structure based on the position of the audit data to be stored in the target storage queue.

[0014] If the first target hash shard is a second type of target hash shard, then the audit data to be stored is stored in the first target hash shard of the target data structure in the form of key-value pairs.

[0015] Optionally, if the first target hash shard is a second type of target hash shard, then storing the audit data to be stored in the form of key-value pairs to the first target hash shard of the target data structure includes:

[0016] If the first target hash shard is a second type of target hash shard, then the target key length threshold corresponding to the current business scenario is determined based on the target multi-dimensional features corresponding to the current business scenario.

[0017] The target key-value pair corresponding to the audit data to be stored is determined. When the key length of the target key-value pair is greater than the target key length threshold, the target key-value pair is compressed using a target key length compression algorithm to obtain the compressed target key-value pair. The compressed target key-value pair is then stored in the first target hash partition of the target data structure.

[0018] Optionally, accessing the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request includes:

[0019] If the second target hash shard is a first type of target hash shard, then the second target hash shard is accessed based on the target data lookup method to obtain the target audit data corresponding to the audit data access request; the target data lookup method includes a linear search method and a binary search method;

[0020] If the second target hash shard is a second type of target hash shard, then the second target hash shard is accessed directly using the hash value contained in the target access request to obtain the target audit data corresponding to the audit data access request.

[0021] Optionally, after accessing the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request, the method further includes:

[0022] Update the access timestamp data corresponding to the target audit data;

[0023] Correspondingly, the data access method of the real-time auditing system also includes:

[0024] By utilizing a target time window and a target data cleanup mechanism, the accessible data is cleaned based on the access timestamp data corresponding to the accessible data stored in the target data structure.

[0025] Optionally, the step of using a target time window and a target data cleanup mechanism to clean up the accessible data based on the access timestamp data corresponding to the accessible data stored in the target data structure includes:

[0026] Using the target time window and the target data cleanup mechanism, the accessible data is cleaned up based on the access timestamp data corresponding to the accessible data stored in the target data structure and the memory usage rate corresponding to each hash shard of the target data structure.

[0027] Secondly, this application discloses a data access device for a real-time auditing system, comprising:

[0028] The shard index determination module is used to obtain the audit data to be stored from the target real-time audit system from memory, determine the target hash value corresponding to the audit data to be stored using the target hash algorithm, and determine the target shard index corresponding to the audit data to be stored based on the target hash value.

[0029] The input data storage module is used to determine the first target hash shard from all available hash shards corresponding to the current business scenario of the target real-time audit system based on the target shard index, and to determine the target storage rule corresponding to the first target hash shard based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule;

[0030] The access data acquisition module is used to acquire audit data access requests, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access requests, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request.

[0031] The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid storage structure of Map and Vector built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure.

[0032] Thirdly, this application discloses an electronic device, including:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement the aforementioned data access method of the real-time auditing system.

[0035] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data access method of the real-time auditing system.

[0036] In this application, when using a real-time auditing system to access data, the system retrieves audit data to be stored from memory, determines the target hash value corresponding to the audit data using a target hash algorithm, and determines the target shard index corresponding to the audit data based on the target hash value. Based on the target shard index, it determines a first target hash shard from all available hash shards corresponding to the current business scenario of the target real-time auditing system, and determines the target storage rule corresponding to the first target hash shard based on a target rule list. The system then saves the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule. Finally, it obtains an audit data access request based on the... The audit data access request determines a second target hash shard from all hash shards corresponding to the target data structure, determines the target access rule corresponding to the second target hash shard based on the target rule list, and accesses the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request. The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid Map and Vector storage structure constructed based on the storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure. Therefore, this application uses a target hash algorithm to obtain the target shard index corresponding to the audit data to be stored, then uses this target shard index to determine the first target hash shard from all available hash shards of the target real-time audit system in the current business scenario, and stores the audit data to be stored based on the corresponding target storage rules, thereby mapping the data to a specific shard according to the hash value. Upon receiving an audit data access request, the system first determines the corresponding second target hash shard based on the request. Then, it uses the target access rules corresponding to the second target hash shard to access it, thereby obtaining the corresponding target audit data. Each shard has an independent lock; when accessing certain data, only the shard to which that data belongs needs to be locked, without blocking access from other shards. This fine-grained locking mechanism greatly improves the system's concurrent throughput. The hybrid Map and Vector storage structure built based on the storage rule list allows for the selection of the optimal data structure according to the business scenario. Vector guarantees timeliness, while Map provides deduplication functionality, enabling the real-time audit system to simultaneously meet multiple business requirements. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This application discloses a flowchart of a data access method for a real-time auditing system.

[0039] Figure 2 This is a schematic diagram of the data access device structure of a real-time auditing system disclosed in this application;

[0040] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] In modern high-concurrency systems, the efficiency of data structure access directly impacts overall system performance. Traditional data access methods face the following challenges: when implementing concurrency control mechanisms, if a global lock is used, all threads share a single lock, easily leading to severe serialization bottlenecks; while read-write locks allow multiple read operations to run concurrently, write operations still require exclusive access. Furthermore, common data structures used to store data are often single data structures like Vector or Map, which cannot simultaneously meet the requirements of sequential access and deduplication. To address these technical issues, this application discloses a data access method for a real-time auditing system, enabling fast access to data stored in data structures under high-concurrency environments.

[0043] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a data access method for a real-time auditing system, including:

[0044] Step S11: Obtain the audit data to be stored from the target real-time audit system in memory, determine the target hash value corresponding to the audit data to be stored using the target hash algorithm, and determine the target shard index corresponding to the audit data to be stored based on the target hash value.

[0045] In this embodiment, after obtaining the audit data to be stored from the target implementation audit system from memory, the target hash algorithm is used to determine the target hash value corresponding to the audit data to be stored, and then the target shard index corresponding to the audit data to be stored is determined based on the target hash value. The target hash algorithm can be the FNV-1a (Fowler-Noll-Vo hash function variant 1a) fast hash algorithm, the CityHash algorithm, or the xxHash algorithm.

[0046] In one specific implementation, if the FNV-1a fast hash algorithm is determined as the target hash algorithm, the specific steps for determining the target hash value include:

[0047] 1. Initialize the hash value. Set an initial hash base value (offset_basis), which is usually 14695981039346656037 (64-bit).

[0048] 2. Iterate through each byte of the input data. For each byte of the input string or data, perform the following operations:

[0049] a. XOR operation: Perform an XOR operation (XOR) on the lower 8 bits of the current hash value and the value of the current byte.

[0050] b. Multiplication operation: Multiply the result of the XOR operation by a specific prime number (FNV_prime), usually 1099511628211 (64-bit).

[0051] 3. Repeat step 2 until all bytes have been processed.

[0052] 4. Generate the final hash value. The hash value obtained after traversal is the FNV-1a hash value of the data, which is also the target hash value.

[0053] The corresponding code example is shown below:

[0054] uint64_t StatisticRuleMatch::fnv1a_hash(const char* data, size_t len){

[0055] const uint64_t FNV_OFFSET_BASIS = 14695981039346656037ULL;

[0056] const uint64_t FNV_PRIME = 1099511628211ULL;

[0057] uint64_t hash = FNV_OFFSET_BASIS;

[0058] for (size_t i = 0; i < len; ++i) {

[0059] hash ^= (uint64_t)(unsigned char)data[i];

[0060] hash *= FNV_PRIME;

[0061] Through the above steps, the number of hash shards can be limited to the number of CPU cores × 2 - 4, thus dynamically adapting to hardware configuration. The generated target hash value is moduloed by the total number of shards (SHARD_COUNT) (Hash Value % SHARD_COUNT), and the result is the shard index to which the audit data to be stored belongs. This distributes the audit data evenly across the shards. By tightly coupling the FNV-1a calculation process with the shard lock selection, the calculated hash value is directly mapped to the shard index, stabilizing the time complexity of data location at O(1). Based on this, each hash shard is independently locked, effectively reducing lock contention. Simultaneously, the FNV-1a algorithm's computational efficiency is approximately 80% higher than MD5, significantly improving sharding efficiency. Furthermore, for string-type keys (such as IP addresses and rule content), the high hashing characteristics of the FNV-1a algorithm solve the "hotspot sharding" problem that traditional modulo sharding easily generates under consecutive IP segments or similar rule content.

[0062] In one specific implementation, to address the lock contention bottleneck in a multi-threaded environment, this embodiment employs a fixed-slot sharding concurrency architecture for the target real-time auditing system. Specifically, a fixed number (e.g., 1000) of statistical slots (statisticInfoVec) are pre-allocated during system initialization. Each slot independently maintains a mutex, forming a pre-allocated sharding pool. Data streams are distributed across different slots via hash mapping. Data read and write operations on different slots are completely parallel and do not interfere with each other. When accessing specific data, only the shard to which that data belongs needs to be locked, without blocking access to other shards. This fine-grained concurrency control design based on hash sharding reduces the probability of global lock contention by three orders of magnitude, achieving near-lockless high concurrency performance and significantly improving the system's concurrent throughput.

[0063] Step S12: Based on the target shard index, determine the first target hash shard from all available hash shards corresponding to the current business scenario of the target real-time audit system, and determine the target storage rule corresponding to the first target hash shard based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule.

[0064] In this embodiment, Vector (time-series) or Map (deduplication) can be intelligently selected based on the current business scenario. Available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid Map and Vector storage structure built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure. It can be understood that the first type of hash shard is used to store data in time-series business scenarios, and the second type of hash shard is used to store data in deduplication business scenarios. Before determining the first target hash shard from all available hash shards corresponding to the current business scenario based on the target shard index in the target real-time audit system, the method further includes: extracting target multi-dimensional features from the audit log data based on business configuration data to determine the current business scenario based on the target multi-dimensional features; the target multi-dimensional features include policy ID, rule ID, source IP, destination IP, account information, and protocol type. In other words, when processing high-concurrency audit data, the system does not simply use a single field as an index. Instead, it constructs a context-aware multi-dimensional key-value generation strategy. Based on the business configuration (MatchStatisticConfig), it dynamically extracts multi-dimensional features from the audit logs (record_info), including policy ID, rule ID, source / destination IP, account information, protocol type (proto_app), and rule-specific content, and dynamically combines them to determine the current business scenario. In one specific implementation, the system internally adopts a dual hybrid storage structure of "Map+Vector" and "Map+Map," adaptively switching according to the business scenario. For time-series / streaming business, an "outer Map index + inner Vector storage" structure is used. The outer Map quickly locates statistical objects, while the inner Vector records event streams (statisticVec) in chronological order, supporting sliding statistics based on time windows. For classification / deduplication business, an "outer Map index + inner Map aggregation" structure is used. The inner Map (statisticMap) is used to quickly deduplicate and count specific dimensions (such as file MD5, data tags) to avoid duplicate calculations. This adaptive data structure ensures the system's functional completeness, allowing a single system to simultaneously meet multiple business needs. A code example of the target data structure using the selection mechanism is shown below:

[0065] / / Enumeration optimization to avoid string comparison

[0066] if (m_matchTypeEnum == MatchTypeEnum::FREQUENCY_VISIT) {

[0067] / / Vector structure: suitable for time series statistics

[0068] iter->second.statisticQueue.push_back(new_hold_data);

[0069] } else if (m_matchTypeEnum == MatchTypeEnum::DATA_TAG_CLASS) {

[0070] / / Map structure: suitable for deduplication

[0071] mapInfo.statisticMap[itemValue.c_str()] = _stHoldDataStatistic(config);

[0072] };

[0073] For example, a deque is used to implement sliding window statistics for frequency access scenarios, a map is used for data label classification scenarios for deduplication and classification, and a map with MD5 as the key can be used for file transfer statistics. It should be noted that the system automatically switches storage strategies at runtime based on the configured rules. For order-sensitive or full-scan scenarios, it automatically downgrades to a vector, while for random lookup scenarios, it upgrades to a map.

[0074] In this embodiment, after determining the current business scenario, the first target hash shard can be determined from all available hash shards corresponding to the current business scenario in the target real-time audit system based on the target shard index, and the target storage rule corresponding to the first target hash shard can be determined based on the target rule list. Saving the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule includes: if the first target hash shard is a first type of target hash shard, then the audit data to be stored is sequentially saved to the target storage queue based on the timestamp data corresponding to the audit data to be stored, and the audit data to be stored is saved to the first target hash shard of the target data structure based on the position of the audit data to be stored in the target storage queue; if the first target hash shard is a second type of target hash shard, then the audit data to be stored is stored in the first target hash shard of the target data structure in the form of key-value pairs. In a specific implementation, the system first defines a rule information structure, including the rule name and rule content. The rule content can be stored using a vector or a map, depending on the type. If the rule type is Vector, the data is stored sequentially, suitable for scenarios requiring preservation of insertion order or traversal. If the rule type is Map, the data is stored as key-value pairs, suitable for scenarios requiring fast lookup and deduplication. The target rule list is a list of rules maintained by the system in the statistics manager. When adding a rule, the corresponding storage structure is constructed based on the passed rule information. When checking rules, different search strategies are adopted according to the rule type: for Vector type, linear search or binary search is used to determine if the content exists; for Map type, hash lookup is used directly to locate the content. A specific code example is as follows:

[0075] string StatisticRuleMatch::GetOptimizedVDoingMapKey(struct record_info *ri, const MatchStatisticConfig &config) {

[0076] string key;

[0077] key.reserve(256); / / Pre-allocate memory to reduce reallocation

[0078] / / Use enumeration comparison instead of string comparison

[0079] if (m_ipMethodEnum == IpMethodEnum::ALL ||

[0080] m_ipMethodEnum == IpMethodEnum::IP_DICT) {

[0081] key += ri->sip;

[0082] }

[0083] / / Using empty() for comparison is more efficient than != "".

[0084] if (!config.strRuleContent.empty()) {

[0085] key += "_" + config.strRuleContent;

[0086] }

[0087] getFastHashString(key); / / Fast hash

[0088] return key;

[0089] };

[0090] It is understandable that using enumeration instead of string comparison can significantly improve the efficiency of conditional judgment. In this embodiment, if the first target hash shard is a second type of target hash shard, the audit data to be stored is stored in the first target hash shard of the target data structure in the form of key-value pairs. This includes: if the first target hash shard is a second type of target hash shard, determining the target key-value length threshold corresponding to the current business scenario based on the target multi-dimensional features corresponding to the current business scenario; determining the target key-value pairs corresponding to the audit data to be stored, and when the key-value length of the target key-value pairs is greater than the target key-value length threshold, compressing the target key-value pairs using a target key-value length compression algorithm to obtain compressed target key-value pairs, and storing the compressed target key-value pairs in the first target hash shard of the target data structure. That is to say, in this embodiment, the system has a built-in length detection mechanism for the generated composite key. When the key-value length exceeds a preset threshold (such as 16 bytes), the MD5 digest algorithm is automatically enabled to compress the long string into a fixed-length fingerprint, thereby achieving intelligent compression and significantly reducing memory usage and hash calculation overhead while ensuring uniqueness.

[0091] Step S13: Obtain an audit data access request, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access request, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request.

[0092] In this embodiment, after obtaining the audit data access request, the second target hash fragment to be accessed can be determined from all hash fragments corresponding to the target data structure based on the audit data access request. Then, the target access rule corresponding to the second target hash fragment can be determined based on the target rule list, and the second target hash fragment can be accessed based on the target access rule to obtain the target audit data corresponding to the audit data access request. Specifically, if the second target hash fragment is a first type of target hash fragment, the second target hash fragment is accessed based on the target data lookup method to obtain the target audit data corresponding to the audit data access request; the target data lookup method includes a linear search method and a binary search method; if the second target hash fragment is a second type of target hash fragment, the second target hash fragment is directly accessed using the hash value contained in the target access request to obtain the target audit data corresponding to the audit data access request.

[0093] In this embodiment, after accessing the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request, it may further include: updating the access timestamp data corresponding to the target audit data. It is understood that this embodiment may also utilize a target time window and a target data cleanup mechanism to clean up the accessible data based on the access timestamp data corresponding to the accessible data stored in the target data structure. For example, using a target time window and a target data cleanup mechanism, based on the access timestamp data corresponding to the accessible data stored in the target data structure, and the memory occupancy rate corresponding to each hash shard of the target data structure, the accessible data is cleaned up. The target data cleanup mechanism in this embodiment is a dynamic memory cleanup mechanism, not just a simple timed cleanup, but a dual cleanup mechanism combining an LRU (Least Recently Used) caching strategy and a time window, specifically including:

[0094] 1. Access timestamp update: Whenever data is accessed (read or updated), the system will update the timestamp of the data to the current time in real time to ensure that active data is not accidentally deleted.

[0095] 2. Dual Threshold Trigger: The cleanup operation is triggered not only by a fixed time interval (CLEAN_INTERVAL), but also by monitoring the memory usage of the current fragment. When memory usage exceeds a preset warning threshold, the cleanup process will be forcibly started, even if the scheduled cleanup time has not yet arrived.

[0096] 3. Incremental Cleanup: To avoid system stoppages caused by cleaning up large amounts of data at once, this invention employs an incremental cleanup strategy. Each cleanup operation processes only a small portion of the oldest data or scans only a subset of shards. Overall memory reclamation is achieved through multiple small cleanup operations, which are performed independently within each shard. Cleanup operations on different shards do not interfere with each other, thus ensuring stable system response latency under high concurrency.

[0097] 4. Intelligent Expiration Detection: During cleanup, the system not only checks if data has exceeded its maximum lifespan (EXPIRE_TIME), but also considers the data's access frequency. Data with extremely low access frequency will be prioritized for removal even if it hasn't expired, especially when memory is low.

[0098] This refined memory management strategy is significantly different from the traditional periodic full cleanup, effectively solving the memory jitter problem in high-concurrency scenarios.

[0099] In this embodiment, a strategy combining "write-on-demand checking" and "periodic scanning" is adopted for memory management. Write-on-demand defense means that before each data insertion, the system checks the current memory level through a global memory limiter (behav_1_hour). Once the threshold is reached, a circuit breaker or log recording is immediately triggered to prevent memory overflow. When accessing data nodes, their timestamps are checked. If data timeout (TTL) is found, it is immediately removed from memory, achieving lazy expiration cleanup. A background thread periodically traverses all shards, clearing "zombie data" that has been inactive for a long time, and ensures the cyclical use of memory resources through periodic deep scanning. In one specific implementation, expired data is cleaned up every 120 seconds using a time window, and half of the least used data is cleaned up when the cache is full using an LRU strategy. Memory thresholds are used to control the memory level, such as a maximum cache of 10,000 records. A specific code example is as follows:

[0100] void StatisticRuleMatch::getFastHashString(string &strKey) {

[0101] / / Check cache

[0102] auto cache_it = m_hashCache.find(strKey);

[0103] if (cache_it != m_hashCache.end()) {

[0104] strKey = cache_it->second;

[0105] return;

[0106] }

[0107] / / LRU cleanup policy

[0108] if (m_hashCache.size() >= MAX_CACHE_SIZE) {

[0109] auto it = m_hashCache.begin();

[0110] for (size_t i = 0; i < MAX_CACHE_SIZE / 2 && it != m_hashCache.end(); ++i) {

[0111] it = m_hashCache.erase(it);

[0112] }

[0113] }

[0114] };

[0115] Understandably, by combining the LRU caching strategy with time-window cleanup, expired data can be promptly cleaned up, memory usage can be controlled, thereby preventing memory leaks and improving system stability. Furthermore, this embodiment can also implement memory management through reference counting and pre-allocated memory pools.

[0116] As can be seen, this application utilizes a target hash algorithm to obtain the target shard index corresponding to the audit data to be stored. Then, using this target shard index, it determines the first target hash shard from all available hash shards in the current business scenario's target real-time audit system. Based on the corresponding target storage rules, it stores the audit data to be stored, thus mapping the data to a specific shard according to the hash value. Upon receiving an audit data access request, it first determines the corresponding second target hash shard based on the request, and then uses the target access rules corresponding to the second target hash shard to access it, thereby obtaining the corresponding target audit data. Each shard has an independent lock; when accessing certain data, only the shard to which that data belongs needs to be locked, without blocking access to other shards. This fine-grained locking mechanism greatly improves the system's concurrent throughput. The hybrid Map and Vector storage structure built based on the storage rule list allows for the selection of the optimal data structure according to the business scenario. Vector guarantees timeliness, while Map provides deduplication functionality, enabling the target real-time audit system to simultaneously meet multiple business requirements.

[0117] See Figure 2 As shown, this application discloses a data access device for a real-time auditing system, comprising:

[0118] The shard index determination module 11 is used to obtain the audit data to be stored from the target real-time audit system from memory, determine the target hash value corresponding to the audit data to be stored using the target hash algorithm, and determine the target shard index corresponding to the audit data to be stored based on the target hash value.

[0119] The input data storage module 12 is used to determine the first target hash shard from all available hash shards corresponding to the current business scenario of the target real-time audit system based on the target shard index, and to determine the target storage rule corresponding to the first target hash shard based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule;

[0120] Access data acquisition module 13 is used to acquire audit data access request, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access request, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request;

[0121] The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid storage structure of Map and Vector built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure.

[0122] As can be seen, this application utilizes a target hash algorithm to obtain the target shard index corresponding to the audit data to be stored. Then, using this target shard index, it determines the first target hash shard from all available hash shards in the current business scenario's target real-time audit system. Based on the corresponding target storage rules, it stores the audit data to be stored, thus mapping the data to a specific shard according to the hash value. Upon receiving an audit data access request, it first determines the corresponding second target hash shard based on the request, and then uses the target access rules corresponding to the second target hash shard to access it, thereby obtaining the corresponding target audit data. Each shard has an independent lock; when accessing certain data, only the shard to which that data belongs needs to be locked, without blocking access to other shards. This fine-grained locking mechanism greatly improves the system's concurrent throughput. The hybrid Map and Vector storage structure built based on the storage rule list allows for the selection of the optimal data structure according to the business scenario. Vector guarantees timeliness, while Map provides deduplication functionality, enabling the target real-time audit system to simultaneously meet multiple business requirements.

[0123] In one specific embodiment, the device may further include:

[0124] The business scenario determination module is used to extract target multi-dimensional features from audit log data based on business configuration data, so as to determine the current business scenario based on the target multi-dimensional features; the target multi-dimensional features include policy ID, rule ID, source IP, destination IP, account information, and protocol type.

[0125] In one specific embodiment, the input data storage module 12 may include:

[0126] The first data storage submodule is used to, if the first target hash shard is a first type of target hash shard, sequentially save the audit data to be stored to the target storage queue based on the timestamp data corresponding to the audit data to be stored, and save the audit data to be stored to the first target hash shard of the target data structure based on the position of the audit data to be stored in the target storage queue.

[0127] The second data storage submodule is used to store the audit data to be stored in the form of key-value pairs in the first target hash partition of the target data structure if the first target hash partition is a second type of target hash partition.

[0128] In one specific implementation, the second data storage submodule may specifically include:

[0129] The length threshold determination unit is used to determine the target key length threshold corresponding to the current business scenario based on the target multi-dimensional features corresponding to the current business scenario if the first target hash partition is a second type of target hash partition.

[0130] The key-value pair storage unit is used to determine the target key-value pair corresponding to the audit data to be stored, and when the key length corresponding to the target key-value pair is greater than the target key length threshold, to compress the target key-value pair using a target key length compression algorithm to obtain the compressed target key-value pair, and to store the compressed target key-value pair in the first target hash partition of the target data structure.

[0131] In one specific embodiment, the access data acquisition module 13 may include:

[0132] The first data acquisition submodule is used to access the second target hash shard based on the target data lookup method if the second target hash shard is a first type of target hash shard, so as to obtain the target audit data corresponding to the audit data access request; the target data lookup method includes a linear search method and a binary search method;

[0133] The second data acquisition submodule is used to directly access the second target hash shard using the hash value contained in the target access request if the second target hash shard is a second type of target hash shard, so as to obtain the target audit data corresponding to the audit data access request.

[0134] In one specific embodiment, the device may further include:

[0135] The timestamp update module is used to update the access timestamp data corresponding to the target audit data;

[0136] Accordingly, the device may further include:

[0137] The data cleaning module is used to clean the accessible data based on the access timestamp data corresponding to the accessible data stored in the target data structure, using a target time window and a target data cleaning mechanism.

[0138] In one specific implementation, the data cleaning module may include:

[0139] The data cleaning unit is used to clean the accessible data based on the access timestamp data corresponding to the accessible data stored in the target data structure and the memory occupancy rate corresponding to each hash shard of the target data structure, using the target time window and the target data cleaning mechanism.

[0140] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0141] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data access method of the real-time auditing system disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0142] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0143] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0144] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data access method of the real-time auditing system executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0145] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the data access method of the aforementioned real-time auditing system. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0147] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0149] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data access method for a real-time auditing system, characterized in that, include: The system retrieves audit data to be stored from memory, uses a target hash algorithm to determine the target hash value corresponding to the audit data to be stored, and determines the target shard index corresponding to the audit data to be stored based on the target hash value. Based on the target shard index, the first target hash shard is determined from all available hash shards corresponding to the current business scenario in the target real-time audit system, and the target storage rule corresponding to the first target hash shard is determined based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule; Obtain an audit data access request, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access request, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request; The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid storage structure of Map and Vector built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure.

2. The data access method of the real-time auditing system according to claim 1, characterized in that, Before determining the first target hash shard from all available hash shards corresponding to the current business scenario in the target real-time audit system based on the target shard index, the method further includes: Based on business configuration data, target multi-dimensional features are extracted from audit log data to determine the current business scenario; the target multi-dimensional features include policy ID, rule ID, source IP, destination IP, account information, and protocol type.

3. The data access method of the real-time auditing system according to claim 1, characterized in that, The step of saving the audit data to be stored to the first target hash partition of the target data structure based on the target storage rule includes: If the first target hash shard is a first type of target hash shard, then the audit data to be stored is sequentially saved to the target storage queue based on the timestamp data corresponding to the audit data to be stored, and the audit data to be stored is saved to the first target hash shard of the target data structure based on the position of the audit data to be stored in the target storage queue. If the first target hash shard is a second type of target hash shard, then the audit data to be stored is stored in the first target hash shard of the target data structure in the form of key-value pairs.

4. The data access method of the real-time auditing system according to claim 3, characterized in that, If the first target hash partition is a second type of target hash partition, then storing the audit data to be stored in the first target hash partition of the target data structure in the form of key-value pairs includes: If the first target hash shard is a second type of target hash shard, then the target key length threshold corresponding to the current business scenario is determined based on the target multi-dimensional features corresponding to the current business scenario. The target key-value pair corresponding to the audit data to be stored is determined. When the key length of the target key-value pair is greater than the target key length threshold, the target key-value pair is compressed using a target key length compression algorithm to obtain the compressed target key-value pair. The compressed target key-value pair is then stored in the first target hash partition of the target data structure.

5. The data access method of the real-time auditing system according to claim 3, characterized in that, The step of accessing the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request includes: If the second target hash shard is a first type of target hash shard, then the second target hash shard is accessed based on the target data lookup method to obtain the target audit data corresponding to the audit data access request; the target data lookup method includes a linear search method and a binary search method; If the second target hash shard is a second type of target hash shard, then the second target hash shard is accessed directly using the hash value contained in the target access request to obtain the target audit data corresponding to the audit data access request.

6. The data access method of the real-time auditing system according to claim 1, characterized in that, After accessing the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request, the method further includes: Update the access timestamp data corresponding to the target audit data; Correspondingly, the data access method of the real-time auditing system also includes: By utilizing a target time window and a target data cleanup mechanism, the accessible data is cleaned based on the access timestamp data corresponding to the accessible data stored in the target data structure.

7. The data access method of the real-time auditing system according to claim 6, characterized in that, The step of using a target time window and target data cleanup mechanism to clean up the accessible data based on the access timestamp data corresponding to the accessible data stored in the target data structure includes: Using the target time window and the target data cleanup mechanism, the accessible data is cleaned up based on the access timestamp data corresponding to the accessible data stored in the target data structure and the memory usage rate corresponding to each hash shard of the target data structure.

8. A data access device for a real-time auditing system, characterized in that, include: The shard index determination module is used to obtain the audit data to be stored from the target real-time audit system from memory, determine the target hash value corresponding to the audit data to be stored using the target hash algorithm, and determine the target shard index corresponding to the audit data to be stored based on the target hash value. The input data storage module is used to determine the first target hash shard from all available hash shards corresponding to the current business scenario of the target real-time audit system based on the target shard index, and to determine the target storage rule corresponding to the first target hash shard based on the target rule list, so as to save the audit data to be stored to the first target hash shard of the target data structure based on the target storage rule; The access data acquisition module is used to acquire audit data access requests, determine a second target hash shard from all hash shards corresponding to the target data structure based on the audit data access requests, determine the target access rule corresponding to the second target hash shard based on the target rule list, and access the second target hash shard based on the target access rule to obtain the target audit data corresponding to the audit data access request. The available hash shards include a first type of hash shard and a second type of hash shard. The target data structure is a hybrid storage structure of Map and Vector built based on a storage rule list. The first type of hash shard corresponds to the Vector storage structure in the target data structure, and the second type of hash shard corresponds to the Map storage structure in the target data structure.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data access method of the real-time auditing system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the data access method of the real-time auditing system as described in any one of claims 1 to 7.

Images