Dynamic weak password detection method and system based on pre-filtering and layered compression

Abstract

The invention relates to a dynamic weak password detection method and system based on pre-filtering and layered compression, and the method comprises the steps: responding to a received password input by a user, carrying out the pre-filtering of the input password through a global Bloom filter, so as to judge whether the input password is a non-weak password or not; screening out potential weak passwords to perform hierarchical matching on the potential weak passwords, and dynamically updating the weak password library through an incremental updating mechanism; and dynamically adjusting the storage hierarchy of the weak password library based on the password access frequency, and migrating cold data to optimize memory utilization. According to the method, non-weak passwords are quickly screened through the global bloom filter, the matching efficiency of high-frequency to low-frequency weak passwords is optimized in combination with hot, warm and cold layered compression storage architectures, and memory occupation and query delay are remarkably reduced. And shadow fragmentation and atomic switching are adopted in an incremental updating mechanism, so that non-perception dynamic updating is realized, and the real-time performance of the weak password library is ensured. And the performance, the accuracy and the expandability of weak password detection in a concurrent scene are integrally improved.

Description

Technical Field

[0001] The present invention relates to the technical field of weak password detection, and in particular to a dynamic weak password detection method and system based on pre-filtering and layered compression. Background Art

[0002] Weak password detection is a core function of intrusion prevention systems (IPS). It mitigates security risks by matching user-entered passwords against a predefined weak password library. Traditional technologies often use hash tables, Trie trees, or Bloom filters to store weak password libraries. However, as the size of weak password libraries grows exponentially, existing solutions face the following drawbacks:

[0003] Memory expansion problem: Trie trees and hash tables consume huge amounts of memory when storing massive amounts of weak passwords. For example, a password database with tens of millions of passwords requires several GB of memory. Although Bloom filters save space, they cannot support dynamic deletion and have a high false positive rate.

[0004] Matching performance bottleneck: Incremental updates to weak password libraries require global locks to protect data structures, leading to service interruptions or increased latency. Matching performance bottleneck: Traditional solutions require full traversal of data structures, making single-core throughput difficult to exceed 10,000 QPS, unable to meet real-time IPS traffic processing requirements.

[0005] Lack of distribution-aware optimization: In real-world scenarios, most passwords are not weak, with only a small number being weak, and even then, only a small number of weak passwords are concentrated in the high-frequency range. Traditional solutions fail to design layered filtering and compression mechanisms to address this distribution characteristic, resulting in wasted resources. Summary of the Invention

[0006] The purpose of the present invention is to provide a dynamic weak password detection method and system based on pre-filtering and layered compression to solve the problems of memory occupation, update blockage and performance bottleneck in weak password detection in the prior art.

[0007] To achieve one of the above-mentioned objects, an embodiment of the present invention provides a dynamic weak password detection method based on pre-filtering and layered compression, which is characterized by comprising:

[0008] In response to receiving a password input by a user, pre-filtering the input password through a global Bloom filter to determine whether the input password is a non-weak password and screen out potential weak passwords;

[0009] Performing hierarchical matching on potential weak passwords and dynamically updating the weak password library through an incremental update mechanism;

[0010] Dynamically adjust the storage tier of weak password libraries based on password access frequency and migrate cold data to optimize memory utilization.

[0011] As a further improvement of an embodiment of the present invention, the method further includes initializing a hierarchical compression storage architecture including a hot data layer, a warm data layer, and a cold data layer to construct a weak password library, and initializing a global Bloom filter based on the weak password library;

[0012] Initializing the weak password library includes,

[0013] Storing high-frequency weak passwords using a compressed double-array Trie in the hot data layer;

[0014] Storing medium-frequency weak passwords using a sharded Cuckoo Filter array in the warm data layer;

[0015] Storing low-frequency weak passwords using a disk bitmap index combined with an in-memory Bloom filter in the cold data layer;

[0016] Initializing the global Bloom filter includes,

[0017] Configuring the global Bloom filter as a bit array with a preset size, performing hash calculations using multiple independent hash functions, and aligning according to the processor cache line to reduce the cache miss rate.

[0018] As a further improvement of an embodiment of the present invention, the method further includes that the pre-filtering of the input password by the global Bloom filter includes,

[0019] Calculating the hash value of the input password using the multiple independent hash functions, and mapping the hash value to the corresponding position of the bit array of the global Bloom filter;

[0020] If the value of any corresponding position of the bit array is 0, it is determined that the input password is a non-weak password; otherwise, it is determined that the input password is a potential weak password.

[0021] As a further improvement of an embodiment of the present invention, the method further includes that the hierarchical matching of the potential weak password includes,

[0022] Using the hash value of the potential weak password, matching high-frequency weak passwords using a double-stack backtracking algorithm through a compressed double-array Trie in the hot data layer, and allowing character difference matching within a predetermined range through support for fuzzy hashing detection. If the match is successful, it is determined as a weak password;

[0023] If the matching in the hot data layer is not successful, parallelly query medium-frequency weak passwords through a sharded Cuckoo Filter array in the warm data layer using a load balancing strategy. If a matching fingerprint is found, it is determined as a weak password;

[0024] If the warm data layer fails to match successfully, a pre-judgment is made in the cold data layer through a memory Bloom filter, and when the pre-judgment indicates that it may exist, a disk bitmap index is triggered to query for low-frequency weak passwords. If the query hits, it is determined as a weak password; otherwise, it is determined as a non-weak password.

[0025] As a further improvement of an embodiment of the present invention, the method further includes that the dynamic update of the weak password library through the incremental update mechanism includes

[0026] Writing the determined weak passwords into a shadow shard through a background thread, and using differential compression to record the differences between the shadow shard and the main shard;

[0027] Verifying the integrity of the shadow shard data, and after passing the verification, synchronously inserting the hash values of the newly added weak passwords into the bitmap of the global Bloom filter by the background thread;

[0028] Replacing the shadow shard with the main shard through an atomic pointer switch;

[0029] Asynchronously releasing the memory resources of the old main shard to reduce memory fragmentation.

[0030] As a further improvement of an embodiment of the present invention, the method further includes that the dynamic adjustment of the storage level of the weak password library includes

[0031] Using a sliding window algorithm to count the password access frequencies within a preset time period to dynamically divide the hot data layer, warm data layer, and cold data layer;

[0032] Based on the LRU-K algorithm, migrating the low-frequency weak passwords that have not been accessed for more than the preset time period to the compressed bitmap index on the disk to release memory resources.

[0033] As a further improvement of an embodiment of the present invention, the method further includes taking a snapshot of the bitmap status of the global Bloom filter regularly and persisting it to a non-volatile storage medium;

[0034] Recording the hash values of the newly added weak passwords and their corresponding position update operations in a log, and reconstructing the bitmap by replaying the log during system recovery to support the continuous operation of the pre-filtering and hierarchical matching.

[0035] To achieve one of the above invention purposes, an embodiment of the present invention further provides a dynamic weak password detection system based on pre-filtering and hierarchical compression, characterized in that it includes a global Bloom filter, a weak password library, an update module, and a cold data migration module;

[0036] The global Bloom filter is used to receive the hash values of passwords input by users, generate filtering results to screen potential weak passwords, and transfer them to the weak password library;

[0037] The weak password library includes a hot data layer, a warm data layer, and a cold data layer. The hot data layer receives potential weak passwords from the global Bloom filter, matches high-frequency weak passwords through a compressed double-array Trie, and outputs the unmatched passwords to the warm data layer. The warm data layer queries medium-frequency weak passwords in parallel through a sharded Cuckoo Filter array and passes the unmatched passwords to the cold data layer. The cold data layer queries low-frequency weak passwords through an in-memory Bloom filter and a disk bitmap index to output the final matching results.

[0038] The update module is used to process newly added weak passwords from public data sets or real-time collections, generate differential compressed data to update the warm data layer and the global Bloom filter.

[0039] The cold data migration module is used to migrate infrequently accessed weak passwords from the warm data layer to disk storage in the cold data layer according to access frequency statistics to optimize resource utilization.

[0040] To achieve one of the above invention purposes, an embodiment of the present invention further provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the program is executed on the processor, the steps in the above-mentioned dynamic weak password detection method based on pre-filtering and hierarchical compression are implemented.

[0041] To achieve one of the above invention purposes, an embodiment of the present invention further provides a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned dynamic weak password detection method based on pre-filtering and hierarchical compression are implemented.

[0042] Compared with the prior art, the present invention provides a dynamic weak password detection method and system based on pre-filtering and hierarchical compression. By quickly screening non-weak passwords through a global Bloom filter and combining a hot, warm, and cold hierarchical compression storage architecture to optimize the matching efficiency of high-frequency to low-frequency weak passwords, it significantly reduces memory occupancy and query latency. The incremental update mechanism uses shadow sharding and atomic switching to achieve seamless dynamic updates and ensure the real-time nature of the weak password library. The cold data migration based on access frequency further optimizes resource utilization, and combined with snapshot and log recovery, it enhances the system's robustness, overall improving the performance, accuracy, and scalability of weak password detection in high-concurrency scenarios, and is applicable to the fields of network security and identity authentication. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flowchart of the dynamic weak password detection method based on pre-filtering and hierarchical compression described in the present invention.

[0044] Figure 2It is a schematic diagram of a multi-level matching pipeline of the dynamic weak password detection method based on pre-filtering and hierarchical compression according to the present invention.

[0045] Figure 3 It is a flowchart of incremental update of the dynamic weak password detection method based on pre-filtering and hierarchical compression according to the present invention.

[0046] Figure 4 It is a schematic diagram of the architecture of the dynamic weak password detection system based on pre-filtering and hierarchical compression according to the present invention. Detailed implementation manners

[0047] The present invention will be described in detail below in conjunction with the specific implementation manners shown in the accompanying drawings. However, these implementation manners do not limit the present invention, and any structural, methodical, or functional transformation made by those of ordinary skill in the art based on these implementation manners is included within the protection scope of the present invention.

[0048] The following details the implementation manners of the present invention. The examples of the implementation manners are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The implementation manners described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0049] In the first embodiment of the present invention, the present invention provides a dynamic weak password detection method based on pre-filtering and hierarchical compression. As Figure 1 shown, the method includes

[0050] S1: In response to receiving the password input by the user, pre-filter the input password through a global Bloom filter to determine whether the input password is a non-weak password and screen out potential weak passwords;

[0051] S2: Perform hierarchical matching on the potential weak passwords, and dynamically update the weak password library through an incremental update mechanism;

[0052] S3: Dynamically adjust the storage level of the weak password library based on the password access frequency, and migrate cold data to optimize memory utilization.

[0053] In a specific implementation manner of the present invention, a hierarchical compression storage architecture including a hot data layer, a warm data layer, and a cold data layer is initialized to construct a weak password library, and a global Bloom filter is initialized based on the weak password library;

[0054] Initializing the weak password library includes

[0055] Storing high-frequency weak passwords in the hot data layer using a compressed double-array Trie;

[0056] Store medium-frequency weak passwords in the warm data layer using a sharded Cuckoo Filter array;

[0057] Store low-frequency weak passwords in the cold data layer using a disk bitmap index combined with a memory Bloom filter;

[0058] The initialization of the global Bloom filter includes,

[0059] Configure the global Bloom filter as a bit array with a preset size, perform hash calculations using multiple independent hash functions, and align it according to the processor cache line to reduce cache miss rates.

[0060] It should be noted that the weak password library is constructed based on a public dataset and real-time collected user passwords. The storage and query efficiency are optimized through a hierarchical compression storage architecture, which is divided into a hot data layer, a warm data layer, and a cold data layer. High-frequency, medium-frequency, and low-frequency weak passwords are stored according to the password access frequency. The specific implementation is as follows:

[0061] Hot data layer: Use a compressed double-array Trie (CDA-Trie) to store high-frequency weak passwords (the top 20% of access frequencies). The CDA-Trie realizes state transition through a Base array and a Check array. The initial length of the Base array is 1024, which stores the base address of the state and has a dynamic expansion step size that is a power of 2. The length of the Check array is 4096, and each element occupies 12 bits, including an 8-bit character value and a 4-bit state transition address. To optimize memory occupancy, high-frequency nodes use a 4-bit compressed state machine to store the state offset, and low-frequency nodes retain a 12-bit complete state. The matching process uses a double-stack backtracking algorithm. Stack 1 records the Base index of the current matching path, and Stack 2 caches the Check states to be backtracked. If the matching fails, the most recent valid state is popped from Stack 2 and the matching continues, supporting fast and accurate queries and fuzzy hash detection.

[0062] Warm data layer: Use a sharded Cuckoo Filter array to store medium-frequency weak passwords (access frequencies from 20% to 50%). Generate a 64-bit hash value through MurmurHash3, divide it into N shards according to the hash value range, each shard contains 100,000 entries, each shard is configured with 4 hash table buckets, and each bucket stores 2 8-bit fingerprints. The shards support lock-free operations. Insertion and deletion are achieved through CAS (Compare-and-Swap) instructions to achieve shard-level atomicity. When updating, the shard pointer is switched through an atomic pointer to ensure that the business is unaware. Dynamically load hot shards into memory based on the access frequency, and unload cold shards to disk to balance memory and storage efficiency.

[0063] Cold data layer: Disk bitmap index combined with in-memory Bloom filter is used to store low-frequency weak passwords (access frequency lower than 50%). For disk storage, Roaring Bitmap is used to map password hash values to chunked bitmaps, with each chunk containing 2^16 entries. Disk space occupancy is reduced by compressing sparse data, and the storage medium is SSD hard disk to support high-speed random reading. The in-memory Bloom filter is configured as a 1MB bit array, and 3 independent hash functions (FNV1a, Jenkins, CRC32) are used for existence pre-judgment to reduce the false positive rate and unnecessary disk accesses.

[0064] It should be noted that the global Bloom filter is generated based on the weak password library, covering all weak passwords (including those in the hot, warm, and cold data layers), and is used to quickly filter out non-weak passwords, reducing the computational pressure of subsequent hierarchical matching. The specific implementation is as follows:

[0065] The bit array is configured to a preset size, and the hash values of weak passwords are calculated through multiple independent hash functions (including MurmurHash3, FNV1a, CRC32, Jenkins, CityHash) and mapped to the bit array. The number of hash functions is optimized to balance the false positive rate and computational efficiency. The bit array is aligned according to the processor cache line and divided into multiple chunks to reduce cache miss rate and improve access speed. The AVX-512 instruction set is used to implement SIMD (Single Instruction Multiple Data) acceleration to calculate the hash values of multiple passwords in parallel, significantly improving the initialization and filtering throughput. To ensure data persistence and system robustness, the status of the bit array is snapshotted every 5 minutes and persisted to the SSD storage medium, and at the same time, the hash values of newly added weak passwords and their bit position update operations are recorded in the log. When the system restarts or recovers from a failure, the bitmap is reconstructed by replaying the log to ensure data consistency. During the initialization process, the hash values of all weak passwords in the weak password library are calculated one by one and inserted into the bit array to generate a bitmap structure covering the hot, warm, and cold data layers, providing efficient support for subsequent pre-filtering.

[0066] Furthermore, to adapt to high-concurrency scenarios, during the initialization process, the hierarchical storage of the weak password library and the bitmap generation of the Bloom filter are processed in parallel by background threads. The in-memory resident shards of the hot data layer and the warm data layer are preferentially loaded, and the cold data layer data is asynchronously written to the disk. After initialization, the system uses the sliding window algorithm to count the password access frequency and dynamically adjusts the division boundaries of the hot, warm, and cold data layers to ensure that the storage architecture matches the actual access pattern, further optimizing the query performance and resource utilization.

[0067] In a specific embodiment of the present invention, the input password is pre-filtered through the global Bloom filter. Specifically,

[0068] Calculate the hash value of the input password using the multiple independent hash functions, and map the hash value to the corresponding position of the bit array of the global Bloom filter;

[0069] If the value of any corresponding position in the bit array is 0, determine that the input password is a non-weak password; otherwise, determine that the input password is a potential weak password.

[0070] It should be noted that the hash values are calculated for the password input by the user using multiple independent hash functions, generating multiple hash values and mapping each hash value to the corresponding position of the bit array of the global Bloom filter. The bit array of the global Bloom filter is configured with a preset size and stored in memory to ensure high-speed access. The hash calculation process uses the AVX-512 instruction set to achieve SIMD (Single Instruction Multiple Data) acceleration, processing the hash values of multiple passwords in parallel, significantly improving the filtering throughput. To optimize the memory access efficiency, the bit array is aligned according to the processor cache line and divided into multiple blocks to reduce the cache miss rate.

[0071] Further, judge the values of all mapped positions in the bit array: if the value of any corresponding position is 0, determine that the input password is a non-weak password, directly return the security result and terminate the subsequent matching; otherwise, if the values of all mapped positions are 1, determine that the input password is a potential weak password and pass it to the subsequent hierarchical matching process (hot data layer, warm data layer, cold data layer). To ensure the robustness of the filtering, the selection of the hash function is optimized to minimize the collision probability, and the false positive rate is controlled within a preset range.

[0072] Further, to support high-concurrency scenarios, the filtering process is executed in parallel by multiple threads. The tasks are allocated to different CPU cores according to the hash values of the input passwords, and the round-robin load balancing strategy is adopted to avoid thread competition. The cache of the filtering results is stored in the CPU L1 Cache to reduce the memory access latency and improve the real-time performance. In addition, the status of the bit array is regularly checked, persisted to the SSD storage medium through snapshots, and the logs of the filtering operations (including the hash values of the input passwords and the judgment results) are recorded to support the data consistency verification during system failure recovery.

[0073] Further, if a potential weak password enters the subsequent matching, the system will reuse the hash values generated by the pre-filtering and pass them to the compressed double-array Trie in the hot data layer for high-frequency weak password matching, avoiding repeated calculations and improving the overall processing efficiency.

[0074] In a specific embodiment of the present invention, the hierarchical matching of potential weak passwords is specifically as follows

[0075] Using the hash value of the potential weak password, in the hot data layer, the double-stack backtracking algorithm is adopted by the compressed double-array Trie to match high-frequency weak passwords, and fuzzy hashing detection is supported to allow character difference matching within a predetermined range. If the match is successful, it is determined as a weak password;

[0076] If the match in the hot data layer is not successful, in the warm data layer, the sharded Cuckoo Filter array is used to query medium-frequency weak passwords in parallel by the load balancing strategy. If a matching fingerprint is found, it is determined as a weak password;

[0077] If the match in the warm data layer is not successful, in the cold data layer, a memory Bloom filter is used for pre-judgment, and when it is pre-judged that there may be a match, a disk bitmap index is triggered to query low-frequency weak passwords. If the query hits, it is determined as a weak password, otherwise it is determined as a non-weak password.

[0078] It should be noted that the hash value of the potential weak password generated by the global Bloom filter is used to match high-frequency weak passwords in the hot data layer by the double-stack backtracking algorithm with the compressed double-array Trie (CDA-Trie). The CDA-Trie uses a Base array (initial length 1024, storing the base address of the state, and the dynamic expansion step is a power of 2) and a Check array (length 4096, each element is 12 bits, including an 8-bit character value and a 4-bit state transition address) to achieve state transition. To reduce memory occupancy, high-frequency nodes use a 4-bit compressed state machine to store the state offset, and low-frequency nodes retain the 12-bit complete state. The double-stack backtracking algorithm records the Base index of the current matching path in stack 1 and caches the Check state to be backtracked in stack 2. If the current character match fails, the most recent valid state is popped from stack 2 and the matching continues. The time complexity is O(m), where m is the password length. The matching process supports fuzzy hashing detection, allowing character difference matching within a predetermined range, and is implemented through the longest prefix matching and character replacement rules. If the match is successful, the potential weak password is determined as a high-frequency weak password, an alarm is triggered, and subsequent matching is terminated; otherwise, the un-matched potential weak password is passed to the warm data layer.

[0079] Further, if the hot data layer fails to match successfully, the medium-frequency weak passwords are queried in parallel in the warm data layer through the sharded Cuckoo Filter array using a load balancing strategy. The sharded Cuckoo Filter array generates 64-bit hash values based on MurmurHash3 and is divided into N shards according to the hash value range. Each shard contains 4 hash table buckets, and each bucket stores 2 8-bit fingerprints. The query process is executed in parallel through multiple threads. Tasks are assigned to CPU cores according to the hash values, and the round-robin load balancing strategy is used to avoid thread competition. It is checked whether the fingerprints of the target shard exist. If a matching fingerprint is found, the potential weak password is determined to be a medium-frequency weak password, an alarm is triggered, and subsequent matching is terminated; otherwise, the unmatched potential weak password is passed to the cold data layer. To support high concurrency, the sharded query uses the CAS (Compare-and-Swap) instruction to implement a lock-free operation, and the query results are cached in the CPU L1 Cache to reduce memory access latency.

[0080] Further, if the warm data layer fails to match successfully, a pre-judgment is performed in the cold data layer through a memory Bloom filter, and when the pre-judgment indicates that it may exist, a disk bitmap index is triggered to query for low-frequency weak passwords. The memory Bloom filter is configured as a 1MB bit array, and 3 independent hash functions (FNV1a, Jenkins, CRC32) are used to calculate hash values, which are mapped to the bit array for existence pre-judgment, and the false positive rate is controlled within a preset range. If the pre-judgment result is "does not exist", the potential weak password is directly determined to be a non-weak password, and a security result is returned and the matching is terminated; if the pre-judgment is "may exist", then a disk bitmap index query is triggered. The disk storage uses Roaring Bitmap to map the password hash values to the chunked bitmaps, reduces the disk space occupancy by compressing sparse data, and is stored on an SSD hard disk to support high-speed random reading. The query process uses the AVX-512 instruction set to calculate multiple hash values in parallel, processing 16 passwords at a time to improve throughput. If the disk query hits, the potential weak password is determined to be a low-frequency weak password, and an alarm is triggered; otherwise, it is determined to be a non-weak password, and a security result is returned. To optimize disk access, the query results are stored in memory through an LRU cache to reduce the overhead of repeated queries.

[0081] Preferably, to improve the matching efficiency, the hash values generated by the pre-filtering are reused for the queries in the hot data layer, warm data layer, and cold data layer, avoiding repeated calculations. The entire hierarchical matching process is designed with a multi-stage pipeline, giving priority to processing high-frequency weak passwords and filtering them step by step to low-frequency weak passwords, significantly reducing the computational complexity and query latency. At the same time, fuzzy hash detection is supported to adapt to password variants, enhancing the flexibility and accuracy of detection. The matching results are recorded in the log (including the password hash value, matching level, and determination result) for subsequent analysis and system optimization.

[0082] In a specific implementation scenario of the present invention, such asFigure 2 As shown, assuming the password "P@ssw0rd" entered by the user, the system needs to determine whether it is a weak password. The specific steps are as follows:

[0083] Step 1: Extract the password

[0084] Extract the password entered by the user, generate the corresponding hash values (such as 0x5C4C3A... generated by MurmurHash3), and cache the hash calculation results in the CPU L1 Cache to reduce memory access latency.

[0085] Step 2: Global Bloom filter pre-judgment

[0086] Use the hash values generated in step 1 to match the positions in the bit array calculated by 5 hash functions. If any position is 0, return "safe"; otherwise, enter the hot data layer.

[0087] Step 3: Matching in the hot data layer

[0088] The double-stack backtracking algorithm matches the CDA-Trie. Stack 1 matches the Base index, and stack 2 matches the cached Check status. Case-insensitive swapping is allowed (e.g., "P@ssw0rd" and "p@ssword"). If the match is successful (the password belongs to the high-frequency weak passwords), an alarm is immediately triggered; otherwise, it enters the warm data layer for verification.

[0089] Step 4: Verification in the warm data layer

[0090] Use the hash values to quickly locate the target shard, query the Cuckoo Filter fingerprints in multiple threads, poll the load balancing strategy, and check whether the fingerprints exist. If the match is successful (the password belongs to the medium-frequency weak passwords), an alarm is triggered; otherwise, it enters the cold data layer for verification.

[0091] Step 5: Fallback in the cold data layer

[0092] The in-memory Bloom filter makes a pre-judgment. If it returns "may exist", a disk bitmap index query is triggered. Otherwise, it directly returns "safe"; after triggering the disk bitmap index, if the disk query hits (the password belongs to the low-frequency weak passwords), an alarm is triggered; otherwise, it returns "safe".

[0093] In a specific embodiment of the present invention, the weak password library is dynamically updated through an incremental update mechanism. Specifically,

[0094] The determined weak passwords are written into the shadow shard through a background thread, and the differences between the shadow shard and the main shard are recorded using differential compression;

[0095] Verify the integrity of the shadow shard data, and after the verification passes, the hash values of the newly added weak passwords are synchronously inserted into the bitmap of the global Bloom filter by the background thread.

[0096] Replace the shadow shard with the main shard through atomic pointer switching;

[0097] Reduce memory fragmentation by asynchronously releasing the memory resources of the old main shard.

[0098] It should be noted that the weak passwords determined (weak passwords from public data sets or identified through real-time user input detection) are written into the shadow shard by a background thread, and differential compression is used to record the differences between the shadow shard and the main shard. The newly added weak passwords are first subjected to hash calculation (generate 64-bit hash values using MurmurHash3), and are located to the target shard in the warm data layer according to the hash value range (shard Cuckoo Filter array, 100,000 entries per shard, including 4 hash table buckets, and 2 8-bit fingerprints are stored in each bucket). The background thread asynchronously inserts the weak passwords into the shadow shard (shard N+1), and realizes shard-level atomic operations through CAS (Compare-and-Swap) instructions to ensure thread safety and prevent multi-threaded concurrent write conflicts. Differential compression only records the newly added entries of the shadow shard relative to the main shard, eliminates duplicate data, and optimizes the compression ratio to a preset range, significantly reducing the storage overhead. To avoid affecting business threads, the priority of the write thread is lowered to give priority to ensuring the foreground query performance.

[0099] Furthermore, verify the integrity of the shadow shard data, and after the verification passes, the background thread synchronously inserts the hash values of the newly added weak passwords into the bitmap of the global Bloom filter. The integrity verification confirms that the data is not damaged or lost by comparing the differential data between the shadow shard and the main shard and using the CRC32 checksum algorithm. If the verification fails, the system triggers a retry mechanism or records error logs for subsequent analysis; if the verification passes, the background thread inserts the hash values of the newly added weak passwords into the bit array of the global Bloom filter. The insertion process uses the AVX-512 instruction set to calculate multiple hash values in parallel, processing 16 passwords at a time to improve the synchronization efficiency. To ensure that the bitmap update does not affect the foreground filtering, the synchronization operation is managed through an asynchronous queue, and the update results are temporarily stored in the memory buffer and periodically submitted to the bit array in batches.

[0100] Furthermore, the shadow shard is replaced with the main shard through atomic pointer switching to achieve imperceptible updates of the weak cryptographic library. The atomic switch uses the std::atomic memory barrier instruction to ensure that the main shard pointer switches seamlessly from shard N to shard N+1. The switching process is completed in nanoseconds and business queries are uninterrupted. Before the switch, the system confirms that the shadow shard has been written and verified. After the switch, the main shard takes effect immediately and supports subsequent hierarchical matching queries. To ensure data consistency, the switching operation is implemented through a double buffering mechanism to prevent the query thread from accessing incomplete data. After the switch is completed, the update log records the switch timestamp and shard ID to facilitate system auditing and troubleshooting.

[0101] Furthermore, memory fragmentation is reduced by asynchronously releasing the memory resources of the old primary shard. After the atomic switch, the old primary shard is marked as reclaimable, and a background thread asynchronously executes the memory release operation to ensure that no active query threads reference the old shard before release (verified by reference counting or locking mechanisms). The release process uses memory pool management, prioritizing the return of memory blocks to the system memory pool to avoid frequent calls to the operating system's memory allocation interface and reduce fragmentation. The release operation has a lower priority than business queries and shadow shard writes to ensure overall system performance. To support high-concurrency scenarios, memory release results are logged (including the size and time of the released shard), facilitating memory utilization monitoring and optimizing resource allocation strategies.

[0102] To improve update efficiency, the incremental update mechanism uses multiple threads to parallelize shadow shard writing, data verification, and Bloom filter synchronization. Tasks are assigned to CPU cores based on shard hash values, and a round-robin load balancing strategy is used to reduce thread contention. After the update is complete, the system recalculates password access frequency using a sliding window algorithm and dynamically adjusts the division of hot, warm, and cold data tiers to ensure that the storage location of newly added weak passwords matches actual access patterns, further optimizing query performance and memory utilization.

[0103] In a specific implementation scenario of the present invention, Figure 3 As shown in the figure, the incremental update process includes the following steps:

[0104] Step 1: Add a new weak password and write it to the shadow shard (shard N+1).

[0105] After the newly added password is hashed, the target shard is located based on the hash value range. A background thread inserts the password into the Cuckoo Filter of the shadow shard, while recording the difference data. CAS instructions are used to ensure thread safety.

[0106] Step 2: The background thread completes the shard data compression and verification.

[0107] Compare the shadow shard with the primary shard (shard N), and only retain the newly added entries. Verify data integrity through CRC32 checksum. Among them, the priority of the compression thread is lowered to avoid affecting business threads.

[0108] Step 3: The background thread inserts the newly added password hash into the global Bloom filter to ensure real-time performance.

[0109] Step 4: The atomic pointer switches to shard N+1, and the old shard (shard N) is released asynchronously.

[0110] Use a memory barrier instruction (such as std::atomic) to atomically replace the primary shard pointer. The old shard (shard N) is marked as recyclable, and the background thread releases the memory asynchronously. At the same time, before releasing the memory, ensure that there are no active query references to the old shard. The entire process is imperceptible to the business traffic.

[0111] In a specific embodiment of the present invention, the storage level of the weak password library is dynamically adjusted. Specifically,

[0112] Adopt a sliding window algorithm to count the password access frequency within a preset time period to dynamically divide the hot data layer, warm data layer, and cold data layer;

[0113] Based on the LRU-K algorithm, migrate the low-frequency weak passwords that have not been accessed for more than the preset time period to the compressed bitmap index on the disk to release memory resources.

[0114] It should be noted that a sliding window algorithm is used to count the access frequency of weak passwords within a preset time period to dynamically divide the hot data layer, warm data layer, and cold data layer to ensure that the storage architecture matches the actual access pattern. The sliding window algorithm records the timestamp and access count of each weak password query through a fixed-size time window. The window slides forward in seconds, and the data beyond the window is removed in real time to maintain the timeliness of the statistics. The access frequency is calculated based on the query count. The hot data layer stores the top ʺ20%ʺ of the weak passwords with the highest access frequency (high-frequency weak passwords), the warm data layer stores the weak passwords with a frequency of ʺ20% - 50%ʺ (medium-frequency weak passwords), and the cold data layer stores the weak passwords with a frequency lower than ʺ50%ʺ (low-frequency weak passwords). The statistical process is executed asynchronously by the background thread, and a hash table (with the password hash value as the key and the access count as the value) is used to efficiently record the frequency data. The capacity of the hash table is dynamically adjusted to avoid memory overflow. To support high concurrency, the statistical tasks are distributed to multiple CPU cores according to the password hash value range, and the round-robin load balancing strategy is combined to reduce thread contention. The division result is updated regularly and synchronized to the hierarchical storage architecture of the weak password library through atomic operations to ensure that the dynamic adjustment of the hot, warm, and cold data layers does not affect the foreground query.

[0115] It should be noted that based on the LRU-K algorithm, the low-frequency weak passwords that have not been accessed for more than a preset time period are migrated to the compressed bitmap index on the disk, releasing memory resources to optimize system performance. The LRU-K algorithm determines whether a password is infrequently accessed data by tracking the last K access timestamps of the weak password. If a weak password has not been accessed within 24 hours (i.e., both of the last two access times exceed the window), it is marked as a low-frequency weak password, triggering the migration operation. The migration process moves the infrequently accessed weak passwords (stored in the sharded Cuckoo Filter array) in the warm data layer to the disk storage in the cold data layer. Specifically, the password hash value is mapped to a chunk bitmap through a Roaring Bitmap, and sparse data compression technology is used to reduce disk space occupancy. The storage medium is an SSD hard disk to support high-speed random reading. The migration operation is executed asynchronously by a background thread with a lower priority than the foreground query and incremental update. Before migration, the target shard is locked through a CAS instruction to ensure thread safety. After migration, the corresponding shard data in memory is marked as recyclable and released to the memory pool to reduce memory fragmentation. To avoid frequent migrations, the LRU-K algorithm sets an access threshold, and the migration is triggered only when the password access count is lower than the threshold. The migration results are logged (including the password hash value, shard identifier, and timestamp) for system monitoring and resource optimization analysis.

[0116] In a specific embodiment of the present invention, the bitmap status of the global Bloom filter is periodically snapshotted and persisted to a non-volatile storage medium;

[0117] The hash value of the newly added weak password and its corresponding position update operation are recorded in a log, and the bitmap is reconstructed by replaying the log during system recovery to support the continuous operation of the pre-filtering and hierarchical matching.

[0118] It should be noted that the bitmap status of the global Bloom filter is periodically snapshotted and persisted to a non-volatile storage medium to prevent data loss due to system failures or restarts. The snapshot operation is executed every 5 minutes to generate a complete status backup of the bit array of the global Bloom filter. The snapshot process is executed asynchronously by a background thread, writing the bit array data in memory to the SSD storage medium, and using the high-speed random write performance of the SSD to ensure that the snapshot generation time is controlled within milliseconds. To reduce I / O overhead, the snapshot adopts an incremental backup strategy, only recording the bit array blocks that have changed since the last snapshot, and reducing the storage space occupancy through differential compression. Before snapshot generation, a memory barrier instruction is used to ensure the consistency of the bit array status and prevent data corruption caused by concurrent updates. After the snapshot is completed, the system records the snapshot timestamp and storage path to the metadata file for quick location of the latest snapshot during fault recovery. To support high-concurrency scenarios, the priority of the snapshot thread is lower than the foreground filtering and matching operations to avoid affecting the real-time performance of the system.

[0119] It should be noted that the hash value of the newly added weak password and its corresponding bit position update operation are recorded in the log, and the bitmap is reconstructed by replaying the log during system recovery to support the continuous operation of pre-filtering and hierarchical matching. During each incremental update, the system records the hash value of the newly added weak password and its mapping position (multiple bit indexes) in the bit array to the log file. The log is written in append mode and stored on the SSD medium. Each log record contains the weak password hash value, bit position index, update timestamp, and operation type (insert or delete). The CRC32 checksum is used to ensure the integrity of the log data. The log writing is asynchronously executed by a background thread and submitted in batches to reduce I / O overhead. The writing priority is lower than that of snapshots and foreground queries. After a system failure or restart, the recovery process first loads the latest snapshot to restore the state of the bit array, and then replays the log in timestamp order, executing the hash value insertion operation one by one to reconstruct the bitmap to the state before the failure. The replay process uses the AVX-512 instruction set to calculate multiple hash values in parallel, processing 16 log entries at a time to accelerate the recovery efficiency. To ensure recovery consistency, the system verifies the integrity of the log before replay. If corruption is detected, it rolls back to the previous snapshot. The recovered bitmap directly supports pre-filtering and subsequent hierarchical matching without re-initializing the weak password library, ensuring the continuous operation of the system.

[0120] In the second embodiment of the present invention, the present invention provides a dynamic weak password detection system based on pre-filtering and hierarchical compression, as Figure 4 shown. The system includes a global Bloom filter 100, a weak password library 200, an update module 300, and a cold data migration module 400;

[0121] The global Bloom filter 100 is used to receive the hash value of the password input by the user, generate a filtering result to screen potential weak passwords, and transfer them to the weak password library 200;

[0122] The weak password library 200 includes a hot data layer 201, a warm data layer 202, and a cold data layer 203. The hot data layer 201 receives potential weak passwords from the global Bloom filter 100, matches high-frequency weak passwords through a compressed double-array Trie, and outputs the unmatched passwords to the warm data layer 202. The warm data layer 202 queries medium-frequency weak passwords in parallel through a sharded Cuckoo Filter array and transfers the unmatched passwords to the cold data layer 203. The cold data layer 203 queries low-frequency weak passwords through an in-memory Bloom filter and a disk bitmap index to output the final matching result;

[0123] The update module 300 is used to process newly added weak passwords from public data sets or real-time collections, generate differential compression data to update the warm data layer 202 and the global Bloom filter 100;

[0124] The cold data migration module 400 is used to migrate infrequently accessed weak passwords from the warm data layer 202 to the disk storage of the cold data layer 203 according to access frequency statistics, so as to optimize resource utilization.

[0125] In the third embodiment of the present invention, the present invention provides an electronic device, including a memory and a processor, characterized in that a computer program that can run on the processor is stored in the memory, and when the program is executed on the processor, the steps in the above-mentioned dynamic weak password detection method based on pre-filtering and hierarchical compression are implemented.

[0126] In the fourth embodiment of the present invention, the present invention provides a storage medium, the storage medium stores a computer program, characterized in that when the computer program is executed by a processor, the steps in the above-mentioned dynamic weak password detection method based on pre-filtering and hierarchical compression are implemented.

[0127] In summary, the present invention provides a dynamic weak password detection method and system based on pre-filtering and hierarchical compression. By quickly screening non-weak passwords through a global Bloom filter and combining a hot, warm, and cold hierarchical compression storage architecture to optimize the matching efficiency of high-frequency to low-frequency weak passwords, the memory occupancy and query latency are significantly reduced. The incremental update mechanism uses shadow sharding and atomic switching to achieve seamless dynamic updates and ensure the real-time nature of the weak password library. The cold data migration based on access frequency further optimizes resource utilization, and combining snapshots and log recovery enhances the system's robustness, overall improving the performance, accuracy, and scalability of weak password detection in high-concurrency scenarios, and is applicable to the fields of network security and identity authentication.

[0128] It should be understood that although this specification is described according to embodiments, not each embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0129] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described modules can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, in each embodiment of the present application, each functional module can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0132] The above integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules stored in a storage medium include several instructions for causing a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor to execute some steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0133] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A dynamic weak password detection method based on pre-filtering and hierarchical compression, characterized in that: including In response to receiving a password input by a user, pre-filter the input password through a global Bloom filter to determine whether the input password is a non-weak password and screen out potential weak passwords; Perform hierarchical matching on the potential weak passwords, and dynamically update the weak password library through an incremental update mechanism; Dynamically adjust the storage level of the weak password library based on the password access frequency, and migrate cold data to optimize memory utilization.

2. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 1, characterized in that: It also includes Initialize a hierarchical compressed storage architecture including a hot data layer, a warm data layer, and a cold data layer to build a weak password library, and initialize a global Bloom filter based on the weak password library; Initializing the weak password library includes Using a compressed double-array Trie to store high-frequency weak passwords in the hot data layer; Using a sharded Cuckoo Filter array to store medium-frequency weak passwords in the warm data layer; Using a disk bitmap index combined with a memory Bloom filter to store low-frequency weak passwords in the cold data layer; Initializing the global Bloom filter includes Configure the global Bloom filter as a bit array with a preset size, perform hash calculations using multiple independent hash functions, and align according to the processor cache line to reduce cache miss rates.

3. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 2, characterized in that: The pre-filtering of the input password through the global Bloom filter includes Calculate the hash value of the input password using the multiple independent hash functions, and map the hash value to the corresponding position of the bit array of the global Bloom filter; If the value of any corresponding position in the bit array is 0, it is determined that the input password is a non-weak password; otherwise, it is determined that the input password is a potential weak password.

4. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 2 or 3, characterized in that: The hierarchical matching of the potential weak passwords includes Using the hash value of the potential weak password, match high-frequency weak passwords in the hot data layer through a compressed double-array Trie using a double-stack backtracking algorithm, and allow character difference matching within a predetermined range through support for fuzzy hash detection. If the match is successful, it is determined as a weak password; If the matching in the hot data layer is not successful, parallelly query medium-frequency weak passwords in the warm data layer through a sharded Cuckoo Filter array using a load balancing strategy. If a matching fingerprint is found, it is determined as a weak password; If the matching in the warm data layer is not successful, make a preliminary judgment in the cold data layer through a memory Bloom filter, and trigger a disk bitmap index query for low-frequency weak passwords when the preliminary judgment indicates that it may exist. If the query hits, it is determined as a weak password; otherwise, it is determined as a non-weak password.

5. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 4, characterized in that: The dynamic update of the weak password library through the incremental update mechanism includes Write the determined weak passwords into the shadow shard through a background thread, and use differential compression to record the differences between the shadow shard and the main shard; Verify the integrity of the shadow shard data, and after verification passes, synchronously insert the hash values of the newly added weak passwords into the bitmap of the global Bloom filter by the background thread; Replace the shadow shard with the main shard through an atomic pointer switch; Asynchronously release the memory resources of the old main shard to reduce memory fragmentation.

6. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 5, characterized in that: The dynamic adjustment of the storage level of the weak password library includes Adopt a sliding window algorithm to count the password access frequency within a preset time period to dynamically divide the hot data layer, the warm data layer, and the cold data layer; Based on the LRU-K algorithm, migrate the low-frequency weak passwords that have not been accessed for more than the preset time period to the compressed bitmap index on the disk to release memory resources.

7. The dynamic weak password detection method based on pre-filtering and hierarchical compression according to claim 1, characterized in that: It also includes Periodically take a snapshot of the bitmap status of the global Bloom filter and persist it to a non-volatile storage medium; Record the hash values of the newly added weak passwords and their corresponding position update operations to a log, and reconstruct the bitmap by replaying the log during system recovery to support the continuous operation of the pre-filtering and hierarchical matching.

8. A dynamic weak password detection system based on pre-filtering and hierarchical compression, characterized in that: It includes a global Bloom filter, a weak password library, an update module, and a cold data migration module; The global Bloom filter is used to receive the hash value of the password input by the user, generate a filtering result to screen potential weak passwords and pass them to the weak password library; The weak password library includes a hot data layer, a warm data layer, and a cold data layer; The hot data layer receives the potential weak passwords from the global Bloom filter, matches the high-frequency weak passwords through a compressed double-array Trie and outputs the unmatched passwords to the warm data layer; The warm data layer queries the medium-frequency weak passwords in parallel through a sharded Cuckoo Filter array and passes the unmatched passwords to the cold data layer; the cold data layer queries the low-frequency weak passwords through an in-memory Bloom filter and a disk bitmap index to output the final matching result; The update module is used to process the newly added weak passwords from the public dataset or real-time collection, generate differential compressed data to update the warm data layer and the global Bloom filter; The cold data migration module is used to migrate the infrequently accessed weak passwords from the warm data layer to the disk storage of the cold data layer according to the access frequency statistics to optimize resource utilization.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the dynamic weak password detection method based on pre-filtering and hierarchical compression according to any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps in the dynamic weak password detection method based on pre-filtering and hierarchical compression according to any one of claims 1-7.

Images