A data multi-element storage and full-text retrieval system

Abstract

This application relates to the field of computer big data processing and retrieval technology, and discloses a multi-dimensional data storage and full-text retrieval system, comprising: a homogeneous hash ingestion module that truncates the input data stream into logical data blocks, extracts terms to generate homogeneous hash packets, and distributes them to the underlying storage; a status monitoring module that maintains the heat potential and query fingerprints, updates the status based on scan feedback, and issues a trigger signal when the heat exceeds the dimensionality increase threshold; a differential projection module that responds to the signal, reads the homogeneous hash packets, and constructs a differential inverted index by calculating the intersection of the query fingerprint and the hash packets; and a cascading retrieval module that performs a memory lookup when the index is mounted, and if no match is found, scans the logical data blocks and feeds the results back to the status monitoring module. This invention tracks the heat potential of data blocks through the status monitoring module, triggering index construction only when the accumulated queries reach the dimensionality increase threshold, significantly reducing the cold start resource consumption of the system.

Description

Technical Field

[0001] This invention relates to the field of computer big data processing and retrieval technology, specifically a multi-dimensional data storage and full-text retrieval system. Background Technology

[0002] In the daily operation and maintenance of large-scale distributed clusters, application logs, audit logs, and monitoring metrics are constantly being generated. These data streams contain crucial clues for system fault diagnosis and security auditing. When business operations are disrupted or abnormal alarms are triggered, operations engineers need to accurately locate specific error codes or user IDs from massive amounts of historical data. The speed and accuracy of this retrieval and location directly affect the efficiency of incident recovery and the stability of business continuity.

[0003] To ensure real-time query response, mainstream log retrieval systems typically employ a write-as-index architecture. The moment data flows into the storage cluster, the system immediately performs word segmentation on all text content and builds an inverted index. This approach ensures that any keyword query can directly locate the target document through the pre-defined index structure. For business scenarios requiring high-frequency interaction and real-time visualization dashboards, this architecture provides an extremely smooth user experience and guarantees immediate data visibility.

[0004] However, machine data exhibits a strong long-tail distribution, with the vast majority of data never being accessed again after being written. Building a full index on all written data indiscriminately consumes several times the storage space of the original data and incurs significant computational costs during the data ingestion phase. If data is only compressed and archived to save resources, subsequent query operations degenerate into extremely inefficient brute-force scans, occupying the I / O bus for extended periods. Even more problematic is that when archived cold data becomes hot due to unforeseen events, the existing architecture lacks a smooth index replenishment mechanism. The system often needs to reread the original compressed file, decompress it, and then perform tedious word segmentation and hash calculations again. This repetitive work not only leads to severe read amplification but also results in a secondary waste of computational resources. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-dimensional data storage and full-text retrieval system. It solves the problems of wasted storage and computing resources caused by the full-index construction mode for low-frequency data in existing technologies, and excessive system I / O and computing power overhead caused by the need to repeatedly perform original data decompression, scanning and word segmentation calculations due to the lack of intermediate structured feature support during the process of converting cold data to hot data.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-source data storage and full-text retrieval system, comprising:

[0007] The isomorphic hash ingestion module is used to truncate the input data stream into logical data blocks, extract tokens from the logical data blocks and generate isomorphic hash packets, and distribute the logical data blocks and the isomorphic hash packets to the underlying storage layer.

[0008] The status monitoring module is used to maintain the heat potential energy index and query fingerprint set of the logical data block. When receiving the scan feedback for the logical data block, it updates the query fingerprint set and the heat potential energy index, and issues a trigger signal when the heat potential energy index exceeds the dimensionality upgrade threshold.

[0009] The differential projection module is used to respond to the trigger signal, read the isomorphic hash packet from the underlying storage layer, and calculate the intersection of the query fingerprint set and the isomorphic hash packet to construct a differential inverted index;

[0010] The cascading retrieval module is used to receive query requests, perform memory lookups when the differential inverted index is attached to the logical data block, scan the logical data block when the differential inverted index is not found, and send the scan results as the scan feedback to the status monitoring module.

[0011] Preferably, the isomorphic hash ingestion module includes:

[0012] The logical data block is parsed into a sequence of binary tuples containing word elements and physical location offsets using a word segmentation algorithm, and the hash feature value of the word elements is calculated using a non-cryptographic hash function.

[0013] Load a preset global noise table, check whether the hash feature value is included in the global noise table. If the hash feature value is included in the global noise table, it is reserved for the setting calculation of the Bloom filter. If the hash feature value is not included in the global noise table, it is reserved for subsequent processing.

[0014] Extract the unique valid hash value within the logical data block to construct a block-level local dictionary, and collect a list of physical location information corresponding to each unique valid hash value;

[0015] Differential encoding and variable-length integer compression are performed on the physical location information list, and the block-level local dictionary is combined with the compressed differential sequence to construct the isomorphic hash packet.

[0016] Preferably, the status monitoring module includes:

[0017] The heat potential energy index is initialized for each logical data block, and the heat potential energy index is periodically updated using a time sliding window decay model.

[0018] Monitor the execution path of the cascading retrieval module, and when a query request for the logical data block is detected to trigger a scanning operation on the original data storage unit, activate the query fingerprint capture mechanism.

[0019] Extract the query term hash values ​​from the query request, filter them using a global noise table, and merge the valid query hash values ​​into the query fingerprint set corresponding to the logical data block;

[0020] The updated heat potential index and the query fingerprint set are persistently stored, and the heat potential index is periodically compared with the dimensionality increase threshold.

[0021] Preferably, the differential projection module includes:

[0022] The isomorphic hash package is located and loaded using the unique identifier of the logical data block. The isomorphic hash package includes a header index area and a data payload area.

[0023] Calculate the intersection of the query fingerprint set and the block-level local dictionary in the isomorphic hash packet, and filter out the target feature set that exists both in the query record and inside the logical data block;

[0024] For each hash feature value in the target feature set, the corresponding position difference sequence is extracted and decompressed from the isomorphic hash packet, and the absolute position list is restored using the prefix sum algorithm;

[0025] All hash feature values ​​in the target feature set and the list of absolute positions corresponding to the hash feature values ​​are assembled into the differential inverted index, and the differential inverted index is loaded into the high-speed storage unit.

[0026] Preferably, the cascading retrieval module includes:

[0027] The hash feature value of the query keyword in the query request is calculated using the hash function defined in the isomorphic hash ingestion module;

[0028] Check whether the logical data block has been mounted with the differential inverted index. If the logical data block has been mounted with the differential inverted index, search for the hash feature value in memory. If a match is found, return the absolute position list directly.

[0029] If the hash feature value is not found in the differential inverted index, the hash feature value is compared with the global noise table. If the hash feature value belongs to the global noise table, the process is terminated.

[0030] If the hash feature value does not belong to the global noise table, load the Bloom filter corresponding to the logical data block to perform probabilistic existence detection. If the detection result is possible existence, backtrack to read and scan the logical data block, and generate the scan feedback.

[0031] Preferably, the isomorphic hash ingestion module constructs the isomorphic hash package by:

[0032] By utilizing the monotonically increasing property of the physical location information list, the difference between adjacent elements in the physical location information list is calculated, and the absolute location sequence is converted into a difference sequence.

[0033] A variable-length integer encoding algorithm is used to perform binary compression encoding on each difference value in the difference sequence, and only a single byte is used to store the difference value that is smaller than a preset value;

[0034] The isomorphic hash packet is divided into a header index area and a data payload area according to its physical structure. The starting byte offset of the compressed differential sequence in the data payload area is calculated, and the mapping relationship between the hash value and the starting byte offset is written into the header index area.

[0035] Preferably, the status monitoring module updates the heat potential energy index using a time-sliding window decay model, including:

[0036] Obtain the historical heat potential energy of the logical data block at the end of the previous time window;

[0037] Calculate the product of the time span within the current time window and the decay coefficient, and subtract the product from the historical heat potential energy. The decay coefficient is used to balance the weights of recent access frequency and historical access records.

[0038] The number of times the original data scan operation was actually triggered for the logical data block within the current time window is counted;

[0039] The number of executions is accumulated and added to the decayed historical heat potential energy to obtain the heat potential energy index of the logical data block at the current moment.

[0040] Preferably, the status monitoring module updates the query fingerprint set by:

[0041] Obtain the set of query requests that caused the scan, and calculate the hash value of each query term in the set of query requests;

[0042] The hash value is compared with the global noise table, and the hash value existing in the global noise table is discarded;

[0043] Check whether the current capacity of the query fingerprint set has reached the maximum capacity threshold;

[0044] If the maximum capacity threshold is reached, old fingerprints are eliminated using a first-in-first-out or least recently used strategy, and the non-noise valid query hash value is combined with the historical query fingerprint set to generate the updated query fingerprint set.

[0045] Preferably, the differential projection module uses a prefix sum algorithm to reconstruct the absolute position list, including:

[0046] Based on the byte offset provided by the header index area, locate the variable-length encoded data segment of the target hash value in the data payload area;

[0047] The variable-length encoded data segment is decoded to recover the position difference sequence, which is composed of multiple positive integer difference values;

[0048] The first difference value in the position difference sequence is taken as the first absolute physical position. Each subsequent absolute physical position is obtained by adding the previous absolute physical position to the current difference value, thereby restoring the list of absolute positions.

[0049] Preferably, the cascading retrieval module backtracks to read and scan the logical data block, and generates the scan feedback including:

[0050] When the hash feature value does not match the differential inverted index and does not belong to the global noise table, read the Bloom filter bit array corresponding to the logical data block;

[0051] The process involves using multiple independent hash mapping functions to check whether the hash feature value might exist, and if it returns that it does not exist, the process ends directly.

[0052] If the return value is possible, a read request is sent to the underlying storage layer to obtain the compressed logical data block. After decompression in the memory buffer, the query keywords are scanned using a string search algorithm.

[0053] Regardless of whether the scan result is found, a status feedback packet containing a logical data block identifier, query term hash value, and scan confirmation bit is constructed and sent to the status monitoring module.

[0054] This invention provides a multi-source data storage and full-text retrieval system. It has the following beneficial effects:

[0055] 1. This invention tracks the heat potential of data blocks through a status monitoring module and triggers index construction only when the cumulative query reaches the dimensionality threshold, which significantly reduces the cold start resource consumption of the system. Traditional full-text search systems usually build the inverted index in its entirety during the data writing stage, resulting in storage space being occupied by a large amount of cold data with low frequency of access. This solution effectively solves the problem of wasted computing resources and storage space under the full index mode.

[0056] 2. The isomorphic hash ingestion module of this invention pre-generates an isomorphic hash package containing a block-level local dictionary and a position difference sequence. Based on this, the differential projection module quickly assembles the differential inverted index through set intersection operation, avoiding the tedious process of re-parse words and calculate physical positions from the original data. Compared with the reconstruction method of the existing technology that requires a second full scan and word segmentation of the original file, this solution greatly reduces the IO overhead and CPU calculation latency during index construction.

[0057] 3. The cascading retrieval module of this invention combines global noise table filtering and Bloom filter detection to build a multi-level interception defense layer when the index is not hit. While ensuring the retrieval recall rate, it improves the system throughput by quickly eliminating invalid query requests. Existing log retrieval solutions often degenerate into brute-force full data scanning when there is a lack of indexes, which is very easy to cause system congestion due to high frequency of invalid queries. This technical solution solves the bottleneck of invalid scanning competing for the system bus in cold data query scenarios. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall architecture of a multi-source data storage and full-text retrieval system according to an embodiment of the present invention;

[0059] Figure 2 This is a data processing workflow diagram according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the data processing flow of the isomorphic hash ingestion module according to an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of the logical processing flow of the status monitoring module according to an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the logic processing flow of the differential projection module according to an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the logical processing flow of the cascading retrieval module according to an embodiment of the present invention;

[0064] Figure 7 This is a simulation curve of the change of the thermal potential energy of the logic data block over time according to an embodiment of the present invention;

[0065] Figure 8 This is a simulated bar chart comparing retrieval delays according to an embodiment of the present invention.

[0066] Among them: 110, homogeneous hash ingestion module; 120, status monitoring module; 130, differential projection module; 140, cascade retrieval module; 210, raw data storage unit; 220, intermediate state storage unit; 230, high-speed storage unit. Detailed Implementation

[0067] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Reference Figure 1 This invention provides a multi-dimensional data storage and full-text retrieval system, which is used for block storage, lifecycle management, and full-text retrieval of massive unstructured text data. The system can dynamically switch between the original compressed storage format and the differential index storage format based on the access frequency and query history characteristics of data blocks.

[0069] like Figure 1 As shown, the data multi-dimensional storage and full-text retrieval system includes: a homogeneous hash ingestion module 110, a status monitoring module 120, a differential projection module 130, and a cascaded retrieval module 140. Furthermore, the data multi-dimensional storage and full-text retrieval system is connected to an underlying storage layer, which includes a raw data storage unit 210 for storing raw data blocks, an intermediate storage unit 220 for storing intermediate metadata, and a high-speed storage unit 230 for storing hotspot indexes and Bloom filters.

[0070] The homogeneous hash ingestion module 110 receives a continuous data stream from external input and truncates it into discrete logical data blocks. The module performs lexical analysis on the text within each logical data block to extract word sequences and calculates the hash feature values ​​of the words. It then filters the hash feature values ​​using a pre-set global noise table, removing high-frequency, undiscriminative words. The module further extracts unique hash values ​​within each logical data block to construct a block-level local dictionary and collects the physical location information corresponding to each hash value. It performs differential encoding and variable-length compression on the physical location information to generate a homogeneous hash packet containing the block-level local dictionary and differential location sequences. The module compresses the original logical data block and writes it to the original data storage unit 210, writes the homogeneous hash packet to the intermediate storage unit 220, and writes the generated Bloom filter to the high-speed storage unit 230.

[0071] The status monitoring module 120 asynchronously maintains the status metadata of each logical data block during system operation. Based on a time-sliding window model, the status monitoring module 120 calculates the access heat potential of the logical data block, reflecting the frequency with which the logical data block has been recently retrieved. The status monitoring module 120 also maintains a query fingerprint set for each logical data block. When a retrieval request hits a specific logical data block and triggers a raw data scan, the status monitoring module 120 extracts the query term hash value from the retrieval request and appends the query term hash value to the query fingerprint set corresponding to that logical data block.

[0072] The differential projection module 130 is used to change the storage format of logical data blocks according to changes in access heat potential. When the status monitoring module 120 detects that the access heat potential of a logical data block exceeds a preset threshold, it sends a trigger signal to the differential projection module 130. In response to the trigger signal, the differential projection module 130 reads the homogeneous hash packet corresponding to the logical data block from the intermediate state storage unit 220. The differential projection module 130 calculates the intersection of the query fingerprint set and the block-level local dictionary in the homogeneous hash packet to determine the target feature set. The differential projection module 130 decompresses the corresponding differential position sequence from the homogeneous hash packet and restores it to an absolute position list only for the hash values ​​in the target feature set, constructing a differential inverted index in memory. The differential projection module 130 loads the constructed differential inverted index into the high-speed storage unit 230.

[0073] The cascading retrieval module 140 processes user-initiated query requests and executes a hierarchical retrieval strategy based on the index type currently mounted on the target logical data block. The cascading retrieval module 140 first receives the query term and calculates its hash value. It then determines whether the target logical data block has a differential inverted index mounted. If mounted, the cascading retrieval module 140 directly searches for the query term hash value in the differential inverted index. If not mounted or not found, the cascading retrieval module 140 determines whether the query term hash value belongs to the global noise table. If it belongs to the global noise table, the cascading retrieval module 140 skips the Bloom filter check. If it does not belong to the global noise table, the cascading retrieval module 140 queries the Bloom filter in the high-speed storage unit 230 to verify the existence of the query term hash value. After passing the above checks, the cascading retrieval module 140 retrieves and decompresses the original logical data block from the original data storage unit 210, performs a brute-force scan matching using a single instruction multiple data stream instruction set, and returns the matching result to the user.

[0074] Reference Figure 2 This invention provides a method for multi-dimensional data storage and full-text retrieval, comprising the following steps:

[0075] S100, the homogeneous hash ingestion module 110 receives the continuous data stream and divides it into logical data blocks, calculates the hash feature value of the words in the logical data block and constructs a homogeneous hash package, and stores the original data block, the homogeneous hash package and the Bloom filter to the underlying storage layer respectively.

[0076] S200, the cascade retrieval module 140 receives query requests for logical data blocks, and the status monitoring module 120 monitors the query process. When a query request triggers a scan of the original data block, the query term features are extracted, the query fingerprint set is updated, and the access heat potential is accumulated.

[0077] S300, when the access popularity potential exceeds the preset threshold, the differential projection module 130 reads the isomorphic hash packet and the query fingerprint set, calculates the intersection of the two and decompresses the location data from the isomorphic hash packet in a targeted manner, and generates a differential inverted index containing historical query hot words.

[0078] S400, when the cascading retrieval module 140 receives a new query request, it first searches for the query terms in the differential inverted index. If the differential inverted index is not found, it sequentially performs a Bloom filter check and a scan of the original data block.

[0079] Reference Figure 3 The homogeneous hash ingestion module 110 is mainly configured to convert unstructured raw text streams into structured, multi-dimensional storage formats, particularly generating intermediate homogeneous hash packets for subsequent parsing-free index construction. In this embodiment, the processing procedure of the homogeneous hash ingestion module 110 specifically includes the following steps:

[0080] S101, the homogeneous hash ingestion module 110 receives a continuous input data stream and logically truncates the input data stream according to a preset capacity threshold to generate a discrete logical data block sequence.

[0081] In this embodiment, the preset capacity threshold is mainly set based on the system memory page size and the I / O throughput characteristics of the underlying storage layer, with a preferred value range of 64MB to 256MB. The technical principle behind setting this threshold is that excessively small data blocks lead to severe metadata fragmentation, significantly increasing random I / O overhead during subsequent retrieval; while excessively large data blocks result in excessive memory buffer usage during subsequent decompression and index construction, affecting system concurrency performance. The homogeneous hash ingestion module 110 monitors the data volume in the input buffer in real time. When the accumulated data volume reaches the capacity threshold, it closes the current logical data block and assigns a unique block identifier to it.

[0082] S102, the isomorphic hash ingestion module 110 performs lexical analysis on the text content within the logical data block, extracts the word sequence, and calculates the hash feature value of the word.

[0083] Specifically, the homogeneous hash ingestion module 110 uses a word segmentation algorithm to parse the text stream into a sequence of binary tuples consisting of tokens and their physical position offsets. In this embodiment, the word segmentation strategy is based on predefined non-alphanumeric delimiters (including but not limited to spaces, punctuation marks, tabs, and newlines). The physical position offset represents the starting byte position of the token within the logical data block. The homogeneous hash ingestion module 110 uses a non-cryptographic hash function to map each token to a fixed-length integer feature value. This non-cryptographic hash function is preferably an algorithm with low computational overhead and good avalanche effect, such as MurmurHash V3 or xxHash algorithms, with an output length configured to be 64 bits or 32 bits to achieve a balance between hash collision probability and storage space. In this embodiment, the token sequence is represented as: ;in Representing the Each word element, Representing the The physical position offset of each word element This represents the total number of tokens within the logical data block. The formula for calculating the hash feature value is: ;in, Indicates the first The hash feature value of each word element. This represents a non-cryptographic hash function.

[0084] S103, the homogeneous hash ingestion module 110 loads the preset global noise table and uses the global noise table to filter and split the hash feature values.

[0085] In this embodiment, the global noise table is a set of hash values ​​containing high-frequency words with low retrieval discrimination. This global noise table is generated through word frequency analysis (TF-IDF) on a large-scale historical log corpus, covering stop words, common prepositions, and punctuation marks whose frequency exceeds a preset high-frequency threshold (e.g., 1% of the total number of words). The isomorphic hash ingestion module 110 checks whether each hash feature value is included in the global noise table. If the hash feature value is included in the global noise table, the isomorphic hash ingestion module 110 uses this hash feature value only for subsequent Bloom filter setting calculations, and does not include it in the isomorphic hash package construction process. This step is based on the principle of information entropy; removing low-entropy high-frequency words can significantly reduce the storage volume of intermediate data and prevent metadata expansion. If the hash feature value is not included in the global noise table, the isomorphic hash ingestion module 110 retains the hash feature value for subsequent processing.

[0086] S104, the homogeneous hash ingestion module 110 extracts the unique valid hash value within the logical data block to construct a block-level local dictionary, and collects a list of physical location information corresponding to each unique valid hash value.

[0087] Because log data has extremely high content repetition, a large number of terms appear repeatedly in the same data block. The isomorphic hash ingestion module 110 performs deduplication on the retained hash feature values ​​to generate a block-level local dictionary. The block-level local dictionary is defined as follows:

[0088] ;

[0089] in, Represents a block-level local dictionary. Represents a unique hash feature value. This represents the global noise table.

[0090] For each hash feature value in the block-level local dictionary The isomorphic hash ingestion module 110 traverses the word sequence, collects all its occurrences in the logical data block, and generates an ascending list of positions:

[0091] ;

[0092] in, Represents hash feature value Location list, This indicates that the eigenvalue is the first... The absolute physical position offset that appears for the first time. This indicates the frequency of occurrence of the feature value within the block.

[0093] S105, the homogeneous hash ingestion module 110 performs differential encoding and variable-length integer compression on the position list to generate a homogeneous hash packet.

[0094] Because the absolute physical location offset is relatively large (for example, an offset close to 64MB might occupy 4 bytes), direct storage would consume a significant amount of space. Utilizing the monotonically increasing property of the location list, the homogeneous hash ingestion module 110 calculates the difference between adjacent elements in the location list, converting the absolute location sequence into a difference sequence. The formula for calculating the difference sequence is as follows:

[0095] ;

[0096] The conversion rules are as follows:

[0097] ;

[0098] in, Represents a difference sequence. This represents the difference value. For log-intensive data, the distance between adjacent terms is usually small, therefore most... The value is extremely small.

[0099] Subsequently, the homogeneous hash ingestion module 110 uses a variable-length integer encoding algorithm to perform binary compression encoding on each difference value in the difference sequence. In this embodiment, the variable-length integer encoding algorithm (such as VarInt) uses the highest bit of each byte as a continuous flag bit, and the lower 7 bits store the two's complement. For difference values ​​less than 128, only 1 byte is needed for storage, instead of the fixed 4 or 8 bytes. The homogeneous hash ingestion module 110 combines the hash feature values ​​in the block-level local dictionary with the corresponding compressed difference sequence to construct a homogeneous hash packet.

[0100] ;

[0101] in, This represents the isomorphic hash packet corresponding to the logical data block. This indicates variable-length integer encoding operations. To support subsequent random access, the homogeneous hash ingestion module 110 divides the homogeneous hash packet into a header index area and a data payload area according to its physical structure when constructing the homogeneous hash packet. The homogeneous hash ingestion module 110 calculates the starting byte offset of the compressed differential sequence corresponding to each hash feature value in the data payload area and writes the mapping relationship between hash values ​​and offsets into the header index area. The homogeneous hash packet, as an intermediate data structure, records all the topological information required to construct the inverted index, but its size is significantly smaller than the original text data and the full inverted index.

[0102] S106, the homogeneous hash ingestion module 110 generates a Bloom filter using all hash feature values ​​and distributes the original data, homogeneous hash packets and Bloom filters to the underlying storage layer.

[0103] Specifically, the homogeneous hash ingestion module 110 utilizes all hash features, including noisy words. The Bloom filter's bit array is set to ensure it can respond to all lexical queries present within the block. For data distribution, the homogeneous hash ingestion module 110 compresses the unparsed raw logical data blocks using a lossless compression algorithm (such as Zstd or Snappy) and writes them to the raw data storage unit 210. The raw data storage unit 210 is configured as an object storage medium, suitable for low-cost, high-capacity storage. The homogeneous hash ingestion module 110 writes the generated homogeneous hash packets to the intermediate storage unit 220. The intermediate storage unit 220 is configured as a high-throughput cold storage medium or erasure coding storage pool. The homogeneous hash ingestion module 110 writes the generated Bloom filter to the high-speed storage unit 230. The high-speed storage unit 230 is configured as a key-value pair storage medium or a solid-state drive to support high-frequency metadata queries. Through the above distribution strategy, the system achieves physical separation and logical association of data content.

[0104] Reference Figure 4 The status monitoring module 120 is configured to asynchronously track the access activity and query characteristics of each logical data block during system operation, constructing a dynamic profile of the data access pattern, thereby providing a decision-making basis for the storage form evolution of the differential projection module 130. In this embodiment, the processing procedure of the status monitoring module 120 specifically includes the following steps:

[0105] S201, the status monitoring module 120 initializes the heat potential energy index for each logical data block and periodically updates the heat potential energy index using a time sliding window decay model.

[0106] In this embodiment, when the logical data block is written to the system for the first time, the status monitoring module 120 sets its initial thermal potential. Set to 0. This heat potential energy index is used to quantify the retrieval value of a specific logical data block at the current moment. The status monitoring module 120 adopts a time-sliding window decay model, the physical meaning of which is to simulate the principle of temporal locality of data access, that is, data that has been frequently accessed recently has a higher probability of being accessed again in a short period of time in the future. This model can balance the weight of recent access frequency and historical access records, so that data blocks that have been frequently accessed recently can quickly accumulate potential energy, while the potential energy of data blocks that have not been accessed for a long time will decay naturally over time.

[0107] The status monitoring module 120 calculates the logical data block's status according to the following formula. The potential energy of heat at any moment:

[0108] ;

[0109] in, Represents logical data blocks At the present moment The potential energy of heat This represents the heat potential of the logical data block at the end of the previous time window. This indicates the length of the time sliding window. This parameter is set according to the statistical period of the system, with a preferred range of 10 minutes to 60 minutes. This represents the attenuation coefficient, and its value range is... The attenuation coefficient setting depends on the business's sensitivity to identifying hot data. The larger the value, the higher the weight of historical popularity, and the more the system tends to hold the index for a long time; The smaller the value, the faster the system reacts to sudden traffic spikes, but the faster it also forgets the data. In this embodiment, to balance stability and sensitivity, The preferred setting is 0.85. Indicates the current time window Within this, the number of times the original data scan operation was actually triggered for this logical data block.

[0110] S202, the status monitoring module 120 monitors the execution path of the cascading retrieval module 140. When it detects that a query request for a specific logical data block has triggered a scanning operation on the original data storage unit 210, it activates the query fingerprint capture mechanism.

[0111] The status monitoring module 120 only records query fingerprints when an I / O-intensive operation, "raw data scan," is triggered. This design follows the technical principle of "recording on demand": if a query request is only intercepted by a Bloom filter or directly hits an existing differential inverted index, it means that the current system's storage structure is sufficient to efficiently handle the request, and no additional metadata recording is required. Only when the existing index structure cannot meet the query requirements, causing the system to perform a rollback scan, is the query considered an "unoptimized request" and captured.

[0112] S203, the status monitoring module 120 extracts the query term hash value from the query request, filters it using a global noise table, and merges the valid query hash values ​​into the query fingerprint set corresponding to the logical data block.

[0113] Specifically, the status monitoring module 120 first obtains the set of query requests that caused the scan. For each query term in the set Calculate its hash value Subsequently, the status monitoring module 120 compares the hash value with the global noise table used in the homogeneous hash ingestion module 110. If the hash value exists in the global noise table, it means that the query term is a high-frequency word without distinguishability (such as common words like "error" and "log"), and the status monitoring module 120 discards it and does not record it.

[0114] For valid query hash values ​​that are not noise, the status monitoring module 120 performs a set merging operation to update the query fingerprint set. To prevent metadata from growing indefinitely, the status monitoring module 120 sets a maximum capacity threshold for the fingerprint set for each logical data block. (For example, 1024 hash values). When the set size reaches... When updating, old fingerprints are replaced using either a First-In-First-Out (FIFO) or Least Recently Used (LRU) strategy. The update logic is as follows:

[0115] ;

[0116] in, This represents the updated set of query fingerprints. This represents the set of historical query fingerprints before the update. The hash value representing the query term. This represents the global noise table. This represents the capacity constraint function. Through the union operation of sets, the query fingerprint set can automatically remove duplicate query records, retaining only unique feature words that have been of business significance throughout the history of that data block.

[0117] S204, the status monitoring module 120 persistently stores the updated heat potential energy and query fingerprint set in the high-speed storage unit 230, and periodically compares the heat potential energy with the dimensionality upgrade threshold.

[0118] In this embodiment, the status monitoring module 120 utilizes a high-speed storage unit 230 (such as a memory-based key-value database) to maintain the heat potential and query fingerprint set. The status monitoring module 120 has a preset dimensionality threshold. The dimensionality threshold The determination of the weighted number of scans is based on the break-even point analysis of the cost of original data scanning and index building, and is usually set to 5 to 20 times. That is, within the current time window, if the weighted number of brute-force scans on a data block exceeds this threshold, it indicates that the benefit of building the index has exceeded its storage cost. When the logical data block... The value exceeds the dimensionality threshold At that time, the status monitoring module 120 generates and sends a trigger signal to the differential projection module 130 to start the storage format conversion process.

[0119] Reference Figure 5The differential projection module 130 is configured to dynamically construct a memory-level inverted index based on intermediate metadata when the access frequency of a logical data block reaches a threshold, thereby achieving index dimensionality upgrade without reading the original data or repeating word segmentation. In this embodiment, the processing procedure of the differential projection module 130 specifically includes the following steps:

[0120] S301, in response to the trigger signal issued by the status monitoring module 120, the differential projection module 130 reads the homogeneous hash packet corresponding to the target logical data block from the intermediate state storage unit 220, and at the same time obtains the query fingerprint set in the high-speed storage unit 230.

[0121] In this embodiment, the trigger signal is generated when the heat potential energy of the logical data block exceeds a preset dimensionality increase threshold. The differential projection module 130 locates and loads the isomorphic hash packet using the unique identifier of the logical data block. The isomorphic hash packet is compressed metadata stored in binary format. Specifically, the isomorphic hash packet physically includes a header index area and a data payload area. The header index area records the starting byte offset of each hash feature value in the block-level local dictionary within the data payload area. Since the isomorphic hash packet removes redundant information from the original text and performs differential compression, its size is typically only 5% to 15% of the original data block size. The I / O overhead of reading the isomorphic hash packet is much smaller than that of reading the original data block. This step utilizes the hardware characteristic that the sequential read speed of memory is much higher than the random read speed of disk, avoiding large bandwidth occupation of the underlying object storage.

[0122] S302, the differential projection module 130 calculates the intersection of the query fingerprint set and the block-level local dictionary in the isomorphic hash packet to determine the target feature set to be indexed.

[0123] Specifically, the differential projection module 130 performs a set intersection operation to filter out hash feature values ​​that exist both in historical high-frequency query records and are actually present within the current data block. The formula for calculating the target feature set is:

[0124] ;

[0125] in, Represents the target feature set, This represents the block-level local dictionary parsed from the header index area of ​​the isomorphic hash packet. This represents the set of query fingerprints corresponding to this data block. If the calculated... If the data block is empty, it means that the current query hotspots do not appear in this data block. The differential projection module 130 will terminate the subsequent index building process and reset the hotspot potential of this data block to avoid unnecessary memory usage. Through this step, the system can filter out those unpopular terms that exist in the block but have never been queried.

[0126] S303, the differential projection module 130 extracts and decompresses the corresponding position difference sequence from the isomorphic hash packet for each hash feature value in the target feature set, and restores the absolute position list using the prefix sum algorithm.

[0127] In this embodiment, the differential projection module 130 directly locates the target hash value based on the byte offset provided by the header index area. Random access is achieved by using variable-length encoded data segments in the data payload area. The differential projection module 130 first decodes the variable-length encoded data to recover the differential sequence. Then, the prefix sum algorithm is used to restore the relative displacement to the absolute physical byte offset of the term in the original data block. The restoration formula is as follows:

[0128] ;

[0129] in, Represents hash feature value No. The absolute physical location of the next occurrence. Represents the first in the difference sequence There are several differences. Because of the differences... All integers are positive integers. This restoration process only involves integer addition operations at the CPU register level and does not involve any string matching or floating-point operations, thus having extremely high execution efficiency.

[0130] S304, the differential projection module 130 assembles all hash feature values ​​and their corresponding absolute position lists in the target feature set into a differential inverted index, and loads the differential inverted index into the high-speed storage unit 230.

[0131] The differential inverted index is a memory-resident data structure whose logical structure is a hash map. To optimize memory space, the differential projection module 130 dynamically selects the storage container based on the density of the position list: for a sparse position list, i.e., a low-frequency term list, it directly uses an integer array to store the absolute positions; for a dense position list, i.e., a high-frequency term list, it uses a compressed bitmap index (e.g., a Roaring Bitmap). After construction, the differential projection module 130 writes the index structure to the high-speed storage unit 230 and updates the metadata status flag of the logical data block, marking it as "differential index mounted".

[0132] S305, after the index is loaded, the differential projection module 130 may optionally trigger a clearing or resetting operation of the query fingerprint set for the logical data block according to the system load.

[0133] To adapt to the dynamic changes in query hotspots, the differential projection module 130 can, after completing one index projection, convert the query fingerprint set... The system clears or decays the index. This allows the status monitoring module 120 to restart collecting a new round of missed query features. If new hot query terms appear later and are not covered by the current differential inverted index, the system will accumulate potential energy through a new round of scanning, triggering the differential projection module 130 to incrementally update or rebuild the index again, thereby achieving adaptive evolution of the index structure as business needs change.

[0134] Reference Figure 6 The cascading retrieval module 140 is configured to receive external query requests and, based on the current index construction status of the data block and the term frequency characteristics of the query terms, execute an optimized routing strategy among multi-level storage modalities. In this embodiment, the cascading retrieval module 140 processing procedure specifically includes the following steps:

[0135] S401, the cascading retrieval module 140 receives a query request containing at least one keyword and calculates the hash feature value of the query keyword using the hash function defined in the isomorphic hash ingestion module 110.

[0136] The cascading search module 140 first parses the text string entered by the user and converts it into a sequence of query terms. For each query term The cascading retrieval module 140 calls a preset unencrypted hash function. Calculate its hash value To ensure consistency in retrieval, the hash algorithm, seed value, and output bit width used in this embodiment are completely consistent with the hash configuration used to generate the homogeneous hash packet in step S102.

[0137] S402, the cascading search module 140 checks whether the target logical data block has been mounted with a differential inverted index. If it has been mounted, an exact search is performed in memory.

[0138] The cascading retrieval module 140 accesses the metadata status table in the high-speed storage unit 230. If the target logical data block is marked as "mounted with differential index", the cascading retrieval module 140 directly searches for the hash value in the memory-resident differential inverted index. If the hash value Upon a match in the index, the cascading retrieval module 140 reads the corresponding value (for bitmap storage formats, this is immediately decoded into an integer sequence) and obtains a list of absolute positions. This is then returned as the final result. The entire latency of this path is typically in the microsecond range. If the hash value... If the term is not found in the index, the cascading retrieval module 140 does not directly determine that the term does not exist. This is because the differential inverted index is a subset index built based on "historical query fingerprints," and the term that is not found may be a "new term" that exists in the data block but has never been queried. Therefore, in this case, the cascading retrieval module 140 passes the logical control flow to the subsequent S403 step to continue performing the existence determination.

[0139] S403, the cascading retrieval module 140 uses a global noise table to bypass the query hash value, thereby blocking invalid access to the underlying storage layer by high-frequency noise words.

[0140] Cascade retrieval module 140 will use hash values With global noise table Perform a comparison. If... This indicates that the query term is a high-frequency stop word with no retrieval value. In this case, the cascading search module 140 directly terminates the subsequent processing for this term and returns a "no result" or "ignore this term" response code, skipping the subsequent Bloom filter check and original data scan. The technical effect of this bypass decision logic is to eliminate high-frequency noise interference in "long-tail queries" and prevent the system from performing meaningless decompression and scanning operations on all data blocks due to the Bloom filter's mandatory neutrality of noise words, thereby protecting the system's I / O throughput.

[0141] S404, the cascaded retrieval module 140 loads the Bloom filter corresponding to the target logical data block and performs probabilistic existence detection.

[0142] If the query term does not match the differential inverted index and is not a global noise term, the cascading retrieval module 140 reads the Bloom filter bit array corresponding to the data block from the high-speed storage unit 230. The cascading retrieval module 140 utilizes... Detection of individual hash mapping functions Is it possible that the term exists in the data block? If the Bloom filter returns "not found", the cascading search module 140 determines that the term absolutely does not exist in the original data block, directly returns the "not found" result, and ends the process. If the Bloom filter returns "possibly exists", considering that the Bloom filter has a false positive rate (typically designed to be 0.1% to 1%), the cascading search module 140 cannot directly confirm the actual existence of the data, therefore triggering the fallback mechanism and proceeding to step S405.

[0143] S405, the cascade retrieval module 140 reads back from the original data storage unit 210 and scans the compressed original data block, and asynchronously feeds back the scan results to the status monitoring module 120.

[0144] The cascading retrieval module 140 initiates a read request to the original data storage unit 210 to obtain the compressed object of the target logical data block. The cascading retrieval module 140 decompresses the data block in the memory buffer and uses a fast string search algorithm (such as the Boyer-Moore algorithm or a substring search algorithm based on the SIMD instruction set) to scan for query terms in the restored text stream. If the scan confirms that the query term exists, the cascading search module 140 records the position offsets of all its occurrences and returns them as results; if the scan confirms that it does not exist (i.e., the Bloom filter misjudged), an empty result is returned.

[0145] After completing the above scanning operation (regardless of whether a result is found), the cascading retrieval module 140 constructs a status feedback packet. And send it to the status monitoring module 120. The data structure of the status feedback packet is defined as follows:

[0146] ;

[0147] in, A unique identifier for a logical data block. The hash value of the query term that caused the scan. This is the current timestamp. This is the scan confirmation bit (fixed to True). This feedback mechanism ensures that the status monitoring module 120 can capture this I / O-intensive query that is "not covered by the index", thereby accumulating heat potential energy and updating the query fingerprint, triggering the differential projection module 130 to include the word in the index in the future, realizing the system's self-iterative optimization.

[0148] To further clarify the collaborative working process of the technical solution of this invention, a specific example of server cluster log analysis will be used below. This embodiment assumes that the input data stream is the real-time running log generated by the server cluster, which includes timestamps, server Internet Protocol addresses, error codes, and detailed error information.

[0149] During the initial system operation phase, the homogeneous hash ingestion module 110 receives the log data stream and segments it into logical data blocks with the identifier ID-1024. The homogeneous hash ingestion module 110 then constructs a homogeneous hash packet for logical data block ID-1024 and compresses and stores the original data. At this time, the status monitoring module 120 sets the initial heat potential of logical data block ID-1024 to 0.

[0150] Subsequently, the cascading retrieval module 140 receives the first query request for keyword timeout. The cascading retrieval module 140 checks and finds that logical data block ID-1024 does not yet have a differential inverted index mounted, and the hash value of the keyword timeout does not belong to the global noise table. The cascading retrieval module 140 accesses the Bloom filter in the high-speed storage unit 230, and the Bloom filter returns possible results. The cascading retrieval module 140 then reads and decompresses logical data block ID-1024 from the original data storage unit 210, performs a brute-force scan, and successfully matches the keyword timeout. The cascading retrieval module 140 sends a status feedback packet containing the keyword timeout hash value to the status monitoring module 120. In response to this status feedback packet, the status monitoring module 120 adds the keyword timeout hash value to the query fingerprint set and accumulates the heat potential of logical data block ID-1024 according to a formula. Figure 7 As shown in the 0 to 10 minute interval, the heat potential energy shows a step-like increase, but has not yet reached the dimensionality threshold.

[0151] As time progresses, the cascading search module 140 receives multiple query requests for keyword timeouts and keyword anomalies within a short period. This typically corresponds to intensive troubleshooting operations by maintenance personnel. The status monitoring module 120 continuously captures these query characteristics and updates the query fingerprint set. For example... Figure 7 As shown at the 20-minute mark, the heat potential of logical data block ID-1024 increased sharply due to frequent scan feedback and exceeded the preset dimensionality increase threshold. At this time, the status monitoring module 120 immediately sent a trigger signal to the differential projection module 130.

[0152] In response to a trigger signal, the differential projection module 130 reads the homogeneous hash packet of logical data block ID-1024 from the intermediate state storage unit 220. The differential projection module 130 reads the query fingerprint set maintained by the state monitoring module 120, which currently contains the hash values ​​of the two high-frequency query terms: timeout and exception. After calculating the intersection, the differential projection module 130 extracts only the differential position sequences corresponding to these two keywords from the homogeneous hash packet and restores them to an absolute position list in memory, constructing a differential inverted index. The differential projection module 130 loads this differential inverted index into the high-speed storage unit 230 and marks logical data block ID-1024 as having a differential index mounted.

[0153] After logical data block ID-1024 completes index mounting, the cascading retrieval module 140 receives another query request regarding keyword timeout. The cascading retrieval module 140 detects that logical data block ID-1024 has a differential inverted index mounted, and directly locates the hash value of the keyword timeout in the memory index of the high-speed storage unit 230 and obtains the location list. For example... Figure 8As shown, in the cold data scanning stage without an index, the average latency is relatively high due to disk input / output operations and decompression operations. However, in the hot data retrieval stage after the differential index is built, the average latency is significantly reduced because it is based entirely on memory operations and does not require decompression of the original data, achieving a fast response at the millisecond level.

[0154] During subsequent operation, if the access frequency of logical data block ID-1024 decreases, the status monitoring module 120 will perform attenuation processing on the heat potential energy based on the time sliding window model. For example... Figure 7 As shown in the curve after 40 minutes, the thermal potential energy gradually decreases. When the thermal potential energy is lower than the preset dimensionality reduction threshold, the system can release the corresponding differential inverted index space in the high-speed storage unit 230, thereby achieving dynamic balancing and optimized configuration of storage resources.

[0155] This embodiment verifies, through the aforementioned collaborative working process, that the data multi-dimensional storage and full-text retrieval system can ensure low-cost storage of massive amounts of data while automatically indexing and accelerating performance for dynamically generated hot data, effectively resolving the contradiction between storage cost and query efficiency in traditional log retrieval systems.

[0156] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data multi-element storage and full-text retrieval system, characterized in that, include: The isomorphic hash ingestion module is used to truncate the input data stream into logical data blocks, extract tokens from the logical data blocks and generate isomorphic hash packets, and distribute the logical data blocks and the isomorphic hash packets to the underlying storage layer. The status monitoring module is used to maintain the heat potential energy index and query fingerprint set of the logical data block. When receiving the scan feedback for the logical data block, it updates the query fingerprint set and the heat potential energy index, and issues a trigger signal when the heat potential energy index exceeds the dimensionality upgrade threshold. The differential projection module is used to respond to the trigger signal, read the isomorphic hash packet from the underlying storage layer, and calculate the intersection of the query fingerprint set and the isomorphic hash packet to construct a differential inverted index; The cascading retrieval module is used to receive query requests, perform memory lookups when the differential inverted index is attached to the logical data block, scan the logical data block when the differential inverted index is not found, and send the scan results as the scan feedback to the status monitoring module.

2. The data multi-store and full-text search system according to claim 1, wherein, The homogeneous hash ingestion module includes: The logical data block is parsed into a sequence of binary tuples containing word elements and physical location offsets using a word segmentation algorithm, and the hash feature value of the word elements is calculated using a non-cryptographic hash function. Load a preset global noise table, check whether the hash feature value is included in the global noise table. If the hash feature value is included in the global noise table, it is reserved for the setting calculation of the Bloom filter. If the hash feature value is not included in the global noise table, it is reserved for subsequent processing. Extract the unique valid hash value within the logical data block to construct a block-level local dictionary, and collect a list of physical location information corresponding to each unique valid hash value; Differential encoding and variable-length integer compression are performed on the physical location information list, and the block-level local dictionary is combined with the compressed differential sequence to construct the isomorphic hash packet.

3. The data multi-source storage and full-text retrieval system according to claim 1, characterized in that, The status monitoring module includes: The heat potential energy index is initialized for each logical data block, and the heat potential energy index is periodically updated using a time sliding window decay model. Monitor the execution path of the cascading retrieval module, and when a query request for the logical data block is detected to trigger a scanning operation on the original data storage unit, activate the query fingerprint capture mechanism. Extract the query term hash values ​​from the query request, filter them using a global noise table, and merge the valid query hash values ​​into the query fingerprint set corresponding to the logical data block; The updated heat potential index and the query fingerprint set are persistently stored, and the heat potential index is periodically compared with the dimensionality increase threshold.

4. The data multi-source storage and full-text retrieval system according to claim 1, characterized in that, The differential projection module includes: The isomorphic hash package is located and loaded using the unique identifier of the logical data block. The isomorphic hash package includes a header index area and a data payload area. Calculate the intersection of the query fingerprint set and the block-level local dictionary in the isomorphic hash packet, and filter out the target feature set that exists both in the query record and inside the logical data block; For each hash feature value in the target feature set, the corresponding position difference sequence is extracted and decompressed from the isomorphic hash packet, and the absolute position list is restored using the prefix sum algorithm; All hash feature values ​​in the target feature set and the list of absolute positions corresponding to the hash feature values ​​are assembled into the differential inverted index, and the differential inverted index is loaded into the high-speed storage unit.

5. The data multi-store and full-text search system according to claim 1, wherein, The cascaded retrieval module includes: The hash feature value of the query keyword in the query request is calculated using the hash function defined in the isomorphic hash ingestion module; Check whether the logical data block has been mounted with the differential inverted index. If the logical data block has been mounted with the differential inverted index, search for the hash feature value in memory. If a match is found, return the absolute position list directly. If the hash feature value is not found in the differential inverted index, the hash feature value is compared with the global noise table. If the hash feature value belongs to the global noise table, the process is terminated. If the hash feature value does not belong to the global noise table, load the Bloom filter corresponding to the logical data block to perform probabilistic existence detection. If the detection result is possible existence, backtrack to read and scan the logical data block, and generate the scan feedback.

6. The data multi-store and full-text search system according to claim 2, wherein, The homogeneous hash ingestion module constructs the homogeneous hash package by: By utilizing the monotonically increasing property of the physical location information list, the difference between adjacent elements in the physical location information list is calculated, and the absolute location sequence is converted into a difference sequence. A variable-length integer encoding algorithm is used to perform binary compression encoding on each difference value in the difference sequence, and only a single byte is used to store the difference value that is smaller than a preset value; The isomorphic hash packet is divided into a header index area and a data payload area according to its physical structure. The starting byte offset of the compressed differential sequence in the data payload area is calculated, and the mapping relationship between the hash value and the starting byte offset is written into the header index area.

7. The data multi-store and full-text search system according to claim 3, wherein, The status monitoring module updates the heat potential energy index using a time-sliding window decay model, including: Obtain the historical heat potential energy of the logical data block at the end of the previous time window; Calculate the product of the time span within the current time window and the decay coefficient, and subtract the product from the historical heat potential energy. The decay coefficient is used to balance the weights of recent access frequency and historical access records. The number of times the original data scan operation was actually triggered for the logical data block within the current time window is counted; The number of executions is accumulated and added to the decayed historical heat potential energy to obtain the heat potential energy index of the logical data block at the current moment.

8. The data multi-store and full-text search system according to claim 3, wherein, The status monitoring module updates the query fingerprint set by: Obtain the set of query requests that caused the scan, and calculate the hash value of each query term in the set of query requests; The hash value is compared with the global noise table, and the hash value existing in the global noise table is discarded; Check whether the current capacity of the query fingerprint set has reached the maximum capacity threshold; If the maximum capacity threshold is reached, old fingerprints are eliminated using a first-in-first-out or least recently used strategy, and the non-noise valid query hash value is combined with the historical query fingerprint set to generate the updated query fingerprint set.

9. The data multi-store and full-text search system according to claim 4, wherein, The differential projection module uses a prefix sum algorithm to reconstruct the absolute position list, including: Based on the byte offset provided by the header index area, locate the variable-length encoded data segment of the target hash value in the data payload area; The variable-length encoded data segment is decoded to recover the position difference sequence, which is composed of multiple positive integer difference values; The first difference value in the position difference sequence is taken as the first absolute physical position. Each subsequent absolute physical position is obtained by adding the previous absolute physical position to the current difference value, thereby restoring the list of absolute positions.

10. The data multi-store and full-text search system according to claim 5, wherein, The cascading retrieval module backtracks to read and scan the logical data block, and generates the scan feedback including: When the hash feature value does not match the differential inverted index and does not belong to the global noise table, read the Bloom filter bit array corresponding to the logical data block; The process involves using multiple independent hash mapping functions to check whether the hash feature value might exist, and if it returns that it does not exist, the process ends directly. If the return value is possible, a read request is sent to the underlying storage layer to obtain the compressed logical data block. After decompression in the memory buffer, the query keywords are scanned using a string search algorithm. Regardless of whether the scan result is found, a status feedback packet containing a logical data block identifier, query term hash value, and scan confirmation bit is constructed and sent to the status monitoring module.

Images