Data retrieval method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides a data retrieval method and device, electronic equipment, a storage medium and a computer program product, and relates to the technical field of data management.The method comprises the steps that a retrieval request is obtained, and a to-be-retrieved key value pair file is determined based on the retrieval request; wherein the key value pair file is determined by combining a plurality of key value pair sub-files by using other equipment, or is determined locally by using a background thread; the foreground thread and the background thread are respectively bound with different central processing unit computing power resources; based on the retrieval request, performing filtering retrieval on a filtering block in the file in combination with the key value to determine first retrieval data, and performing retrieval in a memory to determine second retrieval data; and determining target retrieval data based on merging of the first retrieval data and the second retrieval data.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data retrieval method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] In related technologies, taking object storage as an example, current mainstream multi-version solutions employ key-value (KV) storage systems or structured query language (SQL) storage systems to manage the metadata of the object storage. An object in object storage has one or more metadata records, storing and managing the object's name, bucket, permissions, attributes, and data information. In multi-version solutions, multiple metadata records are used to manage the latest and historical version information; the implementation of multi-versioning is handled by the object storage itself. During retrieval, KV storage or SQL provides full object metadata, including some invalid metadata, resulting in relatively low query efficiency. Summary of the Invention

[0003] This application provides a data retrieval method, apparatus, electronic device, storage medium, and computer program product.

[0004] The technical solution of this application is implemented as follows: This application provides a data retrieval method, including: A search request is obtained, and the key-value pair file to be searched is determined based on the search request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining it locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources; Based on the search request, the first search data is determined by filtering the filter blocks in the file in combination with the key value, and the second search data is determined by searching in memory; Based on the merging of the first search data and the second search data, the target search data is determined.

[0005] In the above scheme, the key-value pair file includes: data block, index block, metadata block, statistics block, file information block, and tail block; The data block is used to store key-value pair data; the index block is used to store index information of the keys; the metadata block is used to determine whether the key-value pair file stores the specified key and version information; the statistics block is used to store statistical information of the keys and version information in the data block; the file information block is used to store the metadata of the key-value pair file; and the tail block is used to store the basic information of the key-value pair file.

[0006] In the above scheme, the step of determining the first search data by filtering the filter blocks in the file based on the search request and the key value includes: Based on the retrieval type represented by the retrieval request, the filter block is determined among multiple blocks of the key-value pair file; Based on the filter block and the search conditions in the search request, target data blocks that meet the search conditions are filtered and determined. Read the first search data that meets the search criteria from the target data block.

[0007] In the above scheme, determining the filter block from multiple blocks of the key-value pair file based on the retrieval type represented by the retrieval request includes: Based on the predetermined string in the search request, determine the search type of the search request; When the search type is a specified search, the metadata block, the index block, and the statistics block are determined as the filter block; when the search type is a range search, the index block and the statistics block are determined as the filter block.

[0008] In the above scheme, when the search type is a specified search, the step of filtering and determining the target data block that meets the search conditions based on the filter block and the search conditions in the search request includes: Based on the metadata block, the search conditions are processed to determine whether the target data block exists in the key-value pair file; When the inspection result indicates existence, the target data block identifier that meets at least one of the first key condition and the first version information condition is found based on the index block, and invalid data blocks in the data block indicated by the target data block identifier are filtered based on the statistical information of the key and version information stored in the statistical information block to determine the target data block; the first key condition and the first version information condition belong to the search conditions.

[0009] In the above scheme, when the search type is a range search, the step of filtering and determining the target data block that meets the search conditions based on the filter block and the search conditions in the search request includes: Based on the index block, find the target data block identifier range that meets the second key condition; wherein, the second key condition belongs to the search condition; Based on the statistical information of keys and version information stored in the statistical information block, invalid data blocks in the data blocks indicated by the target data block identifier range are filtered out to determine the target data block.

[0010] This application also provides a data retrieval device, including: An acquisition unit is used to acquire a retrieval request and determine the key-value pair file to be retrieved based on the retrieval request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining it locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources; The retrieval unit is configured to, based on the retrieval request, filter and retrieve the filter blocks in the file using the key value to determine the first retrieval data, and retrieve and determine the second retrieval data in memory; The merging unit is used to determine the target retrieval data based on the merging of the first retrieval data and the second retrieval data.

[0011] This application also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps in the above-described method.

[0012] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method.

[0014] In this embodiment, a search request is obtained, and the key-value pair file to be searched is determined based on the search request. The key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining them locally using a background thread. The foreground thread and background thread are bound to different CPU computing resources. Based on the search request, filtering is performed using filter blocks in the key-value pair file to determine the first search data, and a second search is performed in memory to determine the second search data. The target search data is determined by merging the first and second search data. This approach, using other devices to merge or using a background thread to determine the key-value pair file, separates the CPU, network, and memory required for offline operations from the online machine because the foreground and background threads are bound to different CPU computing resources. This not only improves the utilization of machine resources and reduces online read / write latency, but also improves search efficiency. Furthermore, by using filter blocks in the key-value pair file to filter invalid data to determine the first search data, this approach of sinking the search operator to the key-value pair file for data filtering avoids the transfer of large amounts of invalid data between KV storage, object storage, and file storage, thereby improving search efficiency. Attached Figure Description

[0015] Figure 1Flowchart of the data retrieval method provided in the embodiments of this application Figure 1 ; Figure 2 Illustration of the effect of the data retrieval method provided in the embodiments of this application Figure 1 ; Figure 3 Illustration of the effect of the data retrieval method provided in the embodiments of this application Figure 2 ; Figure 4 Flowchart of the data retrieval method provided in the embodiments of this application Figure 2 ; Figure 5 Illustration of the effect of the data retrieval method provided in the embodiments of this application Figure 3 ; Figure 6 A schematic diagram of the structure of the data retrieval device provided in the embodiments of this application; Figure 7 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application.

[0016] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] This application implements a key-value (KV) storage system based on a log structured merge tree (LSM), systematically solving the technical problems of this application. In this embodiment, some metadata operations from upper layers (object storage, file storage, etc.) are pushed down to the KV layer of the metadata management system. To address the impact of compaction on KV storage requests, remote compaction technology is used to separate online requests from backend requests, which not only improves the machine resource utilization of KV storage and reduces the latency of online requests, but also improves retrieval efficiency.

[0019] This application provides a data retrieval method. Please refer to the embodiments provided. Figure 1 The following is a flowchart illustrating the data retrieval method provided in the embodiments of this application. Figure 1 , will combine Figure 1 The steps shown are explained below: S101. Obtain a retrieval request and determine the key-value pair file to be retrieved based on the retrieval request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining it locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources.

[0020] In this embodiment, the server obtains a retrieval request, determines the physical region where key-value pair files are stored based on the key conditions in the retrieval request, and determines that the key-value pair files in the physical region are the key-value pair files to be retrieved.

[0021] The search request can correspond to different types of search requests, including specified search requests and range search requests. Specified search requests are used to retrieve a single piece of data, while range search requests are used to retrieve data within a specified range of conditions. Specified search requests can include: `Get(Key, VersionID, Operator(Limit1, Condition))`. Range search requests can include: `Scan(Key1, Key2, Operator(Latest, Limit1, Condition))`.

[0022] The key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by merging them locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources.

[0023] In this embodiment, as Figure 2 As shown, the KV storage system adopts an LSM architecture, divided into memory and file components. Memory data (KV pairs) is ordered and managed using a skip list. When the amount of memory data reaches a threshold, it is written to disk to generate key-value pair subfiles. Data within a single key-value pair subfile is also ordered, and keys may overlap between multiple subfiles. Multiple key-value pair subfiles are merged through a background merge operation to generate a larger key-value pair file. Write requests first write to the log, then to memory; read requests merge the results retrieved from memory and the KV file (KVFile) before returning them to the client. Data management in memory uses a Skip List, arranged lexicographically by key. Each key node manages its own multiple versions of data, arranged in reverse lexicographical order by version identifier (VersionID), with the latest version placed at the front, also managed using a Skip List.

[0024] In this embodiment, the latency of the LSM-based KV storage system is affected by background operations such as flushing and compaction, especially compaction. Once started, compaction significantly impacts the latency of online requests, potentially leading to uncontrollable latency. To ensure the quality of online requests, a common practice is to reserve some Central Processing Unit (CPU) memory for background operations, but this reduces the overall utilization of machine resources. This embodiment offers two solutions to address these issues.

[0025] Option 1: Group the threads within the process into foreground threads (e.g., read / write threads, communication threads) and background threads (e.g., flush, compaction threads). Foreground and background threads are then bound to different CPU computing resources to mitigate the impact of sudden offline operations on online requests. Tests after binding the CPU to specific cores showed that when background operations suddenly increased, CPU utilization briefly exceeded 60%, but still met the latency requirements for online access. Under normal circumstances, CPU utilization remained around 40%.

[0026] Option 2: Separate the compaction operation for key-value pair subfiles to other devices to further improve overall CPU utilization. The LSM write amplification problem mainly stems from compaction, which combines multiple data files into one, consuming a significant amount of CPU and impacting the quality of online access. By placing the compaction operation on a separate device, further separating online operations (foreground threads) from offline operations (background threads), CPU utilization increases from the usual 40% across all machines to 50% for online machines and 80% for offline machines. Combined with efficient load balancing capabilities, cluster CPU utilization can be further improved.

[0027] In this way, the compaction operation for key-value pair subfiles separates the CPU, network, and memory required for offline operations from the online machine, which not only improves the utilization of machine resources, but also greatly reduces latency spikes in online read and write operations, thereby improving the efficiency of data retrieval.

[0028] S102. Based on the search request, filter the filter blocks in the file using the key value to determine the first search data, and search the memory to determine the second search data.

[0029] In this embodiment, the key-value pair file includes: a data block, an index block, a metadata block, a statistics block, a file information block, and a tail block. The data block stores key-value pair data; the index block stores index information for the keys; the metadata block determines whether the key-value pair file stores a specified key and version information; the statistics block stores statistical information about the keys and version information in the data block; the file information block stores metadata about the key-value pair file; and the tail block stores basic information about the key-value pair file. Based on the search type corresponding to the search request, a filtering block can be determined from multiple blocks to filter data. Then, based on the filtering block and the search conditions in the search request, the corresponding first search data can be determined. Furthermore, a second search data can be determined by searching in memory based on the search conditions in the search request.

[0030] In this embodiment, the filtering block for key-value pair files may include an index block, a metadata block, and a statistics block. The filtering block is used to filter invalid metadata information or invalid data blocks based on the search criteria in the search request. In this embodiment, when the search request is for a specified search type, the filtering block may include a metadata block, an index block, and a statistics block. When the search request is for a range search type, the filtering block may include an index block and a statistics block.

[0031] In this embodiment, the scheme for determining the second search data may include: Scheme 1: Perform a search based on the search request: Get(Key, VersionID, Operator(Limit1, Condition)), and search in memory. Search for the Key node in the SkipList in memory based on the Key (search condition), search for the specified version in the Key node based on the VersionID, and determine whether it meets the Condition condition. If it exists, return the second search data. If it does not exist, continue searching in the file. Scheme 2: Perform a search based on the search request: Scan(Key1, Key2, Operator(Latest, Limit1, Condition)), and search in memory. Based on Key1 and Key2, confirm the start node of Key1 and the end node of Key2 in the SkipList in memory, and sequentially retrieve the latest version of each node from the start node to the end node to obtain the second search data.

[0032] In this embodiment, the structure of a key-value file (KV File) is as follows: Figure 3 As shown, it includes the following parts: 1. Data Block 100 stores multiple key-value pairs. It has a fixed size, typically 64KB, and is stored in compressed form. It is the smallest unit for reading and writing file data, and during reading, the entire block is loaded into the memory block cache.

[0033] 2. Index Block 200 stores the index of the key. It consists of three levels: the root index block, intermediate index blocks, and leaf index blocks, forming a balanced multi-way search tree (BTree). The root index block resides in memory, while the intermediate and leaf index blocks are loaded into memory before read and write operations. Index Block 200 allows for quick retrieval of a specified key, filtering out invalid data blocks and invalid keys.

[0034] 3. Meta Block 300: Bloom filter block, storing Bloom filters for key and version information. Bloom filters allow for quick determination of whether a key-value pair file contains the specified key and version information. If the key and version information are not found in the key-value pair, the search can be quickly terminated without reading any index blocks or data blocks.

[0035] 4. Statistics Block 400: Records statistical information about the keys and version information of the data blocks. It is organized by data block and stored in the order of the data blocks. It is loaded into memory before data is read, and the data blocks are arranged in lexicographical order according to their block identifiers, managed using a skip list. Before reading data from a data block, the statistics are used to confirm whether the data in the data block meets the read criteria. Data blocks that do not meet the criteria are immediately skipped and do not need to be loaded into memory or read.

[0036] 5. File Info Block 500: Meta-information and statistics of key-value file, average length of keys and values, number of keys, key comparator, time to live (TTL), etc.

[0037] 6. Trailer Block 600: Basic information about the key-value file, including offsets, addressing information, and file version for each part.

[0038] S103. Based on the merging of the first search data and the second search data, determine the target search data.

[0039] In this embodiment, the first search data in the key-value pair file and the second search data in memory are merged, that is, the search data in memory and the file are merged. The Operator and Condition conditions are executed, and the target search data that meets the search conditions is returned.

[0040] In this embodiment, a retrieval request is obtained, and the key-value pair file to be retrieved is determined based on the retrieval request. The key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining them locally using a background thread. The foreground thread and background thread are bound to different CPU computing resources. Based on the retrieval request, invalid data is filtered using filtering blocks in the key-value pair file to determine the first retrieval data, and second retrieval data is determined by retrieving data from memory. The target retrieval data is determined by merging the first and second retrieval data. This approach, using other devices to merge or using a background thread to determine the key-value pair file, separates the CPU, network, and memory required for offline operations from the online machine because the foreground and background threads are bound to different CPU computing resources. This not only improves the utilization of machine resources and reduces online read / write latency, but also improves retrieval efficiency. Furthermore, by using filtering blocks in the key-value pair file to filter invalid data to determine the first retrieval data, this approach, which decentralizes the retrieval operator to the key-value pair file for data filtering, avoids the transfer of large amounts of invalid data between KV storage, object storage, and file storage, thereby improving retrieval efficiency.

[0041] Please see Figure 4 The following is a flowchart illustrating the data retrieval method provided in the embodiments of this application. Figure 2 , Figure 1 S102 in the process can also be implemented through S201 to S203, combining Figure 4 The steps shown are explained below: S201. Based on the retrieval type represented by the retrieval request, determine the filter block among multiple blocks of the key-value pair file.

[0042] In this embodiment, the search request can be determined as either a specified search type or a range search type based on predetermined fields in the search request. Then, based on the type of search request, the corresponding filter block is determined from multiple blocks in the key-value pair file.

[0043] The types of search requests in this embodiment will be explained below with reference to Table 1. The types of search requests can include two types: Get (specified search) and Scan (range search).

[0044]

[0045] Table 1 S202. Based on the filter block and the search conditions in the search request, filter and determine the target data block that meets the search conditions.

[0046] In this embodiment, the search criteria in the search request may include: key criteria, version information criteria, and quantity criteria. Invalid data blocks can be filtered based on the information stored in the filter block, combined with the search criteria, thereby determining the target data blocks that meet the search criteria.

[0047] In this embodiment, search conditions, or operators for short, refer to the parameters specified during Get (retrieving, indicating a specific search) or Scan (scanning, indicating a range search) operations, such as Key, VersionID, requiring only the latest version or all versions, number of objects or versions, and additional search conditions. During data retrieval, operators push down to the data blocks at the key-value pair file level, quickly filtering out invalid data blocks, keys, and version information. After the operator pushdown, invalid object and version data will not be returned. The retrieval process is as follows: Figure 5 As shown, searches can be performed in the order of object service, key-value client, data partition, key-value pair file and data block. Invalid data blocks can be quickly skipped based on the statistical information blocks in the key-value pair file, which contain statistical information about keys and version information.

[0048] S203. Read the first search data that meets the search conditions from the target data block.

[0049] In this embodiment, the target data block can be loaded and the first search data that meets the search criteria can be read. That is, the values ​​that meet the quantity criteria, version information criteria, and key criteria can be read.

[0050] In this embodiment, based on the retrieval type represented by the retrieval request, the filter block is determined from multiple blocks in the key-value pair file. Based on the filter block and the retrieval conditions in the retrieval request, the target data block that meets the retrieval conditions is determined. The first retrieval data that meets the retrieval conditions is read from the target data block. In this way, by performing operator pushdown filtering retrieval in the key-value pair file based on retrieval conditions, the data received by the upper layer (object storage, file storage, etc.) is all valid data, eliminating the need for secondary filtering. KV and the upper layer do not need to transmit invalid data, thereby improving retrieval efficiency.

[0051] In this embodiment, S201 shown can also be implemented by S301 to S302, which will be described in conjunction with the steps: S301. Determine the retrieval type of the retrieval request based on the predetermined string in the retrieval request.

[0052] In this embodiment, upon receiving a search request, the search type of the search request can be determined based on a predetermined string in the search request that indicates the type of search request.

[0053] In this embodiment, the search type of the search request can be determined based on the string "Get" in the search request Get(Key, VersionID, Operator(Limit1, Condition)). Alternatively, the search type of the search request can be determined based on the string "Scan" in the search request Scan(Key1, Key2, Operator(Latest, Limit1, Condition)).

[0054] S302. When the search type is a specified search, determine the metadata block, the index block, and the statistics block as the filter block; and when the search type is a range search, determine the index block and the statistics block as the filter block.

[0055] In this embodiment, when the search type is a specified search, the metadata block, the index block, and the statistics block are determined as the filter block; and when the search type is a range search, the index block and the statistics block are determined as the filter block.

[0056] In this embodiment, when the search type is a specified search, the metadata block, the index block, and the statistics block are determined as the filter blocks; and when the search type is a range search, the index block and the statistics block are determined as the filter blocks. This allows for filtering in the key-value file through operator pushdown based on search conditions and filter blocks corresponding to different search types. This ensures that the data received by the upper layers (object storage, file storage, etc.) is always valid, eliminating the need for secondary filtering. Furthermore, the KV (key-value pair) and the upper layers do not need to transmit invalid data, thereby improving search efficiency.

[0057] In this embodiment, S202 shown can also be implemented by S401 to S402, which will be described in conjunction with the steps: S401. Based on the metadata block, process the search conditions to determine whether the target data block exists in the key-value pair file.

[0058] In this embodiment, when the retrieval request is a specified retrieval type, the retrieval request specifies a specific version of the specified object, i.e., Get(Key, VersionID, Operator(Limit1, Condition)). The retrieval process of the KV storage is as follows: Based on the Key, determine the Bloom filter block in the specific metadata block; check if the data exists based on Key + VersionID; if not, return directly. In other words, determine if there are retrieval results for the target data block.

[0059] S402. When the inspection result indicates existence, based on the index block, find the target data block identifier that meets at least one of the first key condition and the first version information condition, and based on the statistical information of the key and version information stored in the statistical information block, filter the invalid data blocks in the data block indicated by the target data block identifier to determine the target data block.

[0060] In this embodiment, if the search result representation exists, the target data block identifier that meets at least one of the first key condition and the first version information condition is found based on the index block. Based on the statistical information of the key and version information stored in the statistical information block, it is confirmed whether the data block indicated by the target data block identifier meets the requirements of Key and VersionID. If it does, the target data block is determined, and the target data block is loaded into the Block Cache; the data is read, and it is determined whether it meets the Condition condition.

[0061] In this embodiment, when the inspection result characterization exists, the target data block identifier that meets at least one of the first key condition and the first version information condition is searched based on the index block. Then, based on the statistical information of the key and version information stored in the statistical information block, invalid data blocks in the data block indicated by the target data block identifier are filtered to determine the target data block. In this way, invalid data in data blocks that meet the search conditions can be filtered through the statistical information block, ensuring that the data received by the upper layer (object storage, file storage, etc.) is all valid data, eliminating the need for secondary filtering. KV and the upper layer do not need to transmit invalid data, thereby improving retrieval efficiency.

[0062] In this embodiment, S202 shown can also be implemented by S501 to S502, which will be described in conjunction with the steps: S501. Based on the index block, find the target data block identifier range that meets the second key condition; wherein, the second key condition belongs to the search condition.

[0063] In this embodiment, when the retrieval request is a range retrieval, the retrieval request specifies the latest version of the object, i.e., Scan(Key1,Key2,Operator(Latest,Limit1,Condition)). The retrieval process of KV storage is as follows: Based on the range of (Key1,Key2), the index block index is used to determine the identifier of the start data and the identifier of the end DataBlock.

[0064] S502. Based on the statistical information of the keys and version information stored in the statistical information block, filter out invalid data blocks in the data blocks indicated by the target data block identifier range, and determine the target data block.

[0065] In this embodiment, invalid data blocks are filtered out based on statistical information in the statistical information block, such as Key, VersionID, and DeletedMarker. A specified number of data blocks are read from the valid data blocks, and it is determined whether they meet the search criteria. After reading data that meets the quantity and latest version requirements, invalid data is skipped. Valid data is merged, and it is determined whether the result data meets the search criteria. Data that meets the requirements is returned. Data retrieved from memory and files is merged, the Operator and Condition conditions are executed, and data that meets the conditions is returned.

[0066] If the statistical information in the statistical information block indicates that some data blocks in the target data block identifier range have a Deleted Marker, then the data block can be determined to be an invalid data block, thereby achieving the filtering effect.

[0067] In this embodiment, based on the statistical information of keys and version information stored in the statistical information block, invalid data blocks in the data blocks indicated by the target data block identifier range are filtered to determine the target data block. In this way, invalid data in data blocks that meet the search criteria can be filtered through the statistical information block, ensuring that the data received by the upper layers (object storage, file storage, etc.) is all valid data, eliminating the need for secondary filtering. KV and the upper layers do not need to transmit invalid data, thereby improving search efficiency.

[0068] Please see Figure 6 This is a schematic diagram of the structure of the data retrieval device provided in the embodiments of this application.

[0069] This application also provides a data retrieval device 800, including: an acquisition unit 801, a retrieval unit 802, and a merging unit 803.

[0070] The acquisition unit 801 is used to acquire a retrieval request and determine the key-value pair file to be retrieved based on the retrieval request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by merging them locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources; The retrieval unit 802 is configured to, based on the retrieval request, filter invalid data in the filter blocks of the file using the key value to determine the first retrieval data, and retrieve and determine the second retrieval data in memory; The merging unit 803 is used to determine the target retrieval data based on the merging of the first retrieval data and the second retrieval data.

[0071] In this embodiment, the key-value pair file includes: a data block, an index block, a metadata block, a statistics block, a file information block, and a tail block; The data block is used to store key-value pair data; the index block is used to store index information of the keys; the metadata block is used to determine whether the key-value pair file stores the specified key and version information; the statistics block is used to store statistical information of the keys and version information in the data block; the file information block is used to store the metadata of the key-value pair file; and the tail block is used to store the basic information of the key-value pair file.

[0072] In this embodiment, the retrieval unit 802 in the data retrieval device 800 is used to determine the filter block among multiple blocks of the key-value pair file based on the retrieval type characterized by the retrieval request. Based on the filter block and the search conditions in the search request, target data blocks that meet the search conditions are filtered and determined. Read the first search data that meets the search criteria from the target data block.

[0073] In this embodiment, the retrieval unit 802 in the data retrieval device 800 is used to determine the retrieval type of the retrieval request based on a predetermined string in the retrieval request; When the search type is a specified search, the metadata block, the index block, and the statistics block are determined as the filter block; when the search type is a range search, the index block and the statistics block are determined as the filter block.

[0074] In this embodiment, the retrieval unit 802 in the data retrieval device 800 is used to process the retrieval conditions based on the metadata block to determine whether the target data block exists in the key-value pair file; When the inspection result indicates existence, the target data block identifier that meets at least one of the first key condition and the first version information condition is found based on the index block, and invalid data blocks in the data block indicated by the target data block identifier are filtered based on the statistical information of the key and version information stored in the statistical information block to determine the target data block; the first key condition and the first version information condition belong to the search conditions.

[0075] In this embodiment, the retrieval unit 802 in the data retrieval device 800 is used to search for a target data block identifier range that meets the second key condition based on the index block; wherein, the second key condition belongs to the retrieval condition; Based on the statistical information of keys and version information stored in the statistical information block, invalid data blocks in the data blocks indicated by the target data block identifier range are filtered out to determine the target data block.

[0076] It should be noted that, in the embodiments of this application, if the above-described data retrieval method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a data retrieval device (which may be a personal computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0077] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the method of the data retrieval device 800.

[0078] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0079] It should be noted that, Figure 7 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, this application embodiment provides an electronic device 900, including a memory 902 and a processor 901. The memory 902 stores a computer program that can run on the processor 901. When the processor 901 executes the program, it implements the steps in the above-described method, wherein; Processor 901 typically controls the overall operation of electronic device 900.

[0080] The memory 902 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 901 and various modules in the electronic device 900. It can be implemented by flash memory or random access memory (RAM).

[0081] Correspondingly, this application embodiment also provides a computer program product, including a computer program that can be executed by the processor 901 of the electronic device 900 to complete the steps in the method on one side of the data retrieval device 800.

[0082] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data retrieval method, characterized in that, include: A search request is obtained, and the key-value pair file to be searched is determined based on the search request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining it locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources; Based on the search request, the first search data is determined by filtering the filter blocks in the file in combination with the key value, and the second search data is determined by searching in memory; Based on the merging of the first search data and the second search data, the target search data is determined.

2. The data retrieval method according to claim 1, characterized in that, The key-value pair file includes: data blocks, index blocks, metadata blocks, statistics blocks, file information blocks, and tail blocks; The data block is used to store key-value pair data; the index block is used to store index information of the key; the metadata block is used to determine whether the specified key and version information are stored in the key-value pair file; the statistics block is used to store statistical information of the key and version information in the data block; the file information block is used to store the metadata of the key-value pair file; and the tail block is used to store the basic information of the key-value pair file.

3. The data retrieval method according to claim 2, characterized in that, The step of filtering and retrieving the first search data based on the search request and the key value in the file includes: Based on the retrieval type represented by the retrieval request, the filter block is determined among multiple blocks of the key-value pair file; Based on the filter block and the search conditions in the search request, target data blocks that meet the search conditions are filtered and determined. Read the first search data that meets the search criteria from the target data block.

4. The data retrieval method according to claim 3, characterized in that, The step of determining the filter block among multiple blocks of the key-value pair file based on the retrieval type represented by the retrieval request includes: Based on the predetermined string in the search request, determine the search type of the search request; When the search type is a specified search, the metadata block, the index block, and the statistics block are determined as the filter block; when the search type is a range search, the index block and the statistics block are determined as the filter block.

5. The data retrieval method according to claim 4, characterized in that, When the search type is a specified search, the step of filtering and determining the target data block that meets the search conditions based on the filter block and the search conditions in the search request includes: Based on the metadata block, the search conditions are processed to determine whether the target data block exists in the key-value pair file; When the inspection result indicates existence, the target data block identifier that meets at least one of the first key condition and the first version information condition is searched based on the index block, and invalid data blocks in the data block indicated by the target data block identifier are filtered based on the statistical information of the key and the version information stored in the statistical information block to determine the target data block; the first key condition and the first version information condition belong to the search conditions.

6. The data retrieval method according to claim 4, characterized in that, When the search type is a range search, the step of filtering and determining the target data block that meets the search conditions based on the filter block and the search conditions in the search request includes: Based on the index block, find the target data block identifier range that meets the second key condition; wherein, the second key condition belongs to the search condition; Based on the statistical information of the key and the version information stored in the statistical information block, invalid data blocks in the data blocks indicated by the target data block identifier range are filtered out to determine the target data block.

7. A data retrieval device, characterized in that, include: An acquisition unit is used to acquire a retrieval request and determine the key-value pair file to be retrieved based on the retrieval request; wherein, the key-value pair file is determined by merging multiple key-value pair sub-files using other devices, or by determining it locally using a background thread; the foreground thread and the background thread are respectively bound to different central processing unit computing resources; The retrieval unit is configured to, based on the retrieval request, filter and retrieve the filter blocks in the file using the key value to determine the first retrieval data, and retrieve and determine the second retrieval data in memory; The merging unit is used to determine the target retrieval data based on the merging of the first retrieval data and the second retrieval data.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images