Data query method, device, equipment and medium for storage system

Abstract

The present application discloses a data query method, apparatus, device, and medium for a storage system, which obtains an address range of data to be queried; determines at least one target partition corresponding to the address range based on pre-stored data partition information of the storage system; when performing a data query on a first partition of at least one target partition, determines the next second partition to be queried corresponding to the first partition based on a pre-stored transition probability matrix, and pre-reads data on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix are used to represent the probability of pre-reading between various data in the storage system; and summarizes at least one query result to obtain a target query result corresponding to the data to be queried. By indexing at least one partition separately, the performance of range queries is significantly optimized, and random query latency is reduced. In addition, data pre-reading through the transition probability matrix can effectively improve query efficiency and reduce access waiting time.

Description

Technical Field

[0001] The present application relates to the technical field of storage systems, and in particular to a data query method, apparatus, device, and medium for a storage system. Background Art

[0002] The storage system stores a large amount of data. During operation, data may continue to be written, existing data may be modified, or data may be queried and read from the storage system at any time. Frequent access to the storage system may result in long access wait times due to global lock restrictions, and traditional query methods may have certain data transmission delays in storage scenarios. Summary of the Invention

[0003] The present application provides a data query method, apparatus, device and medium for a storage system to at least solve the problem that frequent access to the storage system may result in long access waiting time due to global lock restrictions, and traditional query methods may have certain data transmission delays in storage scenarios.

[0004] The present application provides a data query method for a storage system, comprising: obtaining an address range of data to be queried;

[0005] Determine at least one target partition corresponding to the address interval based on pre-stored data partition information of the storage system, where the data partition information is obtained by pre-dividing the data of the storage system based on the logical block address range;

[0006] When performing a data query on a first partition of at least one target partition, determining a next second partition to be queried corresponding to the first partition based on a pre-stored transition probability matrix, and pre-reading data on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix are used to represent the probability of pre-reading between respective data in the storage system;

[0007] Aggregate at least one query result to obtain a target query result corresponding to the data to be queried.

[0008] The present application also provides a data query device for a storage system, comprising: an acquisition module, configured to acquire an address range of data to be queried;

[0009] a processing module, configured to determine at least one target partition corresponding to an address interval based on pre-stored data partition information of the storage system, wherein the data partition information is obtained by pre-dividing the data of the storage system according to a logical block address range;

[0010] The processing module is further configured to, when performing a data query on a first partition of at least one target partition, determine a second partition to be queried next corresponding to the first partition based on a pre-stored transition probability matrix, and pre-read data on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix are used to represent the probability of pre-reading between respective data in the storage system;

[0011] The processing module is further used to aggregate at least one query result to obtain a target query result corresponding to the data to be queried.

[0012] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned storage system data query methods when executing the computer program.

[0013] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data query methods for the storage system are implemented.

[0014] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data query methods for the storage system when the computer program is executed by a processor.

[0015] Through this application, the address range of the data to be queried is obtained; based on the pre-stored data partition information of the storage system, at least one target partition corresponding to the address range is determined, and the data partition information is obtained by pre-dividing the data of the storage system according to the logical block address range; when performing a data query on the first partition of at least one target partition, the next second partition to be queried corresponding to the first partition is determined based on the pre-stored transition probability matrix, and the data of the second partition is pre-read to obtain at least one query result, and the elements in the transition probability matrix are used to represent the probability of pre-reading between each data in the storage system; at least one query result is summarized to obtain the target query result corresponding to the data to be queried. In this solution, by indexing at least one partition separately, the range query performance is significantly optimized and the random query delay is reduced. In addition, by implementing data pre-reading through the transition probability matrix, the query efficiency can be effectively improved and the access waiting time can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1The process of a data query method of a storage system provided in an embodiment of the present application is Figure 1 ;

[0018] Figure 2 The process of a data query method of a storage system provided in an embodiment of the present application is Figure 2 ;

[0019] Figure 3 A structural diagram of a data query device for a storage system provided in an embodiment of the present application;

[0020] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0023] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0024] Modern storage systems may face three core challenges:

[0025] (1) Lock contention under high concurrency: Traditional indexes (such as B+ trees and LSM-Trees) suffer from a sharp drop in performance due to global locks in distributed and centralized scenarios.

[0026] (2) Adaptability to mixed loads: Random writes (such as database transactions) and sequential reads (such as data analysis) require different optimization strategies, and existing systems are difficult to take into account both.

[0027] (3) Memory and storage wall: DRAM access latency (about 100ns) and NVMe SSD latency (about 10μs) differ by two orders of magnitude, requiring efficient caching and pre-reading mechanisms.

[0028] Industry pain points:

[0029] (1) Ceph: It uses the Controlled Replication Under Scalable Hashing (CRUSH) algorithm to implement distributed storage, but metadata management relies on a global map, and the random write throughput is less than 500K IOPS (4K blocks).

[0030] (2) Storage Performance Development Kit (SPDK): Improves performance through user-mode polling, but lacks intelligent indexing, and range query efficiency is only 30% of point query.

[0031] (3) Aerospike: uses a hybrid index (memory hash + disk B+ tree), but the memory fragmentation rate is as high as 15%-20%.

[0032] It's important to note that a B+ tree, or Balanced Plus Tree, is widely used in databases and file systems as an index structure to optimize data storage and retrieval efficiency. Compared to a B-tree, a B+ tree stores all data records (or pointers to data records) in its leaf nodes. Leaf nodes are connected by pointers, forming an ordered linked list. This makes range queries and sequential access more efficient.

[0033] It should be noted that LSM-Tree stands for Log-Structured Merge-Tree. LSM-Tree optimizes high-throughput write scenarios through sequential writes and layered merging. It is particularly suitable for big data applications with high write throughput and low read throughput (such as log storage and time series data). Leveraging the fact that sequential write performance on disk far outperforms random writes, all update operations (inserts, deletes, and modifications) are converted to append writes to avoid random I / O. Data is divided into an in-memory component (MemTable) and a disk component (SSTable). Periodic merging (compaction) reduces redundant data and improves storage efficiency. Write operations only involve in-memory appends, without requiring immediate disk updates, making it suitable for high-concurrency scenarios.

[0034] It should be noted that a global lock is a mechanism used in databases or distributed systems to control concurrent access to shared resources. It ensures data consistency and atomicity of operations by locking the entire system or critical resources. A global lock is a coarse-grained locking mechanism that prevents concurrent access to protected resources by other transactions or processes by locking the entire database, system, or critical resources. While a global lock is in effect, any attempt to access the protected resource is blocked until the lock is released. This mechanism is often used in scenarios where strict data consistency must be guaranteed, such as database backups, system maintenance, or atomic updates of critical data structures.

[0035] It should be noted that DRAM can also refer to dynamic random access memory (DRAM). DRAM stores data in a row-column matrix. Accessing data requires first activating the target row (row address strobe, RAS) and then locating the specific cell using column address strobe (CAS). This process involves reading and amplifying the capacitor charge, resulting in row activation delay (tRCD) and column access delay (tCAS). DRAM requires periodic refresh to maintain data integrity (dynamic characteristics). Refresh operations occupy the memory bus, causing other access requests to queue. The process quality of the memory chip directly affects the charge and discharge speed. Advanced processes can shorten capacitor charge and discharge times, reducing latency. In addition, attenuation and interference in signal transmission (such as bus length and noise) can also increase latency.

[0036] It should be noted that NVMe SSDs can refer to non-volatile memory express (SSDs). NVMe SSDs typically offer random read latencies between 20 and 80 microseconds (μs), random write latencies between 50 and 150 μs, and total latency within 0.02 to 0.15 milliseconds (ms). This latency is orders of magnitude lower than traditional SATA SSDs and HDDs, which typically have latencies of several milliseconds. The low latency of NVMe SSDs makes them particularly effective in latency-sensitive scenarios such as database operations, virtual machine operations, and real-time analytics.

[0037] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0038] like Figure 1 As shown, Figure 1 A flowchart of a data query method for a storage system provided in an embodiment of the present application, the method may include the following steps:

[0039] 101. Obtain the address range of the data to be queried.

[0040] In the embodiment of the present application, the address range of the data to be queried can be expressed as [Lstart, Lend], which can refer to an interval range consisting of a continuous logical block address.

[0041] It should be noted that Logical Block Addressing (LBA) is an addressing method used for disk storage devices such as hard drives and solid-state drives. LBA maps physical blocks on a disk to logical numbers, making it easier for the operating system to access and manage data on the disk. LBA numbers all physical blocks on the disk sequentially, starting at 0, and each physical block has a unique logical address. The operating system uses these logical addresses to access data on the disk without worrying about the data's actual physical location on the disk. Each LBA corresponds to a fixed-size data block (such as 4KB) and is often used as a key in hash indexes.

[0042] In some embodiments, the method for obtaining the address range of the data to be queried may include at least the following implementation methods:

[0043] Implementation method 1: User request.

[0044] When a user or application needs to read or write a continuous section of data, a starting address (L start ) and the end address (L end For example, if a user needs to read a portion of a file, and the file is stored between LBA 1000 and LBA 2000, then the address range is [1000, 2000].

[0045] Implementation method 2: system task.

[0046] The system may need to perform some background tasks, such as data migration, backup, or garbage collection. These tasks usually involve a continuous LBA range. For example, if the system needs to migrate the data of a partition to another node, it will specify an interval [Lstart, Lend] to represent the data range to be migrated.

[0047] Implementation method three: pre-reading mechanism.

[0048] To improve performance, the system may predict the data the user will need next based on access patterns and read a contiguous LBA range in advance. For example, if the system predicts that the user will read data from LBA 1000 to LBA 1500, the interval would be [1000, 1500].

[0049] 102. Determine at least one target partition corresponding to the address range according to pre-stored data partition information of the storage system.

[0050] In an embodiment of the present application, a storage system stores a large amount of data. To optimize storage and enable faster data location, query, and retrieval, the storage system can be partitioned. The specific data partition information is obtained by pre-dividing the storage system data according to the logical block address range. After determining the address range of the data to be queried, it is necessary to determine the partition corresponding to the address range. The number of partitions can be at least one, that is, to determine at least one target partition covered by the address range.

[0051] In some embodiments, the system needs to know [L start , L end ] range is stored in which partitions, in order to correctly read or write data, for example: if the interval [1000, 2000] is distributed in partitions S1, S2 and S3, then the system needs to read data from these partitions respectively. In addition, the query interval can help the system understand the uniformity of data distribution, so as to perform load balancing. For example: if the amount of data in a partition is too large, the system can migrate part of the data to other partitions to avoid hot spots. At the same time, the query interval can help the system optimize data access performance, for example: if [L start , L end ] spans multiple partitions, the system can minimize cross-partition access and improve performance; if the interval [L start , L end ] is distributed across multiple partitions, and the system can pre-read data from these partitions in advance to reduce latency. Also, during data migration or system expansion, query intervals can help the system determine the data range that needs to be migrated. For example, if the system needs to migrate the data of a partition to a new node, it is necessary to query the distribution of all intervals in the partition. Finally, when a system failure occurs, query intervals can help the system quickly locate the affected data range and perform recovery. For example, if a partition fails, the system can query the distribution of all intervals in the partition and restore the data from the backup.

[0052] 103. When performing data query on a first partition in at least one target partition, determine a next second partition to be queried corresponding to the first partition according to a pre-stored transition probability matrix, and pre-read data on the second partition to obtain at least one query result.

[0053] In an embodiment of the present application, after determining at least one target partition corresponding to the address range, it is necessary to perform data query in the at least one target partition. During the data query process, the B+ tree structure in each target partition can be traversed in parallel, that is, the data in each target partition in at least one target partition is traversed at the same time, thereby obtaining at least one query result. The at least one query result can correspond to at least one target partition, that is, each target partition can obtain a query result.

[0054] It should be noted that when querying data on each target partition, a pre-read mechanism can be used to ensure data access efficiency. Pre-reading can be simply understood as predicting which partition's data will be queried next and pre-fetching it when querying the current partition's data. That is, when querying data on the first partition, the next partition likely to be queried can be predicted and pre-read. If the prediction is correct, data can be queried directly without waiting. To improve the accuracy of data pre-read prediction, a pre-stored transition probability matrix can be used to specifically determine the partition. The elements in this transition probability matrix can be used to represent the probability of pre-reading between each data in the storage system. That is, the transition probability matrix stores the probability of pre-reading between the first partition and each other partition. When querying data on the first partition, the probability of pre-reading between the first partition and each other partition can be used to determine the second partition and pre-read it. Of course, when querying other partitions, the steps are the same: the pre-read mechanism is used to pre-read the data of the next partition likely to be queried, thereby obtaining at least one query result, which can correspond one-to-one with the partition.

[0055] 104. Aggregate at least one query result to obtain a target query result corresponding to the data to be queried.

[0056] In an embodiment of the present application, after each partition in at least one target partition is queried to obtain at least one query result, the at least one query result may be aggregated to obtain a final target query result corresponding to the data to be queried.

[0057] For example, the data query process described above can be transformed into a scenario where you search for books in a library. Specifically, a library has many bookshelves (partitions), each of which contains many books (data). Each book has a unique ID (LBA) that allows it to be quickly found. Now, you need to find a batch of consecutively numbered books, such as books numbered 1000 to 2000. These books may be distributed across different shelves, so you need to search for the books on each relevant shelf and then organize the found books into a single list to generate the final result.

[0058] During the book search process, several assistants (parallel tasks) can simultaneously search different bookshelves (traversing partitions in parallel). For example, Assistant A searches for bookshelf 1, Assistant B searches for bookshelf 2, and Assistant C searches for bookshelf 3. Each assistant has a map (B+ tree) that allows them to quickly find books (data) numbered between 1000 and 2000. For example, Assistant A finds books {D1, D2}, Assistant B finds books {D3, D4}, and Assistant C finds book {D5}. Finally, after all the books (data) have been found, the books (data) found by all assistants are aggregated to form the final result (the target query result), resulting in a stack of books numbered from 1000 to 2000: {D1, D2, D3, D4, D5}.

[0059] In an embodiment of the present application, by indexing at least one partition separately, the range query performance is significantly optimized and the random query latency is reduced. In addition, data pre-reading is achieved through the transition probability matrix, which can effectively improve query efficiency and reduce access waiting time.

[0060] like Figure 2 As shown, Figure 2 Another flowchart of a data query method for a storage system provided in an embodiment of the present application, the method may include the following steps:

[0061] 201. Obtain the address range of the data to be queried.

[0062] In the embodiment of the present application, for the description of step 201, please refer to the detailed description of step 101 in the above embodiment, and the embodiment of the present application will not be repeated.

[0063] 202. Obtain data to be written.

[0064] In an embodiment of the present application, before performing operations such as querying and reading the data, the data must first be written into the storage system. This writing can be understood as writing new data or modifying the stored data.

[0065] 203. Map the data to be written according to a preset hash function, store the data to be written in the corresponding partition, and obtain data partition information.

[0066] In an embodiment of the present application, since the storage system includes multiple partitions, when writing data, it is necessary to determine which specific partition each data needs to be written to. At this time, it is necessary to map the data to be written and determine the partition corresponding to the data to be written, so as to store the data to be written in the corresponding partition. Since the number of data to be written may be at least one, the corresponding partition may also be at least one partition. Each data will correspond to a partition to be stored. After all storage is completed, the data partition information can be obtained, and the data partition information can indicate which data is stored in each partition of the storage system.

[0067] It should be noted that when mapping the data to be written, it can be achieved by using a preset hash function. The preset hash function can be an improved Fibonacci hash function. The improved Fibonacci hash function can be expressed as:

[0068]

[0069] Among them, k can represent the original key value (such as LBA address), which is input by the outside (user); N can represent the total number of partitions. It is the golden ratio, a fixed constant that can be used to reduce hash conflicts.

[0070] In some embodiments, h(k) represents the result of computing a hash function H on an input key k. Specifically, h(k) is a function that maps a key k to a hash value. This hash value is typically used to determine the storage location of data in a hash table. The primary function of h(k) is to map the key k to a partition number, thereby determining the partition in which the data should be stored.

[0071] In some embodiments, a compare-and-swap (CAS) insertion can be performed during data writes. CAS insertion is a common lock-free concurrency control mechanism primarily used to implement thread-safe atomic operations. When referring to "CAS insertion," it generally refers to safely inserting elements into data structures (such as linked lists, trees, and hash tables) using the CAS mechanism in a concurrent environment. A thread first reads the current value of the target memory location (or a node in the data structure) and attempts to update the target memory location to the new value. However, this operation is conditional: the update succeeds only if the current value of the target memory location is the same as the value previously read by the thread. This is typically implemented using a special CPU instruction (such as CMPXCHG in the x86 architecture). If the update succeeds, it indicates that no other thread has modified the target memory location in the interim, and the insertion is complete. If the update fails, it indicates that another thread has modified the target memory location in the interim, and the thread must reread the current value and potentially retry the insertion (typically implemented in a loop until the insertion succeeds or another termination condition is met).

[0072] In some embodiments, the storage system pre-implements partitioning, assuming that the total LBA range of the storage space is [0, L max ], the number of partitions is N, then the partition S i The managed LBA subspace can be expressed as:

[0073]

[0074] Among them, L max It can represent the maximum LBA address, N can represent the total number of partitions, initialized according to the cluster size (such as 1024 partitions), and i can represent the partition index (0 to N-1).

[0075] In some embodiments, when writing data, since the number of partitions is limited, there may be partition conflicts between the data. Specifically, the data to be written is mapped according to a preset hash function, and the data to be written is stored in the corresponding partition to obtain data partition information. This may include: mapping the data to be written according to the preset hash function to obtain a mapping result; if the mapping result indicates that there is a partition conflict between at least two conflicting data, merging the at least two conflicting data to obtain a data merge interval; updating the version number according to the data merge interval, and storing the data merge interval to obtain data partition information.

[0076] It should be noted that data conflicts may manifest as partition mapping conflicts, intra-bucket conflicts, and linked list or tree structure degradation. A partition mapping conflict occurs when different keys are mapped to the same partition. Assuming the hash function h(k) maps keys to partition numbers, and the number of partitions N = 1024, if there are two keys k1 = 123456 and k2 = 654321, and h(k1) = h(k2) = 512, then k1 and k2 are mapped to the same partition S512, which is a partition mapping conflict. An intra-bucket conflict occurs when different keys are mapped to the same bucket within the same partition. Assuming partition Si has multiple buckets, each capable of storing a certain number of keys, if two keys k1 and k2 are mapped to the same partition Si and then further mapped to the same bucket, this is an intra-bucket conflict. Linked list or tree structure degradation means that when a conflict occurs, the hash table may degenerate into a linked list or tree structure, resulting in a decrease in query efficiency. If the buckets within partition Si use linked lists to handle conflicts, when multiple keys are mapped to the same bucket, the length of the linked list will increase, and the query time complexity will degenerate from O(1) to O(n).

[0077] When a hash collision occurs (i.e., different LBAs are mapped to the same partition or bucket), directly storing these LBAs will lead to the following problems: low storage efficiency: each LBA needs to be stored separately, increasing metadata overhead; poor query performance: multiple independent LBAs need to be traversed, increasing query time.

[0078] Merging adjacent LBAs can reduce the number of hash collisions. If two LBAs are adjacent (for example, LBA1 = 1000 and LBA2 = 1001), they likely belong to the same data block or contiguous data blocks. Merging them into a single range (for example, [1000, 1001]) reduces hash collisions because only one range needs to be stored instead of multiple independent LBAs. Merging adjacent LBAs allows multiple LBAs to be stored as a single range, reducing metadata overhead such as pointers and index entries. For example, storing the range [1000, 1001] requires only one index entry, while storing two independent LBAs requires two. Merging adjacent LBAs allows queries to quickly locate the entire range without traversing multiple independent LBAs. For example, a query for LBA = 1000 can directly locate the range [1000, 1001] without having to query LBA1 = 1000 and LBA2 = 1001 separately. In practical applications, many access patterns are sequential (such as continuously reading multiple data blocks). Merging adjacent LBAs can better adapt to this sequential access pattern and reduce the overhead of pre-reading and caching.

[0079] Therefore, if there is a partition conflict between at least two conflicting data, the conflicting data can be merged and then stored. At the same time, the partition version number needs to be updated synchronously to avoid system errors.

[0080] In some embodiments, data may not necessarily conflict when written. The probability of conflict is determined by the number of partitions, the size of the buckets within the partitions, and the number of keys. Specifically, the probability of conflict can be expressed as:

[0081] Among them, N can represent the total number of partitions, B can represent the number of hash buckets managed by each partition, the default value is 64, and K can represent the current total number of keys. When N=1024, B=64, K=10 8 When P collision <0.05%.

[0082] In some embodiments, to reduce conflicts, the number of drawers (partitions) (N) can be increased. More drawers provide a larger distribution space for items (data), reducing the probability of conflicts. For example, increasing the number of drawers from 512 to 1024 significantly reduces the probability of conflicts. Hash functions can also be optimized. A hash function evenly distributes items (data) across drawers (partitions). Using a better hash function (such as one based on the golden ratio) can achieve a more even distribution and reduce conflicts. The number of items (data) (K) can also be controlled. A smaller number of items (data) reduces the probability of conflicts. For example, when K = 108, the probability of conflicts is much lower than when K = 1000. The hash space (B) can also be increased. A larger hash space provides a more distributed distribution of items (data), reducing the probability of conflicts. For example, a B = 64 reduces conflicts better than a B = 32.

[0083] In some embodiments, a lower collision probability results in higher system performance and efficiency. Specifically, a lower collision probability results in faster data queries because there is no need to process crowded drawers; a lower collision probability results in a more even distribution of items, leading to a more balanced system load; a lower collision probability results in higher resource utilization, as additional resources are not required to handle collisions; and a lower collision probability results in a more stable system, with fewer performance bottlenecks.

[0084] Through this solution, if there is a conflict between data, the data can be merged and then stored, which can reduce the number of hash conflicts, optimize the storage structure by reducing metadata, and quickly locate the interval during query to improve query efficiency.

[0085] In some embodiments, during the data writing process, data may be compressed to save space and reduce data size to improve transmission efficiency. Specifically, storing the data to be written in the corresponding partition and obtaining data partition information may include: detecting data transmission delay in real time; determining a data compression level based on the data transmission delay; and compressing the data to be written according to the data compression level and storing it in the corresponding partition to obtain data partition information.

[0086] It should be noted that the compression level can be dynamically adjusted according to the system load. Specifically, the compression level can be expressed as:

[0087]

[0088] The SLA maximum allowable latency refers to the maximum latency allowed by the service-level agreement (in milliseconds) and can be customized by the user. Current I / O latency can be obtained through real-time monitoring or collected at a specific sampling interval (e.g., 10 seconds). The calculated compression level can be customized, for example, from 9 levels (1 to 9), with 1 being the highest and 9 being the lowest.

[0089] In some embodiments, when compressing data, if some data blocks are small, adjacent small data blocks may be merged before compression. For example, multiple 4KB data blocks may be merged into 1MB data blocks, and then compressed. After compression, the following conditions must be met:

[0090]

[0091] Among them, B merged It can represent the size of the merged data block, B compressed It can represent the size of the compressed data block. 4KB can refer to the size of the minimum storage unit, which is a fixed constant. merged =1MB, the compression ratio is increased to 3.2 times.

[0092] Through this solution, data can be compressed according to the data transmission delay, which can reduce the storage space required for the data, improve the utilization of the storage system, allow more data to be stored, and at the same time ensure the efficiency of data transmission.

[0093] 204. Save data partition information.

[0094] In an embodiment of the present application, after obtaining the data partition information, the data partition information can be stored for subsequent calculations.

[0095] This solution enables partitioned data writing. By introducing hash functions and lock-free design, data conflicts can be effectively avoided. At the same time, the system's writing capabilities can be efficiently expanded in distributed scenarios.

[0096] 205. Determine at least one target partition corresponding to the address range according to pre-stored data partition information of the storage system.

[0097] In the embodiment of the present application, for the description of step 205, please refer to the detailed description of step 102 in the above embodiment, and the embodiment of the present application will not be repeated.

[0098] 206. Obtain the historical access sequence.

[0099] In an embodiment of the present application, the historical access sequence can be used to represent the order of access to each data during the historical data query process. For example, if the access sequence is [LBA1, LBA2, LBA3, LBA2, LBA3, LBA1], it means that the data corresponding to LBA1 is accessed first, then the data corresponding to LBA2 is accessed, then the data corresponding to LBA3 is accessed, then the data corresponding to LBA2 is accessed, then the data corresponding to LBA3 is accessed, and then the data corresponding to LBA1 is accessed.

[0100] In some embodiments, the access sequence can be an access sequence between data or an access sequence between partitions. The historical access sequence is used to calculate the transition probability matrix for subsequent pre-reading. That is to say, when pre-reading, pre-reading can be performed between data or between partitions, and the logical steps are the same.

[0101] 207. Count the number of transfers between the first data and the second data according to the historical access sequence.

[0102] In the embodiment of the present application, the first data and the second data are both arbitrary data in the storage system, that is, it is necessary to count the number of transfers between any two data, and the transfers between the data are different in direction, that is, the transfer of the first data to the second data and the transfer of the second data to the first data are different concepts and need to be counted separately.

[0103] For example, the access sequence is [LBA1, LBA2, LBA3, LBA2, LBA3, LBA1]. Then, it can be seen from the statistics that LBA1 is transferred to LBA2 1 time, LBA1 is transferred to LBA3 0 times, LBA2 is transferred to LBA1 0 times, LBA2 is transferred to LBA3 2 times, LBA3 is transferred to LBA1 1 time, and LBA3 is transferred to LBA2 1 time.

[0104] 208. Determine a pre-reading probability between the first data and the second data according to the number of transfers.

[0105] In the embodiment of the present application, the pre-reading probability can be expressed as the ratio between the number of transfers and the total number of transfers. The pre-reading probability can be expressed as:

[0106]

[0107] Among them, P ij It can be represented from state S i to S j The read-ahead probability. S i Indicates the current status (currently accessed LBA or LBA group), S j Indicates the next state (the next possible access LBA or LBA group), P ij It reflects the pre-reading rules in the LBA access mode and is used to construct the state transition matrix and optimize the pre-reading strategy.

[0108] For example, the access sequence is [LBA1, LBA2, LBA3, LBA2, LBA3, LBA1]. Then, when counting, it can be seen that the number of times LBA1 transfers to LBA2 is 1, the number of times LBA1 transfers to LBA3 is 0, the number of times LBA2 transfers to LBA1 is 0, the number of times LBA2 transfers to LBA3 is 2, the number of times LBA3 transfers to LBA1 is 1, and the number of times LBA3 transfers to LBA2 is 1. After calculation, P 12 =1 (always access LBA2 after LBA1), P 23 =1 (always access LBA3 after LBA2), P 32 =0.5 (50% probability of accessing LBA2 after LBA3), P 31 =0.5 (there is a 50% probability of accessing LBA1 after LBA3).

[0109] 209. Summarize the pre-reading probabilities to obtain a transfer probability matrix.

[0110] In the embodiment of the present application, the pre-reading probabilities obtained by the above calculations can be combined to obtain a transition probability matrix.

[0111] In some embodiments, the data within the partition can also be divided into queues based on the historical access sequence and the current access data, thereby forming a dual LRU queue model, which can specifically include a Hot queue and a Warm queue. The Hot queue can store high-frequency access data, accounting for approximately 20% of the capacity; the Warm queue can store low-frequency access but possible pre-read data, accounting for approximately 80% of the capacity.

[0112] In some embodiments, by dividing the queues described above, a hit rate can be calculated to adjust the queues and the pre-reading strategy. The hit rate can be expressed as:

[0113]

[0114] Among them, λ h and λ w They can represent the real-time access frequency (times / second) of the Hot queue and the Warm queue respectively. h and C w They can represent the capacity (number of entries) of the Hot queue and the Warm queue respectively, and can be fixed constants; It can indicate the proportion of hot data requests and can be obtained through access log analysis.

[0115] By calculating the hit rate, we can analyze whether the queue division is correct and whether the pre-reading strategy is appropriate, and promote the hit rate to adjust the data queue.

[0116] 210. When performing data query on the first partition, retrieve corresponding pre-reading probabilities between the first partition and at least one other partition from the transition probability matrix.

[0117] In an embodiment of the present application, when performing data query on the first partition, in order to achieve pre-reading, the corresponding pre-reading probability between the first partition and at least one other partition can be determined through the transfer probability matrix; of course, in order to ensure the accuracy of pre-reading, the corresponding pre-reading probability between the first partition and all other partitions can be determined.

[0118] In some embodiments, the process of querying data includes point query and range query.

[0119] Point lookup refers to querying a single data item, such as searching for the corresponding value using a key. In hash partitioning, data is distributed across multiple partitions, each of which is located using a hash function. The hash function has a time complexity of O(1) because it directly locates the partition by calculating the hash value, without traversing the partition. Therefore, the time complexity of point lookup is O(1). Suppose you want to query data with the key "A": you calculate the hash value H("A") = Si (partition number), and then directly search for the data in partition Si. The entire process is O(1).

[0120] A range search refers to querying a continuous section of data, such as querying all data with a key in the range [K1, K2]. In a range search, you need to traverse the B+ tree structure to locate the data. The time complexity of the B+ tree is O(logn), where n is the total amount of data; the time complexity of partitioning is O(m). Range searches may involve multiple partitions, where m is the number of partitions covered, and you need to query data in each of these partitions separately. Therefore, the time complexity of a range search is O(logn + m). Suppose you want to query data with a key in the range [1000, 2000]. You traverse the B+ tree to find all data that meets the criteria. The time complexity of this step is O(logn). If this data is distributed across m partitions, you need to query the data in each of these partitions separately. The time complexity of this step is O(m). Therefore, the total time complexity is O(logn + m).

[0121] By calculating time complexity, we can evaluate the query performance of the system, optimize the system design, guide the query strategy, predict the system scalability, and compare the performance of different systems.

[0122] 211. If the pre-read probability between the first partition and the second partition is greater than a preset probability threshold and greater than the pre-read probability between the first partition and the other partitions, pre-read data for the second partition to obtain at least one query result.

[0123] In an embodiment of the present application, when performing pre-reading, it is necessary to determine the next partition to be queried, so the transfer probabilities between the first partition and the other partitions can be compared. The preset probability threshold can be a custom pre-read trigger threshold. That is, as long as there is a probability greater than the preset probability threshold among the pre-read probabilities between the first partition and the other partitions, then the next pre-reading can be performed for the first partition, and then the maximum probability is selected from the various probabilities greater than the preset probability threshold, and the corresponding partition is the second partition to be queried in the next step.

[0124] In some embodiments, the pre-reading condition can be expressed as:

[0125]

[0126] in, It can be expressed as the maximum probability of all possible pre-reading paths in the current state Si. 0.7 is the preset probability threshold, which means that when the probability of a pre-reading path exceeds 70%, the path is considered to be strongly associated and the pre-reading hit rate is high.

[0127] For example, we can convert the data pre-reading into a scenario of reading a book in a library. When reading a book in the library, every time you finish a page, the librarian will guess which page you might read next based on your past reading habits, and copy the contents of those pages in advance and keep them at hand. In this way, when you really turn to that page, you don’t need to run to the bookshelf to find the book, you can read it directly at a very fast speed. Here, the "current state S i " can be understood as the page you are reading, for example, you are reading page 50 (S i = Page 50), the librarian will look through the historical reading records and find that after reading page 50 in the past: there is a 70% probability of continuing to read page 51, a 20% probability of jumping to page 100 (for example, jumping to read the end), and a 10% probability of turning back to page 30 (for example, reviewing the previous content); if a certain probability is particularly high (for example, more than 70%), such as the probability of page 51 here is 70%, the administrator will copy page 51 in advance and put it on the table for use. If the probabilities are very low (for example, less than 50%), there may be no preparation in advance.

[0128] In some embodiments, during pre-reading, pre-reading may be performed according to a preset pre-reading step length, which may be calculated based on an average step length, for example: .

[0129] In some embodiments, during pre-reading, it is also detected whether the pre-read data is valid. Specifically, the CRC or data fingerprint can be checked. If valid, the query data is determined. At the same time, the queues of the pre-read data (Hot queue and Warm queue) can be adjusted and the hit rate can be recalculated.

[0130] 212. Aggregate at least one query result to obtain a target query result corresponding to the data to be queried.

[0131] In the embodiment of the present application, for the description of step 212, please refer to the detailed description of step 104 in the above embodiment, and the embodiment of the present application will not be repeated.

[0132] 213. Periodically detect the partition heat of each partition in the storage system.

[0133] In the embodiment of the present application, access to partitions is involved in each stage of data writing, querying, and reading. If a partition is accessed too many times or the amount of data is too large, the performance of the partition will be affected. Therefore, the partition heat of each partition can be periodically detected according to a certain period.

[0134] In some embodiments, partition heat can be expressed as:

[0135]

[0136] T represents the statistical time window, which can be a custom configuration parameter and has a default value of 300 seconds. Partition capacity represents the maximum data capacity of a single partition, such as 1GB, which is a fixed constant and predefined by the storage policy. 、 and is the weight coefficient, + =1, the optimization can be automatically adjusted by the gradient descent method.

[0137] 214. When the partition heat of the third partition meets the preset migration condition, the data in the third partition is migrated to the fourth partition.

[0138] In the embodiment of the present application, the preset migration condition can be expressed as:

[0139]

[0140] in, It can represent the average heat of all partitions, obtained through periodic detection; It can represent the standard deviation of heat, It can represent the initial adjustment coefficient, the initial value is 1.5, and it can be customized to control the migration frequency.

[0141] It should be noted that if the migration is unsuccessful, a migration warning message may be output to prompt staff to perform maintenance.

[0142] During migration, it is also necessary to determine the partition to be migrated to. The fourth partition can be determined based on the load variance value of at least one controller node. Specifically, it can be expressed as:

[0143]

[0144]

[0145] in, It can represent the heat of the current k-th partition of node j; It can represent the change in the popularity of node j due to migration. It can be understood that only the popularity of the migrated node increases by Hi, while the popularity of other nodes remains unchanged; It can represent the heat value of the partition to be migrated; M can be represented as the number of partitions currently managed by node j, which may change after migration.

[0146] It should be noted that when the partition is migrated to node j, node j will add a new node with a heat of H i Therefore, its total heat will increase by Hi, that is: ΔH j =H i, which means that this node bears the additional load brought by the migration. When the partition is not migrated to node j, the load of other nodes will not be affected by this migration, and their heat remains unchanged, so: ΔH j =0, which means that these nodes are not affected by this migration and the load remains the same.

[0147] It should be noted that in order to accurately evaluate the load distribution of the entire system after migration when making load balancing decisions and implement the core logic of the target node selection algorithm, when selecting the node to migrate to, we can find a node j that has the smallest load variance among all nodes in the entire system after receiving the new partition (that is, the most balanced load). Therefore, it is necessary to determine which node will have an additional H i The load (ie ΔH j =H i ), which nodes will not change (i.e. ΔH j =0). Only in this way can we calculate the new load distribution after migration and compare which migration method is the most balanced. In addition, by j Set to H i or 0, which clearly shows that partition migration is a discrete, one-to-one action that will not have ambiguous effects on multiple nodes, making it easier to model and calculate the global load distribution after migration.

[0148] It should be noted that migration is to optimize system performance and resource utilization. Some partitions may be overloaded (hotspot partitions) while other partitions are underloaded. Through migration, data can be transferred from high-load partitions to low-load partitions to achieve load balancing; the resources of some partitions (such as storage space and computing power) may be over-occupied, while the resources of other partitions are idle. Through migration, resources can be allocated more reasonably and overall utilization can be improved; the query performance of some partitions may be poor (such as long response time). Through migration, data can be transferred to partitions with better performance to improve query efficiency; when a partition fails, data can be transferred to other partitions through migration to ensure high availability of the system; the data distribution of some partitions may be uneven (such as data skew). Through migration, data distribution can be adjusted to enable the system to process queries more efficiently.

[0149] It should be noted that the partition that can be migrated can be a low-load partition with a low load, which can receive additional data without affecting performance; it can be a partition with high resource utilization, with sufficient resources (such as storage space and computing power) to support the storage and processing of new data; it can be a high-performance partition with good performance (such as short response time and high throughput) and can efficiently process new data; it can be a partition with low data skew, with evenly distributed data, which will not cause data skew due to receiving new data; it can be a stable partition, which runs stably, has no failures or performance fluctuations, and can support new data for a long time.

[0150] In this solution, data migration is achieved by detecting the partition heat. By reflecting the changes in access frequency over time, the current heat of the partition can be evaluated more accurately. In addition, data migration can achieve load balancing between partitions and improve data response efficiency.

[0151] In the embodiments of the present application, a core architecture of a storage system is also disclosed to implement the above solution. The core architecture may include:

[0152] (1) Lock-free ASRH hybrid index layer: Based on partition-level atomic operations (CAS) and version number control, it eliminates global lock contention. This is the foundational layer of the entire architecture, responsible for data storage and retrieval. It achieves efficient point and range queries through a combination of hashing and B+ trees.

[0153] (2) Dynamic load balancing layer: A lightweight atomic migration protocol avoids lock blocking during the migration process. This layer relies on the data partition information of the ASRH hybrid index layer. By monitoring the popularity of each partition, it dynamically adjusts the data distribution to optimize system performance.

[0154] (3) Adaptive caching layer: Dual LRU queues + Markov pre-reading model. This layer is closely related to the ASRH hybrid indexing layer and the dynamic load balancing layer. It uses the data access patterns of the indexing layer and the popularity information of the load balancing layer to optimize the data caching strategy and improve data access efficiency.

[0155] (4) Streaming Compression Layer: LZ4 compression and data block merging are jointly optimized. This layer is located at the top of the architecture and compresses data processed by the first three layers to reduce storage space and network transmission overhead. It relies on the data structure of the index layer and the access pattern of the cache layer to optimize the compression strategy.

[0156] In some embodiments, data interaction can be achieved between the above four-layer structures, which can be described by switching scenarios.

[0157] The ASRH indexing process is similar to that of a sorting center, requiring packages to be quickly received (data written) and correctly labeled (LBA addresses). When a shelf (partition) is full, the sorter (CAS operation) notifies the dispatch department (dynamic load balancing layer): "Shelf A is full, and we need to expand or transfer stock!" If there are any package label conflicts (hash collisions), the sorter merges adjacent packages (LBA interval merging) to avoid congestion on the conveyor belt.

[0158] The dynamic load balancing process can be similar to that of a dispatch center, which monitors the busiest shelves (popularity scores) and adjusts resource allocation. Upon receiving a notification from the sorting center that "Shelf A is full," the dispatcher immediately expands capacity by adding a temporary storage area to Shelf A (data migration); then transfers some packages from Shelf A to the available Shelf B (load balancing); and simultaneously notifies the courier (pre-reading engine): "The next customer may need a package from Shelf B, so prepare it in advance!"

[0159] The pre-reading engine process involves intelligent couriers predicting customer demand based on historical orders and pre-packing packages (data pre-reading). When the dispatch center indicates that "shelf B may be hot," the courier immediately checks historical orders and discovers that 70% of customers who buy items from shelf B also purchase items from shelf C. They then pre-stock the inventory by placing packages from shelves B and C together in the delivery vehicle (a buffer). If the prediction is accurate, the delivery vehicle can be dispatched immediately upon customer order placement, increasing delivery speed by three times. If the prediction is incorrect, at most one extra item will be placed on the vehicle, wasting less than 10% of resources.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0161] like Figure 3 As shown, an embodiment of the present application further provides a data query device for a storage system, which may include:

[0162] An acquisition module 301 is used to acquire an address range of data to be queried;

[0163] Processing module 302 is configured to determine at least one target partition corresponding to an address interval based on pre-stored data partition information of the storage system, where the data partition information is obtained by pre-dividing the data of the storage system based on a logical block address range.

[0164] The processing module 302 is further configured to, when performing a data query on a first partition of at least one target partition, determine a second partition to be queried next to the first partition based on a pre-stored transition probability matrix, and pre-read data on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix represent the probability of pre-reading between respective data in the storage system.

[0165] The processing module 302 is further configured to aggregate at least one query result to obtain a target query result corresponding to the data to be queried.

[0166] In some embodiments, the acquisition module 301 is further used to acquire data to be written;

[0167] The processing module 302 is further configured to map the data to be written according to a preset hash function, store the data to be written in a corresponding partition, and obtain data partition information;

[0168] The processing module 302 is further configured to store data partition information.

[0169] In some embodiments, the processing module 302 is specifically configured to map the data to be written according to a preset hash function to obtain a mapping result;

[0170] The processing module 302 is specifically configured to merge the at least two conflicting data to obtain a data merge interval if the mapping result indicates that the at least two conflicting data have a partition conflict;

[0171] The processing module 302 is specifically configured to update the version number according to the data merging interval, store the data merging interval, and obtain data partition information.

[0172] In some embodiments, the processing module 302 is specifically configured to detect data transmission delay in real time;

[0173] The processing module 302 is specifically configured to determine a data compression level according to a data transmission delay;

[0174] The processing module 302 is specifically configured to compress the data to be written according to the data compression level and store the compressed data in the corresponding partition to obtain data partition information.

[0175] In some embodiments, the processing module 302 is further configured to periodically detect the partition heat of each partition in the storage system;

[0176] The processing module 302 is further configured to migrate data in the third partition to a fourth partition when the partition heat of the third partition meets a preset migration condition, where the fourth partition is determined according to a load variance value of at least one controller node.

[0177] In some embodiments, the acquisition module 301 is further configured to acquire a historical access sequence;

[0178] The processing module 302 is further configured to count the number of transfers between the first data and the second data according to the historical access sequence, where the first data and the second data are both arbitrary data in the storage system;

[0179] The processing module 302 is further configured to determine a pre-read probability between the first data and the second data according to the number of transfers;

[0180] The processing module 302 is further configured to aggregate the pre-read probabilities to obtain a transition probability matrix.

[0181] In some embodiments, the processing module 302 is specifically configured to retrieve corresponding pre-reading probabilities between the first partition and at least one other partition from the transition probability matrix when performing a data query on the first partition;

[0182] The processing module 302 is specifically configured to pre-read data for the second partition to obtain at least one query result if the pre-read probability between the first partition and the second partition is greater than a preset probability threshold and greater than the pre-read probability between the first partition and other partitions.

[0183] In the embodiments of the present application, the description of the features in the embodiments corresponding to the data query device of the storage system can be found in the relevant description of the embodiments corresponding to the data query method of the storage system, and will not be repeated here.

[0184] like Figure 4 As shown, an embodiment of the present application further provides an electronic device, including a memory 401 and a processor 402, wherein the memory 401 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above-mentioned data query method embodiments of the storage system.

[0185] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data query method embodiments of the storage system when running.

[0186] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0187] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned data query method embodiments for the storage system are implemented.

[0188] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned data query method embodiments of the storage system.

[0189] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] The above is a detailed introduction to the process monitoring of a storage system provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core ideas of this application. It should be pointed out that, for those skilled in the art, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A data query method for a storage system, characterized in that: The method comprises: Get the address range of the data to be queried; Determining at least one target partition corresponding to the address interval according to pre-stored data partition information of the storage system, wherein the data partition information is obtained by pre-dividing the data of the storage system according to the logical block address range; When performing a data query on a first partition of the at least one target partition, a second partition to be queried next corresponding to the first partition is determined based on a pre-stored transition probability matrix, and data is pre-read on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix are used to represent the probability of pre-reading between each data in the storage system, the transition probability matrix is ​​determined based on the number of transfers between each data in a historical access sequence, and the historical access sequence is used to represent the access order between each data during the historical data query process, the second partition is the partition corresponding to the maximum probability among the pre-read probabilities corresponding to the first partition and at least one other partition in the transition probability matrix, and the pre-read probability between the first partition and the second partition is greater than a preset probability threshold, and the at least one query result is obtained by traversing the data in the at least one target partition in parallel, and each target partition corresponds to one query result; The at least one query result is aggregated to obtain a target query result corresponding to the data to be queried.

2. The method according to claim 1, characterized in that Before determining at least one target partition corresponding to the address range based on pre-stored data partition information of the storage system, the method further includes: Get the data to be written; Mapping the data to be written according to a preset hash function, storing the data to be written in a corresponding partition, and obtaining the data partition information; The data partition information is saved.

3. The method according to claim 2, characterized in that Mapping the data to be written according to a preset hash function, storing the data to be written in a corresponding partition, and obtaining the data partition information includes: Mapping the data to be written according to the preset hash function to obtain a mapping result; If the mapping result indicates that there is a partition conflict between the at least two conflicting data, merging the at least two conflicting data to obtain a data merging interval; The version number is updated according to the data merging interval, and the data merging interval is stored to obtain the data partition information.

4. The method according to claim 2, characterized in that The storing the data to be written into the corresponding partition to obtain the data partition information includes: Real-time detection of data transmission delays; determining a data compression level according to the data transmission delay; According to the data compression level, the data to be written is compressed and then stored in the corresponding partition to obtain the data partition information.

5. The method according to claim 1, characterized in that The method further comprises: Periodically detecting the partition heat of each partition in the storage system; When the partition heat of the third partition meets a preset migration condition, data in the third partition is migrated to a fourth partition, where the fourth partition is determined according to a load variance value of at least one controller node.

6. The method according to claim 1, characterized in that When performing a data query on a first partition of the at least one target partition, determining a next second partition to be queried corresponding to the first partition according to a pre-stored transition probability matrix, and pre-reading data on the second partition before obtaining at least one query result, the method further includes: Get historical access sequence; counting the number of transfers between first data and second data according to the historical access sequence, where the first data and the second data are both arbitrary data in the storage system; determining a pre-reading probability between the first data and the second data according to the number of transfers; The pre-reading probabilities are summarized to obtain the transition probability matrix.

7. The method according to claim 6, characterized in that The method further comprises: when performing a data query on a first partition of the at least one target partition, determining a next second partition to be queried corresponding to the first partition according to a pre-stored transition probability matrix, and pre-reading data on the second partition to obtain at least one query result, including: When performing data query on the first partition, retrieving corresponding pre-reading probabilities between the first partition and at least one other partition from the transition probability matrix; If the pre-read probability between the first partition and the second partition is greater than a preset probability threshold and greater than the pre-read probability between the first partition and other partitions, data is pre-read on the second partition to obtain the at least one query result.

8. A data query device for a storage system, characterized in that: The device comprises: An acquisition module is used to obtain the address range of the data to be queried; a processing module, configured to determine at least one target partition corresponding to the address interval based on pre-stored data partition information of the storage system, wherein the data partition information is obtained by pre-dividing the data of the storage system according to a logical block address range; The processing module is further configured to, when performing a data query on a first partition of the at least one target partition, determine, based on a pre-stored transition probability matrix, a next second partition to be queried corresponding to the first partition, and pre-read data on the second partition to obtain at least one query result, wherein the elements in the transition probability matrix are used to represent the probability of pre-reading between each data in the storage system, the transition probability matrix is ​​determined based on the number of transfers between each data in a historical access sequence, the historical access sequence is used to represent the access order between each data during a historical data query process, the second partition is the partition corresponding to the maximum probability among the pre-read probabilities corresponding to the first partition and at least one other partition in the transition probability matrix, and the pre-read probability between the first partition and the second partition is greater than a preset probability threshold, and the at least one query result is obtained by traversing the data in the at least one target partition in parallel, and each target partition corresponds to one query result; The processing module is further configured to aggregate the at least one query result to obtain a target query result corresponding to the data to be queried.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data query method for the storage system according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data query method of the storage system according to any one of claims 1 to 7 are implemented.

Images