Request path generation method and device, electronic equipment and storage medium

Abstract

The invention discloses a request path generation method and device, electronic equipment and a storage medium, and relates to the technical field of data storage, and the method comprises the steps: receiving a client request, and obtaining hardware configuration information of a storage cluster; dividing the storage pool into different speed levels based on hardware performance differences; positioning a target cache node through topology perception and obtaining the state of the target cache node; in combination with request features and cache node states, intelligently scheduling request data to a storage pool of a matching speed level; and finally generating and executing an optimized target storage path. According to the method and the device, the problem of poor request path generation performance caused by insufficient intelligent and efficient resource scheduling in related technologies in a heterogeneous hardware storage environment is solved, and the technical effects of improving the utilization efficiency of storage resources, reducing request processing delay and optimizing end-to-end storage access performance are achieved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a request path generation method, device, electronic device and storage medium. Background Art

[0002] Driven by applications like cloud computing and big data, the surge in data size poses a severe challenge to the performance of distributed storage systems. Efficiently processing client requests and generating optimal storage access paths for them is key to improving overall system performance.

[0003] However, in practical applications, the performance of related request path generation technology is often limited, especially when facing complex storage environments composed of heterogeneous hardware, and it is difficult to fully meet the needs of efficient and intelligent resource scheduling. Summary of the Invention

[0004] The present application provides a request path generation method, device, electronic device and storage medium to at least solve the problem in the related art that the request path generation performance is insufficiently optimized in a heterogeneous hardware storage environment and difficult to meet the requirements of efficient and intelligent resource scheduling.

[0005] The present application provides a request path generation method, including: obtaining hardware configuration information of a client request, a data storage pool, and each storage node in a storage cluster; dividing the data storage pool into a first storage pool and a second storage pool according to the media performance difference in the hardware configuration information, wherein the storage speed of the first storage pool is greater than the storage speed of the second storage pool; using a preset topology perception method, determining a target cache node corresponding to the client request and a metadata status of the target cache node; based on the request characteristics and metadata status of the client request, storing the request data carried by the client request in a target storage pool, wherein the target storage pool is the first storage pool or the second storage pool; scheduling the client request to be executed at the target storage pool, and generating a target storage path corresponding to the client request.

[0006] The present application also provides a request path generation device, including: an acquisition module for acquiring client requests, a data storage pool, and hardware configuration information of each storage node in a storage cluster; a division module for dividing the data storage pool into a first storage pool and a second storage pool according to the media performance differences in the hardware configuration information, wherein the storage speed of the first storage pool is greater than the storage speed of the second storage pool; a determination module for determining the target cache node corresponding to the client request and the metadata status of the target cache node by using a preset topology perception method; a storage module for storing the request data carried by the client request to the target storage pool based on the request characteristics and metadata status of the client request, wherein the target storage pool is the first storage pool or the second storage pool; a generation module for scheduling the client request to the target storage pool for execution, and generating a target storage path corresponding to the client request.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned request path generation methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned request path generation methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned request path generation methods when executed by a processor.

[0010] Through this application, since it is possible to divide the data storage pool into speed levels based on the hardware performance differences of the storage nodes, and combine network topology awareness to accurately locate the cache nodes and their status, and then based on the client request characteristics and node status information, intelligently and dynamically schedule the requested data to the most matching speed level storage pool, it can solve the problem of poor request path generation performance caused by insufficiently intelligent and efficient resource scheduling in heterogeneous hardware storage environments, and achieve the technical effects of improving storage resource utilization, reducing request access latency, and optimizing end-to-end storage performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 A flowchart of a request path generation method provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of another request path generation method provided in an embodiment of the present application;

[0014] Figure 3 A flowchart of another request path generation method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of a request path generation system provided in an embodiment of the present application;

[0016] Figure 5 A structural block diagram of a request path generation device provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] Block storage provides efficient storage by directly mapping raw disk space. Storage Area Networks (SANs) are a typical application of this technology, but it lacks the flexibility and scalability of distributed storage. Distributed storage addresses the performance bottlenecks, single points of failure, and scalability issues of traditional centralized storage by distributing data. The core challenge of distributed block storage currently lies in high I / O latency. Mainstream vendors are innovating through hardware acceleration, protocol streamlining, and architectural reconstruction, but their solutions are limited by hardware dependency, compatibility, complexity, reliability, or cost, and generally face a trade-off between protocol stack optimization and hardware dependency.

[0021] In related technologies, a three-tier architecture with separated clients, caches, and storage clusters is adopted to improve performance through layered decoupling and asynchronous collaboration. Its core processes include layered initialization, write-ahead logging (WAL) mechanism and asynchronous IO aggregation and flushing for write requests, cache priority and intelligent prefetching for read requests, and key features such as path separation, media optimization, strong consistency, and adaptive recycling are designed. However, this technology increases the complexity of system collaboration and the difficulty of operation and maintenance due to module separation. It relies on a high-precision prediction and state synchronization framework, and the adaptation layer introduced for compatibility with back-end storage (such as Ceph) may bring protocol parsing overhead in ultra-large-scale clusters, becoming a throughput bottleneck.

[0022] In view of this, the technical solution of the present invention aims at the bottleneck points in the IO software stack in high-performance scenarios, and proposes a distributed block storage IO path optimization method. The storage pool is divided based on the performance differences of hardware media, and the target cache node is accurately located in combination with topology awareness to directly generate the shortest storage path, thereby avoiding the multi-layer protocol stack overhead and hardware binding of the traditional solution, and achieving low latency through pure software scheduling, solving the trade-off contradiction between protocol stack optimization and hardware dependence. Secondly, the data storage decision is converged to a unified storage cluster, and the request is directly scheduled to the target storage pool through metadata status and request characteristics, skipping the adaptation layer conversion between the cache-storage cluster, avoiding the protocol parsing overhead and cross-module collaboration complexity from the root, while maintaining high compatibility (no specific hardware / backend storage dependence).

[0023] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] According to an embodiment of the present invention, an embodiment of a request path generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] In this embodiment, a request path generation method is provided, which can be used in electronic devices, such as servers. Figure 1 is a flow chart of a request path generation method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0026] Step S101: Obtain client request, data storage pool, and hardware configuration information of each storage node in the storage cluster.

[0027] A client request refers to an I / O operation instruction (read / write request) initiated by a computing node (or application host) in a distributed block storage cluster. A data storage pool is a logical storage resource pool formed by aggregating the physical media of multiple storage nodes in a storage cluster. A storage cluster aggregates the storage space in multiple storage devices into a storage system that can provide a unified access interface and management interface to application servers. Hardware configuration information refers to information such as the media type, performance parameters, and network interface of a storage node. Specifically, during the cluster deployment and initialization phase, the media awareness engine automatically scans the hardware configuration of the storage node (such as identifying NVMe SSDs, SCM high-speed media, and slow media such as HDD / SATASSDs), and collects the capacity and topology information of the data storage pool. Client requests are received through the dynamic routing engine, which subscribes to the global cache service map (including node health status, storage pool metadata, etc.) based on bRPC persistent connections to obtain request and cluster resource information in real time.

[0028] Step S102 : dividing the data storage pool into a first storage pool and a second storage pool according to the media performance difference in the hardware configuration information, wherein the storage speed of the first storage pool is greater than the storage speed of the second storage pool.

[0029] Media performance differences refer to the disparity in I / O capabilities between different storage media, such as NVMe SSD latency (microseconds) versus HDD latency (milliseconds), and SCM bandwidth (GB / s) versus SATA SSD bandwidth (hundreds of MB / s). The first storage pool, comprised of high-speed media (such as NVMe SSDs and SCMs), provides low-latency access and is also referred to as the fast storage pool. The second storage pool, comprised of low-speed media (such as HDDs and SATA SSDs), is used for large-capacity storage and is also referred to as the slow storage pool. Specifically, storage pools are dynamically divided based on I / O performance differences identified in hardware scan results. The first storage pool, comprised of high-speed media such as NVMe SSDs and SCMs, utilizes persistent memory programming models (such as PMDK) to achieve memory-level access speeds and establishes cross-node cache coherence via the RDMA network. The second storage pool, comprised of slower media such as HDDs and SATA SSDs, is used for persistent storage of large-capacity data. This division is automatically categorized based on media latency and bandwidth metrics, ensuring that the high-speed pool prioritizes low-latency requests.

[0030] Step S103: using a preset topology awareness method, determining the target cache node corresponding to the client request and the metadata status of the target cache node.

[0031] The preset topology-aware approach refers to a preset routing algorithm based on network and node location. The target cache node is the optimal cache service instance to handle the client request. The metadata state refers to the real-time status of core control information such as data location, version, and cache hit status in the target cache node. Specifically, a preset topology-aware routing algorithm (based on, for example, network latency, node physical location, and switch hop count) is used to analyze the source location of the current client request and the network topology of the storage cluster. This is combined with node health and load information (such as CPU, memory, and network bandwidth usage) obtained from the global cache service graph (obtained through real-time subscription via bRPC persistent connections), as well as cached data location information, to calculate the target cache node with the best performance or the most balanced load for handling the specific client request. Simultaneously, the node's metadata state (core control information such as whether the requested data block is already cached at the node, whether the cached version is up to date, and cache space occupancy) is also obtained in real time from the global service graph or by directly querying the target node, providing a basis for subsequent decision-making.

[0032] Step S104 : based on the request characteristics and metadata status of the client request, storing the request data carried by the client request into a target storage pool, where the target storage pool is the first storage pool or the second storage pool.

[0033] Request characteristics refer to a dynamic set of attributes of client I / O requests, which are used to drive intelligent scheduling decisions. The target storage pool is a physical storage location dynamically selected based on the request characteristics and metadata status. Specifically, the client request characteristics are analyzed, such as the request type (read / write), the requested data size, whether it is random access or sequential access, and the expected access frequency (popularity) of the data. Simultaneously, the requested data is stored in the first or second storage pool based on the metadata status of the target cache node.

[0034] Step S105 : dispatching the client request to the target storage pool for execution, and generating a target storage path corresponding to the client request.

[0035] The target storage path refers to an optimized end-to-end I / O execution link, which includes routing and storage policies. Specifically, based on the selected target cache node and target storage pool, an optimized I / O execution link is dynamically assembled. This includes routing the client request to the selected target cache node; determining the target storage pool where the requested data is to be stored based on the request characteristics and metadata status of the client request; and applying the corresponding storage policy (such as the persistent memory access model PMDK+RDMA for the high-speed pool, or the traditional access method for the slow pool). The resulting end-to-end path - from the client, through the target cache node, to the physical location of the target storage pool where the final data is located and the access protocol used - is the target storage path generated for the request, ensuring that the request is executed efficiently and correctly.

[0036] The request path generation method provided by the present invention combines hardware-aware dynamic partitioning of storage pools with topology-aware target node selection to achieve intelligent optimization of request paths. This method constructs tiered storage pools based on media performance differences, utilizes topology-aware mechanisms to precisely locate cache nodes and obtain metadata status, and then dynamically selects a high-speed or low-speed storage pool to execute requests based on request characteristics, ultimately forming the shortest end-to-end I / O path. This design systematically addresses the core pain points of traditional distributed storage, such as extended multi-layer network forwarding and mismatched resource scheduling, significantly reducing access latency and improving storage efficiency.

[0037] In this embodiment, a request path generation method is provided, which can be used in electronic devices, such as servers. Figure 2 is a flow chart of a request path generation method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0038] Step S201: Obtain client request, data storage pool, and hardware configuration information of each storage node in the storage cluster. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0039] Step S202: Based on the media performance differences in the hardware configuration information, the data storage pool is divided into a first storage pool and a second storage pool. The storage speed of the first storage pool is greater than the storage speed of the second storage pool. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0040] Step S203: using a preset topology awareness method, determining the target cache node corresponding to the client request and the metadata status of the target cache node.

[0041] Specifically, the above step S203 includes:

[0042] Step S2031: parse the client request to determine the logical block address and node topology relationship corresponding to the client request.

[0043] A logical block address is the linear addressing location of data requested by a client and serves as a unique logical identifier for a data block in a block storage system. Node topology refers to the physical and logical connectivity between nodes in a storage cluster. This can include, for example, network connection paths, node physical locations, and logical groupings. Specifically, in a distributed block storage system, client requests follow standard block protocols (such as iSCSI or NVMe-oF). The parsing process consists of two main steps: First, the parser extracts key information from the protocol packet header, including the operation type (read / write), the logical block address (LBA) starting address and data length (for example, LBA 0x1000 to 0x2000), and the volume identifier (Volume ID). This information is then used to associate metadata with the target data volume. Second, the global topology map and metadata cache are used to retrieve node topology information. During system initialization, the global topology map is constructed and monitoring metrics are subscribed to via the bRPC streaming interface to obtain information such as node location, network distance, and storage pool type. The parser queries two levels of metadata cache based on the volume ID: the block protocol layer's LRU cache, which quickly retrieves the volume's storage pool and primary replica node locations; and the global cache layer's persistent snapshot, which is used to load topology information if the cache misses. Ultimately, the parser outputs a mapping between the LBA range and the target volume's topology. For example, LBAs 0x1000 to 0x2000 correspond to storage pool 1, with replicas held by nodes A / B / C.

[0044] Step S2032: Map the client request to the corresponding target cache node in the first storage pool or the second storage pool according to the relationship between the logical block address and the node topology.

[0045] The mapping process relies on a topology-aware routing engine and a dynamic weight algorithm. Specifically, different processing methods are selected based on the routing strategy. If the client and storage node are deployed on the same machine (the service is deployed on the client node), the request is directly processed through the node's intra-loop network to avoid cross-node forwarding. If it is a non-local request, the logical block address (LBA) is mapped to the target cache node. Combined with the topology-aware mechanism, the cache node with the closest network distance (such as the rack) to the request source node is preferentially selected to ensure that the average number of network hops is ≤1.

[0046] Step S2033: Analyze the metadata access characteristics of the target cache node to determine the metadata status of the target cache node.

[0047] Metadata access characteristics refer to the dynamic access properties of metadata in the target cache node. Specifically, in the metadata access characteristics analysis, efficient management is achieved through two-level cache query and dynamic state determination. First, the metadata status (such as dirty data mark, version number) of the target logical block address (LBA) is retrieved in the memory cache of the block protocol layer (based on LRU metadata index). If the memory cache misses, the persistent metadata snapshot of the global cache layer is accessed (stored on high-speed media such as NVMeSSD / SCM). Monitor the node dirty data ratio (such as > threshold). A high ratio indicates that disk flushing is required first, which may increase access latency. At the same time, version conflicts (such as inconsistent version numbers between nodes) are detected through real-time distribution maps to indicate data inconsistency risks.

[0048] In some optional implementations, the above step S2033 includes:

[0049] Step a1: Obtain the persistently stored metadata snapshot and the version number of the metadata snapshot from the global cache layer.

[0050] The global cache layer refers to a persistent metadata storage service across nodes, built on a distributed KV database (such as TeraDB). A metadata snapshot refers to a consistent copy of the metadata state at a certain moment. Specifically, the metadata snapshot corresponding to the target logical block address (LBA) is directly read through the persistent storage interface of the global cache layer (such as the PMDK persistent memory library). Based on a two-level cache architecture (block protocol layer memory cache and global cache layer persistent storage), if the memory cache misses, the storage location of the global cache layer (such as a specific data page on an NVMe SSD / SCM) is located based on the hash value of the logical block address. The metadata snapshot is stored in a versioned structure (containing fields such as data pointer, checksum, version number, etc.). When reading, the structure header field is parsed to obtain the current version number (such as a 64-bit incrementing integer).

[0051] In step a2, the version number of the metadata snapshot is incremented through an atomic operation to obtain an incremented result.

[0052] The increment result is used to characterize the execution status of the atomic compare-and-swap operation. Specifically, the CAS (Compare-and-Swap) instruction is used to implement lock-free incrementing, read the current version number current_version, and calculate the new version number new_version = current_version + 1; call the CAS atomic operation and try to update the version number in memory from current_version to new_version. If the version number in memory has not been modified (still current_version), it is atomically updated to new_version and a success flag is returned, indicating that the version number has been exclusively updated; if it is detected that the version number has been modified by other threads / nodes, a conflict flag (such as the error code ERR_VERSION_CONFLICT) is returned, indicating that there is version contention. This mechanism avoids distributed lock overhead through hardware-level atomic instructions, ensuring thread safety and high performance of version number updates.

[0053] In step a3, when the incremental result represents a version number conflict, an asynchronous metadata reconstruction process is triggered based on the pre-trained event-driven model to obtain the metadata status.

[0054] The event-driven model refers to an asynchronous processing framework that responds to version number conflicts. The asynchronous metadata reconstruction process refers to the metadata consistency repair process executed in the background. Specifically, when CAS returns a failure (i.e., the incremental result indicates a conflict), it indicates that other nodes / threads have modified the metadata version, triggering the event-driven model response. A metadata conflict event is created (including the conflicting logical block address LBA, the current node ID, and the timestamp) and pushed to the high-priority event queue; based on the RAFT / Paxos protocol, the latest metadata snapshot of the LBA is pulled from the majority of healthy nodes; the conflicting versions are merged by timestamp to generate a new snapshot; the snapshot is persisted to the global cache layer and the version number is atomically updated; and the relevant nodes are notified to refresh the memory cache. When successful, METADATA_STABLE is returned (indicating strong consistency recovery); when a timeout or failure occurs, METADATA_RECOVERING is returned (triggering an alarm and retry mechanism). This process uses event-driven and distributed consensus protocols to achieve rapid reconstruction and consistency assurance of conflicting metadata under lock-free conditions.

[0055] In the above implementation, versioned metadata snapshots are obtained from the global cache layer, and lock-free version number incrementing is achieved using atomic operations. When a version conflict is detected, an asynchronous reconstruction process based on a pre-trained event-driven model is intelligently triggered. This not only avoids the contention overhead of traditional distributed locks, but also ensures strong metadata consistency through decoupled fault handling. This design reduces metadata state maintenance latency to microseconds while maintaining zero lock contention, completely resolving the metadata management bottleneck in high-concurrency scenarios.

[0056] Step S204: based on the request characteristics and metadata status of the client request, the request data carried by the client request is stored in the target storage pool, which is the first storage pool or the second storage pool. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0057] Step S205: dispatch the client request to the target storage pool for execution, and generate the target storage path corresponding to the client request. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0058] The request path generation method provided by the embodiment of the present invention realizes direct connection of requests to the target cache node through precise matching of logical block addresses and node topology relationships, eliminating traditional multi-layer routing overhead. It first parses the request to locate the physical ownership of the data, and then maps the request to the target node in the high-speed or low-speed storage pool based on the topological relationship. Finally, it analyzes the metadata access characteristics of the target node in real time to determine the status, forming a closed-loop mechanism of "address resolution-topology mapping-status evaluation", which fundamentally solves the performance loss problem caused by path detours and metadata lag.

[0059] In this embodiment, a request path generation method is provided, which can be used in electronic devices, such as servers. Figure 3 is a flow chart of a request path generation method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0060] Step S301: Obtain client request, data storage pool, and hardware configuration information of each storage node in the storage cluster. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0061] Step S302: Based on the media performance differences in the hardware configuration information, the data storage pool is divided into a first storage pool and a second storage pool. The storage speed of the first storage pool is greater than the storage speed of the second storage pool. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0062] Step S303: Using a preset topology awareness method, determine the target cache node corresponding to the client request and the metadata status of the target cache node. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0063] Step S304 : based on the request characteristics and metadata status of the client request, storing the request data carried by the client request into a target storage pool, where the target storage pool is the first storage pool or the second storage pool.

[0064] Specifically, the above step S304 includes:

[0065] Step S3041 : Based on the request characteristics and metadata status of the client request, the requested data is divided into cold and hot data to obtain cold data and hot data.

[0066] Cold data refers to data with low access frequency (such as data that has not been used for a long time or archived data). Hot data refers to data with high access frequency (such as active business data that is frequently read and written). Specifically, the hot and cold data are dynamically determined through the request feature analysis model and metadata status. Request features include request size, type (read / write) and access mode (such as continuous / random). For example, large requests >256KB are preferentially marked as hot data candidates. The hot and cold analysis model refers to the data access frequency of the past 5 minutes based on the sliding window statistics (such as frequently accessed data is marked as hot data). Metadata status (such as version number, cache hit rate) assists in identifying active data. For example, data with high version numbers or frequently hits is considered hot data. Based on the above characteristics and real-time statistical results, data with high access frequency, small granularity or delay sensitivity is classified as hot data, and vice versa.

[0067] In some optional implementations, step S3041 includes:

[0068] Step b1: Count the access frequency of client requests within a preset time using a sliding window.

[0069] Sliding window refers to a time window algorithm that dynamically counts access frequencies. Access frequency refers to the total number of times data is requested to be accessed within the time period covered by the sliding window (such as 5 minutes). Specifically, a fixed time length (such as a 5-minute time window) is maintained, and the window slides forward as new requests enter. For example, the window records all requests from the current time T to T-5 minutes. Whenever a data block is accessed, its access counter is accumulated within the window; when the request exceeds the window time range (such as earlier than T-5 minutes), its count is automatically removed. At the end of the window, each data block obtains its exact access frequency in the last 5 minutes, which is used as the basis for dividing into hot and cold.

[0070] In step b2, the requested data is divided into cold and hot data according to the access frequency to obtain cold data and hot data.

[0071] Dynamic hot and cold stratification is performed based on the access frequency output by the sliding window. Specifically, the hot data standard is frequently accessed data (such as frequently read and written business data), whose access frequency exceeds the preset threshold (the threshold can be adjusted according to the scenario). The cold data standard is infrequently accessed data (such as archived data and historical logs), whose access frequency is lower than the threshold. Hot data is automatically marked as short-lifecycle data (preferentially stored in a high-speed pool), and cold data is marked as long-lifecycle data (stored in a slow pool). This process does not require manual intervention and adapts to changes in business load in real time.

[0072] In the above implementation, dynamic quantitative identification of hot and cold data is achieved through a sliding window mechanism - the frequency of request access is counted in real time using a preset time window, with objective access frequency as the only classification standard. This not only eliminates the lag of traditional static threshold strategies, but also uses a minimalist computing model to achieve precise stratification of adaptive business fluctuations, ensuring that storage resource allocation always matches data value in real time.

[0073] Step S3042: Store the cold data in the second storage pool, and store the hot data in the first storage pool.

[0074] The storage execution process adopts a media-aware tiering strategy. Specifically, hot data is written directly to the first storage pool (fast pool), which is composed of high-speed media such as NVMe SSD / SCM. It achieves sub-millisecond response through persistent memory programming models (such as PMDK) and ensures cross-node consistency through the RDMA network. Cold data is directed to the second storage pool (slow pool), which uses large-capacity media such as HDD / SATASSD. The system uses an append-only sequential write + IO aggregation algorithm (such as merging random small IOs into 4M / 8M blocks), which significantly improves throughput and avoids disk seek latency.

[0075] In some optional implementations, the above step S304 further includes:

[0076] Step c1: Determine the data storage address corresponding to the client request through a preset physical address pre-allocation strategy.

[0077] Physical address preallocation is a strategy that pre-plans physical storage locations on disks to avoid fragmented writes. The data storage address refers to the actual physical location of data on the disk (e.g., cylinder, track, or sector). Specifically, the physical address preallocation strategy achieves contiguous storage through multi-dimensional scheduling and space pre-occupation, generating the data storage address corresponding to the client request.

[0078] In some optional implementations, the above step c1 includes:

[0079] In step c11 , the client requests are classified into multiple priority queues using a multi-dimensional scheduling method, and each priority queue is bound to an independent disk area.

[0080] Multidimensional scheduling refers to an algorithm that classifies and schedules I / O requests based on multiple dimensions (such as request type, priority, and data hotness). Priority queues are request queues sorted by priority (e.g., queue 0 handles real-time transactions, queue 15 handles background backups). Independent disk areas are dedicated physical disk space allocated to each priority queue (e.g., divided into different tracks or LBA segments). Specifically, the multidimensional scheduling algorithm analyzes client requests based on multi-dimensional characteristics (such as request type, data volume ID, service priority, and I / O size) and dynamically allocates them to multiple independent priority queues (e.g., 16). Each queue is pre-assigned a dedicated physical disk area (e.g., queue 0 is assigned LBA 0x0000-0xFFFF, queue 1 is assigned LBA 0x10000-0x1FFFF) to achieve resource isolation. High-priority services (such as real-time transactions) are assigned to high-order queues (e.g., queues 0-3) and are bound to high-speed media areas (e.g., physical segments of NVMe SSDs). Low-priority requests (such as backup tasks) are assigned to low-order queues (such as queues 12-15) and bound to high-density slow disk areas (such as continuous sectors of an HDD).

[0081] During system initialization, a fixed range of physical addresses (e.g., 32MB / queue) is reserved for each queue. The disk driver layer maps logical addresses to physical areas to ensure that there is no storage space overlap between queues, thus completely avoiding resource contention.

[0082] Step c12: for client requests in any priority queue, allocate requested data of the same volume to adjacent storage spaces in the independent disk area corresponding to the priority queue.

[0083] Contiguous storage space refers to data blocks with consecutive physical addresses on the disk (such as 32MB continuous sectors). Specifically, for requests in the same priority queue, a volume-level continuous space allocation strategy is adopted. Multiple I / O requests for the same data volume (such as 4K random writes for volume A) are cached in the queue in the order of arrival. 32MB of physically continuous space (such as LBA0x20000-0x27FFFF) is allocated for the volume, and all subsequent requests for volume A are forced to write to this interval. Data is filled sequentially through a space sliding pointer (such as the current write offset 0x20100) to ensure that new requests are always adjacent to the previous write location.

[0084] In this implementation, multi-dimensional scheduling and strong physical space constraints achieve ultimate optimization of disk access. Requests are prioritized and assigned to independent queues, each bound to a dedicated disk area to avoid I / O interference. Requests for the same volume are also forced to be allocated to adjacent storage spaces (e.g., a 32MB contiguous area), converting random writes to sequential writes and significantly reducing disk head seek distances.

[0085] Step c2: storing the request data requested by the client according to the data storage address.

[0086] Performs low-fragmentation writes based on pre-assigned physical addresses. Specifically, it directly locates the disk location based on pre-assigned physical addresses (e.g., cylinder C1, track T2, sectors S5-S100). Requested data is written sequentially to consecutive sectors, avoiding random seeks (e.g., multiple 4K requests for the same volume are merged and written to a contiguous 32MB space).

[0087] Step S305: dispatch the client request to the target storage pool for execution, and generate the target storage path corresponding to the client request. Figure 2 Step S205 of the illustrated embodiment will not be described in detail here.

[0088] The request path generation method provided by the present invention achieves precise tiering of storage resources through dynamic dual-factor decision-making, intelligently dividing hot and cold data into hot and cold data by combining client request characteristics and real-time metadata status. Hot data is allocated to high-speed storage pools, while cold data is relegated to low-speed storage pools. This creates a strict match between data value and storage performance, significantly improving high-speed media utilization while avoiding the waste of static policy resources.

[0089] In the following embodiment, the request path generation method is further described from the perspective of the request path generation system. Figure 4 shown.

[0090] Figure 4 This paper describes a hierarchical design of a high-performance distributed storage system. The core modules and their interactions are as follows:

[0091] The client compute node layer is the user-facing application layer, primarily processing database requests. Its components include multipathing software to ensure I / O path redundancy and load balancing, thereby improving system availability. A private client integrates a customized protocol stack to optimize communication efficiency with the storage system. Direct connection to the data plane's block protocol services via the RDMA network enables efficient, low-latency interaction.

[0092] The Control Plane is responsible for global resource scheduling, monitoring, and configuration management. Key modules include a management interface that provides visual operations and policy configuration; a reconstruction engine that responds to node failures or load changes and triggers storage pool reconfiguration, such as data migration; a monitoring system that collects 12 metrics in real time, including storage pool health and cache hit rates, and synchronizes this data to global services via bRPC persistent connections; and media management that dynamically identifies media types such as SCM and NVMe SSD, partitions storage pools, and adapts the corresponding drivers.

[0093] The core function of the data plane is to process data I / O requests to achieve low-latency and high-throughput data access. Its sub-modules include block protocol services, in which host management and load balancing achieve an average hop count of ≤1 through consistent hashing and topology-aware routing; SPDK (NVMe over RoCE) bypasses the kernel through the user-mode protocol stack and provides microsecond-level NVMeoF request processing; LIBDBS uniformly manages multiple storage engines (such as SPDK DBS) and improves concurrency efficiency through the coroutine model. The NOF (Network Optimized Fabric) storage pool module uses bRPC and full-stack RTC coroutines to achieve cross-node communication. Coroutines effectively reduce the overhead of context switching; the metadata engine is designed for small-capacity, low-latency scenarios and uses CAS atomic operations to ensure strong consistency. In terms of storage media, SCM is a memory-level fast storage pool managed through the PMDK persistence programming model; NVMe SSD is managed at a 32KB block granularity to optimize sequential access performance.

[0094] The underlying infrastructure includes computing resources and media tiering. In terms of computing resources, the CPU is responsible for general computing, processing control logic, and lightweight I / O. The DPU is used to offload storage protocol processing (such as data encryption and compression) to improve resource utilization. The network layer uses a full RDMA architecture (RoCEv2 / InfiniBand) to achieve high-bandwidth, low-latency remote memory access. In terms of media tiering, the fast tier (such as SCM and NVMe SSD) is used to process hot data and supports memory-level access. The slow tier (such as HDD and SATA SSD) is used to store cold data and implements automatic data migration through intelligent tiering strategies.

[0095] This architecture diagram illustrates vertical integration from the client request entry point to the underlying storage media, emphasizing the construction of a high-throughput, low-latency distributed storage system through software and hardware collaboration (such as DPU offload and RDMA), tiered storage strategies (SCM / SSD / HDD), and lock-free metadata management. The decoupling of the management and data planes ensures operational flexibility and extreme data performance.

[0096] The request path generation system provided by the embodiments of the present invention achieves localized routing of requests through dynamic topology awareness, combining consistent hashing with node health status. Two-level cache consistency utilizes memory-level LRU caching and persistent version snapshots, and resolves distributed lock conflicts through lock-free CAS technology. I / O scheduling optimization utilizes multi-dimensional priority queues and physical address pre-allocation, effectively reducing disk seek times. Full-stack coroutine technology is implemented throughout all modules from LIBDBS to bRPC, reducing concurrency latency.

[0097] In addition, a hardware abstraction layer (HAL) can be built between the data plane and the control plane to encapsulate heterogeneous hardware capabilities (DPU / FPGA / smart network card, etc.) through standardized interfaces (such as dpu_offload_init(), scm_flush()) to form a universal execution environment for protocol offloading. Based on the device fingerprint library (PCIe ID + firmware version), the driver adapter is automatically matched to enable the same I / O processing logic (such as NVMe-oF data packet parsing) to seamlessly adapt to different hardware platforms. The control plane monitoring system analyzes request characteristics in real time (such as small packet ratio > 70% or dense encryption requests), dynamically orchestrates the protocol processing pipeline (for example: parsing → verification → encryption) and allocates it to the optimal hardware unit (such as idle DPU or FPGA). When a hardware unit fails, the circuit breaker mechanism is triggered to downgrade to software implementation (such as the SPDK encryption module) in real time. At the same time, the faulty device is marked and the global health status map is updated to ensure seamless task migration. This design is centered on hardware independence and dynamic resilience. It not only shields hardware iteration differences through an abstraction layer (compatible with future new devices such as CXL-attached accelerators), but also uses pipeline hot migration and fault fallback mechanisms to break through the limitations of traditional hardware binding solutions, providing a sustainable and evolving technical framework for protocol acceleration.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0099] The embodiment of the present application also provides a request path generation device, such as Figure 5 Shown, including:

[0100] The acquisition module 501 is used to obtain the client request, the data storage pool, and the hardware configuration information of each storage node in the storage cluster;

[0101] A division module 502 is configured to divide the data storage pool into a first storage pool and a second storage pool according to the medium performance difference in the hardware configuration information, wherein the storage speed of the first storage pool is greater than the storage speed of the second storage pool;

[0102] The determination module 503 is configured to determine the target cache node corresponding to the client request and the metadata status of the target cache node using a preset topology awareness method;

[0103] The storage module 504 is configured to store the request data carried by the client request to a target storage pool based on the request characteristics and metadata status of the client request, where the target storage pool is the first storage pool or the second storage pool;

[0104] The generating module 505 is used to dispatch the client request to the target storage pool for execution and generate a target storage path corresponding to the client request.

[0105] In some optional implementations, the determining module 503 includes:

[0106] The parsing submodule is used to parse the client request and determine the logical block address and node topology relationship corresponding to the client request;

[0107] A mapping submodule, configured to map the client request to the corresponding target cache node in the first storage pool or the second storage pool according to the relationship between the logical block address and the node topology;

[0108] The analysis submodule is used to analyze the metadata access characteristics of the target cache node and determine the metadata status of the target cache node.

[0109] In some optional embodiments, the analysis submodule includes:

[0110] The acquisition unit is used to obtain the persistent storage metadata snapshot and the version number of the metadata snapshot from the global cache layer;

[0111] The increment unit is used to increment the version number of the metadata snapshot through atomic operations to obtain an increment result;

[0112] The asynchronous unit is used to trigger the asynchronous metadata reconstruction process based on the pre-trained event-driven model when the incremental result representation version number conflicts and obtain the metadata status.

[0113] In some optional implementations, the storage module 504 includes:

[0114] The partitioning submodule is used to divide the request data into hot and cold data based on the request characteristics and metadata status of the client request, and obtain cold data and hot data;

[0115] The first storage submodule is configured to store cold data in the second storage pool and store hot data in the first storage pool.

[0116] In some optional implementations, the division submodules include:

[0117] A statistics unit, used to use a sliding window to count the access frequency of client requests within a preset time;

[0118] The partitioning unit is used to divide the requested data into cold and hot data according to the access frequency, thereby obtaining cold data and hot data.

[0119] In some optional implementations, the storage module 504 further includes:

[0120] A determination submodule is used to determine the data storage address corresponding to the client request through a preset physical address pre-allocation strategy;

[0121] The second storage submodule is used to store the request data requested by the client according to the data storage address.

[0122] In some optional implementations, the determining submodule includes:

[0123] The classification unit is used to classify client requests into multiple priority queues using a multi-dimensional scheduling method. Each priority queue is bound to an independent disk area.

[0124] The allocation unit is used to allocate the requested data of the same volume to the adjacent storage space in the independent disk area corresponding to the priority queue in response to the client request in any priority queue.

[0125] For the description of the features in the embodiment corresponding to the request path generation device, please refer to the relevant description of the embodiment corresponding to the request path generation method, and will not be repeated here.

[0126] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, it includes a memory 10 and a processor 20, the memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above request path generation method embodiments.

[0127] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned request path generation method embodiments when running.

[0128] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0129] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned request path generation method embodiments are implemented.

[0130] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned request path generation method embodiments are implemented.

[0131] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The above is a detailed introduction to a request path generation method, device, electronic device and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A request path generation method, characterized in that: The method comprises: Obtain client requests, data storage pools, and hardware configuration information of each storage node in the storage cluster; Dividing the data storage pool into a first storage pool and a second storage pool according to the medium performance difference in the hardware configuration information, wherein the storage speed of the first storage pool is greater than the storage speed of the second storage pool; Determine the target cache node corresponding to the client request and the metadata status of the target cache node by using a preset topology awareness method; Based on the request characteristics of the client request and the metadata state, storing the request data carried by the client request in a target storage pool, where the target storage pool is the first storage pool or the second storage pool; The client request is dispatched to the target storage pool for execution, and a target storage path corresponding to the client request is generated.

2. The request path generation method according to claim 1, characterized in that: The determining the target cache node corresponding to the client request and the metadata status of the target cache node by using a preset topology awareness method includes: Parsing the client request to determine the logical block address and node topology relationship corresponding to the client request; Mapping the client request to a corresponding target cache node in the first storage pool or the second storage pool according to the relationship between the logical block address and the node topology; The metadata access characteristics of the target cache node are analyzed to determine the metadata state of the target cache node.

3. The request path generation method according to claim 2, characterized in that: The analyzing the metadata access characteristics of the target cache node to determine the metadata status of the target cache node includes: Obtain a persistently stored metadata snapshot and its version number from the global cache layer; Increment the version number of the metadata snapshot through atomic operations to obtain an incremented result; When the increment result indicates that a conflict occurs in the version number, an asynchronous metadata reconstruction process is triggered based on a pre-trained event-driven model to obtain the metadata state.

4. The request path generation method according to claim 1, wherein: The storing, based on the request characteristics of the client request and the metadata status, the request data carried by the client request into the target storage pool includes: Based on the request characteristics of the client request and the metadata status, the request data is divided into cold and hot data to obtain cold data and hot data; The cold data is stored in the second storage pool, and the hot data is stored in the first storage pool.

5. The request path generation method according to claim 4, characterized in that: The step of dividing the request data into cold and hot data based on the request characteristics of the client request and the metadata status to obtain cold data and hot data includes: Using a sliding window to count the access frequency of the client request within a preset time; The requested data is divided into cold and hot data according to the access frequency to obtain the cold data and the hot data.

6. The request path generation method according to claim 4, characterized in that: Also includes: Determining the data storage address corresponding to the client request through a preset physical address pre-allocation strategy; The request data requested by the client is stored according to the data storage address.

7. The request path generation method according to claim 6, characterized in that: The determining of the data storage address corresponding to the client request by using a preset physical address pre-allocation strategy includes: Using a multi-dimensional scheduling method, the client requests are classified into multiple priority queues, and each priority queue is bound to an independent disk area; For client requests in any of the priority queues, requested data of the same volume is allocated to adjacent storage spaces in the independent disk area corresponding to the priority queue.

8. A request path generation device, characterized in that: include: The acquisition module is used to obtain client requests, data storage pools, and hardware configuration information of each storage node in the storage cluster; a partitioning module, configured to partition the data storage pool into a first storage pool and a second storage pool according to a difference in medium performance in the hardware configuration information, wherein a storage speed of the first storage pool is greater than a storage speed of the second storage pool; A determination module, configured to determine a target cache node corresponding to the client request and a metadata state of the target cache node using a preset topology perception method; a storage module, configured to store the request data carried by the client request in a target storage pool based on the request characteristics of the client request and the metadata state, where the target storage pool is the first storage pool or the second storage pool; A generation module is used to dispatch the client request to the target storage pool for execution and generate a target storage path corresponding to the client request.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the request path generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the request path generation method according to any one of claims 1 to 7 are implemented.

Images