Queue storage management system under RDMA network

Abstract

A queue storage management system under an RDMA network comprises a data management module, a network transmission module, a data storage module and a queue management module which run on an FPGA platform. The data management module is bidirectionally interconnected with the network transmission module, the data storage module and the queue management module respectively; the data management module comprises a main control processing sub-module, a logic control sub-module, a data caching sub-module and a data checking and processing sub-module; the main control processing sub-module is respectively and bidirectionally interconnected with the logic control sub-module, the data caching sub-module and the data checking and processing sub-module, and is used for controlling the overall working logic of the system, including data storage management, scheduling of data transmission, working state control, dynamic allocation of system resources, abnormal state processing and starting of a fault-tolerant mechanism; the method can be more flexibly deployed in various application scenes; for multi-level abstraction of an interface, network layer queue processing and storage layer queue processing are hidden for a user level, a user does not need to care about how to process queues in a data transmission process, and only needs to care about correctness of instruction transmission, so that the system is easy to deploy and use, and has the advantages of high operation efficiency, flexible deployment and high efficiency. And the availability is high.

Description

Technical Field

[0001] The present invention relates to the technical field of data communication, and in particular to a queue storage management system under an RDMA network. Background Art

[0002] When performing network storage work, compared with local storage, the data type of network storage is more complex, and the size and type of data packets may change at any time. This requires that the network storage system should have a certain degree of flexibility and be able to change the system's working mode (such as queue form, quantity, depth, etc.) according to different real-time needs. In addition, network storage not only requires the system to have high data throughput and low data transmission latency, but also requires the system to be small in size and not too difficult to deploy. At the same time, it also requires the system interface to be as simple as possible to improve the ease of use and versatility of the storage system. At the same time, therefore, existing solutions are not suitable for direct application to network data storage work.

[0003] The patent application with application number CN202410160584.2, publication number CN118101777A, and titled "An NVMe-over-RDMA protocol transmission method and electronic device" invented a network storage implementation method based on NVMe-oF, which uses the standard NVMe-oF protocol as the middle layer between the network protocol and the storage protocol. When using standard NVMe-oF as the middleware, it is limited by the standard protocol and cannot achieve autonomous management of each queue pair (including RDMA and NVMe queue pairs), and cannot flexibly allocate queues according to actual application conditions. Therefore, this solution is not applicable in environments with variable and complex data.

[0004] The patent application with application number CN202411171854.6, publication number CN118672953A, and titled "NVMeoF response instruction transmission method based on RDMA completion event" invented a method of tightly coupling data transmission and NVMeoF response instruction transmission to improve the system's request processing efficiency. When using the method of tightly coupling data transmission and NVMeoF response instruction transmission, when it transmits a control instruction without data transmission or there is no data transmission temporarily, the resources used for data transmission cannot be released, which will cause high system resource usage and possible memory management problems.

[0005] The patent application with application number CN202410970696.4, publication number CN118509398A, and titled "A virtual machine communication method, system, device, equipment and storage medium based on RDMA" invented a network storage solution using virtual machine queue exchange as middleware. When using virtual machine queue exchange as middleware, it cannot avoid the performance loss caused by virtual machine scheduling, which will cause the overall performance of the system to decline, and the virtual machine architecture is relatively large, and the requirements for the operating environment are relatively higher.

[0006] The patent application with application number CN202311845696.3, publication number CN117880216A, and titled "A data transmission method, storage medium, and device based on RDMA communication" invented a network storage solution with multiple nodes as middleware, which realizes data transmission and storage through queue transformation between two nodes. When multiple nodes are used as middleware, the control logic is realized through data interaction between nodes, but multiple nodes will inevitably lead to increased data latency, increased volume, and increased deployment difficulty of the system, making it difficult to flexibly apply in various scenarios. Summary of the invention

[0007] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a queue storage management system under an RDMA network. For scenarios where the data transmitted over the network is relatively complex or the data type / volume to be stored / read is not uniform, the present invention is deployed on an FPGA and adopts a variable queue design. The system can flexibly allocate RDMA queue pairs and NVMe queue pairs according to the current data transmission requirements / data storage requirements to ensure that the system can exert full performance under most working conditions without the user actively intervening in the operation queue. The virtual queues within the system include queue types, queue depths, and queue numbers, etc., and multiple receiving queues / completion queues that are not sensitive to performance can be abstracted into the same queue to adapt to data transmission / data access requirements under different working conditions; there are no strict restrictions on specific FPGA models and volumes, and compared with traditional storage servers, the power consumption is relatively low, and it can be more flexibly deployed in various application scenarios; for the multi-level abstraction of the interface, the network layer queue processing and the storage layer queue processing are hidden from the user level, and the user does not need to care about how to process the queue during the data transmission process, but only needs to care about the correctness of the instruction transmission, thereby further improving the ease of deployment and usability of the system, and having the advantages of efficient operation, flexible deployment, and high availability.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A queue storage management system under an RDMA network comprises a data management module, a network transmission module, a data storage module and a queue management module running on an FPGA platform; the data management module is bidirectionally interconnected with the network transmission module, the data storage module and the queue management module respectively; the data management module comprises a main control processing submodule, a data inspection and processing submodule, a data cache submodule and a logic control submodule; the main control processing submodule is bidirectionally interconnected with the logic control submodule, the data cache submodule and the data inspection and processing submodule respectively to control the overall working logic of the system, including data storage management and data transmission scheduling, working state control, dynamic allocation of system resources and abnormal state processing and fault tolerance mechanism startup.

[0010] The network transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule;

[0011] The network protocol stack submodule includes RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack is based on the standard network layer connection. It receives the network frames transmitted by the host computer and parses the data to be stored or packages the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that the devices in the network can connect and exchange data normally, mapping IP addresses to MAC addresses through the ARP protocol, or testing the reachability of devices through the ping operation through the ICMP protocol.

[0012] Among them, the network connection control submodule is used to connect this system with a remote server, switch or user machine; after the QSFP interface detects that the network connection control submodule is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process and actively sends a connection request (REQ) packet, and then, after receiving the host computer connection reply (REP) agreeing to the connection, it actively sends a ready to use (RTU) packet, and the connection establishment is completed at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, and access key, to prepare for data transmission.

[0013] The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule and a storage array control submodule; wherein the storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule respectively;

[0014] Among them, the non-volatile storage medium submodule transfers various storage media, including solid-state drives, mechanical hard drives, and storage arrays, to the FPGA through the FMC interface expansion card;

[0015] Among them, the storage instruction control submodule packages the user data into various storage layer protocols, thereby storing the data in the storage medium or reading the data from the storage medium and unpacking it;

[0016] Among them, the storage array control submodule supports common storage array types including RAID 0, RAID 1, and RAID 5 working mechanisms through built-in support, achieving efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read-write management. It can directly communicate with the storage array and perform data table maintenance and management to get rid of dependence on storage array cards, making the configuration of the storage array more flexible and comprehensive.

[0017] The queue management module includes a virtual queue pool submodule, a network queue management submodule and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule respectively;

[0018] Among them, the network queue management submodule controls the establishment, handshake, link breaking, and queue pair maintenance of RDMA queues; the specific control methods include: ① Using dynamic allocation and release technology, RDMA queues are allocated in the virtual queue pool according to traffic demand. When the traffic increase causes a queue to approach the load limit, the resource allocation behavior is triggered to increase its queue depth or allocate a new queue from the virtual queue pool; when it is detected that a queue has not been used for a long time or the occupancy rate is lower than the set threshold, the queue resources are automatically recycled to reduce memory and processor overhead; ② Set a dedicated register to record the current status of the RDMA completion queue (CQ), including queue length and number of completed tasks; the state machine continuously obtains real-time status data to determine whether there are unfinished tasks or queue interruptions. If a link break is detected, the state machine reinitializes the link break queue; ③ Establish a binding relationship between the RDMA request queue SQ and the completion queue CQ, and realize bidirectional tracking of task status through a shared index pointer; when the request queue SQ issues a new request, the task ID is recorded and the expected completion status of the completion queue CQ is synchronously updated; once a task exception occurs, the binding behavior is retriggered to ensure the reliability of the queue pair;

[0019] Among them, the storage queue management submodule controls the NVMe queue, including the establishment and destruction of the Admin queue and IO queue, as well as the maintenance of the queue pair; the specific control methods include: ① establishing and destroying queues through the NVMe instruction set; ② using the queue resource mapping table to track the usage status of the queue in real time and dynamically adjust resource allocation; ③ by real-time monitoring the update of the task completion flag in the Admin queue, detect whether the task is completed as expected; if it is found that the task is not completed or an error occurs, reinitialize the relevant I / O queue to ensure efficient scheduling;

[0020] Among them, the virtual queue pool submodule controls the network queue management module and the storage queue management module. The specific implementation methods include: ① Based on the dynamic allocation algorithm of the queue resource pool, the physical queue is abstracted into a logical queue to flexibly adjust the queue size and number; ② Using virtualization technology, the queue interface is isolated from the physical memory, and a unified abstract interface is provided for the completion queue and the receiving queue; ③ Through the queue status monitoring module, the operating status of the queues of each module is detected in real time, and queue resources are dynamically allocated to ensure that the system can still run stably under high load; ④ Integrate the load balancing algorithm to give priority to allocating low-latency queue resources in the queue pool.

[0021] The specific method of data storage management is:

[0022] First, through the interaction between the main control processing submodule and the data storage module, as well as real-time monitoring of the storage queue status and data priority, the queue depth and cache strategy are dynamically adjusted to coordinate data read and write operations: when the data read demand continues to exceed the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is reduced; conversely, when the data write demand continues to exceed the read demand, the read instruction queue depth is reduced, the read cache is reduced, and the write cache is increased;

[0023] Secondly, according to the system status including storage load and data priority, key data is processed first and queue scheduling is optimized, and data storage strategies are dynamically adjusted, including cache management and storage queue scheduling. Within a time window, when the storage queue occupancy rate continues to exceed the set Max threshold, high-priority data is processed first, and cache space and queue depth are increased. When the storage queue occupancy rate is lower than the set Min threshold, low-priority tasks are merged and their queue allocation is reduced, the cache is released and the queue depth is reduced to save storage load.

[0024] The time window is a fixed length of time before the current time.

[0025] The specific method of scheduling the data transmission is:

[0026] A dynamic queue scheduling algorithm is used to select high-load queues, increase their queue capacity, and reduce the capacity of idle queues. Network resources are allocated in real time, and data transmission paths are adjusted based on real-time network latency and bandwidth utilization. When network latency continues to increase or bandwidth utilization continues to exceed the set Max threshold, low-latency, high-bandwidth transmission paths are given priority. At the same time, the built-in flow control mechanism of the protocol stack, namely the PFC flow control + ECN congestion management of standard Ethernet, is further combined to ensure efficient and reliable transmission.

[0027] The specific method of working state control is: when the system works in cascade mode, there are two different working modes: host mode and slave mode; in host mode: simultaneous analysis includes user control package, user data package, user management package, data storage work and cascade network management including monitoring cascade network load status, joining / leaving cascade network and task allocation; slave mode: only responsible for data storage but not participating in cascade network management; when working in cascade mode, the main control processing submodule is responsible for monitoring the working state of the entire system, so as to control the state switching between the host and the slave: the specific method is: using the state feedback mechanism to judge the operation status of the host mode and the slave mode, when the host mode operation reaches a certain parameter threshold of the set state feedback mechanism, the slave working mode switching is triggered by state feedback, the slave mode is switched to the host mode and takes over all functions of the host mode, and at the same time the original host mode is switched to the slave mode to enter the standby state.

[0028] The specific method of dynamic allocation of system resources is:

[0029] Prioritize high-priority tasks or tasks initiated by high-priority users through the priority scheduling algorithm to coordinate resource competition between tasks of different priorities;

[0030] Dynamic resource allocation technology is used to allocate more queue resources to currently active users, while reducing resource allocation to idle users to coordinate user fairness in a multi-user environment;

[0031] Combined with the polling strategy, fairness and efficiency in a multi-task environment are ensured. Specifically, for tasks with the same weight, resources are allocated in sequence according to the polling order to ensure fairness. At the same time, under high load conditions, key tasks are given priority.

[0032] The specific method for abnormal state processing and fault tolerance mechanism activation is as follows: during system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted; in the host mode, user packets are parsed and stored while storing data, and in the slave mode, only data is stored and data packets are forwarded, and no other operations are performed.

[0033] The data checking and processing submodule receives the data transmitted by the network transmission module, and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, according to the instructions transmitted by the user through the network transmission module, the data required by the user is sent to the network transmission module;

[0034] The data cache submodule includes a high-speed volatile storage for bidirectionally caching received and to-be-sent data;

[0035] The logic control submodule manages all queue work logics in the system and sends instructions to subsequent queue management modules:

[0036] The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load;

[0037] The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, to meet the performance requirements of different tasks;

[0038] In the third step, after the queue resources are allocated in the second step, a global state synchronization mechanism is introduced to synchronize the load and usage status information of the one-to-one bound RDMA queues and NVMe queues in real time to ensure the information consistency of the one-to-one bound RDMA queues and NVMe queues;

[0039] The fourth step is to regularly obtain queue status information from the subsequent queue management module, including the queue's operating efficiency, task completion status, and error status, and summarize and analyze it in combination with the queue resource information in the first step: when it is found that the processing delay of a queue continues to exceed the threshold or the task completion rate has a significant downward trend, its queue resources are expanded first, including increasing the queue depth and improving its queue priority; when the error rate increases, the cause of the failure is analyzed and resources are reallocated to reduce the task allocation of the affected queue to improve system performance and reliability;

[0040] In the fifth step, combining the data and information in the third and fourth steps, when an abnormal situation is detected, including queue overflow and hardware failure, the fault-tolerant mechanism is immediately triggered to release the tasks of the affected queue, and dynamically assign the tasks to other available queues or switch to the backup queue; when the abnormal situation cannot be recovered, the queue tasks are taken over by switching to the backup resources to ensure that the queue tasks are not interrupted and the system runs stably.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention is deployed on FPGA, and the main control processing submodule of the data management module is responsible for controlling the working logic of the entire system of the present invention; in an environment where the volume / power consumption of the storage system is restricted, there is no strict restriction on the specific FPGA model and volume, and compared with traditional storage servers, the power consumption is relatively low, so the system can be flexibly deployed in various application scenarios, such as storage rooms, ships, aerospace and other places or fields; for the multi-level abstraction of the user interface, the system does not need to be configured when working, and can work completely unattended.

[0043] 2. The data storage module can normally establish queues and access data. When switching different types of non-volatile storage medium models and different storage array models, the present invention can correctly identify and switch to the protocol stack of the corresponding protocol; in view of the current situation where the storage device bus version, protocol version, and interface type are iterated quickly, the present invention can be based on the various different protocols and interface storage devices used by users. The present invention adopts a replaceable storage medium interface universal FMC board, and the speed, volume, capacity, interface type, etc. of the storage medium can be flexibly selected. At the same time, it supports multiple storage arrays such as RAID0 / RAID1 / RAID5, etc. Further improve the ease of use of the system and reduce the cost of use.

[0044] 3. For scenarios where the data transmitted over the network is relatively complex or the data type / volume that needs to be stored / read is not uniform, the queue management module of the present invention adopts a variable queue design. The system can flexibly allocate RDMA queue pairs and NVMe queue pairs according to the current data transmission requirements / data storage requirements to ensure that the system can exert its full performance under most working conditions without the user actively intervening in operating queues, as well as the virtual queue pool within the system (including but not limited to queue type, queue depth, number of queues, etc.), which can abstract multiple receiving queues / completion queues that are not sensitive to performance into the same queue to adapt to data transmission / data access requirements under different working conditions.

[0045] 4. The network transmission module of the present invention uses a standard QSFP optical fiber interface to communicate with a remote host computer, which has less restrictions on the transmission distance and can achieve long-distance high-speed data transmission at low cost and high efficiency. At the same time, due to the mechanism of the variable queue and the existence of the queue pool, the system can access the switch without changing any device protocol stack in the network, which is convenient for multiple users to access at the same time, effectively reducing system resource usage and reducing transmission delays.

[0046] In summary, the present invention is deployed on FPGA, and has no strict restrictions on the specific FPGA model and volume. At the same time, compared with traditional storage servers, the power consumption is relatively low, and it can be more flexibly deployed in various application scenarios; for the multi-level abstraction of the interface, the network layer queue processing and the storage layer queue processing are hidden from the user level. The user does not need to worry about how to process the queue during the data transmission process, but only needs to care about the correctness of the instruction transmission, which further improves the system's ease of deployment and usability, and has the advantages of efficient operation, flexible deployment, and high availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly and effectively illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a system function module relationship diagram of the present invention.

[0049] Figure 2 It is a schematic diagram of the data management module of the present invention.

[0050] Figure 3 It is a schematic diagram of the variable queue design workflow of the present invention.

[0051] Figure 4 It is a schematic diagram of the virtual queue pool of the present invention.

[0052] Figure 5 It is a typical working principle diagram of the present invention.

[0053] Figure 6 This is a schematic diagram of a first typical deployment of the present invention.

[0054] Figure 7 This is a second typical deployment diagram of the present invention.

[0055] Figure 8 This is a third typical deployment diagram of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Reference Figure 1 , Figure 2 A queue storage management system under an RDMA network comprises a data management module, a network transmission module, a data storage module and a queue management module running on an FPGA platform; the data management module is bidirectionally interconnected with the network transmission module, the data storage module and the queue management module respectively; the data management module comprises a main control processing submodule, a data inspection and processing submodule, a data cache submodule and a logic control submodule; the main control processing submodule is bidirectionally interconnected with the logic control submodule, the data cache submodule and the data inspection and processing submodule respectively to control the overall working logic of the system, including data storage management and data transmission scheduling, working state control, dynamic allocation of system resources and abnormal state processing and fault tolerance mechanism startup.

[0058] The network transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule;

[0059] The network protocol stack submodule includes RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack is based on the standard network layer connection. It receives the network frames transmitted by the host computer and parses the data to be stored or packages the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that the devices in the network can connect and exchange data normally, mapping IP addresses to MAC addresses through the ARP protocol, or testing the reachability of devices through the ping operation through the ICMP protocol.

[0060] Among them, the network connection control submodule is used to connect this system with a remote server, switch or user machine; after the QSFP interface detects that the network connection control submodule is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process and actively sends a connection request (REQ) packet, and then, after receiving the host computer connection reply (REP) agreeing to the connection, it actively sends a ready to use (RTU) packet, and the connection establishment is completed at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, and access key, to prepare for data transmission.

[0061] The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule and a storage array control submodule; wherein the storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule respectively;

[0062] Among them, the non-volatile storage medium submodule transfers various storage media, including solid-state drives, mechanical hard drives, and storage arrays, to the FPGA through the FMC interface expansion card;

[0063] Among them, the storage instruction control submodule packages the user data into various storage layer protocols, thereby storing the data in the storage medium or reading the data from the storage medium and unpacking it;

[0064] Among them, the storage array control submodule supports common storage array types including RAID 0, RAID 1, and RAID 5 working mechanisms through built-in support, achieving efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read-write management. It can directly communicate with the storage array and perform data table maintenance and management to get rid of dependence on storage array cards, making the configuration of the storage array more flexible and comprehensive.

[0065] The queue management module includes a virtual queue pool submodule, a network queue management submodule and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule respectively;

[0066] Among them, the network queue management submodule controls the establishment, handshake, link breaking, and queue pair maintenance of RDMA queues; the specific control methods include: ① Using dynamic allocation and release technology, allocate RDMA queues in the virtual queue pool according to traffic demand. When the traffic increase causes a queue to approach the load limit, the resource allocation behavior is triggered to increase its queue depth or allocate a new queue from the virtual queue pool; when it is detected that a queue has not been used for a long time or the occupancy rate is lower than the set threshold (such as 20%), the queue resources are automatically recycled to reduce memory and processor overhead; ② Setting a dedicated register to record the current status of the RDMA completion queue (CQ) includes The state machine continuously obtains real-time status data to determine whether there are unfinished tasks or queue interruptions. If a disconnection is detected (such as the completion flag is not updated), the state machine reinitializes the disconnection queue; ③ Establish a binding relationship between the RDMA request queue SQ and the completion queue CQ, and realize bidirectional tracking of task status through a shared index pointer; When the request queue SQ issues a new request, the task ID is recorded and the expected completion status of the completion queue CQ is synchronously updated; Once a task exception occurs (such as failure to update the completion status in time or an abnormal completion status occurs), the binding behavior is retriggered to ensure the reliability of the queue for work;

[0067] Among them, the storage queue management submodule controls the NVMe queue, including the establishment and destruction of the Admin queue and IO queue, as well as the maintenance of the queue pair; the specific control methods include: ① establishing and destroying queues through the NVMe instruction set; ② using the queue resource mapping table to track the usage status of the queue in real time and dynamically adjust resource allocation; ③ by real-time monitoring the update of the task completion flag in the Admin queue, detect whether the task is completed as expected; if it is found that the task is not completed or an error occurs, reinitialize the relevant I / O queue to ensure efficient scheduling;

[0068] Among them, the virtual queue pool submodule controls the network queue management module and the storage queue management module. The specific implementation methods include: ① Based on the dynamic allocation algorithm of the queue resource pool, the physical queue is abstracted into a logical queue to flexibly adjust the queue size and number; ② Using virtualization technology, the queue interface is isolated from the physical memory, and a unified abstract interface is provided for the completion queue and the receiving queue; ③ Through the queue status monitoring module, the operating status of the queues of each module is detected in real time, and queue resources are dynamically allocated to ensure that the system can still run stably under high load; ④ Integrate the load balancing algorithm to give priority to allocating low-latency queue resources in the queue pool.

[0069] The specific method of data storage management is:

[0070] First, through the interaction between the main control processing submodule and the data storage module, as well as real-time monitoring of the storage queue status and data priority, the queue depth and cache strategy are dynamically adjusted to coordinate data read and write operations: when the data read demand continues to exceed the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is reduced; conversely, when the data write demand continues to exceed the read demand, the read instruction queue depth is reduced, the read cache is reduced, and the write cache is increased;

[0071] Secondly, according to the system status including storage load and data priority, key data is processed first and queue scheduling is optimized, and data storage strategies are dynamically adjusted, including cache management and storage queue scheduling. Within a time window, when the storage queue occupancy rate continues to exceed the set Max threshold, high-priority data is processed first, and cache space and queue depth are increased. When the storage queue occupancy rate is lower than the set Min threshold, low-priority tasks are merged and their queue allocation is reduced, the cache is released and the queue depth is reduced to save storage load.

[0072] The time window is a fixed length of time before the current time.

[0073] The specific method of scheduling the data transmission is:

[0074] A dynamic queue scheduling algorithm is used to select high-load queues, increase their queue capacity, and reduce the capacity of idle queues. Network resources are allocated in real time, and data transmission paths are adjusted based on real-time network latency and bandwidth utilization. When network latency continues to increase or bandwidth utilization continues to exceed the set Max threshold, low-latency, high-bandwidth transmission paths are given priority. At the same time, the built-in flow control mechanism of the protocol stack, namely the PFC flow control + ECN congestion management of standard Ethernet, is further combined to ensure efficient and reliable transmission.

[0075] The specific method of working state control is as follows: when the system works in cascade mode, there are two different working modes: host mode and slave mode; in host mode: simultaneously parse user control packages, user data packages, user management packages, data storage work and cascade network management including monitoring cascade network load status, joining / leaving cascade network and task allocation; in slave mode: only responsible for data storage but not participating in cascade network management; when working in cascade mode, the main control processing submodule is responsible for monitoring the working state of the entire system, so as to control the state switching between the host and the slave: the specific method is: using the state feedback mechanism (such as heartbeat detection, load monitoring) to judge the operation status of the host mode and the slave mode, when the load in the host mode is continuously higher than 80% or the heartbeat detection prompts that the delay is too high, the slave working mode switching is triggered by state feedback, the slave mode is switched to the host mode and takes over all functions of the host mode, and at the same time the original host mode is switched to the slave mode to enter the standby state.

[0076] The specific method for dynamically allocating the system resources is as follows:

[0077] Prioritize high-priority tasks or tasks initiated by high-priority users through the priority scheduling algorithm to coordinate resource competition between tasks of different priorities;

[0078] Dynamic resource allocation technology is used to allocate more queue resources to currently active users, while reducing resource allocation to idle users to coordinate user fairness in a multi-user environment;

[0079] Combined with the polling strategy, fairness and efficiency in a multi-task environment are ensured. Specifically, for tasks with the same weight, resources are allocated in sequence according to the polling order to ensure fairness. At the same time, under high load conditions, key tasks are given priority to ensure efficiency.

[0080] The specific method for abnormal state processing and fault tolerance mechanism activation is as follows: during system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted; in the host mode, user packets are parsed and stored while storing data, and in the slave mode, only data is stored and data packets are forwarded, and no other operations are performed, thereby ensuring system stability.

[0081] The data checking and processing submodule receives the data transmitted by the network transmission module, and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, according to the instructions transmitted by the user through the network transmission module, the data required by the user is sent to the network transmission module;

[0082] The data cache submodule includes a high-speed volatile storage for bidirectionally caching received and to-be-sent data;

[0083] The logic control submodule manages all queue work logics in the system and sends instructions to subsequent queue management modules:

[0084] The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load;

[0085] The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, to meet the performance requirements of different tasks;

[0086] In the third step, after the queue resources are allocated in the second step, a global state synchronization mechanism is introduced to synchronize the load and usage status information of the one-to-one bound RDMA queues and NVMe queues in real time to ensure the information consistency of the one-to-one bound RDMA queues and NVMe queues;

[0087] The fourth step is to regularly obtain queue status information from the subsequent queue management module, including the queue's operating efficiency, task completion status, and error status, and summarize and analyze it in combination with the queue resource information in the first step: when it is found that the processing delay of a queue continues to exceed the threshold or the task completion rate has a clear downward trend, give priority to expanding its queue resources, including increasing the queue depth and improving its queue priority; when the error rate increases, analyze the cause of the failure and reallocate resources to reduce the task allocation of the affected queue to improve system performance and reliability.

[0088] In the fifth step, combining the data and information in the third and fourth steps, when an abnormal situation is detected, including queue overflow and hardware failure, the fault-tolerant mechanism is immediately triggered to release the tasks of the affected queue, and dynamically assign the tasks to other available queues or switch to the backup queue; when the abnormal situation cannot be recovered, the queue tasks are taken over by switching to the backup resources to ensure that the queue tasks are not interrupted and the system runs stably.

[0089] Reference Figure 3, the logic control submodule continuously analyzes the data transmission requests received by the current system. When it is detected that the current data type is a continuous large data packet, the number of queues will be reduced and the queue depth will be increased. This can help improve the system's efficiency and throughput in processing longer data packets, while minimizing the cache and resource usage; when it is detected that the current data type is a scattered small data packet, the number and depth of the queues will be increased at the same time, because although the continuous scattered small data packets are short in length, they will bring a lot of IO consumption. Increasing the number of queues will help reduce the pressure of a single queue processing engine, and increasing the depth of the queue can minimize the possibility of data packet congestion in the system. At the same time, when storing and reading data, scattered data packets will be merged into larger data transmission requests as much as possible to avoid the IOPS performance bottleneck of the storage system. Finally, interact with the storage module to store / read data. When the transmission task is completed, the queue will be restored to the default state and wait for the next transmission task.

[0090] Reference Figure 4 The queues described in the present invention include: RDMA SQ, RDMA RQ, RDMACQ, NVMe Admin SQ / CQ, and NVMe I / O SQ / CQ.

[0091] Among them, the real queue of the RDMA part is SQ, and RQ and CQ are presented by the virtual queue pool. SQ is the receiving queue, which is strongly related to the host computer. Each queue must have an independent physical control to facilitate sending data to the host computer; RQ is the receiving queue. In FPGA, it is actually only necessary to store the correspondence between the serial number, data and queue number of each data packet. In fact, it is not really necessary to completely store the entire content of each data packet; similarly, CQ is the completion queue, which records every completed network transmission command of the system, and is also not strongly related to the host computer. Therefore, using the virtual queue pool to implement RQ and CQ can minimize the waste of system resources and improve the system's operating efficiency without affecting its communication with any other RDMA devices marked with you.

[0092] In addition, the real queue of the MVNe part is SQ and does not distinguish between the Admin queue and the I / O queue. CQ is also implemented by the virtual queue pool. Similarly, CQ is not strongly related to the storage medium under it. The indication marks whether the instruction is completed. Since Admin instructions are not often used during the normal operation of the system, if a pair of queues are saved separately for Admin instructions, it will inevitably cause a waste of resources. Therefore, if the system needs to transmit Admin instructions, the virtual queue pool will control an I / OSQ to temporarily convert it into Admin SQ, and open up a small space for Admin CQ. When the instruction is completed, the queue will be restored and the resources will be released immediately. Such a mechanism will not affect the system's communication with standard NVMe devices while minimizing the waste of resources as much as possible.

[0093] Reference Figure 5 The typical workflow of this system is as follows: after receiving a data transmission request, the RDMA protocol stack first unpacks the data, and then stores the data in the data cache module. At the same time, the request is transferred to the logic processing module to analyze the request and determine whether the queue needs to be dynamically processed. This part of the process can be referred to Figure 3 . While the logic processing module analyzes the request, it will also return a network response packet and analyze the storage array type. After the request analysis is completed, the read / write instructions for the corresponding storage array will be generated and data will be read from the cache. After the data storage / reading process is completed, the storage response packet / read reply packet will be returned to the corresponding queue of the host computer. At this point, a transmission process ends. If there is no data packet to be processed, the system will be silent, otherwise the process will be repeated. It should be emphasized that in this system, all work processes are managed without CPU, which means that each step in such a process is running at any time, rather than in a sequential order. Figure 4 The workflow shown in is for ease of understanding only.

[0094] Reference Figure 6 , 7 8. The deployment method of this system is relatively flexible and can be deployed in a variety of ways:

[0095] like Figure 6 As shown: The first typical deployment method of this system is the simplest deployment method, which only uses one network port to connect to the host computer and one FMC expansion interface to connect to non-volatile storage media such as SSD, HDD, etc.

[0096] like Figure 7As shown in the figure: In the second typical deployment mode of this system, both QSFP network ports are used to connect to the host computer or switch. At the same time, both FMC expansion interfaces are enabled to connect to the storage array or high-speed SSD as a data buffer. There is no restriction on what device each network port should be connected to and what device each FMC interface should be connected to. In other words, each network port and each FMC interface have the same function.

[0097] like Figure 8 As shown in the figure: The third typical deployment mode of this system, both QSFP interfaces are enabled, but one of them is used to cascade multiple systems, and the connection mode of the FMC expansion interface remains unchanged. The maximum number of systems connected in this way is 255, and this connection mode can maximize system performance, whether it is speed, capacity or system robustness.

[0098] The technical effects of the present invention are described in detail below in conjunction with simulation experiments.

[0099] Based on the present invention, a simulated host computer is built, which can simulate the online and offline states of the host computer and different network interface rates. At the same time, a simple model of a virtual non-volatile storage medium is also built to simulate data storage.

[0100] 1. Network transmission verification: When the simulated host computer is detected, the present invention can establish a connection with the simulated host computer normally and establish a queue for communication. When the host computer with different network interface rates is connected, the present invention automatically switches the corresponding interface rate to adapt to the host computer. At a transmission rate of 100Gbps, the present invention works stably. According to the simulation results, the actual unidirectional throughput of the system is >300Gbps, and the actual forwarding packet capacity is >150Mpps (pps: Packet per seconds), which can meet the performance requirements of a dual-channel 100G RDMA channel.

[0101] 2. Data storage verification: When the storage medium access is detected, the present invention can also establish a queue and perform data access normally. When switching different types of non-volatile storage medium models and different storage array models, the present invention can also correctly identify and switch to the protocol stack of the corresponding protocol. Due to the different performances of different non-volatile storage media and storage arrays under actual application conditions, it is impossible to estimate the required storage rate, so a throughput test is performed. According to the simulation results, the maximum unidirectional throughput related to the data storage module reaches 150Gbps.

[0102] 3. Optimization effect of queue management and scheduling: In this part, by simulating different types of task requests (such as storage read and write requests, network data transmission requests), the working performance of the logic control module and the queue management module under different loads is verified. 1) Low load conditions: The system automatically reduces the number and depth of queues, thereby reducing system power consumption and improving working stability. 2) Under continuous high load conditions: The system automatically increases the number and depth of queues, and the virtual queue pool intervenes to abstract the virtual completion queue (storage) and the virtual receiving queue (network). In addition, the queue dynamic priority mechanism is started. In order to verify the performance, a simulation is also performed with the virtual queue pool disabled. According to the simulation results, compared with the case where the virtual queue pool is not enabled, the storage (RAM) resources occupied by the system are reduced by 34%, and compared with the case where the queue dynamic priority mechanism is not enabled, the system transmission delay is reduced by an average of about 10us.

[0103] 4. Coordination capability of the master processing submodule: In this part, two devices are cascaded for simulation, in which one device A works in the master mode. The other device B works in the slave mode. At this time, the load of device A is increased to the limit state. The simulation results show that the master and slave identities are exchanged, A works in the slave mode, and device B works in the master mode, taking over the work of device A. At this time, if the load of device A is continued to be increased, it can be found that device A enters the standby state, only forwarding data packets without processing, and the data packets are processed by device B with a relatively smaller load. In addition, in a simulated multi-user environment, the present invention correctly adjusts the queue according to the task priority to ensure fairness and efficiency and avoid resource competition.

[0104] It can be seen from the above that: 1. The present invention can work normally when equipped with different types of non-volatile storage media. 2. When the network interface rate of the host computer is different, the present invention can correctly identify and adjust its own interface rate. 3. Under high-load conditions, the dynamic queue + virtual queue pool management method of the present invention can effectively reduce system resource occupancy while reducing transmission delays. 4. Under extreme conditions, the present invention can automatically switch working modes and can work independently without supervision. 5. Through the virtual queue pool method, user ease of use can be further improved, and the present invention can be connected to any standard protocol device, reducing the deployment difficulty of the present invention.

[0105] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A queue storage management system under an RDMA network, characterized in that: It includes a data management module, a network transmission module, a data storage module, and a queue management module running on an FPGA platform; the data management module is bidirectionally interconnected with the network transmission module, the data storage module, and the queue management module; the data management module includes a main control processing submodule, a data inspection and processing submodule, a data cache submodule, and a logic control submodule; The main control processing submodule is bidirectionally interconnected with the logic control submodule, data cache submodule, and data checking and processing submodule to control the overall working logic of the system, including data storage management and data transmission scheduling, working status control, dynamic allocation of system resources, abnormal status processing, and fault tolerance mechanism startup.

2. The queue storage management system under the RDMA network according to claim 1, characterized in that: The network transmission module includes a bidirectionally interconnected network protocol stack submodule and a network connection control submodule; The network protocol stack submodule includes RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack is based on the standard network layer connection. It receives the network frames transmitted by the host computer and parses the data to be stored or packages the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that the devices in the network can connect and exchange data normally, mapping IP addresses to MAC addresses through the ARP protocol, or testing the reachability of devices through the ping operation through the ICMP protocol. Among them, the network connection control submodule is used to connect this system with a remote server, switch or user machine; after the QSFP interface detects that the network connection control submodule is inserted and the physical layer connection has been established, the network connection control submodule autonomously starts the connection establishment process and actively sends a connection request (REQ) packet, and then, after receiving the host computer connection reply (REP) agreeing to the connection, it actively sends a ready to use (RTU) packet, and the connection establishment is completed at this time; after the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters of the host computer, including memory address, memory size, and access key, to prepare for data transmission.

3. The queue storage management system in an RDMA network according to claim 1, characterized in that: The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule and a storage array control submodule; The storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule respectively; Among them, the non-volatile storage medium submodule transfers various storage media, including solid-state drives, mechanical hard drives, and storage arrays, to the FPGA through the FMC interface expansion card; Among them, the storage instruction control submodule packages the user data into various storage layer protocols, thereby storing the data in the storage medium or reading the data from the storage medium and unpacking it; Among them, the storage array control submodule supports common storage array types including RAID 0, RAID 1, and RAID5 working mechanisms through built-in support, achieving efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read-write management. It can directly communicate with the storage array and perform data table maintenance and management to get rid of dependence on storage array cards, making the configuration of the storage array more flexible and comprehensive.

4. The queue storage management system in an RDMA network according to claim 1, characterized in that: The queue management module includes a virtual queue pool submodule, a network queue management submodule and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule respectively; Among them, the network queue management submodule controls the establishment, handshake, link breaking, and queue pair maintenance of RDMA queues; the specific control methods include: ① Using dynamic allocation and release technology, RDMA queues are allocated in the virtual queue pool according to traffic demand. When the traffic increase causes a queue to approach the load limit, the resource allocation behavior is triggered to increase its queue depth or allocate a new queue from the virtual queue pool; when it is detected that a queue has not been used for a long time or the occupancy rate is lower than the set threshold, the queue resources are automatically recycled to reduce memory and processor overhead; ② Set a dedicated register to record the current status of the RDMA completion queue (CQ), including queue length and number of completed tasks; the state machine continuously obtains real-time status data to determine whether there are unfinished tasks or queue interruptions. If a link break is detected, the state machine reinitializes the link break queue; ③ Establish a binding relationship between the RDMA request queue SQ and the completion queue CQ, and realize bidirectional tracking of task status through a shared index pointer; when the request queue SQ issues a new request, the task ID is recorded and the expected completion status of the completion queue CQ is synchronously updated; once a task exception occurs, the binding behavior is retriggered to ensure the reliability of the queue pair; Among them, the storage queue management submodule controls the NVMe queue, including the establishment and destruction of the Admin queue and IO queue, as well as the maintenance of the queue pair; the specific control methods include: ① establishing and destroying queues through the NVMe instruction set; ② using the queue resource mapping table to track the usage status of the queue in real time and dynamically adjust resource allocation; ③ by real-time monitoring the update of the task completion flag in the Admin queue, detect whether the task is completed as expected; if it is found that the task is not completed or an error occurs, reinitialize the relevant I / O queue to ensure efficient scheduling; Among them, the virtual queue pool submodule controls the network queue management module and the storage queue management module. The specific implementation methods include: ① Based on the dynamic allocation algorithm of the queue resource pool, the physical queue is abstracted into a logical queue to flexibly adjust the queue size and number; ② Using virtualization technology, the queue interface is isolated from the physical memory, and a unified abstract interface is provided for the completion queue and the receiving queue; ③ Through the queue status monitoring module, the operating status of the queues of each module is detected in real time, and queue resources are dynamically allocated to ensure that the system can still run stably under high load; ④ Integrate the load balancing algorithm to give priority to allocating low-latency queue resources in the queue pool.

5. The queue storage management system in an RDMA network according to claim 1, characterized in that: The specific method of data storage management is: First, through the interaction between the main control processing submodule and the data storage module, as well as real-time monitoring of the storage queue status and data priority, the queue depth and cache strategy are dynamically adjusted to coordinate data read and write operations: when the data read demand continues to exceed the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is reduced; conversely, when the data write demand continues to exceed the read demand, the read instruction queue depth is reduced, the read cache is reduced, and the write cache is increased; Secondly, according to the system status including storage load and data priority, key data is processed first and queue scheduling is optimized, and data storage strategies are dynamically adjusted, including cache management and storage queue scheduling. Within a time window, when the storage queue occupancy rate continues to exceed the set Max threshold, high-priority data is processed first, and cache space and queue depth are increased. When the storage queue occupancy rate is lower than the set Min threshold, low-priority tasks are merged and their queue allocation is reduced, the cache is released and the queue depth is reduced to save storage load. The time window is a fixed length of time before the current time.

6. The queue storage management system in an RDMA network according to claim 1, characterized in that: The specific method for scheduling the data transmission is: A dynamic queue scheduling algorithm is used to select high-load queues, increase their queue capacity, and reduce the capacity of idle queues. Network resources are allocated in real time, and data transmission paths are adjusted based on real-time network latency and bandwidth utilization. When network latency continues to increase or bandwidth utilization continues to exceed the set Max threshold, low-latency, high-bandwidth transmission paths are given priority. At the same time, the built-in flow control mechanism of the protocol stack, namely the PFC flow control + ECN congestion management of standard Ethernet, is further combined to ensure efficient and reliable transmission.

7. The queue storage management system in an RDMA network according to claim 1, characterized in that: The specific method of the working state control is as follows: when the system works in a cascade mode, there are two different working modes: host mode and slave mode; in the host mode: simultaneous parsing includes user control packets, user data packets, user management packets, data storage work, and cascade network management including monitoring cascade network load status, joining / leaving cascade network, and task allocation; Slave mode: only responsible for data storage and not participating in cascade network management; when working in cascade mode, the master control processing submodule is responsible for monitoring the working status of the entire system, so as to control the state switching between the host and the slave: the specific method is: using the state feedback mechanism to judge the operation status of the host mode and the slave mode, when the host mode operation reaches a certain parameter threshold of the set state feedback mechanism, the state feedback is used to trigger the slave working mode switching, and the slave mode is switched to the host mode and takes over all the functions of the host mode, and at the same time the original host mode is switched to the slave mode to enter the standby state.

8. The queue storage management system in an RDMA network according to claim 1, characterized in that: The specific method of dynamic allocation of system resources is: Prioritize high-priority tasks or tasks initiated by high-priority users through the priority scheduling algorithm to coordinate resource competition between tasks of different priorities; Dynamic resource allocation technology is used to allocate more queue resources to currently active users, while reducing resource allocation to idle users to coordinate user fairness in a multi-user environment; Combined with the polling strategy, fairness and efficiency in a multi-task environment are ensured. Specifically, for tasks with the same weight, resources are allocated in sequence according to the polling order to ensure fairness. At the same time, under high load conditions, key tasks are given priority.

9. The queue storage management system in an RDMA network according to claim 1, characterized in that: The specific method of abnormal state processing and fault tolerance mechanism activation is as follows: during system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers the path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted; In the host mode, the user packets are parsed and stored while the data is stored. In the slave mode, only data is stored and data packets are forwarded without any other operations.

10. The queue storage management system in an RDMA network according to claim 1, characterized in that: The data checking and processing submodule receives the data transmitted by the network transmission module, and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, according to the instructions transmitted by the user through the network transmission module, the data required by the user is sent to the network transmission module; The data cache submodule includes a high-speed volatile storage for bidirectionally caching received and to-be-sent data; The logic control submodule comprehensively manages all queue work logics in the system and sends instructions to subsequent queue management modules: The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load; The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, to meet the performance requirements of different tasks; In the third step, after the queue resources are allocated in the second step, a global state synchronization mechanism is introduced to synchronize the load and usage status information of the one-to-one bound RDMA queues and NVMe queues in real time to ensure the information consistency of the one-to-one bound RDMA queues and NVMe queues; The fourth step is to regularly obtain queue status information from the subsequent queue management module, including the queue's operating efficiency, task completion status, and error status, and summarize and analyze it in combination with the queue resource information in the first step: when it is found that the processing delay of a queue continues to exceed the threshold or the task completion rate has a significant downward trend, give priority to expanding its queue resources, including increasing the queue depth and improving its queue priority; When the error rate increases, analyze the cause of the failure and reallocate resources to reduce the task allocation of the affected queue to improve system performance and reliability; In the fifth step, combining the data and information in the third and fourth steps, when an abnormal situation is detected, including queue overflow and hardware failure, the fault-tolerant mechanism is immediately triggered to release the tasks of the affected queue, and dynamically assign the tasks to other available queues or switch to the backup queue; when the abnormal situation cannot be recovered, the queue tasks are taken over by switching to the backup resources to ensure that the queue tasks are not interrupted and the system runs stably.

Images