A queue storage management system in RDMA network

Abstract

A queue storage management system under an RDMA network includes a data management module, a network transmission module, a data storage module, and a queue management module running on an FPGA platform. The data management module is bidirectionally interconnected with the network transmission module, the data storage module, and the queue management module. The data management module includes a main control processing submodule, a logic control submodule, a data caching submodule, and a data checking and processing submodule. The main control processing submodule is bidirectionally interconnected with the logic control submodule, the data caching submodule, and the data checking and processing submodule, controlling the overall working logic of the system, including data storage management and data transmission scheduling, working status control, dynamic allocation of system resources, and abnormal state handling and fault tolerance mechanism activation. It can be deployed more flexibly in various application scenarios. For the multi-level abstraction of the interface, the network layer queue processing and storage layer queue processing are hidden from the user level. Users do not need to care about how the queue is handled during data transmission, but only need to care about the correctness of instruction transmission, which further improves the system's ease of deployment and ease of use. It has the advantages of high efficiency, flexible deployment, and high availability.

Description

Technical Field

[0001] This invention relates to the field of data communication technology, and in particular to a queue storage management system under an RDMA network. Background Technology

[0002] Compared to local storage, network storage involves more complex data types, and the size and type of data packets can change at any time. This necessitates that network storage systems possess a degree of flexibility, capable of adapting their operation (such as queue type, quantity, and depth) to varying real-time demands. Furthermore, network storage requires not only high data throughput and low data transmission latency but also a small system size and minimal deployment complexity. Simultaneously, the system interface should be as simple as possible to improve usability and versatility. Therefore, existing solutions are not suitable for direct application in network data storage.

[0003] Patent application CN202410160584.2, publication number CN118101777A, entitled "An NVMe-over-RDMA Protocol Transmission Method and Electronic Device," discloses a network storage implementation method based on NVMe-oF, which uses the standard NVMe-oF protocol as an intermediate layer between the network protocol and the storage protocol. However, when using standard NVMe-oF as middleware, it is limited by the standard protocol and cannot achieve autonomous management of each queue pair (including RDMA and NVMe queue pairs), nor can it flexibly allocate queues according to actual application conditions. Therefore, this solution is not suitable for environments with variable and complex data.

[0004] Patent application CN202411171854.6, publication number CN118672953A, entitled "NVMeoF Response Command Transmission Method Based on RDMA Completion Event," discloses a method that tightly couples data transmission and NVMeoF response command transmission to improve system request processing efficiency. However, when using this tightly coupled method, resources used for data transmission cannot be released when there are no control commands being transmitted or when there is temporarily no data transmission. This can lead to high system resource consumption and potential memory management problems.

[0005] Patent application CN202410970696.4, publication number CN118509398A, entitled "A Virtual Machine Communication Method, System, Device, Equipment, and Storage Medium Based on RDMA," discloses a network storage solution that uses virtual machine queue switching as middleware. However, using virtual machine queue switching as middleware cannot avoid performance losses caused by virtual machine scheduling, leading to an overall performance degradation of the system. Furthermore, the virtual machine architecture is relatively large and has relatively higher requirements for the operating environment.

[0006] Patent application CN202311845696.3, publication number CN117880216A, entitled "A Data Transmission Method, Storage Medium, and Device Based on RDMA Communication," discloses a network storage solution using multiple nodes as middleware. This solution achieves data transmission and storage through queue switching between two nodes. While using multiple nodes as middleware involves data interaction between nodes to implement control logic, the increased number of nodes inevitably leads to increased system latency, larger system size, and greater deployment complexity, making it difficult to apply flexibly in various scenarios. Summary of the Invention

[0007] To overcome the problems existing in the prior art, the present invention aims to provide a queue storage management system under RDMA networks. For scenarios where network transmission data is relatively complex or the types and volumes of data to be stored / retrieved are inconsistent, this invention is deployed on an FPGA and adopts a variable queue design. The system can flexibly allocate RDMA queue pairs and NVMe queue pairs according to current data transmission / data storage requirements, ensuring that the system can perform at its full potential under most operating conditions without requiring user intervention to operate the queues. The system's internal virtual queues, including queue type, queue depth, and queue quantity, can abstract multiple receiving / completing queues that are not performance-sensitive into a single queue to adapt to different data transmission / data access requirements under different operating conditions. There are no strict limitations on the specific FPGA model or size, and compared to traditional storage servers, the power consumption is relatively low, allowing for more flexible deployment in various application scenarios. The multi-level abstraction of the interface hides network layer queue processing and storage layer queue processing from the user level. Users do not need to worry about how queues are handled during data transmission; they only need to focus on the correctness of command transmission, further improving the system's deployability and usability. It has the advantages of high efficiency, flexible deployment, and high availability.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A queue storage management system under an RDMA network includes a data management module, a network transmission module, a data storage module, and a queue management module running on an FPGA platform. The data management module is bidirectionally interconnected with the network transmission module, the data storage module, and the queue management module. The data management module includes a main control processing submodule, a data checking and processing submodule, a data caching submodule, and a logic control submodule. The main control processing submodule is bidirectionally interconnected with the logic control submodule, the data caching submodule, and the data checking and processing submodule, controlling the overall working logic of the system, including data storage management and data transmission scheduling, working status control, dynamic allocation of system resources, and abnormal state handling and fault tolerance mechanism activation.

[0010] The network transmission module includes a bidirectional interconnected network protocol stack submodule and a network connection control submodule;

[0011] The network protocol stack submodule includes the RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack follows the standard network layer connection method. It accepts network frames transmitted from the host computer and parses out the data that needs to be stored or assembles the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that devices in the network can connect and exchange data normally. It performs IP address to MAC address mapping through the ARP protocol or performs ping operation through the ICMP protocol to test the reachability of devices.

[0012] The network connection control submodule is used for the connection between this system and a remote server, switch, or user machine. After the QSFP interface detects the insertion of the network connection control submodule and the physical layer connection is established, the network connection control submodule autonomously initiates the connection establishment process, actively sending a connection request (REQ) packet. Then, after receiving a connection reply (REP) from the host computer agreeing to the connection, it actively sends a ready-to-use (RTU) packet, at which point the connection establishment is complete. After the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters from the host computer, including memory address, memory size, and access key, in preparation for data transmission.

[0013] The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule, and a storage array control submodule; wherein the storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule, respectively.

[0014] Among them, the non-volatile storage medium submodule connects various storage media, including solid-state drives, hard disk drives, and storage arrays, to the FPGA through the FMC interface expansion card;

[0015] Among them, the storage instruction control submodule packages user data into various storage layer protocols, thereby storing the data in the storage medium or reading and unpacking the data from the storage medium;

[0016] The storage array control submodule supports common storage array types, including RAID 0, RAID 1, and RAID 5, enabling efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read / write management. It can communicate directly with the storage array and maintain and manage data tables, thus eliminating dependence on storage array cards and making storage array configuration more flexible and comprehensive.

[0017] The queue management module includes a virtual queue pool submodule, a network queue management submodule, and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule, respectively.

[0018] The network queue management submodule controls the establishment, handshake, disconnection, and maintenance of RDMA queues. Specific control methods include: ① Using dynamic allocation and release technology, RDMA queues are allocated in the virtual queue pool based on traffic demand. When traffic increases and a queue approaches its load limit, resource allocation is triggered, increasing its queue depth or allocating a new queue from the virtual queue pool. When a queue is detected to have been unused for a long time or its occupancy rate is below a set threshold, the queue resources are automatically reclaimed to reduce memory and processor overhead. ② A dedicated register is set to record the current state of the RDMA completion queue (CQ), including queue length and number of completed tasks. The state machine continuously acquires real-time status data to determine if any tasks are incomplete or queues are interrupted. If a disconnection is detected, the state machine reinitializes the disconnected queue. ③ A binding relationship is established between the RDMA request queue (SQ) and the completion queue (CQ), achieving bidirectional tracking of task status through a shared index pointer. When the request queue (SQ) issues a new request, the task ID is recorded and the expected completion status of the completion queue (CQ) is updated synchronously. If a task exception occurs, the binding behavior is retried to ensure the reliability of the queue pair operation.

[0019] The storage queue management submodule controls NVMe queues, including the creation, destruction, and maintenance of Admin queues and IO queues. Specific control methods include: ① creating and destroying queues using the NVMe instruction set; ② using a queue resource mapping table to track queue usage status in real time and dynamically adjust resource allocation; ③ monitoring the update status of task completion flags in the Admin queue in real time to detect whether tasks are completed as expected; if a task is not completed or an error occurs, the relevant I / O queues are reinitialized to ensure efficient scheduling.

[0020] The virtual queue pool submodule controls the network queue management module and the storage queue management module. Specific implementation methods include: ① using a dynamic allocation algorithm based on the queue resource pool to abstract physical queues into logical queues, allowing for flexible adjustment of queue size and quantity; ② employing virtualization technology to isolate the queue interface from physical memory, providing a unified abstract interface for completing and receiving queues; ③ using a queue status monitoring module to monitor the running status of each module's queues in real time, dynamically allocating queue resources to ensure stable system operation under high load; and ④ integrating a load balancing algorithm to prioritize the allocation of low-latency queue resources in the queue pool.

[0021] The specific method for data storage management is as follows:

[0022] First, by interacting with the main control processing submodule and the data storage module, and by monitoring the storage queue status and data priority in real time, the queue depth and caching strategy are dynamically adjusted to coordinate data read and write operations: if the data read demand continuously exceeds the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is decreased; conversely, if the data write demand continuously exceeds the read demand, the read instruction queue depth is decreased, the read cache is decreased, and the write cache is increased.

[0023] Secondly, based on system status, including storage load and data priority, prioritize the processing of critical data and optimize queue scheduling, dynamically adjusting data storage strategies, including cache management and storage queue scheduling; within a time window, if the storage queue occupancy rate continuously exceeds the set Max threshold, prioritize the processing of high-priority data, increase cache space and queue depth; when the storage queue occupancy rate is lower than the set Min threshold, merge low-priority tasks and reduce their queue allocation, release cache and reduce queue depth to save storage load;

[0024] The time window is a fixed length of time preceding the current time.

[0025] The specific method for scheduling the data transmission is as follows:

[0026] The system utilizes a dynamic queue scheduling algorithm to select high-load queues, increase their capacity, and reduce the capacity of idle queues. This allows for real-time allocation of network resources, and the data transmission path is adjusted based on real-time network latency and bandwidth utilization. When network latency continues to rise or bandwidth utilization consistently exceeds the set maximum threshold, a low-latency, high-bandwidth transmission path is prioritized. Furthermore, the system incorporates the built-in flow control mechanism of the protocol stack, namely PFC flow control and ECN congestion management for standard Ethernet, to ensure efficient and reliable transmission.

[0027] The specific method for controlling the working state is as follows: When this system operates in a cascaded manner, there are two different working modes: master mode and slave mode. In master mode, it simultaneously parses user control packets, user data packets, user management packets, performs data storage, and manages the cascaded network, including monitoring the load status of the cascaded network, joining / leaving the cascaded network, and task allocation. In slave mode, it is only responsible for data storage and does not participate in cascaded network management. When operating in cascaded mode, the master control processing submodule is responsible for monitoring the working state of the entire system, thereby controlling the state switching between master and slave. The specific method is as follows: the system uses a state feedback mechanism to determine the operating status of master mode and slave mode. When the master mode reaches a certain threshold parameter of the set state feedback mechanism, the slave working mode is switched through state feedback. The slave mode is switched to master mode and takes over all functions of master mode, while the original master mode is switched back to slave mode and enters standby mode.

[0028] The specific method for dynamic allocation of system resources is as follows:

[0029] Priority scheduling algorithms are used to prioritize high-priority tasks or tasks initiated by high-priority users to coordinate resource competition among tasks of different priorities.

[0030] By using dynamic resource allocation technology, more queue resources are allocated to currently active users, while reducing resource allocation to idle users, thus coordinating user fairness in a multi-user environment.

[0031] By combining a polling strategy, fairness and efficiency are ensured in a multi-tasking environment. Specifically, for tasks with the same weight, resources are allocated sequentially according to the polling order to ensure fairness; at the same time, critical tasks are processed first under high load conditions.

[0032] The specific method for handling abnormal states and activating the fault tolerance mechanism is as follows: During system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted; in the host mode, user packets are parsed and stored while storing data, and in the slave mode, only data is stored and data packets are forwarded without performing other operations.

[0033] The data inspection and processing submodule receives data transmitted from the network transmission module and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, it sends the data required by the user to the network transmission module according to the instructions transmitted by the user through the network transmission module.

[0034] The data caching submodule includes a high-speed volatile storage that bidirectionally caches received data and data that is about to be sent.

[0035] The logic control submodule manages all queue operation logic in the system and issues instructions to subsequent queue management modules:

[0036] The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load.

[0037] The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, in order to meet the performance requirements of different tasks.

[0038] The third step, after the queue resource allocation in the second step, introduces a global state synchronization mechanism to synchronize the load and usage status information of the one-to-one bound RDMA queue and NVMe queue in real time, ensuring the information consistency between the one-to-one bound RDMA queue and NVMe queue.

[0039] The fourth step involves periodically retrieving queue status information from the subsequent queue management module, including queue operating efficiency, task completion status, and error status. This information is then combined with the queue resource information from the first step for summary and analysis. When a queue's processing delay consistently exceeds a threshold or its task completion rate shows a significant downward trend, its queue resources are expanded first, including increasing queue depth and raising its queue priority. When the error rate increases, the causes of the failures are analyzed and resources are reallocated to reduce the task allocation to affected queues, thereby improving system performance and reliability.

[0040] Fifth, combining the data and information from steps three and four, when an anomaly is detected, including queue overflow or hardware failure, the fault tolerance mechanism is immediately triggered to release the tasks in the affected queue and dynamically allocate the tasks to other available queues or switch them to a standby queue. If the anomaly cannot be recovered, the standby resource is switched to take over the queue tasks to ensure that the queue tasks are not interrupted and the system remains stable.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention is deployed on an FPGA. The main control processing submodule of the data management module is responsible for controlling the working logic of the entire system. In environments where there are limitations on the size and power consumption of the storage system, there are no strict limitations on the specific FPGA model and size. Compared with traditional storage servers, the power consumption is relatively low. Therefore, this system can be flexibly deployed in various application scenarios, such as storage rooms, ships, aerospace and other locations or fields. Due to the multi-level abstraction of the user interface, this system does not require any configuration during operation and can work completely unattended.

[0043] 2. The data storage module can normally establish queues and perform data access. When switching between different types of non-volatile storage media models and different storage array models, this invention can correctly identify and switch to the corresponding protocol stack. Given the rapid iteration of storage device bus versions, protocol versions, and interface types, this invention adopts a replaceable universal FMC board for storage media interfaces, allowing flexible selection of storage media speed, size, capacity, and interface type. It also supports multiple storage arrays such as RAID0 / RAID1 / RAID5, further improving system usability and reducing operating costs.

[0044] 3. For scenarios where network transmission data is relatively complex or the types / volumes of data to be stored / read are inconsistent, the queue management module of this invention adopts a variable queue design. The system can flexibly allocate RDMA queue pairs and NVMe queue pairs according to the current data transmission / data storage requirements, so as to ensure that the system can perform its full performance under most operating conditions without the need for users to actively intervene in the queue operation. In addition, the system's internal virtual queue pool (including but not limited to queue type, queue depth, queue number, etc.) can abstract multiple receiving queues / completion queues that are not sensitive to performance into a single queue to adapt to the data transmission / data access requirements under different operating conditions.

[0045] 4. The network transmission module of this invention uses a standard QSFP fiber optic interface to communicate with a remote host computer, which has fewer limitations on transmission distance and can achieve high-speed data transmission over long distances at low cost and high efficiency. At the same time, due to the variable queue mechanism and the existence of the queue pool, this system can be connected to the switch without changing any device protocol stack in the network, which is convenient for multiple users to access at the same time, effectively reducing system resource consumption and reducing transmission latency.

[0046] In summary, this invention is deployed on an FPGA, which does not have strict limitations on the specific FPGA model or size. Compared with traditional storage servers, it has relatively low power consumption and can be deployed more flexibly in various application scenarios. For the multi-level abstraction of the interface, the network layer queue processing and storage layer queue processing are hidden from the user level. Users do not need to care about how queues are handled during data transmission, but only need to care about the correctness of instruction transmission. This further improves the system's ease of deployment and ease of use, and has the advantages of high efficiency, flexible deployment, and high availability. Attached Figure Description

[0047] To more clearly and effectively illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a diagram showing the relationship between the system functional modules of the present invention.

[0049] Figure 2 This is a schematic diagram of the data management module of the present invention.

[0050] Figure 3 This is a schematic diagram of the variable queue design workflow of the present invention.

[0051] Figure 4 This is a schematic diagram of the virtual queue pool of the present invention.

[0052] Figure 5 This is a typical working principle diagram of the present invention.

[0053] Figure 6 This is a schematic diagram of the first typical deployment of the present invention.

[0054] Figure 7 This is a schematic diagram of the second typical deployment of the present invention.

[0055] Figure 8 This is a schematic diagram of the third typical deployment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0057] Reference Figure 1 , Figure 2 A queue storage management system under an RDMA network includes a data management module, a network transmission module, a data storage module, and a queue management module running on an FPGA platform. The data management module is bidirectionally interconnected with the network transmission module, the data storage module, and the queue management module. The data management module includes a main control processing submodule, a data checking and processing submodule, a data caching submodule, and a logic control submodule. The main control processing submodule is bidirectionally interconnected with the logic control submodule, the data caching submodule, and the data checking and processing submodule, controlling the overall working logic of the system, including data storage management and data transmission scheduling, working status control, dynamic allocation of system resources, and abnormal state handling and fault tolerance mechanism activation.

[0058] The network transmission module includes a bidirectional interconnected network protocol stack submodule and a network connection control submodule;

[0059] The network protocol stack submodule includes the RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack follows the standard network layer connection method. It accepts network frames transmitted from the host computer and parses out the data that needs to be stored or assembles the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that devices in the network can connect and exchange data normally. It performs IP address to MAC address mapping through the ARP protocol or performs ping operation through the ICMP protocol to test the reachability of devices.

[0060] The network connection control submodule is used for the connection between this system and a remote server, switch, or user machine. After the QSFP interface detects the insertion of the network connection control submodule and the physical layer connection is established, the network connection control submodule autonomously initiates the connection establishment process, actively sending a connection request (REQ) packet. Then, after receiving a connection reply (REP) from the host computer agreeing to the connection, it actively sends a ready-to-use (RTU) packet, at which point the connection establishment is complete. After the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters from the host computer, including memory address, memory size, and access key, in preparation for data transmission.

[0061] The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule, and a storage array control submodule; wherein the storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule, respectively.

[0062] Among them, the non-volatile storage medium submodule connects various storage media, including solid-state drives, hard disk drives, and storage arrays, to the FPGA through the FMC interface expansion card;

[0063] Among them, the storage instruction control submodule packages user data into various storage layer protocols, thereby storing the data in the storage medium or reading and unpacking the data from the storage medium;

[0064] The storage array control submodule supports common storage array types, including RAID 0, RAID 1, and RAID 5, enabling efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read / write management. It can communicate directly with the storage array and maintain and manage data tables, thus eliminating dependence on storage array cards and making storage array configuration more flexible and comprehensive.

[0065] The queue management module includes a virtual queue pool submodule, a network queue management submodule, and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule, respectively.

[0066] The network queue management submodule controls the establishment, handshake, disconnection, and maintenance of RDMA queues. Specific control methods include: ① Using dynamic allocation and release technology, RDMA queues are allocated in the virtual queue pool based on traffic demand. When increased traffic causes a queue to approach its load limit, resource allocation is triggered, increasing its queue depth or allocating a new queue from the virtual queue pool. When a queue is detected to have been unused for an extended period or its occupancy rate is below a set threshold (e.g., 20%), the queue resources are automatically reclaimed to reduce memory and processor overhead. ② Setting dedicated registers to record the current status of the RDMA completion queue (CQ) includes... Queue length and number of completed tasks; the state machine continuously acquires real-time status data to determine if any tasks are incomplete or the queue is interrupted. If a break in the chain is detected (e.g., the completion flag is not updated), the state machine reinitializes the broken-chain queue; ③ A binding relationship is established between the RDMA request queue SQ and the completion queue CQ, and bidirectional tracking of task status is achieved through a shared index pointer; when the request queue SQ issues a new request, the task ID is recorded and the expected completion status of the completion queue CQ is updated synchronously; once a task abnormality occurs (e.g., the completion status is not updated in time or an abnormal completion status occurs), the binding behavior is retried to ensure the reliability of the queue for work;

[0067] The storage queue management submodule controls NVMe queues, including the creation, destruction, and maintenance of Admin queues and IO queues. Specific control methods include: ① creating and destroying queues using the NVMe instruction set; ② using a queue resource mapping table to track queue usage status in real time and dynamically adjust resource allocation; ③ monitoring the update status of task completion flags in the Admin queue in real time to detect whether tasks are completed as expected; if a task is not completed or an error occurs, the relevant I / O queues are reinitialized to ensure efficient scheduling.

[0068] The virtual queue pool submodule controls the network queue management module and the storage queue management module. Specific implementation methods include: ① using a dynamic allocation algorithm based on the queue resource pool to abstract physical queues into logical queues, allowing for flexible adjustment of queue size and quantity; ② employing virtualization technology to isolate the queue interface from physical memory, providing a unified abstract interface for completing and receiving queues; ③ using a queue status monitoring module to monitor the running status of each module's queues in real time, dynamically allocating queue resources to ensure stable system operation under high load; and ④ integrating a load balancing algorithm to prioritize the allocation of low-latency queue resources in the queue pool.

[0069] The specific method for data storage management is as follows:

[0070] First, by interacting with the main control processing submodule and the data storage module, and by monitoring the storage queue status and data priority in real time, the queue depth and caching strategy are dynamically adjusted to coordinate data read and write operations: if the data read demand continuously exceeds the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is decreased; conversely, if the data write demand continuously exceeds the read demand, the read instruction queue depth is decreased, the read cache is decreased, and the write cache is increased.

[0071] Secondly, based on system status, including storage load and data priority, critical data is processed first and queue scheduling is optimized. Data storage strategies, including cache management and storage queue scheduling, are dynamically adjusted. Within a time window, if the storage queue occupancy rate continuously exceeds the set Max threshold, high-priority data is processed first, and cache space and queue depth are increased. When the storage queue occupancy rate is lower than the set Min threshold, low-priority tasks are merged and their queue allocation is reduced, cache is released and queue depth is reduced to save storage load.

[0072] The time window is a fixed length of time preceding the current time.

[0073] The specific method for scheduling the data transmission is as follows:

[0074] The system utilizes a dynamic queue scheduling algorithm to select high-load queues, increase their capacity, and reduce the capacity of idle queues. This allows for real-time allocation of network resources, and the data transmission path is adjusted based on real-time network latency and bandwidth utilization. When network latency continues to rise or bandwidth utilization consistently exceeds the set maximum threshold, a low-latency, high-bandwidth transmission path is prioritized. Furthermore, the system incorporates the built-in flow control mechanism of the protocol stack, namely PFC flow control and ECN congestion management for standard Ethernet, to ensure efficient and reliable transmission.

[0075] The specific method for controlling the working state is as follows: When this system operates in a cascaded manner, there are two different working modes: master mode and slave mode. In master mode, it simultaneously parses user control packets, user data packets, user management packets, performs data storage, and manages the cascaded network, including monitoring the load status of the cascaded network, joining / leaving the cascaded network, and task allocation. In slave mode, it is only responsible for data storage and does not participate in cascaded network management. When operating in cascaded mode, the master control processing submodule is responsible for monitoring the working state of the entire system to control the state switching between master and slave. The specific method is as follows: using a status feedback mechanism (such as heartbeat detection and load monitoring) to determine the operating status of master mode and slave mode, when the load in master mode is continuously higher than 80% or the heartbeat detection indicates excessive delay, the slave working mode is switched through status feedback, switching from slave mode to master mode and taking over all functions of master mode, while the original master mode is switched back to slave mode and enters standby mode.

[0076] The specific method for dynamic allocation of system resources is as follows:

[0077] Priority scheduling algorithms are used to prioritize high-priority tasks or tasks initiated by high-priority users to coordinate resource competition among tasks of different priorities.

[0078] By using dynamic resource allocation technology, more queue resources are allocated to currently active users, while reducing resource allocation to idle users, thus coordinating user fairness in a multi-user environment.

[0079] By combining a polling strategy, fairness and efficiency are ensured in a multi-tasking environment. Specifically, for tasks with the same weight, resources are allocated sequentially according to the polling order to ensure fairness; at the same time, critical tasks are prioritized under high load conditions to ensure efficiency.

[0080] The specific method for handling abnormal states and activating the fault tolerance mechanism is as follows: During system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted; in host mode, user packets are parsed and stored while storing data, and in slave mode, only data is stored and data packets are forwarded without performing other operations; ensuring system stability.

[0081] The data inspection and processing submodule receives data transmitted from the network transmission module and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, it sends the data required by the user to the network transmission module according to the instructions transmitted by the user through the network transmission module.

[0082] The data caching submodule includes a high-speed volatile storage that bidirectionally caches received data and data that is about to be sent.

[0083] The logic control submodule manages all queue operation logic in the system and issues instructions to subsequent queue management modules:

[0084] The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load.

[0085] The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, in order to meet the performance requirements of different tasks.

[0086] The third step, after the queue resource allocation in the second step, introduces a global state synchronization mechanism to synchronize the load and usage status information of the one-to-one bound RDMA queue and NVMe queue in real time, ensuring the information consistency between the one-to-one bound RDMA queue and NVMe queue.

[0087] The fourth step involves periodically obtaining queue status information from the subsequent queue management module, including queue operating efficiency, task completion status, and error status. This information is then combined with the queue resource information from the first step for summary and analysis. When a queue's processing delay consistently exceeds a threshold or its task completion rate shows a significant downward trend, its queue resources are expanded first, including increasing queue depth and raising its queue priority. When the error rate increases, the causes of the failures are analyzed and resources are reallocated to reduce the task allocation to affected queues, thereby improving system performance and reliability.

[0088] Fifth, combining the data and information from steps three and four, when an anomaly is detected, including queue overflow or hardware failure, the fault tolerance mechanism is immediately triggered to release the tasks in the affected queue and dynamically allocate the tasks to other available queues or switch them to a standby queue. If the anomaly cannot be recovered, the standby resource is switched to take over the queue tasks to ensure that the queue tasks are not interrupted and the system remains stable.

[0089] Reference Figure 3The logic control submodule continuously analyzes the data transmission requests received by the system. When it detects a continuous large data packet, it reduces the number of queues and increases the queue depth. This helps improve the system's efficiency and throughput when processing longer data packets, while minimizing cache and resource consumption. When it detects a scattered small data packet, it increases both the number and depth of queues. Although continuous scattered small data packets are short, they cause significant IO consumption. Increasing the number of queues helps reduce the pressure on a single queue processing engine, while increasing the queue depth minimizes the possibility of data packet congestion within the system. Simultaneously, when storing or retrieving data, it tries to merge scattered data packets into larger data transmission requests to avoid IOPS performance bottlenecks in the storage system. Finally, it interacts with the storage module to store / retrieve data. After the transmission task is completed, the queue is restored to its default state, awaiting the next transmission task.

[0090] Reference Figure 4 The queues described in this invention include: RDMA SQ, RDMA RQ, RDMACQ, NVMe Admin SQ / CQ, and NVMe I / O SQ / CQ.

[0091] In the RDMA part, the actual queue is SQ, while RQ and CQ are represented by a virtual queue pool. SQ is the receive queue, which is strongly related to the host computer. Each queue must have an independent physical control to send data packets to the host computer. RQ is the receive queue. In the FPGA, it is only necessary to store the correspondence between the sequence number, data, and queue number of each data packet. It is not actually necessary to store the entire content of each data packet. Similarly, CQ is the completion queue, which records each completed network transmission command of the system. It is also not strongly related to the host computer. Therefore, using a virtual queue pool to implement RQ and CQ can minimize the waste of system resources, improve the system's operating efficiency, and not affect its communication with any other RDMA devices.

[0092] Furthermore, the actual queues in the MVNe portion are SQs and do not distinguish between Admin queues and I / O queues. CQs are also implemented by a virtual queue pool. Similarly, CQs are not strongly correlated with their underlying storage media, and their indications signify whether an instruction has been completed. Since Admin instructions are not frequently used during normal system operation, storing a separate pair of queues for Admin instructions would inevitably waste resources. Therefore, if the system needs to transmit an Admin instruction, the virtual queue pool will temporarily convert an I / OQ into an Admin SQ and allocate a small space for the Admin CQ. When the instruction is completed, the queue is immediately restored, and the resources are released. This mechanism minimizes resource waste while ensuring the system can communicate with standard NVMe devices.

[0093] Reference Figure 5 The typical workflow of this system is as follows: Upon receiving a data transmission request, the RDMA protocol stack first unpacks the data, then stores the data in the data cache module. Simultaneously, the request is transferred to the logic processing module for analysis to determine whether dynamic processing of the queue is required. This part of the process can be referred to... Figure 3 The logic processing module analyzes the request and returns a network response packet, along with the storage array type. After request analysis, it generates read / write instructions for the corresponding storage array and reads data from the cache. Once the data storage / read process is complete, a storage response packet / read reply packet is returned to the corresponding queue on the host computer. This completes one transmission cycle. If there are no pending data packets, the system remains silent; otherwise, the process repeats. It is important to emphasize that in this system, all processes are CPU-free, meaning that each step in this process runs continuously, rather than sequentially. Figure 4 The workflow shown is for ease of understanding only.

[0094] Reference Figure 6 , 7 8. This system has a relatively flexible deployment method and can be deployed in multiple ways:

[0095] like Figure 6 As shown: The first typical deployment method of this system is the simplest deployment method, which uses only one network port to connect to the host computer and one FMC expansion interface to connect to non-volatile storage media such as SSD, HDD, etc.

[0096] like Figure 7As shown: The second typical deployment method of this system uses both QSFP network ports, which can be used to connect to the host computer or switch. At the same time, both FMC expansion interfaces are also enabled, which can be used to connect to storage arrays or high-speed SSDs as high-speed data buffers. There are no restrictions on which devices should be connected to each network port or each FMC interface. In other words, each network port and each FMC interface has the same function.

[0097] like Figure 8 As shown: This is the third typical deployment method for this system. Both QSFP interfaces are enabled, but one is used to cascade multiple systems. The connection method of the FMC expansion interface remains unchanged. The maximum number of systems that can be connected in this way is 255, and this connection method can maximize system performance, whether in terms of speed, capacity, or system robustness.

[0098] The technical effects of the present invention will be described in detail below with reference to simulation experiments.

[0099] Based on this invention, a simulated host computer was built, which can simulate the online and offline states of a host computer and different network interface speeds. A simple model of a virtual non-volatile storage medium was also built to simulate data storage.

[0100] 1. Network Transmission Verification: When a simulated host computer is detected to be accessing the network, this invention can establish a normal connection with the simulated host computer and establish a queue for communication. When host computers with different network interface rates are accessed, this invention automatically switches to the corresponding interface rate to adapt to the host computer. At a transmission rate of 100Gbps, this invention operates stably. According to simulation results, the actual one-way throughput of the system is >300Gbps, and the actual forwarding packet capacity is >150Mpps (pps: Packets per second), which can meet the performance requirements of dual-path 100G RDMA.

[0101] 2. Data Storage Verification: When a storage medium is detected, the present invention can establish a queue and perform data access normally. When switching between different types of non-volatile storage media models and different storage array models, the present invention can also correctly identify and switch to the corresponding protocol stack. Since the performance of different non-volatile storage media and storage arrays varies under actual application conditions, the required storage rate cannot be estimated; therefore, throughput testing is performed. Simulation results show that the maximum unidirectional throughput related to the data storage module reaches 150Gbps.

[0102] 3. Optimization Effects of Queue Management and Scheduling: In this section, the performance of the logic control module and queue management module under different loads is verified by simulating different types of task requests (such as storage read / write requests and network data transmission requests). 1) Low load conditions: The system automatically reduces the number and depth of queues, thereby reducing system power consumption and improving operational stability. 2) Continuous high load conditions: The system automatically increases the number and depth of queues, and a virtual queue pool intervenes, abstracting virtual completion queues (storage) and virtual receiving queues (network). In addition, the dynamic priority mechanism for queues is activated. To verify performance, a simulation was also performed with the virtual queue pool disabled. According to the simulation results, compared with the virtual queue pool not being enabled, the system's storage (RAM) resources are reduced by 34%, and compared with the dynamic priority mechanism for queues not being enabled, the average system transmission latency is reduced by approximately 10µs.

[0103] 4. Coordination Capability of the Main Control Processing Submodule: In this section, a simulation is conducted using two cascaded devices, with device A operating in master mode and device B operating in slave mode. When the load on device A is increased to its limit, the simulation results show that the master and slave roles have switched; A operates in slave mode, while device B operates in master mode, taking over the work from device A. Further increasing the load on device A results in it entering standby mode, only forwarding data packets without processing them, while the relatively less loaded device B handles the packet processing. Furthermore, in a simulated multi-user environment, this invention correctly adjusts the queue according to task priority, ensuring fairness and efficiency and avoiding resource contention.

[0104] From the above, it can be seen that: 1. This invention can operate normally when equipped with different types of non-volatile storage media. 2. When the network interface speed of the host computer is different, this invention can correctly identify and adjust its own interface speed. 3. Under high load conditions, the dynamic queue + virtual queue pool management method of this invention can effectively reduce system resource consumption and reduce transmission latency. 4. Under extreme conditions, this invention can automatically switch working modes and can work independently without supervision. 5. The virtual queue pool method can further improve user-friendliness and also enable this invention to interface with any standard protocol device, reducing the deployment difficulty of this invention.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A queue storage management system under an RDMA network, characterized in that, It includes a data management module, a network transmission module, a data storage module, and a queue management module running on an FPGA platform; the data management module is bidirectionally interconnected with the network transmission module, the data storage module, and the queue management module; the data management module includes a main control processing submodule, a data checking and processing submodule, a data caching submodule, and a logic control submodule. The main control processing submodule is bidirectionally interconnected with the logic control submodule, data caching submodule, and data checking and processing submodule to control the overall working logic of the system, including data storage management and data transmission scheduling, working status control, dynamic allocation of system resources, and abnormal state handling and fault tolerance mechanism activation; the queue management module includes a virtual queue pool submodule, a network queue management submodule, and a storage queue management submodule. The virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule. The network queue management submodule controls the establishment, handshake, disconnection, and maintenance of RDMA queues. Specific control methods include: ① using dynamic allocation and release technology to allocate RDMA queues in the virtual queue pool based on traffic demand; ② setting dedicated registers to record the current status of the RDMA completion queue (CQ), including queue length and number of completed tasks; ③ establishing a binding relationship between the RDMA request queue (SQ) and the completion queue (CQ), achieving bidirectional tracking of task status through shared index pointers; when the request queue (SQ) issues a new request, it records the task ID and synchronously updates the expected completion status of the completion queue (CQ). The storage queue management submodule controls NVMe queues, including the establishment, destruction, and maintenance of Admin queues and IO queues. Specific control methods include: ① using NVM... The e-instruction set establishes and destroys queues; ② It uses a queue resource mapping table to track the queue usage status in real time and dynamically adjust resource allocation; ③ It monitors the update status of the task completion flag in the Admin queue in real time to detect whether the task is completed as expected; The virtual queue pool sub-module controls the network queue management module and the storage queue management module. The specific implementation methods include: ① Based on the dynamic allocation algorithm of the queue resource pool, the physical queue is abstracted into a logical queue to flexibly adjust the queue size and number; ② It adopts virtualization technology to isolate the queue interface from physical memory and provide a unified abstract interface for the completion queue and the receiving queue; ③ Through the queue status monitoring module, the running status of each module's queue is detected in real time and queue resources are dynamically allocated; ④ It integrates a load balancing algorithm to prioritize the allocation of low-latency queue resources in the queue pool.

2. The queue storage management system under an RDMA network according to claim 1, characterized in that, The network transmission module includes a bidirectional interconnected network protocol stack submodule and a network connection control submodule; The network protocol stack submodule includes the RoCE v2 network protocol stack, UDP / IP protocol stack, ARP protocol stack, and ICMP protocol stack. The connection between each protocol stack follows the standard network layer connection method. It accepts network frames transmitted from the host computer and parses out the data that needs to be stored or assembles the data into network frames and sends them to the host computer. It is also responsible for the discovery of network devices and their reachability testing, ensuring that devices in the network can connect and exchange data normally. It performs IP address to MAC address mapping through the ARP protocol or performs ping operation through the ICMP protocol to test the reachability of devices. The network connection control submodule is used for the connection between this system and a remote server, switch, or user machine. After the QSFP interface detects the insertion of the network connection control submodule and the physical layer connection is established, the network connection control submodule autonomously initiates the connection establishment process, actively sending a connection request (REQ) packet. Then, after receiving a connection reply (REP) from the host computer agreeing to the connection, it actively sends a ready-to-use (RTU) packet, at which point the connection establishment is complete. After the connection is established, the network connection control submodule actively requests to obtain the data transmission parameters from the host computer, including memory address, memory size, and access key, in preparation for data transmission.

3. The queue storage management system under an RDMA network according to claim 1, characterized in that, The data storage module includes a non-volatile storage medium submodule, a storage instruction control submodule, and a storage array control submodule; The storage instruction control submodule is bidirectionally interconnected with the non-volatile storage medium submodule and the storage array control submodule, respectively. Among them, the non-volatile storage medium submodule connects various storage media, including solid-state drives, hard disk drives, and storage arrays, to the FPGA through the FMC interface expansion card; Among them, the storage instruction control submodule packages user data into various storage layer protocols, thereby storing the data in the storage medium or reading and unpacking the data from the storage medium; The storage array control submodule supports common storage array types, including RAID 0, RAID 1, and RAID 5, through built-in support. This enables efficient compatibility and control of various storage arrays, including array creation, deletion, expansion, and read / write management. It can communicate directly with the storage array and maintain and manage data tables, thus eliminating dependence on storage array cards and making storage array configuration more flexible and comprehensive.

4. A queue storage management system under an RDMA network according to claim 1, characterized in that, The queue management module includes a virtual queue pool submodule, a network queue management submodule, and a storage queue management submodule; wherein the virtual queue pool submodule is bidirectionally interconnected with the network queue management submodule and the storage queue management submodule, respectively. The network queue management submodule controls the establishment, handshake, disconnection, and maintenance of RDMA queues. Specific control methods include: ① Using dynamic allocation and release technology, RDMA queues are allocated in the virtual queue pool based on traffic demand. When traffic increases and a queue approaches its load limit, resource allocation is triggered, increasing its queue depth or allocating a new queue from the virtual queue pool. When a queue is detected to have been unused for a long time or its occupancy rate is below a set threshold, the queue resources are automatically reclaimed to reduce memory and processor overhead. ② A dedicated register is set to record the current state of the RDMA completion queue (CQ), including queue length and number of completed tasks. The state machine continuously acquires real-time status data to determine if any tasks are incomplete or queues are interrupted. If a disconnection is detected, the state machine reinitializes the disconnected queue. ③ A binding relationship is established between the RDMA request queue (SQ) and the completion queue (CQ), achieving bidirectional tracking of task status through a shared index pointer. When the request queue (SQ) issues a new request, the task ID is recorded and the expected completion status of the completion queue (CQ) is updated synchronously. If a task exception occurs, the binding behavior is retried to ensure the reliability of the queue pair operation. The storage queue management submodule controls NVMe queues, including the creation, destruction, and maintenance of Admin queues and IO queues. Specific control methods include: ① creating and destroying queues using the NVMe instruction set; ② using a queue resource mapping table to track queue usage status in real time and dynamically adjust resource allocation; ③ monitoring the update status of task completion flags in the Admin queue in real time to detect whether tasks are completed as expected; if a task is not completed or an error occurs, the relevant I / O queues are reinitialized to ensure efficient scheduling. The virtual queue pool submodule controls the network queue management module and the storage queue management module. Specific implementation methods include: ① using a dynamic allocation algorithm based on the queue resource pool to abstract physical queues into logical queues, allowing for flexible adjustment of queue size and quantity; ② employing virtualization technology to isolate the queue interface from physical memory, providing a unified abstract interface for completing and receiving queues; ③ using a queue status monitoring module to monitor the running status of each module's queues in real time, dynamically allocating queue resources to ensure stable system operation under high load; and ④ integrating a load balancing algorithm to prioritize the allocation of low-latency queue resources in the queue pool.

5. A queue storage management system under an RDMA network according to claim 1, characterized in that, The specific method for data storage management is as follows: First, by interacting with the main control processing submodule and the data storage module, and by monitoring the storage queue status and data priority in real time, the queue depth and caching strategy are dynamically adjusted to coordinate data read and write operations: if the data read demand continuously exceeds the write demand within a time window, the read instruction queue depth is increased, the read cache is increased, and the write cache is decreased; conversely, if the data write demand continuously exceeds the read demand, the read instruction queue depth is decreased, the read cache is decreased, and the write cache is increased. Secondly, based on system status, including storage load and data priority, prioritize the processing of critical data and optimize queue scheduling, dynamically adjusting data storage strategies, including cache management and storage queue scheduling; within a time window, if the storage queue occupancy rate continuously exceeds the set Max threshold, prioritize the processing of high-priority data, increase cache space and queue depth; when the storage queue occupancy rate is lower than the set Min threshold, merge low-priority tasks and reduce their queue allocation, release cache and reduce queue depth to save storage load; The time window is a fixed length of time preceding the current time.

6. A queue storage management system under an RDMA network according to claim 1, characterized in that, The specific method for scheduling the data transmission is as follows: The system utilizes a dynamic queue scheduling algorithm to select high-load queues, increase their capacity, and reduce the capacity of idle queues. It allocates network resources in real time and adjusts data transmission paths based on real-time network latency and bandwidth utilization. When network latency continues to rise or bandwidth utilization continues to exceed the set maximum threshold, it prioritizes low-latency, high-bandwidth transmission paths. Furthermore, it incorporates the built-in flow control mechanism of the protocol stack to ensure efficient and reliable transmission.

7. A queue storage management system under an RDMA network according to claim 1, characterized in that, The specific method for controlling the working status is as follows: When this system works in a cascaded manner, there are two different working modes: including host mode and slave mode; In host mode: it simultaneously parses user control packets, user data packets, user management packets, data storage, and manages the cascaded network, including monitoring the load status of the cascaded network, joining / leaving the cascaded network, and task allocation; Slave mode: Only responsible for data storage and does not participate in cascaded network management; When working in cascaded mode, the master control processing submodule is responsible for monitoring the working status of the entire system, thereby controlling the state switching between the master and slave: The specific method is to use the status feedback mechanism to judge the operation status of master mode and slave mode. When the master mode operation reaches a certain parameter threshold of the set status feedback mechanism, the slave working mode is switched through status feedback, switching the slave mode to master mode and taking over all functions of master mode, while switching the original master mode to slave mode and entering standby state.

8. A queue storage management system under an RDMA network according to claim 1, characterized in that, The specific method for dynamic allocation of system resources is as follows: Priority scheduling algorithms are used to prioritize high-priority tasks or tasks initiated by high-priority users to coordinate resource competition among tasks of different priorities. By using dynamic resource allocation technology, more queue resources are allocated to currently active users, while reducing resource allocation to idle users, thus coordinating user fairness in a multi-user environment. By combining a polling strategy, fairness and efficiency are ensured in a multi-tasking environment. Specifically, for tasks with the same weight, resources are allocated sequentially according to the polling order to ensure fairness; at the same time, critical tasks are processed first under high load conditions.

9. A queue storage management system under an RDMA network according to claim 1, characterized in that, The specific method for handling abnormal states and activating the fault tolerance mechanism is as follows: During system operation, the main control processing submodule identifies potential faults and abnormal states, activates the fault tolerance mechanism, triggers path switching behavior, and switches the affected tasks to the backup path or backup storage to ensure that the tasks are not interrupted. In master mode, user packets are parsed and stored while storing data. In slave mode, only data is stored and data packets are forwarded; no other operations are performed.

10. A queue storage management system under an RDMA network according to claim 1, characterized in that, The data inspection and processing submodule receives data transmitted from the network transmission module and verifies the transmitted data to ensure the integrity and correctness of the received data; at the same time, it sends the data required by the user to the network transmission module according to the instructions transmitted by the user through the network transmission module. The data caching submodule includes a high-speed volatile storage that bidirectionally caches received data and data that is about to be sent. The logic control submodule manages all queue operation logic in the system and issues instructions to subsequent queue management modules: The first step is to monitor the queue resources in the system in real time, including usage status, queue length, and queue load. The second step is to allocate more queue resources to queues with high usage demand based on queue resource information, including usage status, queue length, and queue load, according to different task priorities and queue priorities, in order to meet the performance requirements of different tasks. The third step, after the queue resource allocation in the second step, introduces a global state synchronization mechanism to synchronize the load and usage status information of the one-to-one bound RDMA queue and NVMe queue in real time, ensuring the information consistency between the one-to-one bound RDMA queue and NVMe queue. The fourth step is to periodically obtain queue status information from the subsequent queue management module, including queue operating efficiency, task completion status, and error status. At the same time, the queue resource information from the first step is summarized and analyzed: when it is found that the processing delay of a certain queue continues to exceed the threshold or the task completion rate declines significantly, its queue resources are expanded first, including increasing the queue depth and increasing its queue priority. When the error rate increases, the cause of the failure is analyzed and resources are reallocated to reduce the task allocation to the affected queues in order to improve system performance and reliability. Fifth, combining the data and information from steps three and four, when an anomaly is detected, including queue overflow or hardware failure, the fault tolerance mechanism is immediately triggered to release the tasks in the affected queue and dynamically allocate the tasks to other available queues or switch them to a standby queue. If the anomaly cannot be recovered, the standby resource is switched to take over the queue tasks to ensure that the queue tasks are not interrupted and the system remains stable.

Images