Storage control method and storage device

Abstract

The invention discloses a storage control method and storage device.The method comprises the steps that virtual disks with the same namespace identifiers and namespaces of the virtual disks are created on multiple storage control cards, so that a host recognizes the multiple same namespace identifiers as a target object, and the host distributes IO requests issued by the application layer to the storage control cards and submits the IO requests to respective storage device hardware queues in parallel for processing, so that cross-card parallel IO access is realized, and the throughput and the concurrency performance are improved. Another implementation mode comprises the steps that one of the multiple storage control cards serves as a master card, the other storage control cards serve as slave cards managed by the master card, the master card manages control channels of the slave cards in a PCIe peer-to-peer mapping mode, and only a single card view is presented to the host. The main card can forward the doorbell event triggered by the host to the corresponding target storage control card, so that the target storage control card executes actual IO processing, and multi-card cooperative control and load balancing at a hardware level are realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and in particular, to a storage control method and a storage device. BACKGROUND

[0002] In modern data centers and server systems, Peripheral Component Interconnect Express (PCIe) devices, as one of the core components, are widely used to provide high-speed storage, computing acceleration and network functions. Different types of PCIe devices can provide a variety of service functions for the host, such as Redundant Array of Independent / Inexpensive Disks (RAID) cards, also known as storage control cards, which can implement redundancy and data protection functions for storage devices, Solid State Drives (SSDs) based on the Non-Volatile Memory Express (NVMe) protocol can provide high-performance data access capabilities, and hardware acceleration cards are used for Erasure Coding (EC), encryption and decryption, and other computing offload tasks. These PCIe devices are usually accessed to the host in the form of independent functional units, and the host identifies and calls them through the operating system driver, so as to achieve the purpose of expanding performance, improving system function and efficiency.

[0003] In the prior art, PCIe devices usually work in single card mode, and RAID card vendors provide operating system specific drivers or use operating system standard drivers. The operating system operates and manages the RAID card hardware through these drivers, including adding hard disks to the RAID card, creating virtual disks, monitoring array status, and other operations. The application layer directly accesses the virtual disk, and the RAID card is responsible for the underlying data striping, mirroring or checking operations. This type of way can meet the basic storage management needs in a single card scenario, but when multiple PCIe devices (such as multiple RAID cards or acceleration cards) need to work together in the system, the PCIe devices can only rely on the host side for unified scheduling and coordination, which leads to low overall efficiency and increases the operation and management burden of the host side. SUMMARY

[0004] Based on the above problems, the embodiments of the present application provide a storage control method and a storage device, which aims to realize load balancing and fault isolation at the level of storage control cards by distributing the input / output (IO) requests issued by a host in parallel among multiple storage control cards; secondly, each storage control distributes the received IO requests to the hardware queues of multiple storage devices in a further striping manner, realizes parallel access to the underlying storage medium, and further improves the overall throughput of the storage device.

[0005] In a first aspect, the embodiments of the present application provide a storage control method applied to a storage device connected to a host through a PCIe interface and including at least two storage control cards, which comprises:

[0006] Creating a virtual disk and its namespace with the same namespace identifier on multiple storage control cards, so that the host identifies multiple same namespace identifiers as one target object that can be accessed by an application layer;

[0007] Receiving read / write IO requests issued by the application layer to the target object, distributing the IO requests to each storage control card by the host, and submitting them to the hardware queues of the corresponding storage devices of each storage control card for parallel IO processing.

[0008] In a second aspect, the embodiments of the present application also provide a storage control method applied to a storage device connected to a host through a PCIe interface and including at least two storage control cards, which comprises:

[0009] Taking at least one of the multiple storage control cards as a master card, and taking other storage control cards except the master card as slave cards managed by the master card, the master card is visible to the host, and the slave cards are invisible to the host, so that the host identifies the master card as one target object that can be accessed by an application layer;

[0010] Receiving IO requests issued by the application layer to the target object, writing the IO requests into a target IO queue by the host to trigger a doorbell event; after the master card detects that the doorbell event is triggered, distributing the doorbell event to the corresponding target storage control card according to the target IO queue corresponding to the doorbell event, so that the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and performs IO processing.

[0011] In a third aspect, the embodiments of the present application also provide a storage control device applied to a storage device connected to a host through a PCIe interface and including at least two storage control cards, which comprises:

[0012] a virtual disk creation unit, configured to create a virtual disk and a namespace of the virtual disk with a same namespace identifier on a plurality of the storage control cards, so that the host identifies a plurality of the same namespace identifiers as one target object which can be accessed by an application layer;

[0013] an IO request processing unit, configured to receive a read-write IO request issued by the application layer to the target object, and distribute the IO request to each of the storage control cards by the host and submit the IO request to a corresponding storage device hardware queue of each of the storage control cards for parallel IO processing.

[0014] In a fourth aspect, an embodiment of the present application further provides a storage control device applied to storage devices of at least two storage control cards connected with a host through a PCIe interface, and the device comprises:

[0015] a master-slave card configuration unit, configured to take at least one of the plurality of storage control cards as a master card, and take other storage control cards except the master card as slave cards managed by the master card, the master card is visible to the host, and the slave cards are invisible to the host, so that the host identifies the master card as one target object which can be accessed by an application layer;

[0016] a queue management unit, configured to create a management queue for each slave card managed by the master card, so that the master card manages a control path and a data path of the slave card through a PCIe peer mapping mode;

[0017] an IO request processing unit, configured to receive an IO request issued by the application layer to the target object, and write the IO request into the target IO queue to trigger a doorbell event after the host; after the master card detects that the doorbell event is triggered, the master card distributes the doorbell event to a corresponding target storage control card according to the target IO queue corresponding to the doorbell event, so that the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and performs IO processing.

[0018] In a fifth aspect, an embodiment of the present application further provides a storage device, and the storage device stores instructions, and the instructions are executed by a processor to implement the storage control method described in the first aspect or any specific implementation manner of the first aspect or the second aspect or any specific implementation manner of the second aspect.

[0019] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, performs the storage control method described in the first aspect of the embodiments of the present application or any of the specific implementation manners of the first aspect, or the storage control method described in the second aspect or any of the specific implementation manners of the second aspect.

[0020] In a seventh aspect, the embodiments of the present application further provide a computer program product, having stored thereon a computer program / instruction, which, when executed by a processor, is used to implement the storage control method described in the first aspect of the embodiments of the present application or any of the specific implementation manners of the first aspect, or the storage control method described in the second aspect or any of the specific implementation manners of the second aspect.

[0021] It can be seen from the above technical solutions that the storage control method provided by the first aspect of the embodiments of the present application has the beneficial effects that, by creating a virtual disk and a namespace with the same namespace identifier on multiple storage control cards, the host can aggregate and identify multiple virtual disks and namespaces with the same namespace identifier as a single target object, thereby simplifying the access mode of the application layer and avoiding the complexity of directly managing multiple cards by the application layer. Meanwhile, the host distributes the IO request issued by the application layer to multiple storage control cards in parallel and submits the IO request to the respective storage device hardware queues for processing, thereby realizing parallel IO access across cards, breaking through the single-card performance bottleneck, improving the throughput and concurrent performance of the overall storage system, and improving the utilization rate of system resources.

[0022] The storage control method provided by the second aspect of the embodiments of the present application has the beneficial effects that, by dividing multiple storage control cards into a master card and slave cards and uniformly managing the slave cards by the master card, the host only needs to identify the master card to access the overall storage resource, thereby reducing the device management complexity on the host side and reducing the host scheduling burden. Meanwhile, the master card creates a management queue for the slave cards and manages the control path and data path of the slave cards by using the PCIe peer mapping mode, thereby realizing direct cooperation between multiple cards, avoiding the frequent involvement of the host in the interaction between multiple cards, and improving the overall processing efficiency. In addition, the master card uniformly monitors the doorbell operation from the host, and for the doorbell event associated with the master card itself, the master card performs corresponding processing. For the doorbell event associated with the slave card, the master card can forward the doorbell event to the corresponding slave card and further process the doorbell event by the slave card, thereby realizing coordination and load balancing between multiple storage control cards, and further improving the overall performance of the storage device. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A connection relationship diagram of a PCIe device in a server is provided for the embodiments of the present application.

[0024] Figure 2 A traditional RAID card-based storage system architecture schematic diagram provided for an embodiment of the present application;

[0025] Figure 3 A PCIe peer mapping data path of a DRAID card based on a direct memory access data path storage system architecture schematic diagram provided for an embodiment of the present application;

[0026] Figure 4 A multi-PCIe device cooperative system architecture schematic diagram applied to the method of the present application provided for an embodiment of the present application;

[0027] Figure 5 A storage control method flowchart schematic diagram provided for an embodiment of the present application;

[0028] Figure 6 A system architecture and workflow schematic diagram of DRAID-based multi-path namespace creation provided for an embodiment of the present application;

[0029] Figure 7 A storage control method flowchart schematic diagram provided for an embodiment of the present application;

[0030] Figure 8 A multi-storage control card IO queue distribution and workflow schematic diagram provided for an embodiment of the present application;

[0031] Figure 9 A multi-storage control card IO queue address allocation and mapping schematic diagram provided for an embodiment of the present application;

[0032] Figure 10 A storage control device structure schematic diagram provided for an embodiment of the present application;

[0033] Figure 11 Another storage control device structure schematic diagram provided for an embodiment of the present application;

[0034] Figure 12 A storage device structure schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the prior art, various PCIe devices are deployed as independent functional modules inside a server, such as Figure 1As shown, a typical PCIe device connection block diagram in a server is provided in an embodiment of the present application. The server system generally includes a central processing unit (CPU), a memory (DDR memory), and multiple PCIe devices connected through a peripheral component interconnect express (PCIe) system bus. Specifically, the CPU interacts with multiple PCIe devices through the PCIe system, and the DDR memory establishes a high-speed data exchange channel with the CPU as the host memory. Different types of PCIe devices are in the form of independent functional units and are connected to the PCIe system, such as a RAID card (or a storage control card) for providing disk array management and data redundancy protection functions, an SSD based on the NVMe protocol (NVMe SSD) for providing high-performance data access capabilities, a hardware acceleration card supporting EC erasure code calculation offload or supporting encryption and decryption task offload for reducing CPU burden, and other PCIe devices for providing network interfaces, computing acceleration, or IO expansion functions for the host. Through the above architecture, the server can realize the expansion and cooperation of multiple functional modules under a unified PCIe interconnection framework, thereby improving the overall performance and flexibility of the system.

[0036] As shown, Figure 2 Figure 2 A traditional RAID card working principle diagram is provided in an embodiment of the present application, including an application layer, an operating system layer, and a hardware layer. An application program in the application layer directly accesses a virtual disk interface (VD) provided by the operating system, without perceiving the physical hard disk structure of the underlying layer, and shielding the complexity of the underlying RAID. The operating system layer manages the RAID card through an operating system specific driver provided by the RAID card manufacturer, or uses an operating system standard driver (hereinafter referred to as an OS RAID Card Driver) to manage the RAID card. The OS RAID Card Driver exposes multiple VDs to the application layer for the application layer to access. At the same time, the OS RAID Card Driver is responsible for converting requests from the application layer into control commands that can be recognized and executed by the RAID card hardware, such as adding hard disks to the RAID card, creating a virtual disk VD, monitoring the array running state, etc. The RAID card in the hardware layer is connected to the hard disk backplane by the RAID controller through the connector, and further connects multiple hard disks. The RAID card performs data striping, mirroring, and checking operations at the hardware layer to realize data redundancy and performance optimization, but as can be seen from the figure, its working mode is usually limited to a single card scenario.

[0037] As shown, Figure 3 ​As shown, it is a working schematic diagram of a new type of RAID card (referred to as DRAID card) based on a direct memory access data path provided by an embodiment of the present application. The application program in the application layer directly accesses data through the VD provided by the operating system without perceiving the underlying hard disk type or connection mode. The operating system layer manages the DRAID card through the OS RAID Card Driver drive, and the OS RAID Card Drive exposes multiple virtual disk interfaces to the application layer for direct access. All virtual disk interfaces converge to the same physical RAID controller. Although the host layer sees multiple independent virtual disks, all IO requests and RAID management functions (such as striping, mirroring, and checking) are processed by the single RAID card. Generally, the DRAID card can establish a direct channel with hard disks (SAS / SATA / NVMe) of different interface types through a PCIe system, where SAS (Serial Attached SCSI), SATA (Serial Advanced Technology Attachment), and NVMe are currently common hard disk or solid state disk interface types. However, unlike the traditional centralized forwarding mode of "hard disks are first converged to a single RAID card, and then accessed to the host through an uplink by the card", Figure 2 the traditional centralized forwarding mode of "hard disks are first converged to a single RAID card, and then accessed to the host through an uplink by the card", Figure 3 In the architecture shown, each DRAID card and NVMe SSD is directly connected to the host PCIe level, and the DRAID card uses PCIe peer mapping technology to complete control and coordination. Data is transmitted along the independent PCIe link between each DRAID card and the host, thereby avoiding the limitations of uplink bandwidth and processing capacity of the traditional single RAID card, fully utilizing the native performance of multiple NVMe SSDs, and achieving higher overall throughput and concurrency capability. Figure 3 The medium red bidirectional arrow represents the data flow path of PCIe access between the DRAID card and each hard disk.

[0038] In summary, from Figures 1 to 3 It can be seen that the PCIe device under the existing architecture is usually connected to the host in a standard interface mode, relying on the host operating system and driver to complete operation, management, and resource scheduling. Although this mode can meet the basic functional requirements in a single card scenario, there is a lack of direct interaction mechanism between different PCIe devices, and they are in a state of mutual isolation. For example, Figure 2 and Figure 3For example, the traditional RAID card or the new DRAID card can only provide a virtual disk interface to the host as an independent unit, and when cooperation between multiple cards is required, new host-side software must be developed from 0 for unified management and scheduling. This not only increases the complexity of software development and adaptation, but also significantly increases the burden of operation and debugging. Therefore, the existing single-card management mode cannot efficiently support complex application scenarios such as load dynamic balancing and resource elastic expansion required by multi-card cooperation.

[0039] The various embodiments of the present application will be described in further detail below with reference to the accompanying drawings. The embodiments of the present application provide a storage control method, as shown in Figure 5 The method includes steps S501-S502.

[0040] S501: Create virtual disks and their namespaces with the same namespace identifier on multiple storage control cards, so that the host identifies multiple identical namespace identifiers as a target object that can be accessed by the application layer.

[0041] The namespace identifier (NSID, Namespace Identifier) is identification information used to uniquely identify the virtual disk namespace created by the storage control card, which can include but is not limited to the extended unique identifier (EUI, Extended Unique Identifier), the globally unique identifier (GUID, Globally Unique Identifier), the universally unique identifier (UUID, Universally Unique Identifier), and the command set identifier (CSI, Command Set Identifier). When multiple storage control cards create virtual disks, the same namespace identifier is assigned to these virtual disks in a pre-set manner, so that multiple virtual disks appear as logically consistent namespaces on the host side. In other words, although the virtual disks are created on different storage control cards, the namespace identifier remains consistent, and the host will aggregate and identify them as a unified target object when identifying this type of virtual disk. This target object is a logical storage unit that can be directly accessed by the application layer, and the application layer does not need to be aware of the number, distribution, or type of underlying physical storage control cards when performing data read / write operations, but only needs to initiate an IO request based on the unified target object. In this way, by creating virtual disks and their namespaces with the same namespace identifier on multiple storage control cards, the virtual disks on multiple storage control cards are unified at the logical namespace level, effectively solving the problem of scattered storage resources in the prior art and the need for individual management on the host side. The aggregation and unified presentation of the storage capabilities provided by multiple storage control cards are achieved, thereby simplifying the access path of the application layer.

[0042] S502: Receive the read-write IO request issued by the application layer to the target object, and distribute the IO request to each storage control card by the host, and submit to the storage device hardware queue corresponding to each storage control card for parallel IO processing.

[0043] When an application program of an application layer initiates read-write access to a target object, a block layer / multi-path layer of an operating system first receives a read-write IO request, which at least contains information such as a logical block address (LBA, Logical Block Addressing) or a range thereof, a data length, and the like. Since the target object actually corresponds to a same namespace identifier set exposed by multiple storage control cards, the multi-path management software maintains a path mapping table, which records the correspondence between the target object and the underlying multiple storage control card namespaces. For each received IO request, the software will distribute the multiple independent IO requests arriving according to the load state of the current paths and the path selection strategy adopted (for example, any implementable strategy such as polling, by weight, by current queue depth, or by historical latency feedback, which does not affect the implementation principle of this step) to a specific storage control card. After the storage control card receives the IO request, the firmware program or hardware control module in the storage control card submits the IO request to the storage device hardware queue corresponding to the storage control card according to the predefined rule to complete the subsequent IO processing. The storage device hardware queue here refers to the hardware-level queue structure possessed by the physical storage device (such as an NVMe SSD) managed by the storage control card for storing pending IO commands. Preferably, the structure of a "submission queue / completion queue pair" can be adopted, for example, the NVMe protocol's Submission Queue (abbreviated as SQ) and Completion Queue (abbreviated as CQ), usually collectively referred to as an SQ / CQ queue pair (Submission / Completion Queue Pair), which has a determined starting address, queue depth, and queue identifier (Queue ID, abbreviated as qid) in the host address space. When the host distributes a certain IO request to the target storage control card, the host writes a command entry containing the LBA, length, and direction fields into the submission queue SQ corresponding to the card according to the queue pointer, and performs a write operation on the doorbell register corresponding to the qid to trigger queue consumption; each storage control card accordingly takes the command entry on its own side in parallel and executes read-write operations on the back-end storage device, and after completion, writes a completion entry into the corresponding completion queue CQ and updates the queue state. Since different IO requests are distributed to different storage control cards and their respective hardware IO queues, the IO traffic originally sent to a single logical entity is effectively dispersed to multiple storage control cards and subsequently distributed to the storage devices corresponding to each storage control card, thereby realizing true parallel processing at the hardware queue level of each storage device and significantly improving the overall throughput and actual read-write times per second (IOPS, Input / Output Operations Per Second) performance of the storage system.Meanwhile, since the IO requests are evenly distributed to different hardware paths, the single storage control card or single storage device is avoided from becoming a performance bottleneck, and the load balancing capability of the system is enhanced. The application layer always faces only a single target object, and does not need to perceive the selection and submission details of the underlying path and queue, so that the access interface of the application layer and the upper layer of the operating system does not change due to the complexity of the underlying multi-card architecture, and the additional burden of application development, system adaptation and operation and maintenance is avoided.

[0044] In an embodiment, referring to Figure 6 , the entire system can be composed of a host, at least two storage control cards, and physical storage resources managed by the storage control cards. The host side runs an operating system and its NVMe driver, and can optionally install multipath management software such as NVMe-multipath or dm-multipath, which is used to aggregate and identify namespaces from different storage control cards but with the same namespace identifier (NSID, including EUI, GUID, UUID, CSI field) as a target object available for application access. Taking the Linux system as an example, the target object can be a block device. The storage control card can be a new type of RAID card based on a direct memory access data path (Direct Memory Access Data Path), namely a DRAID card, which is respectively denoted as a first storage control card DRAID_1 and a second storage control card DRAID_N. The two are interconnected with the host through a PCIe bus and support PCIe peer mapping between each other, so as to directly exchange control and metadata information without forwarding through the host memory. Using the above system architecture, a virtual disk and its namespace with the same name and identifier can be created on multiple storage control cards. Thus, the host operating system will not identify multiple virtual disks and their namespaces as independent storage devices, but through the multipath management software, these seemingly identical and repetitive namespaces are aggregated as a target object that can be accessed by the application layer, so as to achieve the goals of load balancing and device high availability.

[0045] On the basis of the above architecture, a virtual disk and its namespace with the same namespace identifier are created on multiple storage control cards, specifically including: configuring a shared buffer for metadata synchronization between multiple storage control cards; in response to a virtual disk creation command issued by the host to the first storage control card, creating a virtual disk and its namespace locally on the first storage control card, and writing the metadata generated in the process of locally creating the virtual disk and its namespace into the shared buffer, so that the second storage control card obtains the same metadata by accessing the shared buffer, and creates the same virtual disk and its namespace locally on the second storage control card.

[0046] To ensure the metadata consistency of multiple storage control cards in the virtual disk creation process, a shared buffer for metadata synchronization between multiple storage control cards is configured. In a feasible implementation, the shared buffer can include: registering, by the second storage control card, a specific address space in the local memory of the second storage control card to the first storage control card, and performing, by the first storage control card, PCIe peer mapping configuration on the address space registered by the second storage control card, so that the address space becomes a shared buffer directly accessible by the first storage control card.

[0047] Specifically, the second storage control card DRAID_N can select a continuous specific address space in its local memory and register the address space to the first storage control card DRAID_1. After receiving the registration information, the first storage control card DRAID_1 performs PCIe peer mapping configuration on the address space registered by the second storage control card DRAID_N, so that the address space is directly accessible by the first storage control card DRAID_1, and becomes a shared buffer (hereinafter referred to as meta_buffer) agreed by the second storage control card DRAID_N and the first storage control card DRAID_1. The meta_buffer is used to carry the structured metadata generated in the virtual disk creation and namespace establishment process, so as to ensure the consistency of the business data and the metadata.

[0048] When the host issues a command line interface (CLI, Command Line Interface) management tool via the NVMe driver to the first storage control card DRAID_1 to create a virtual disk interface VD (corresponding to Figure 6 As shown in flow ①), DRAID_1 as the primary receiver of the creation request first performs the creation process of VD locally. In the process, DRAID_1 completes the logical volume arrangement of VD according to the preset RAID strategy and resource allocation rule, and establishes the corresponding NVMe namespace on the storage medium controlled by it. After the establishment of the namespace, DRAID_1 triggers the asynchronous event of NVMe namespace change according to the NVMe protocol specification, so that the host side can perceive the newly established namespace entity from DRAID_1 (corresponding to Figure 6 As shown in flow ②).

[0049] After DRAID_1 completes the local VD and namespace creation, it immediately writes all necessary metadata generated in the creation process to the aforementioned meta_buffer (corresponding to Figure 6The metadata can include, for example, a capacity parameter of the VD, an identification of the RAID group to which the VD belongs (RAID group ID), and NSID information of the namespace, i.e., an EUI, a GUID, a UUID, a CSI, or the like. Since the DRAID_1 ensures that the creation process of the local virtual disk has been completed before writing, the metadata stored in the meta_buffer is complete and valid. When other storage control cards poll the shared buffer for reading, the metadata information can be used to locally replicate the virtual disk and the namespace consistent with the DRAID_1, so as to ensure that the virtual disk and the namespace created by all storage control cards in the host are completely consistent in configuration and identification. In comparison, if the first storage control card writes the unformed metadata to the shared buffer before the virtual disk is completely established, other storage control cards may perform the creation of the virtual disk based on the incomplete or incorrect metadata when polling, thereby causing the risk of inconsistency of the virtual disk and the namespace in the host, and causing the host to identify and access the target virtual disk abnormally or incorrectly.

[0050] It should be noted that the metadata of the embodiments of the present application can be organized in a pre-set structured form, so that any storage control card can parse and identify each field according to a unified rule when accessing the shared buffer to read the metadata, thereby avoiding ambiguity or misreading.

[0051] At the same time, the second storage control card DRAID_N, as a consumer of the metadata, accesses the meta_buffer at a pre-set polling period. When detecting the creation record written by the DRAID_1 and passing the verification, the DRAID_N locally executes a complete mirror virtual disk creation process according to the record, specifically, creating a VD of the same specification as the DRAID_1, and reusing the NSID information (EUI, GUID, UUID, CSI) obtained from the meta_buffer to establish a one-to-one NVMe namespace corresponding to the VD. It should be noted that, to ensure that the host side can identify the namespaces created by the DRAID_1 and the DRAID_N as equivalent, the DRAID_N will not regenerate or rewrite the NSID field obtained from the shared buffer, but directly reuse the field value to ensure that the host can identify the namespaces on the two storage control cards as the same target object.

[0052] It should be emphasized that the above process is only used as an example of a two-card scenario of DRAID_1 and DRAID_N, and in actual applications, it is not limited to the scenario of two storage control cards. When there are any number of storage control cards, metadata consistency can be achieved through the above-mentioned shared buffer meta_buffer. For example, when there are only two storage control cards, DRAID_N corresponds to DRAID_2, and in the scenario of more cards, all subsequent storage control cards can act as metadata consumers and complete metadata synchronization and virtual disk creation.

[0053] Taking three storage control cards as an example, DRAID_1 acts as a metadata creator, and after completing the creation of a virtual disk and a namespace locally, it writes complete metadata to meta_buffer. Subsequently, DRAID_2 and DRAID_3 act as metadata consumers, respectively, and detect the metadata creation record by polling meta_buffer, and based on the obtained metadata information, they complete the creation of the same virtual disk and namespace as DRAID_1 locally. Finally, the namespaces of the three storage control cards are completely consistent at the NSID information level, and the host side can identify them as the same target object, but when processing IO requests, they can be distributed to different storage control cards to achieve multi-card collaboration.

[0054] Exemplarily, regarding the configuration mode of meta_buffer, the following distributed shared buffer can be used, that is, each storage control card reserves a segment of address space as its local meta_buffer. Specifically, after DRAID_1 completes local creation, it needs to write the generated metadata to the local meta_buffer of DRAID_2, DRAID_3, and DRAID_N simultaneously or sequentially, and each consumer polls and reads its local shared buffer to complete metadata synchronization. Distributed shared buffer can reduce the performance bottleneck of a single global shared buffer to some extent and can effectively disperse concurrent access pressure.

[0055] After DRAID_N completes the creation of a virtual disk and a namespace locally, it will trigger an asynchronous event of "namespace change" according to the NVMe protocol specification, so that the host can perceive and identify the namespace entity from DRAID_N (corresponding to Figure 6The namespace created by DRAID_1 and the namespace created by DRAID_N are completely consistent in the identification information level, and belong to the respective storage control cards in the actual processing path of the IO request. Therefore, the host side recognizes them as the same target object, but when the IO request is actually executed, the request can be distributed to the corresponding storage control card for processing according to the binding relationship between the namespace and the specific storage control card. It can be understood that when each storage control card triggers an asynchronous event of namespace change, the host establishes a mapping and binding relationship between the storage control card and the corresponding namespace, so as to ensure that the target storage control card can be accurately found when the request is distributed, and the balanced processing and path correctness of the IO request are realized.

[0056] After receiving the namespace change asynchronous events from DRAID_1 and DRAID_N respectively, the host side determines according to the NSID (including EUI, GUID, UUID, and CSI) by the multi-path management software, and recognizes the newly created namespaces on the two cards as different access paths of the same logical device (corresponding to Figure 6 The flowchart V) shows that the multi-path aggregation, state monitoring, and fault switching strategies are configured, and the aggregated target object (corresponding to Figure 6 The flowchart VI) shows that in the Linux system, the target object is an aggregated block device, which provides unified disk capacity information and standardized read-write interfaces to the outside, and the application program can directly perform IO read-write operations on the VD (corresponding to Figure 6 The flowchart VII) shows that in the hardware layer, the multi-path software automatically determines the path of the IO request, distributes the load to different storage control cards, and detects the path health status in real time, and automatically switches the path when a problem occurs, so as to realize the decoupling of the application layer and the underlying path. Moreover, since the application layer is oriented to a single target object, the request only needs to be issued to the target object, and there is no need to develop a special host-side management software from 0, but the mature open-source host-side multi-path management software can be directly used to realize the distribution and processing of the IO request, thereby reducing the development difficulty and the difficulty of system integration.

[0057] In another embodiment, the consistency of the metadata is no longer dependent on the inter-card shared buffer, but is completed uniformly by the host side, that is, the embodiment of the application creates a virtual disk and its namespace with the same namespace identifier on multiple storage control cards, including: receiving a virtual disk creation command issued by the host to each storage control card respectively; wherein each virtual disk creation command carries the same metadata, and the metadata includes namespace identifier information; creating a virtual disk and its namespace with the same namespace identifier locally on each storage control card according to the metadata carried by the received virtual disk creation command.

[0058] Specifically, the CLI management tool run by the host determines an identifier set (that is, NSID information, including EUI, GUID, UUID, CSI) used to identify the same namespace in advance based on business planning, and writes the metadata containing the namespace identifier information as a creation parameter into the creation request. Subsequently, the host issues virtual disk creation commands with the same content to each storage control card. The consistency here actually means that the metadata carried by each command is the same in field name and field value, and at least includes the NSID field. If necessary, to ensure that the virtual disks presented on each card are equivalent in capacity and geometric properties, the metadata can also include parameters related to capacity, stripe width or other logical arrangement.

[0059] Each storage control card executes the creation process locally based on the metadata carried by the virtual disk creation command from the host after receiving the virtual disk creation command. Specifically, the storage control card creates a corresponding virtual disk VD on its managed physical resources, and establishes a one-to-one corresponding NVMe namespace accordingly. During the namespace establishment process, the control card directly uses the given EUI, GUID, UUID, SCI in the command as the identifier information of the local namespace without any change, and after the namespace creation is completed, the storage control card triggers an asynchronous event of namespace change according to the NVMe protocol, so that the host can perceive the latest available namespace entity on the storage control card.

[0060] When two or more storage control cards successively complete local creation and each trigger a namespace change asynchronous event, the NVMe driver on the host side completes discovery and enumeration, and the multi-path management software reads the NSIDs of each namespace. Since each control card creates a namespace according to the same set of NSIDs, the host can consider that these namespaces belong to different access paths of the same logical device, and thus performs path aggregation to generate a single aggregation object entity.

[0061] In an embodiment, in order to efficiently allocate the storage device hardware queues connected with the host among multiple storage control cards, the determination manner of the storage device hardware queues corresponding to each storage control card comprises: exchanging memory address information for allocating the storage device hardware queues among the multiple storage control cards through a shared buffer; and based on the exchanged memory address information, the multiple storage control cards distribute the storage device hardware queues connected with the host according to a predefined rule.

[0062] In the embodiments of the present application, the storage device hardware queues take the IO queue pairs of NVMe SSD as an example, and each storage control card interacts and cooperates with the memory address information based on the meta_buffer. Since the host can perceive multiple DRAID cards in parallel, the creation, binding and attribution of the hardware IO queues can all be allocated and managed by using the shared buffer, so as to avoid the same NVMe SSD queue resource from being repeatedly occupied or from being allocated in conflict.

[0063] In the hardware queue allocation process, DRAID_1 writes a request mark (such as a get_ioqpair_mem_address request) for obtaining the memory address of the IO queue in the meta_buffer, DRAID_N accesses the meta_buffer according to a preset polling period, and after detecting the request from DRAID_1, allocates the memory space for the NVMe SSD IO queue locally, and writes the start address and length parameters of the space back to the meta_buffer. In this way, DRAID_1 can read the memory space address information fed back by DRAID_N when polling the meta_buffer again.

[0064] In the actual NVMe SSD hardware IO queue creation process, DRAID_1 and DRAID_N continue to interact based on meta_buffer. For example, when multiple hardware queues need to be distributed to different storage control cards in a multi-card system, the first storage control card DRAID_1 and the second storage control card DRAID_N can write and read the address space information provided by each other in the meta_buffer to ensure that each IO queue meets the pre-defined rules in terms of logical number, length, etc., and to ensure that IO queues on the same NVMe SSD are not repeatedly created or cross-occupied by multiple cards. Taking the N=2 scenario as an example, if there are two DRAID cards in the system, the hardware queues can be divided in a parity allocation manner: the IO queues with odd numbers are responsible for allocation and management by DRAID_1, and the IO queues with even numbers are responsible for allocation and management by DRAID_N. In this way, the two cards can perform their respective functions at the hardware queue level, avoid queue ownership conflicts with minimal coordination cost, and achieve reasonable utilization and equal distribution of resources. In the N>2 scenario, the Round-robin (round-robin scheduling) method can also be used to sequentially allocate multiple IO requests to multiple DRAID cards, such as IO queue 1 to DRAID_1, IO queue 2 to DRAID_2, IO queue 3 to DRAID_3, IO queue 4 back to DRAID_1, IO queue 5 to DRAID_2, IO queue 6 to DRAID_3, and so on until all hardware queues of the storage devices are allocated.

[0065] On the basis of the above embodiment, the IO request initiated by the host application layer or the file system is first received by the multi-path management tool of the operating system layer, the multi-path management tool determines the NVMe namespace to which the request should be submitted according to the path mapping table, and the IO request is forwarded to the DRAID card associated with the namespace for processing by the NVMe namespace. It can be understood that since each storage control card creates a virtual disk (VD) with the same namespace identifier (NSID) locally and triggers an asynchronous event when the namespace state changes, the host side synchronously generates a corresponding NVMe namespace, thereby forming a path mapping table in the multi-path management tool, which records the correspondence between each namespace on the host side and each DRAID card at the bottom. Therefore, when the operating system receives the IO request of the application layer, it can determine the namespace corresponding to the IO request and its associated DRAID card according to the path mapping table, and realize dynamic selection and scheduling of the IO path. Subsequently, the IO request is distributed to the NVMe SSD hardware queue managed by the DRAID card by the firmware or hardware module of the DRAID card to perform the underlying data access operation.

[0066] In a further optimized solution, the cooperation and communication between PCIe devices can not be limited to the metadata synchronization of the shared buffer as described above, but can be extended to the unified management of the control channel and the data channel. In other words, a certain PCIe device can directly manage the control and data interaction of other PCIe devices through PCIe peer mapping or other hardware interconnection methods, so as to present only a logical view of a single PCIe device on the host side. In this way, the host can complete load balancing and unified access at the hardware level without relying on the multi-path management software of the software layer. For example, in a DRAID application scenario, one DRAID card can be selected as a master card, which centrally manages the control channel and the data channel of other slave cards through PCIe peer mapping technology, and the host only needs to identify and operate the master card to transparently access the underlying multi-card resources. Therefore, in another aspect, the application also provides a storage control method applied to at least two storage control cards connected with a host through a PCIe interface, as shown in the following figure. Figure 7 The method comprises steps S701-S702.

[0067] S701: At least one of the plurality of storage control cards is selected as a master card, and the other storage control cards except the master card are selected as slave cards managed by the master card. The master card is visible to the host, and the slave card is invisible to the host, so that the host identifies the master card as a target object that can be accessed by the application layer.

[0068] Specifically, the master card is directly connected with the host through a PCIe bus, and after the host operating system side completes the detection and identification of the hardware devices one by one connected to the PCIe bus, a target object visible to the application layer is exposed in the host. The slave card is not directly exposed to the host operating system in the architecture of the embodiment, that is, the host side cannot separately identify or access the physical existence of the slave card, and the logical object perceived by the host is only one device entity corresponding to the master card. The IO request of the application layer is issued to the master card through the logical interface exposed by the master card, and the master card coordinates and schedules the resources of the slave card through the internal control logic and data transmission channel. The master card has access capability to the control channel and the data channel of the slave card, such as communication based on PCIe peer mapping technology. In this way, the master card can initialize and configure the slave card, monitor the state of the slave card, and perform IO request distribution and collection operations without relying on the host software for multi-device management. For example, if there are three DRAID cards in the system, one of which is designated as a master card and the other two are designated as slave cards, the host can only identify the target object exposed by the master card at the operating system layer, and the two slave cards are completely controlled by the master card, and the host cannot perceive their existence.

[0069] In this way, the host only needs to identify and operate the single device entity of the master card to transparently access the storage resources actually managed by multiple storage control cards in cooperation, which not only eliminates the dependence on the host software layer multipath management tool (such as nvme-multipath or dm-multipath), but also avoids the drive complexity and path management burden caused by multiple device enumeration in the traditional multi-card scenario, simplifying the interaction between the operating system and the application layer. Secondly, since the slave card is no longer directly exposed to the host, but is uniformly managed through the master card, the host side software no longer needs to develop a complex multi-path scheduling module to achieve transparent use of multi-card resources, greatly reducing the complexity of system integration and operation. For example, in actual deployment, when the host issues an IO request, the master card is responsible for monitoring the doorbell operation of the host to each target IO queue. If the target IO queue is managed by the master card, the master card directly executes the corresponding IO processing; if the target IO queue is managed by the slave card, the master card forwards the doorbell event to the corresponding slave card, and the slave card reads the IO request from the host memory according to the doorbell event and executes the corresponding IO processing, and the final result is returned to the host. When a new slave card is added, only the identification and management of the master card are needed, and the host side can use the new storage control card without any modification, which has good scalability.

[0070] S702: receiving the IO request issued by the application layer to the target object, writing the IO request into the target IO queue by the host to trigger a doorbell event, and monitoring the doorbell event triggered by the master card, according to the target IO queue corresponding to the doorbell event, distributing the doorbell event to the corresponding target storage control card, so that the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and executes the IO processing.

[0071] The application layer issues an IO request to the target object identified by the host side, which corresponds to the master card that is the only one visible to the host in the system. After the host issues the IO request, the IO request is written into the host memory region mapped by the target IO queue (i.e. hardware queue SQ / CQ) corresponding to the target virtual disk VD, and the corresponding doorbell operation is performed to trigger a doorbell event. After the master card monitors the doorbell event, according to the corresponding relationship between the target IO queue and the storage control card, the doorbell event is distributed to the corresponding target storage control card for execution, and the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and executes the corresponding IO processing.

[0072] After the IO request is submitted to the hardware queue, the corresponding storage device takes out the request entry in turn according to the queue scheduling mechanism and completes the actual data read-write operation. For the IO completion notification, if the host VD driver works in the polling mode (i.e., poll mode), the slave card does not need to report an interrupt request to the master card after completing the IO, and the host can directly poll the completion queue CQ managed by the slave card to obtain the completion event; if the host VD driver works in the interrupt mode, the slave card sends an interrupt request to the master card after completing the IO, and the master card writes an interrupt message to the interrupt address recorded in its MSI-X table after receiving the interrupt request, thereby triggering the host to generate an interrupt. The host thereby learns the completion status of the IO operation, so that the application layer can transparently obtain the execution result.

[0073] As can be seen from the above, in the embodiment, the master card is responsible for unified monitoring and forwarding of the doorbell event of the host. In the perspective of the application layer and the operating system of the host, the entire storage system only appears as a target object that can be directly accessed. The application layer does not need to perceive that there are multiple control cards at the bottom layer, nor does it need to rely on the multi-path software at the host end to select a path or manage fault tolerance, thereby simplifying the complexity of application development and operation and maintenance. Moreover, through the correspondence between the target IO queue and the storage control card, the master card can distribute the doorbell event for different target IO queues, can fully utilize the parallel processing capability of all storage control cards and their back-end storage devices, can disperse the IO traffic originally concentrated on a single card, and can thereby significantly improve the IOPS and throughput of the overall system and reduce the risk of a single card becoming a performance bottleneck. In addition, the IO data is directly transmitted between the host memory and each NVMe SSD through the DMA (Direct Memory Access) mechanism, and the host CPU does not need to participate in the specific data transfer process, thereby reducing the processing load and access delay at the host side.

[0074] For reference Figure 8 To understand the method of the embodiments of the present application, before the host recognizes the master card as a target object that can be accessed by the application layer, it is necessary to first establish the management relationship between the master card and the slave cards among multiple storage control cards and complete the pre-creation and distribution of IO queues at the hardware level. For example, the master card brings at least one slave card into its management range through PCIe peer mapping and establishes a management queue (admin queue) that can be used for the control plane for the slave card (corresponding to process ① shown in Figure 8 On this basis, the host CLI sends a VD creation request to DRAID_1 (corresponding to process ② shown in Figure 8 After receiving the VD creation request, DRAID_1 starts the queue configuration process according to the preset allocation strategy (odd-even allocation or round-robin allocation) (corresponding to process ③ shown in Figure 8The flowchart ③) shows that on one hand, the DRAID_1 establishes a corresponding queue management object (qpair) for the queue allocated to itself locally, and records the address information and related attributes of the queue in the host memory in its internal register or memory table item, and forms an association with the queue structure in the host (SQ / CQ in the host memory) to realize the management and access capability of the IO queue; on the other hand, the master card issues a management instruction of creating an IO queue to the slave card through the admin queue of the slave card, requests the slave card to create and allocate a queue management object (qpair) corresponding to the allocation strategy locally, and records the context information of the queue in the hardware register of the slave card, so as to have the capability of directly accessing the queue in the host memory through PCIe DMA. In order to ensure that the queue sets created by the cards do not overlap, are unique in numbering and consistent with the planning, the master card generates and maintains a global queue allocation table at the same time of issuing the instruction, and records the one-to-one mapping relationship of “qid ↔ storage control card”.

[0075] In the specific implementation of the IO queue allocation, strategies such as odd-even allocation or round-robin allocation can be adopted. For example, in the scenario of only two cards, the master card can be responsible for managing the IO queues with qid of 1, 3, 5, …, 2n+1, and the slave card is responsible for managing the IO queues with qid of 2, 4, 6, …, 2n; in the scenario of multiple slave cards, the qid can be evenly allocated to different slave cards in a modulo or round-robin manner, so as to avoid contention or repeated creation of the same qid by multiple cards.

[0076] In an embodiment, before the host identifies the master card as a target object that can be accessed by the application layer, the method of the embodiment of the application further includes: creating a management queue for each slave card managed by the master card, so that the master card manages the control path and data path of the slave card through PCIe peer mapping; creating an IO queue allocated to the master card itself according to a preset allocation strategy, and issuing an IO queue creation request to the slave card managed by the master card through the management queue, so as to create an IO queue allocated to the slave card; wherein the master card establishes a virtual mapping relationship for the IO queue managed by the slave card, and forwards a doorbell event to the corresponding slave card through the management queue based on the virtual mapping relationship.

[0077] It can be understood that, in order to realize the unified management of the master card to the slave card, it is necessary to create an independent management queue for each slave card managed by the master card in the master card. The management queue here refers to a logical channel for interacting control commands and data information between the master card and the slave card, including two types of structures of command submission queue and completion queue, which are respectively used for sending control instructions and receiving execution results. In specific implementation, the master card accesses the internal register group and buffer of the slave card through PCIe peer mapping, to realize low-delay communication between cards. For this purpose, a point-to-point mapping relationship needs to be established between the master card and the slave card, that is, the master card allocates unique management queue identifier (Queue ID), queue start address and queue depth and other parameters for each slave card, and completes the registration and validation of the mapping through the PCIe configuration space. In this way, when the master card needs to issue a control instruction (such as initialization, state query or data acquisition, etc.), the master card directly writes the instruction into the submission queue of the corresponding slave card, and the slave card writes the result status in the corresponding completion queue after completing the processing. In this way, each slave card maintains an independent management queue pair with the master card, and the master card can accurately distinguish and schedule the operation requests of different slave cards, to ensure the communication order and efficiency in the multi-card environment. For example, if the system contains one master card and two slave cards, the master card will establish management queues for the two slave cards, so that the master card can issue instructions and receive responses to different slave cards in parallel.

[0078] Under the multi-storage control card cooperative architecture involved in the embodiment, the host end considers that there is only one logical controller in the system, that is, the target object corresponding to the master card. When the host initiates a virtual disk VD creation request command, the command is first received and analyzed by the master card. The master card determines the storage control card to which each IO queue of the virtual disk belongs according to a preset IO queue allocation strategy. If the queue identifier (qid) of the queue belongs to the allocation range of the master card itself, the master card establishes a queue management object (namely, a qpair object) corresponding to the queue locally, and records the address information and related attributes of the queue in the host memory in its internal register or memory table item, thereby forming the management and access ability to the IO queue.

[0079] If the queue identifier (qid) of an IO queue is assigned to a slave card according to the assignment strategy, the host card sends an IO queue creation request to the slave card through the established Admin Queue, which carries at least the identifier (qid) of the target IO queue, the queue depth, the host memory physical address, and other information. After receiving the IO queue creation request, the slave card establishes a queue management object (qpair) corresponding to the queue by its firmware or hardware logic unit, and records the context information of the queue in its own hardware register, thereby possessing the ability to directly access the host memory of the queue through PCIe DMA. Thereafter, when the host submits a request on the IO queue, the slave card can directly read the command entry in the host memory and perform the corresponding IO operation according to the pre-recorded queue information.

[0080] It should be noted that although the physical space of all IO queues on the host side is allocated in the host memory, under the architecture of the present embodiment, the management rights of different queues are distributed among different storage control cards. Taking the odd-even assignment strategy as an example, the host card is responsible for the odd-numbered queues, and the slave card is responsible for the even-numbered queues. The host card maintains the correspondence between each IO queue and the specific storage control card through a global queue assignment table (a mapping table containing qid), thereby realizing the logical unification and management division of IO resources among multiple cards. In this way, the host still only interacts with the host card in a logical sense, while the underlying multiple storage control cards can manage the IO queues they are responsible for through the doorbell events forwarded by the host card, realizing parallel access and efficient cooperation among multiple cards, and avoiding the complexity brought by the host directly managing multiple cards.

[0081] Further, the master card establishes a virtual mapping relationship for all IO queues (i.e. IO request processing queues) belonging to the slave card, so that the master card has the same visibility and operability as the local IO queue of the host. The virtual mapping relationship includes at least two aspects: first, the master card reserves a unified access entry for all qids in its register / memory mapping space, so that the host only faces the master card for doorbell register knocking, queue attribute configuration and queue memory registration of any qid (including the qid belonging to the slave card). Second, for the qid belonging to the slave card, the master card does not consume the request entry written by the host to the corresponding SQ, but performs processing according to the global queue allocation table (i.e. the correspondence between the target IO queue and the storage control card), for example, encapsulates the doorbell event as a management message according to the mapping relationship, and forwards it to the target slave card through the established slave card management queue (admin queue), which carries at least the target qid. The slave card triggers an actual doorbell action after receiving the management message, and directly reads the SQ command entry from the host memory through PCIe DMA according to the queue memory address information registered by the host when creating the queue, and then the slave card parses and executes the corresponding IO command, and submits the IO command to the NVMe SSD hardware queue managed by the slave card for execution.

[0082] In an embodiment, assuming that the virtual disk VD drive of the current host works in the interrupt mode, after the back-end storage device completes and feeds back the processing result, in combination with the architecture of "the master card is visible to the host, and the slave card is invisible to the host" of the embodiment of the application, in order to make the host still face only the master card in logic, and realize the unified perception of the multi-card system, the completion notification of the IO request adopts the interrupt link reported by the slave card and sent by the master card. Specifically, after detecting that the NVMe SSD hardware queue belonging to the slave card completes a command, the slave card forms corresponding completion event description information locally, and sends an interrupt notification request to the master card through the established slave card management queue (e.g. admin queue), which carries at least the identifier (qid) of the target queue. After receiving the interrupt notification request, the master card writes the interrupt message (i.e. Message Data) conforming to the PCIe MSI-X specification to the host side interrupt receiving address (i.e. Message Address) in the form of PCIe memory write transaction (Memory Write TLP) according to the interrupt vector information corresponding to the qid in the locally configured MSI-X table (MSI-X Table), so as to trigger a hardware interrupt response at the host interrupt controller side.

[0083] After the host operating system receives the interrupt signal, an interrupt service routine (ISR) bound to the interrupt vector is triggered, and the host directly reads the completion entry of the qid queue from the corresponding CQ area in the host memory according to the memory address and pointer information of the completion queue (CQ) registered by the driver in the queue creation stage. After the host parses the completion entry, it can confirm that the IO request has been successfully processed, allowing the upper layer to continue performing subsequent data processing or task scheduling operations.

[0084] In another embodiment, if the host's virtual disk driver works in poll mode, the slave card does not need to report an interrupt notification after detecting the completion of the IO request. The host can directly obtain the IO completion event by periodically polling the content of the corresponding completion queue CQ in the hardware queue pair (SQ / CQ) managed by the slave card.

[0085] After the completion queue is pre-created, allocated, and the virtual mapping relationship is established, the host card can report the readiness state of the logical volume / naming space to the host (for example, through an asynchronous event notification), and the host identifies and updates the corresponding target entity object (which can be represented as an NVMe block device in the Linux environment) accordingly (corresponding to Figure 8 Flow 4). At this point, any IO access to the target object by the application layer (corresponding to Figure 8 Flow 5) can be distributed between the IO queues of the host card and the IO queues of the slave cards managed by the host card according to the established global queue allocation table and scheduling strategy, and submitted to the corresponding storage device hardware queues for IO processing, which not only reduces the complexity and CPU involvement of the host-side software, but also improves the throughput by utilizing multi-card parallel processing.

[0086] In an embodiment, the step S702 specifically includes: in response to an IO request issued by the application layer to the target object, writing the IO request to the target IO queue of the virtual disk corresponding to the target object by the host, and completing the doorbell operation of the target IO queue to trigger a doorbell event; if the target IO queue corresponding to the doorbell event is managed by the host card, reading the IO request from the target IO queue in the host memory by the host card and submitting it to the storage device hardware queue corresponding to the host card for IO processing after processing; if the target IO queue corresponding to the doorbell event is managed by the target slave card corresponding to the host card, forwarding the doorbell event to the corresponding target slave card by the host card through the management queue, and reading the IO request from the target IO queue in the host memory by the target slave card and submitting it to the storage device hardware queue corresponding to the target slave card for IO processing after processing. (corresponding to Figure 8 Flow 6).

[0087] When the application program of the application layer initiates read-write access to the target object, the host performs a doorbell operation on the virtual disk queue corresponding to the target object according to the IO queue (i.e., SQ) maintained by the block layer or the NVMe driver. The doorbell operation is essentially that the host writes trigger information into the register corresponding to the queue to inform the storage control card to consume the newly submitted command in the queue. Since the host can only recognize and operate the register space mapped by the host card, all doorbell operations are formally performed on the host card. After receiving the doorbell write, the host card first performs attribution determination on the target queue identifier qid according to the queue allocation table it maintains.

[0088] In one case, if the target IO queue corresponding to the doorbell event is a queue directly managed by the host card, for example, according to the predefined odd-even allocation rule, the host card is responsible for the IO queue with qid being 1, 3, 5, 2n+1, after detecting the doorbell operation, the host card directly takes out the command entry from the corresponding submission queue in the host memory, and parses and processes the IO request. The parsing and processing process can specifically include: parsing the original IO command submitted by the host, command splitting or merging, and generating read-write instructions recognizable by the underlying storage device, etc. The IO request after parsing and processing is submitted to the storage device hardware queue managed by the host card for execution. The hardware queue can adopt the submission queue / completion queue pair structure specified in the NVMe protocol, and the host card is responsible for converting the request into a read-write operation executable by the underlying hardware. After completion, the host card writes the result in the completion queue according to the completion entry returned by the underlying device, and feeds back the completion status to the host.

[0089] In another case, if the target IO queue corresponding to the doorbell event is not directly managed by the master card, but belongs to the allocation range of a certain slave card, for example, the IO queues with qid of 2, 4, 6, 2n, the master card will not consume the request by itself after detecting the doorbell register write, but forward the doorbell event to the target slave card through the management queue (admin queue) established by the master card for the slave card. Specifically, when the host rings the doorbell, the doorbell write is actually a virtual doorbell in the register space of the master card, and the master card will not really consume the IO request, but forward the doorbell event to the target slave card through the management queue; after receiving the doorbell event, the target slave card performs a real doorbell operation again in the controller of the target slave card to write the real doorbell register of the controller to trigger the local DMA to read the host memory to obtain the IO request. After reading the IO request, the target slave card further analyzes and processes the IO request in the local firmware or hardware logic, including but not limited to: analyzing the original IO command submitted by the host, command splitting or merging, and generating read-write instructions recognizable by the underlying storage device, etc. After completing the above processing, the IO request is submitted to the storage device hardware queue corresponding thereto for processing, and finally the corresponding data read-write operation is completed.

[0090] In this way, the host always interacts with the master card in logic, but all IO requests are still distributed to the master card and multiple slave cards according to the queue number rule in physics, so as to realize parallel IO processing and resource balanced utilization across cards. In the architecture of the embodiment of the present application, the host only needs to face a single target object when submitting a request, without the need to maintain independent drivers or interaction logic for different control cards, thereby reducing the system complexity. At the same time, relying on the proxy management of the master card to the slave card, the accurate transmission of the doorbell operation and the consistency of the queue consumption can be ensured, thereby ensuring the correctness and efficiency of the IO processing.

[0091] In an embodiment, after the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and performs IO processing, the method of the embodiment of the present application further comprises: sending an interrupt notification request to the master card through the management queue, and the master card generates an interrupt message according to the information of the interrupt notification request and writes it to the interrupt address of the host to trigger the host interrupt response.

[0092] Considering that the storage control card needs to feedback the completion to the host after finishing the IO request processing. In one embodiment, if the host's virtual disk driver works in polling mode, the card does not need to report the interrupt notification after detecting the completion of the IO request, and the host can directly obtain the IO completion event by periodically polling the content of the corresponding completion queue CQ in the hardware queue pair (SQ / CQ) managed by the card.

[0093] But considering that if the host works in polling mode, it will cause the host to occupy CPU resources for a long time to repeatedly query the completion status, resulting in performance waste and response delay. To avoid this inefficient way, the host usually works in interrupt mode, so that the storage control card actively sends an interrupt notification to the host when the request is truly completed, so as to trigger the host response without continuous polling, improve CPU utilization and shorten the perceived time of IO completion.

[0094] And when the host's virtual disk VD driver works in interrupt mode, each storage control card needs to initiate a completion notification to the host after finishing the IO processing it is responsible for. Since the host can only see the master card and only map the master card's registers and interrupt resources, the slave card cannot directly trigger an interrupt to the host, but needs to report an interrupt notification request through the existing admin queue between the master card. Specifically, after writing the corresponding completion entry into the local completion queue (CQ) and updating the queue pointer, the slave card constructs an asynchronous event message, delivers the interrupt notification request containing the target queue identifier qid, the write pointer or entry index of the completion queue CQ, the phase flag, and the optional error code / return code, etc. to the admin queue of the master card (such as Figure 6 The master card continuously retrieves the request from the admin queue and performs semantic analysis, by which it confirms the attribute and interrupt vector mapping relationship of the qid that triggers the completion.

[0095] When the host card receives the interrupt notification request from the slave card, it first retrieves the mapping according to the qid carried in the request, and parses out the Message Signaled Interrupts eXtended (MSI-X) target address to which the message should be written. The parsing process can be performed by the host card by pre-maintaining a "qid→MSI-X vector" mapping and the interrupt message address information corresponding thereto (determined by the MSI-X table of the host card). Subsequently, the host card writes the interrupt message (message) in the memory mapping area corresponding to the MSI-X message address / data register, thereby triggering the host interrupt response under the MSI-X mechanism. The host operating system enters the interrupt service process accordingly, reads the completion queue entry according to the queue view presented by the host card, and recovers the IO request. It should be noted that for the qid managed locally by the host card, the host card can directly write the MSI-X message to the host according to the same mapping rule after completing the IO processing of the local device and writing the completion entry; for the qid belonging to the slave card, the host card only serves as the proxy trigger party of the interrupt, and does not consume the completion entry ready for the slave card again. Taking the two-card odd-even allocation as an example, when the DRAID_N (slave card) completes the request on qid 2, 4, 6, …, 2n, the DRAID_N first writes the completion in the local CQ and updates the pointer, and then sends an asynchronous event to the DRAID_1 (host card) through the admin queue as an interrupt notification request; the DRAID_1 receives the request, locates the corresponding MSI-X target address according to the qid, writes the interrupt message to trigger the host interrupt, and then the host processes the completion according to the queue space uniformly exposed by the DRAID_1. For the request on qid 1, 3, 5, …, 2n+1, the DRAID_1 can directly write the MSI-X message after local completion, without going through the admin queue.

[0096] Please refer to Figure 9 On the basis of the above host card and slave card embodiment, the NVMe SSD management process is as follows:

[0097] Since the master card DRAID_1 holds the management queue of the slave card DRAID_N, the communication between multiple cards can be realized by using the management queue, so as to jointly complete the operation on the NVMe SSD. Specifically, the host CLI first issues a management operation, and includes the DRAID_N and the NVMe SSD in the management range of the DRAID_1. As shown in flow ①, the host CLI registers the BAR address space of the DRAID_N and the NVMe SSD to the DRAID_1, the DRAID_1 controls these BAR spaces through the PCIe peer mapping access mode, and creates the management queue (admin queue) corresponding to the DRAID_N and the NVMe SSD. On this basis, the DRAID_1 can interact with the DRAID_N and the NVMe SSD through the management queue.

[0098] In the process of management information interaction, the DRAID_1 needs to obtain the memory address range of the DRAID_N specially used for storing the NVMe SSD IO queue. Specifically, the DRAID_1 can send a get_ioqpair_mem_address request to the management queue of the DRAID_N to obtain the start address and length of the memory space allocated by the DRAID_N for the NVMe SSD IO queue, as shown in flow ②.

[0099] After receiving the memory address information, the DRAID_1 starts to create the IO queue of the NVMe SSD. Taking the number N of DRAID cards as 2 for example, the IO queue of the NVMe SSD can be evenly distributed to the two DRAID cards according to the odd-even allocation rule, that is, the DRAID_1 is responsible for creating the IO queue with an odd ID, and the DRAID_2 is responsible for creating the IO queue with an even ID. When the DRAID_1 creates the IO queue with an ID of 1, it issues an IO queue creation command to the NVMe SSD through the local memory address; when the DRAID_1 creates the IO queue with an ID of 2, it issues a creation command to the NVMe SSD by using the previously obtained DRAID_2 dedicated memory address, and synchronizes the information (including the start address, length and ID, etc.) of the IO queue to the DRAID_2 through the management queue of the DRAID_2. After receiving the information, the DRAID_2 creates a qpair object corresponding thereto in the local to operate the actual NVMe SSD hardware queue, as shown in flow ③. For more number of DRAID cards, other queue allocation rules can be used, for example, the IO queue is evenly distributed in each DRAID card based on the round-robin mode.

[0100] After the above process is completed, DRAID_1 and DRAID_N both hold the control right of the hardware IO queue of the NVMe SSD, so as to be able to directly issue read-write commands to the NVMe SSD, as shown in process ④. With the help of direct memory access (DMA) technology, the NVMe SSD and the host memory can realize direct transmission of data, thereby avoiding the performance overhead caused by intermediate copying, as shown in process ⑤.

[0101] The complete IO processing flow is shown in process ⑥. When the host issues an IO request to a target object, the IO request is first written by the host to the host memory area mapped by the corresponding virtual disk queue (i.e., the target IO queue), and a corresponding doorbell operation is performed. After DRAID_1 (the master card) detects the doorbell operation, the ownership of the target IO queue is determined according to the queue ID (qid) corresponding to the doorbell event: if the qid corresponds to a virtual disk qpair managed by DRAID_1, DRAID_1 directly reads the IO command entry in the queue from the host memory, and submits it to the NVMe SSD hardware queue managed by itself for IO processing; if the qid corresponds to a virtual disk qpair managed by DRAID_N, DRAID_1 does not directly consume the request, but forwards the doorbell event to DRAID_N through the management queue of DRAID_N. After DRAID_N receives the doorbell event from DRAID_1, a real doorbell operation is performed in the local control logic of DRAID_N, so as to trigger the local DMA controller to read the IO command entry of the target IO queue in the host memory, and submit the read IO request to the NVMe SSD hardware queue managed by DRAID_N itself for IO processing.

[0102] Through the above process, multiple DRAID cards realize distributed management of the NVMe SSD hardware IO queue under the unified scheduling of the master card, which not only ensures balanced utilization of queue resources, but also realizes collaborative operation across devices without increasing the complexity of the host side, thereby effectively improving the IO parallelism and overall system performance, while maintaining the single-card view of the host and realizing load balancing and fault tolerance.

[0103] In order to realize the storage control method of the embodiments of the present application, the embodiments of the present application also provide a storage control device applied to a storage device of at least two storage control cards connected with a host through a PCIe interface, as shown in Figure 10As shown, the apparatus comprises a virtual disk creation unit 1001 configured to create a virtual disk and a namespace of the virtual disk with the same namespace identifier on a plurality of storage control cards, so that the host identifies a plurality of same namespace identifiers as one target object accessible by an application layer; and an IO request processing unit 1002 configured to receive a read / write IO request issued by the application layer to the target object, and distribute the IO request to each storage control card by the host, and submit the IO request to a storage device hardware queue corresponding to each storage control card for parallel IO processing.

[0104] In an embodiment, the storage control cards comprise a first storage control card and a second storage control card; and the virtual disk creation unit 1001 is specifically configured to: configure a shared buffer for metadata synchronization between the plurality of storage control cards; and in response to a virtual disk creation command issued by the host to the first storage control card, create a virtual disk and a namespace of the virtual disk locally on the first storage control card, and write metadata generated in the process of creating the virtual disk and the namespace locally into the shared buffer, so that the second storage control card obtains the same metadata by accessing the shared buffer, and creates the same virtual disk and the namespace locally on the second storage control card.

[0105] In an embodiment, the virtual disk creation unit 1001 is specifically configured to: receive a virtual disk creation command issued by the host to each storage control card respectively; wherein each virtual disk creation command carries the same metadata, and the metadata comprises namespace identifier information; and create a virtual disk and a namespace of the virtual disk with the same namespace identifier locally on each storage control card according to the metadata carried by the received virtual disk creation command.

[0106] In an embodiment, the virtual disk creation unit 1001 is specifically configured to: register, by the second storage control card, a specific address space in a local memory of the second storage control card to the first storage control card, and perform PCIe peer mapping configuration on the address space registered by the second storage control card to the first storage control card, so that the address space becomes a shared buffer directly accessible by the first storage control card.

[0107] In an embodiment, the apparatus further comprises a hardware queue determination unit, which is specifically configured to: interact, through the shared buffer, memory address information for allocating storage device hardware queues between the plurality of storage control cards; and based on the obtained memory address information, the plurality of storage control cards distribute the storage device hardware queues connected with the host according to a predefined rule.

[0108] In order to implement the storage control method of the embodiments of the present application, the embodiments of the present application further provide a storage control apparatus, which comprises the virtual disk creation unit 1001 and the IO request processing unit 1002. Figure 11As shown, the device is applied to storage devices of at least two storage control cards connected with a host through a PCIe interface, and the device comprises: a master-slave card configuration unit 1101, configured to take at least one of the plurality of storage control cards as a master card, and take other storage control cards except the master card as slave cards managed by the master card, the master card is visible to the host, and the slave card is invisible to the host, so that the host identifies the master card as a target object which can be accessed by an application layer; an IO request processing unit 1102, configured to receive an IO request issued by the application layer to the target object, and write the IO request into a target IO queue to trigger a doorbell event after the host; after the master card detects that the doorbell event is triggered, the doorbell event is distributed to the corresponding target storage control card according to the target IO queue corresponding to the doorbell event, so that the target storage control card reads the IO request from the target IO queue in the host memory according to the doorbell event and performs IO processing.

[0109] In an embodiment, the device further comprises: a queue management unit; and the queue management unit 1103 is specifically configured to: create a management queue for each slave card managed by the master card, so that the master card manages the control path and the data path of the slave card through PCIe peer mapping; create an IO queue assigned to the slave card according to a preset assignment strategy, and issue an IO queue creation request to the slave card managed by the master card through the management queue, so as to create an IO queue assigned to the slave card; wherein the master card establishes a virtual mapping relationship for the IO queue managed by the slave card, and forwards the doorbell event to the corresponding slave card through the management queue based on the virtual mapping relationship.

[0110] In an embodiment, the IO request processing unit 1102 is specifically configured to: in response to the IO request issued by the application layer to the target object, write the IO request into the target IO queue of the virtual disk corresponding to the target object by the host, and complete the doorbell operation corresponding to the target IO queue to trigger the doorbell event; if the target IO queue corresponding to the doorbell event is managed by the master card, read the IO request from the target IO queue in the host memory by the master card and submit to the storage device hardware queue corresponding to the master card for IO processing after processing; if the target IO queue corresponding to the doorbell event is managed by the target slave card corresponding to the master card, forward the doorbell event to the corresponding target slave card by the master card through the management queue, and read the IO request from the target IO queue in the host memory by the target slave card and submit to the storage device hardware queue corresponding to the target slave card for IO processing after processing.

[0111] In an embodiment, the IO request processing unit 1102 is specifically configured to: send an interrupt notification request to the master card through the management queue, and generate an interrupt message according to the information of the interrupt notification request and write the interrupt message to an interrupt address of the host, so as to trigger a host interrupt response.

[0112] It should be noted that the above embodiments provide the storage control device to perform the storage control method, and only the division of the above program modules is exemplified, and in actual application, the above processing can be completed by different program modules according to needs, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the storage control device and the storage control method provided by the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0113] Based on the hardware implementation of the above program modules, and in order to implement the storage control method provided by the embodiments of the present application, the embodiments of the present application also provide a storage device, as shown in Figure 12 The storage device 1200 includes a central processor 1201, a memory 1202, and an input / output interface 1203; the memory 1202 is a transient storage memory or a persistent storage memory; the central processor 1201 is configured to communicate with the memory 1202 and execute instruction operations in the memory 1202 to execute any one of the above storage control methods.

[0114] Of course, in actual application, each component in the storage device 1200 is coupled together through a bus system 1204. It can be understood that the bus system 1204 is used to realize the connection and communication between the components. The bus system 1204 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the bus system 1204 in Figure 12 The memory 1202 in the embodiments of the present application is used to store various types of data to support the operation of the storage device 1200. Examples of these data include any computer programs used to operate on the storage device 1200.

[0115] It can be understood that when the processor in the above-mentioned storage device executes the computer program, the functions of each unit in the corresponding device embodiments described above can also be realized, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory and executed by the processor to complete the embodiments of the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which is used to describe the execution process of the computer program in the storage device. For example, the computer program can be divided into the units in the above-mentioned storage device, and each unit can realize the specific functions as described above in the corresponding storage device.

[0116] The storage device can be a solid state drive (SSD), a storage control card, a distributed RAID control module, an NVMe controller, a storage array node, a PCIe expansion storage module, and the like. The storage device can include a processor and a memory. Those skilled in the art can understand that the processor and the memory are only examples of the storage device and do not constitute a limitation on the storage device. The storage device can further include more or fewer components, or combine certain components, or adopt a different component structure. For example, the storage device can further include a bus interface module, a direct memory access (DMA) control unit, a cache management unit, an error checking and correction (ECC) unit, a PCIe interface controller, an NVMe protocol stack processing unit, a power consumption management module, and the like, for supporting high-speed transmission of storage data, access scheduling, and system-level resource coordination.

[0117] The central processor 1201 can be a dedicated storage control processor, such as an embedded multi-core controller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, for performing storage control logic, PCIe transmission management, queue scheduling, data path control, and error recovery operations. The memory 1202 is used to store instruction codes, control parameters, and runtime state information, and can include volatile memory and non-volatile memory. The volatile memory can be a dynamic random access memory (DRAM) for storing command queues, meta buffers, and data caches; the non-volatile memory can be a flash memory (NAND Flash) or other solid state storage medium for storing firmware programs, system logs, or configuration information. The input / output interface 1203 is a communication interface between the storage device and the host, which can adopt a PCIe bus interface, for establishing a high-speed data transmission channel with the host system and supporting PCIe peer mapping access mode to realize direct data and control path interaction between master cards and slave cards.

[0118] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to perform the storage control method described above.

[0119] The embodiment of the present application further provides a computer program product, which stores a computer program / instruction. When the computer program / instruction is executed by a processor, the computer program / instruction is used for implementing the storage control method described in the first aspect of the embodiment of the present application or any specific implementation manner of the first aspect.

[0120] Those skilled in the art should understand that the system and method disclosed in the present application can be realized by software, hardware or a combination of software and hardware. For example, the above-mentioned functional modules can be integrated in the same storage control chip, or divided into multiple independent modules according to the function logic and distributed on different hardware units. The functional modules can be realized in the form of hardware circuit, or through firmware or executable program. When realized in the form of software or firmware, the instruction code of the functional modules can be stored in a computer readable storage medium. The computer readable storage medium can be a flash memory, an electrically erasable programmable read-only memory (EEPROM), an embedded read-only memory (Embedded Read-Only Memory), a solid state disk control chip memory area or other medium capable of storing program codes. When the processor reads and executes the program, all or part of the steps of the storage control method described in the present application can be realized.

Claims

1. A storage control method characterized by comprising: The application is applied to at least two storage control cards connected with a host through a PCIe interface, and comprises: creating a virtual disk and a namespace with the same namespace identifier on multiple storage control cards, so that the host identifies multiple same namespace identifiers as a target object which can be accessed by an application layer; receiving read-write IO requests issued by the application layer to the target object, distributing the IO requests to each storage control card by the host, and submitting the IO requests to a corresponding storage device hardware queue of each storage control card for parallel IO processing.

2. The method of claim 1, wherein, The storage control card comprises a first storage control card and a second storage control card; the creating of the virtual disk and the namespace with the same namespace identifier on multiple storage control cards comprises: configuring a shared buffer for metadata synchronization between multiple storage control cards; in response to a virtual disk creation command issued by the host to the first storage control card, creating a virtual disk and a namespace locally on the first storage control card, and writing metadata generated in the process of creating the virtual disk and the namespace locally into the shared buffer, so that the second storage control card obtains the same metadata by accessing the shared buffer, and creates the same virtual disk and namespace locally on the second storage control card.

3. The method of claim 1, wherein, The creating of the virtual disk and the namespace with the same namespace identifier on multiple storage control cards comprises: receiving a virtual disk creation command issued by the host to each storage control card respectively; wherein each virtual disk creation command carries the same metadata, and the metadata comprises namespace identifier information; creating a virtual disk and a namespace with the same namespace identifier locally on each storage control card according to the metadata carried by the received virtual disk creation command.

4. The method of claim 2, wherein, The configuration of the shared buffer for metadata synchronization between multiple storage control cards comprises: registering, by the second storage control card, a specific address space in the local memory of the second storage control card with the first storage control card, and performing PCIe peer mapping configuration on the address space registered by the second storage control card by the first storage control card, so that the address space becomes a shared buffer which can be directly accessed by the first storage control card.

5. The method according to any of claims 2 or 4, characterized in that, The determination of the storage device hardware queue corresponding to each storage control card comprises: interacting, through the shared buffer, memory address information for allocating storage device hardware queues between multiple storage control cards; based on the obtained memory address information, the multiple storage control cards distribute the storage device hardware queues connected with the host according to a predefined rule.

6. A storage control method characterized by comprising: The application is applied to at least two storage control cards connected with a host through a PCIe interface, and comprises: At least one of the plurality of storage control cards is a master card, and other storage control cards are slave cards managed by the master card, the master card is visible to the host, and the slave card is invisible to the host, so that the host identifies the master card as a target object that can be accessed by an application layer; The application layer issues an IO request to the target object, and the host writes the IO request into a target IO queue and triggers a doorbell event; the master card monitors the triggering of the doorbell event, and distributes the doorbell event to a corresponding target storage control card according to the target IO queue corresponding to the doorbell event, so that the target storage control card reads the IO request from the target IO queue in the host memory and performs IO processing.

7. The method of claim 6, wherein, Before the host identifies the master card as a target object that can be accessed by an application layer, the method further comprises: Creating a management queue for each slave card managed by the master card, so that the master card manages the control path and data path of the slave card through PCIe peer mapping; According to a preset allocation strategy, an IO queue allocated to the master card itself is created, and an IO queue creation request is issued to the slave card managed by the master card through the management queue, so as to create an IO queue allocated to the slave card; wherein the master card establishes a virtual mapping relationship for the IO queue managed by the slave card, and forwards the doorbell event to the corresponding slave card through the management queue based on the virtual mapping relationship.

8. The method of claim 7, wherein, The application layer issues an IO request to the target object, and the host writes the IO request into the target IO queue and triggers a doorbell event; After the master card monitors the triggering of the doorbell event, the doorbell event is distributed to a corresponding target storage control card according to the target IO queue corresponding to the doorbell event, so that the target storage control card reads the IO request from the target IO queue in the host memory and performs IO processing, comprising: In response to the IO request issued by the application layer to the target object, the host writes the IO request into the target IO queue of the virtual disk corresponding to the target object, and completes the doorbell operation corresponding to the target IO queue to trigger a doorbell event; If the target IO queue corresponding to the doorbell event is managed by the master card, the master card reads the IO request from the target IO queue in the host memory and processes it, and then submits it to the storage device hardware queue corresponding to the master card for IO processing; If the doorbell event corresponds to a target IO queue managed by a target slave card corresponding to the master card, the master card forwards the doorbell event to the corresponding target slave card through the management queue, and the target slave card reads the IO request in the target IO queue in the host memory and processes it, and then submits it to the target storage device hardware queue for IO processing.

9. The method of claim 7, wherein, After the target storage control card reads the IO request in the target IO queue in the host memory according to the doorbell event and performs IO processing, the method further comprises: sending an interrupt notification request to the master card through the management queue, and the master card generates an interrupt message according to the information of the interrupt notification request and writes it to the interrupt address of the host to trigger the host interrupt response.

10. A storage device having stored thereon instructions, the storage device characterized in that, The instructions are executed by the processor to implement the method of any one of claims 1-5 or claims 6-9.

Images