Method, system and non-transitory processor-readable media for data processing using non-volatile storage device offloading of host tasks

Abstract

Various implementations involve a non-volatile memory device receiving a host command from a host, comprising device content information of the non-volatile memory device. The device content includes the address of a buffer for each of the non-volatile memory devices. In response to receiving the host command, portions of host data are partitioned among the non-volatile memories. The non-volatile memory device sends a transfer request to the host, instructing that each portion of the host data be transferred to a corresponding one of the non-volatile memory devices. The non-volatile memory device sends a peer command to another non-volatile memory device based on the device content information.

Description

Technical Field

[0001] This disclosure generally relates to systems, methods, and nontransitory processor-readable media for data processing using multiple nonvolatile memory devices. Prior Technology

[0002] A typical system providing data storage may include a host computer coupled to multiple non-volatile memory devices via buses such as Peripheral Component Interconnect Express (PCIe) buses. The host computer may include processing units such as a Central Processing Unit (CPU) coupled to memory cells such as Dynamic Random Access Memory (DRAM). The CPU is coupled to the bus via a PCIe root complex. Redundant Array of Inexpensive Drives (RAID) can be implemented on non-volatile memory devices to provide protection against drive failures. Summary of the Invention

[0003] The various configurations disclosed herein relate to a host command for receiving device content information of a second non-volatile memory device from a host via a first non-volatile memory device, a buffer for transmitting data corresponding to the host command from the host to the first non-volatile memory device via the first non-volatile memory device, and a system, method, apparatus, and non-transitory processor-readable medium for sending peer commands from the first non-volatile memory device to the second non-volatile memory device based on device content information, wherein the device content includes the address of the buffer of the second non-volatile memory device. Simple Explanation of the Diagram

[0004] [Figure 1] shows a block diagram of an example of a system including a non-volatile memory device and a host according to some implementations.

[0005] [Figures 2A, 2B, 2C and 2D] are schematic diagrams illustrating example copying methods based on various configurations.

[0006] [Figure 2E] is a flowchart illustrating the copying methods according to various configurations.

[0007] [Figures 3A, 3B, 3C, 3D, 3E and 3F] are schematic diagrams illustrating example propagation methods based on various configurations.

[0008] [Figure 3G] is a flowchart illustrating the propagation methods according to various configurations.

[0009] [Figures 4A, 4B and 4C] are schematic diagrams illustrating example partitioning methods based on various configurations.

[0010] [Figure 4D] is a flowchart illustrating the partitioning methods based on various configurations.

[0011] [Figure 5A] is a flowchart illustrating example methods for group-guided computation based on various configurations.

[0012] [Figure 5B] is a diagram illustrating the grouping of non-volatile memory devices according to various configurations. Implementation

[0013] The host includes a CPU and memory devices such as DRAM. The host can be coupled to multiple non-volatile memory devices via buses (e.g., PCIe buses) and a PCIe root complex. The PCIe root complex provides the host's CPU with knowledge of the addresses of all non-volatile memory devices connected to the bus. The CPU can explicitly instruct the non-volatile memory devices to perform various operations. The host's memory devices can be used to perform critical data communication between the host's CPU and the non-volatile memory devices. This critical data communication must be traversed through interfaces (e.g., buses and PCIe root complexes). Traditionally, non-volatile memory devices do not communicate with each other, and for write operations, the host's CPU issues a write command to each of the non-volatile memory devices. In response to the write command, each of the non-volatile memory devices performs a DMA transfer from the same DRAM in the host to write data to the non-volatile memory (e.g., flash media) of each of the non-volatile memory devices. When a non-volatile memory device performs a DMA transfer, the data must be transferred through the PCIe root complex and bus.

[0014] The controller of a non-volatile memory device has processing power (e.g., CPU), a direct memory access (DMA) engine, and memory between other components. The controller of a traditional non-volatile memory device is not exposed to the host, and the non-volatile memory device cannot be directly addressed.

[0015] The configurations disclosed herein allow non-volatile memory devices to be directly addressed by the host and other devices. In some examples, a controller memory buffer (CMB), subsystem level memory (SLM), or persistent memory region (PMR) can be implemented to allow non-volatile memory devices to be directly addressed by the host and other devices. Such implementations allow for improved autonomy of non-volatile memory devices and reduce some of the burden on the host's CPU in managing non-volatile memory. The CPU cost on the host side for managing non-volatile memory devices can be reduced accordingly, and bottlenecks in the data path can be reduced.

[0016] In some configurations, a host processor (e.g., CPU) configured by device driver code running on it issues commands to at least one of a plurality of non-volatile memory devices. The commands include device content information (e.g., a set of device contents) instructing one or more of the plurality of non-volatile memory devices on how to operate the other plurality of non-volatile memory devices. In some examples, the device content information may specify one or more methods for contacting at least one of the plurality of non-volatile memory devices. The plurality of non-volatile memory devices may be part of a RAID protection scheme.

[0017] Figure 1 shows a block diagram of a system including non-volatile memory devices 100a, 100b, ..., 100n (collectively referred to as non-volatile memory devices 100) coupled to a host 101, according to some examples. The host 101 can be a user device operated by a user or an autonomous central controller of the non-volatile memory devices 100, wherein the host 101 and the non-volatile memory devices 100 correspond to a storage subsystem or storage device. Examples of such storage subsystems or devices include all-flash array (AFA) or network-attached storage (NAS) devices. As shown, the host 101 includes memory 102, a processor 104, and a bus 106. The processor 104 is operatively coupled to the memory 102. The processor 104 and the memory 102 are operatively coupled to the bus 106 via a root complex 105. Processor 104 is sometimes referred to as the CPU of host 101 and is configured to perform the processing of host 101 as described herein.

[0018] Memory 102 is local memory of host 101. In some examples, memory 102 is or includes a buffer, sometimes referred to as a host buffer. In some examples, memory 102 includes at least one of volatile memory or non-volatile persistent memory. Examples of memory 102 include, but are not limited to, random access memory (RAM), DRAM, static RAM (SRAM), magnetic RAM (MRAM), phase change memory (PCM), and so on.

[0019] Bus 106 includes one or more software, firmware, and hardware that allow processor 104, network card, storage device, memory 102, graphics card, non-volatile memory device 100, etc., to communicate with each other. In some examples, non-volatile memory device 100 is directly attached to or communicatively coupled to bus 106. Bus 106 is one or more of a serial, PCIe bus or network, internal switch (e.g., PCIe switch), etc.

[0020] In some examples, root complex 105 (e.g., PCIe root complex) includes one or more software, firmware, and hardware components that connect memory 102 and processor 104 to bus 106. In this regard, root complex 105 may include at least one of one or more controllers, one or more physical connectors, one or more data transfer protocols including namespaces, one or more ports, one or more switches, one or more bridges, one or more transport mechanisms, their connectivity, etc. Root complex 105 can establish transaction requests for operational tasks of processor 104 and send them via bus 106 to non-volatile memory device 100 based on the address of non-volatile memory device 100 on bus 106. In some examples, root complex 105 may be implemented on the hardware (e.g., chip) of processor 104. In some examples, root complex 105 and bus 106 can be collectively referred to as interface 140 between host processor 104 / memory 102 and non-volatile memory device 100.

[0021] During startup, root compound 105 scans bus 106 to obtain any attached devices (e.g., physically connected or connected via a network such as a network structure) and to obtain the device addresses of non-volatile memory device 100, processor 104, and memory 102. In some examples, root compound 105 also scans bus 106 to obtain buffer 112 on non-volatile memory device 100. Non-volatile memory device 100, buffer 112, and memory 102 are each allocated an address space within the logical address space of processor 104. In some examples, SLM and PMR namespaces can be used to address buffer 112. Therefore, processor 104 can use the logical address space to perform operations such as read and write. Thus, the address of buffer 112 is exposed to processor 104 and non-volatile memory device 100. Other methods for exposing the address of buffer 112 can be implemented similarly, such as memory mapping (e.g., memory-mapped input / output (I / O) space). Memory-mapped I / O space allows any memory coupled to bus 106 to be mapped to an address recognizable by processor 104.

[0022] Processor 104 can execute an operating system (OS) that provides a file system and applications that use the file system. Processor 104 can communicate with non-volatile memory devices 100 (e.g., controllers 110 of each of the non-volatile memory devices 100) via a communication link or network. In this regard, processor 104 can use interface 140 to the communication link or network to send data to and receive data from one or more non-volatile memory devices 100. Although the connection between host 101 and non-volatile memory devices 100 is shown as a direct link, in some implementations, the link may include a network structure that may include network components such as bridges and switches.

[0023] To send and receive data, processor 104 (and the software or file system running thereon) uses a storage data transfer protocol running on interface 140 to communicate with non-volatile memory device 100. Examples of protocols include, but are not limited to, SAS, Serial ATA (SATA) and NVMe protocols. In some examples, interface 140 includes hardware (e.g., a controller) implemented on or operatively coupled to bus 106, non-volatile memory device 100 (e.g., controller 110), or another device operatively coupled to bus 106 and / or non-volatile memory device 100 via one or more suitable networks. Interface 140 and the storage protocols running thereon also include software and / or firmware executing on such hardware.

[0024] In some examples, processor 104 can communicate with a communication network via bus 106 and a network interface. Other host systems or non-volatile memory devices attached to or communicatively coupled to the communication network can communicate with host 101 using suitable network storage protocols, examples of which include, but are not limited to, NVMe over Fabrics (NVMeoF), Internet Small Computer System Interface (iSCSI), Fibre Channel (FC), Network File System (NFS), Server Message Block (SMB), etc. The network interface allows software running on processor 104 (e.g., storage protocols or file systems) to communicate with external hosts attached to the communication network via bus 106. In this way, network storage commands can be issued by external hosts and processed by processor 104, which can issue storage commands to non-volatile memory device 100 as needed. Therefore, data can be exchanged between the external host and the non-volatile memory device 100 via a communication network.

[0025] In some examples, the non-volatile memory device 100 is located in a data center (not shown for simplicity). The data center may include one or more platforms or rack units, each supporting one or more storage devices (e.g., but not limited to, the non-volatile memory device 100). In some implementations, a host 101 forms a storage node together with the non-volatile memory device 100, and the host 101 acts as a node controller. One or more storage nodes within the platform are connected to a Top of Rack (TOR) switch. Each storage node is connected to the TOR via one or more network connections, such as Ethernet, Fiber Channel, or InfiniBand, and can communicate with each other via the TOR switch or another suitable intra-platform communication mechanism. In some implementations, the non-volatile memory device 100 may be a network-attached storage device (e.g., an Ethernet SSD) connected to a TOR switch, and the host 101 is also connected to the TOR switch and can communicate with the non-volatile memory device 100 via the TOR switch. In some implementations, at least one router may facilitate communication between non-volatile memory devices 100 in storage nodes on different platforms, racks, or cabinets via a suitable network architecture. Examples of non-volatile memory devices 100 include, but are not limited to, solid-state drives (SSDs), Ethernet-attached SSDs, non-volatile dual in-line memory modules (NVDIMMs), universal flash storage (UFS), secure digital (SD) devices, compute express link (CXL) devices, and so on.

[0026] Each of the non-volatile memory devices 100 includes at least a controller 110 and a memory array 120. For simplicity, other components of the non-volatile memory device 100 are not shown. The memory array 120 includes NAND flash memory devices 130a-130n. Each of the NAND flash memory devices 130a-130n contains one or more individual NAND flash dies, which are NVMs capable of retaining data without power. Therefore, NAND flash memory devices 130a-130n refer to a plurality of NAND flash memory devices or dies within the flash memory device 100. Each of the NAND flash memory devices 130a-130n includes one or more dies, each of which has one or more planes. Each plane has multiple blocks, and each block has multiple pages.

[0027] While NAND flash memory devices 130a-130n are shown as examples of memory array 120, other examples of non-volatile memory technologies used to implement memory array 120 include, but are not limited to, non-volatile (battery-backed) DRAM, magnetic random access memory (MRAM), phase change memory (PCM), ferro-electric RAM (FeRAM), and the like. The configuration described herein can be similarly implemented in memory systems using such memory technologies and other suitable memory technologies.

[0028] Examples of controller 110 include, but are not limited to, SSD controllers (e.g., client SSD controllers, data center SSD controllers, enterprise SSD controllers, etc.), UFS controllers, or SD controllers, etc.

[0029] Controller 110 can combine raw data storage in a plurality of NAND flash memory devices 130a-130n, such that those NAND flash memory devices 130a-130n operate logically as a single unit of storage. Controller 110 may include a processor, microcontroller, buffer memory (e.g., buffers 112, 114, and / or 116), error correction system, data encryption system, flash translation layer (FTL) and flash interface module. Such functionality can be implemented in hardware, software, and firmware, or any combination thereof. In some configurations, the software / firmware of controller 110 may be stored in memory array 120 or any other suitable computer-readable storage medium.

[0030] Controller 110 includes suitable processing and storage capabilities for performing the functions described herein, as well as other functions. As described above, controller 110 manages various features of NAND flash memory devices 130a-130n, including, but not limited to, I / O handling, reading, writing / programming, erasing, monitoring, logging, error handling, garbage collection, wear leveling, logical and physical address mapping, data protection (encryption / decryption, cyclic redundancy check (CRC)), error correction coding (ECC), data binding, and the like. Therefore, controller 110 provides visibility of NAND flash memory devices 130a-130n.

[0031] Buffer memory is a memory device located locally on and operatively coupled to controller 110. For example, buffer memory may be on-chip SRAM memory located on the chip of controller 110. In some implementations, buffer memory may be implemented using memory devices of storage device 110 external to controller 110. For example, buffer memory may be on-chip DRAM located outside the chip of controller 110. In some implementations, buffer memory may be implemented using memory devices both internal and external to controller 110 (e.g., on-chip and off-chip of controller 110). For example, buffer memory may be implemented using both internal SRAM and external DRAM, which are transparent / exposed and accessible via bus 106 by other devices, such as host 101 (with the assistance of root complex 105) and other non-volatile memory devices 100. In this example, controller 110 includes an internal processor using memory addresses within a single address space and a memory controller that controls the internal SRAM and external DRAM. Controller 110 selects whether to place data on the internal SRAM and external DRAM based on efficiency. In other words, the internal SRAM and external DRAM are addressed as if they were a single memory. Buffered memory includes at least one of buffer 112, write buffer 114, or read buffer 116.

[0032] Controller 110 includes buffer 112, sometimes referred to as a drive buffer, examples of which may be CMB, SLM, or PMR. In addition to being accessible by controller 110, buffer 112 can also be accessed via bus 106 by other devices, such as host 101 and other non-volatile memory devices 100a, 100b, ... 100n. In this way, buffer 112 (e.g., addresses of memory locations within buffer 112) is exposed between buses 106, and any device operatively coupled to bus 106 can issue commands (e.g., read commands, write commands, etc.) using addresses corresponding to memory locations within buffer 112 to read data from and write data to those memory locations within buffer 112. In some examples, buffer 112 is volatile memory. In some examples, buffer 112 is non-volatile persistent memory, which can provide improved protection against accidental power failures of one or more non-volatile memory devices 100. Examples of buffer 112 include, but are not limited to, RAM, DRAM, SRAM, MRAM, PCM, etc. As described herein, buffer 112 may refer to multiple buffers, each configured to store different types of data.

[0033] In some implementations, as shown in FIG1, buffer 112 is local memory of controller 110. For example, buffer 112 may be on-chip SRAM memory located on the chip of controller 110. In some implementations, buffer 112 may be implemented using memory devices of storage device 110 external to controller 110. For example, buffer 112 may be on-chip DRAM located outside the chip of controller 110. In some implementations, buffer 112 may be implemented using memory devices both inside and outside controller 110 (e.g., on-chip and off-chip of controller 110). For example, buffer 112 may be implemented using both internal SRAM and external DRAM, which are transparent / exposed and accessible via bus 106 by other devices 100, such as host 101 and other non-volatile memory devices 100. In this example, controller 110 includes an internal processor using memory addresses within a single address space and a memory controller that controls both internal SRAM and external DRAM, controller 110 selecting whether to place data on internal SRAM and external DRAM based on efficiency. In other words, the internal SRAM and external DRAM are addressed like a single memory module.

[0034] In an example of a write operation, in response to receiving data from host 101 (via host interface 140), controller 110 acknowledges the write command to host 101 after writing the data to write buffer 114. In some implementations, write buffer 114 may be implemented in a separate memory different from buffer 112, or write buffer 114 may be a defined region or portion of the memory containing buffer 112, wherein other devices can only access the CMB, SLM, or PMR portions of the memory, but not write buffer 114. Controller 110 may write the data stored in write buffer 114 to memory array 120 (e.g., NAND flash memory devices 130a-130n). Once the physical address of the data in memory array 120 has been written, FTL updates the mapping between the logical address (e.g., Logical Block Address (LBA)) used by host 101 to associate with the data and the physical address used by controller 110 to identify the physical location of the data. In another example regarding read operations, controller 110 includes another buffer 116 (e.g., a read buffer), different from buffers 112 and 114, to store data read from memory array 120. In some implementations, read buffer 116 may be implemented in a separate memory, different from buffer 112, or read buffer 116 may be a defined region or portion of the memory containing buffer 112, wherein other devices may only access the CMB, SLM, or PMR portions of the memory, but not the read buffer 116.

[0035] Although this document presents non-volatile memory devices (e.g., NAND flash memory devices 130a-130n) as examples, the disclosed solutions can be implemented on any storage system or device connected to host 101 via an interface, wherein such a system temporarily or continuously stores data of host 101 for later retrieval.

[0036] In some examples, the non-volatile memory devices 100 form a RAID group for parity protection. That is, one or more of the non-volatile memory devices 100 store parity data (parity bits) for the data stored on those devices and / or the data stored on other non-volatile memory devices 100.

[0037] In some implementations, the processor 104 of host 101 sends a host command to a first non-volatile memory device (e.g., non-volatile memory device 100a) in the non-volatile memory device group 100. The host command includes device content information of at least one other non-volatile memory device 100 in the group. The first non-volatile memory device may be referred to as the source device. In addition to executing the host command, the first non-volatile memory device is also responsible for executing the command on at least one other non-volatile memory device 100, encompassing all further communications within the PCIe subsystem that do not reach the processor 104 or memory 102.

[0038] Figures 2A, 2B, 2C, and 2D are schematic diagrams illustrating example copying methods 200 according to various configurations. Figure 2E is a flowchart illustrating copying method 200 according to various configurations. Figures 2A-2E show the components of the non-volatile memory device 100 and the host 101 in Figure 1; for clarity, some components are omitted in Figures 2A-2E. In some examples, method 200 involves copying data from the host between multiple non-volatile memory devices 100a-100n. The method 200 shown in Figure 2E can be performed by the non-volatile memory device 100a.

[0039] At 210, the controller 110 of the first non-volatile memory device (e.g., non-volatile memory device 100a) receives a host command from the host 101 via an interface 140 including a root complex 105 and a bus 106. In some examples, the host command is a write command (e.g., a first write command) addressing the non-volatile memory device 100a (e.g., its buffer 112). The root complex 105 can route the host command to the controller 110 of the non-volatile memory device 100a on the bus 106 using the address of the non-volatile memory device 100a (e.g., its buffer 112). The host command includes the address of memory 102. The address of memory 102 of host 101 includes a buffer address, address descriptor, identifier, index, or another suitable indicator identifying memory 102 of host 101.

[0040] The host command includes device content information, which may be a data structure including various information (e.g., the addresses of non-volatile memory devices 100b-100n) related to at least one other non-volatile memory device, referred to as the second non-volatile memory device. The second non-volatile memory device may be referred to as the source non-volatile memory device. In some examples, the device content information may include the address of the second non-volatile memory device.

[0041] In some examples, device content information may include the address of buffer 112 of a second non-volatile memory device. The address of buffer 112 may be a CMB address, SLM address, PMR address, address descriptor, identifier, index, or another suitable indicator identifying the buffer 112 of the non-volatile memory device. The address of buffer 112 of non-volatile memory device 100 is stored in a shared address register (e.g., a shared PCIe base address register) known to host 101 through root complex 105. The address can be used to send data, instructions, commands, etc., on bus 106. For example, in NVMe, the CMB is defined by the NVMe controller memory buffer location (CMBLOC) and the controller memory buffer size (CMBSZ), the former defining the starting PCI address location of the CMB and the latter defining the size of the CMB. SLM and PMR addresses can be implemented similarly. These controller registers can reside in root complex 105, bus 106, or another suitable entity coupled to root complex 105.

[0042] In some examples, the device content information may include permission information for at least one of the second non-volatile memory devices or their buffers 112. The permission information for at least one of the second non-volatile memory devices or their buffers 112 includes information relating to whether or how the first non-volatile memory device may access at least one of the second non-volatile memory devices or their buffers 112. For example, the permission information may include one or more types of operations or tasks (e.g., read, write, copy, spread, partition, reduce, etc.) for access to at least one of the second non-volatile memory devices or their buffers 112. This permission may include authentication credentials for access to at least one of the second non-volatile memory devices or their buffers 112, such that after the first non-volatile memory device provides authentication credentials (along with any commands or instructions described herein) to at least one of the second non-volatile memory devices or their buffers 112, the second non-volatile memory device may perform an operation according to the command or instruction. In some examples, security tokens may be used to implement authentication credentials. Security tokens may be issued separately for different operations and tasks. In some examples, authentication credentials can be implemented using access control lists (e.g., whitelists, blacklists, or a combination of both) that specify whether a first non-volatile memory device can access a second non-volatile memory device and whether the first non-volatile memory device can access the second non-volatile memory device for certain operations and tasks.

[0043] In some examples, the device content information may include priority information for processing data associated with a host command by the second non-volatile memory device. For example, the priority information may indicate a priority level associated with the data, such that if the data corresponding to the host command has a higher priority level than another data (e.g., the data associated with the host command is processed faster), the second non-volatile memory device may process the data corresponding to the host command before processing the other data. If the data corresponding to the host command has a lower priority level than another data (e.g., the data associated with the host command is processed slower), the second non-volatile memory device may process the other data before processing the data corresponding to the host command. In some examples, the priority information may indicate a priority level for communication between the first and second non-volatile memory devices regarding data related to a host command. For example, if the data corresponding to a host command has a higher priority level than another data (e.g., the data associated with the host command is processed faster), the first non-volatile memory device can prioritize processing the data corresponding to the host command (e.g., sending a peer command as described herein) before processing the other data (e.g., sending a peer command for the other data). If the data corresponding to the host command has a lower priority level than another data (e.g., the data associated with the host command is processed slower), the first non-volatile memory device can prioritize processing the other data before processing the data corresponding to the host command.

[0044] At 220, the controller 110 of the non-volatile memory device 100a transfers data corresponding to a host command from the host to the buffer 112 of the non-volatile memory device 100a via interface 140. For example, the controller 110 of the non-volatile memory device 100a can perform a DMA transfer by issuing a DMA request to the host 101 to transfer data stored in memory 102 (e.g., a host buffer) of the host 101 to the buffer 112, wherein the DMA request includes the address of memory 102. The PCI root complex 105 can use the address of memory 102 to route the DMA request to memory 102 on bus 106.

[0045] At 230, the controller 110 of the non-volatile memory device 100a writes data from the buffer 112 of the non-volatile memory device 100a into its non-volatile memory (e.g., memory array 120). For example, the controller 110 of the non-volatile memory device 100a issues an internal command to write the buffer 112 into its own flash media (e.g., memory array 120). In some examples, the non-volatile memory device 100a writes data into the memory array 120 while other non-volatile memory devices 100b-100n write data into their respective memory arrays 120, for example at 260. In some examples, 230 may occur immediately in response to 220, simultaneously with 240, after 240, simultaneously with 250, or after 250. In some examples, 230 may occur at any time before data is deleted or erased from the buffer 112.

[0046] At 240, the controller 110 of non-volatile memory device 100a sends a peer command via bus 106 to a second non-volatile memory device (e.g., non-volatile memory devices 100b-100n) based on device content information.

[0047] For example, the controller 110 of non-volatile memory device 100a can send a peer command to the controller 110 of each of non-volatile memory devices 100b to 100n based on (for example, in) the address of each of the non-volatile memory devices 100b to 100n contained in the device content information received from host 101 at 210.

[0048] For example, the controller 110 of the non-volatile memory device 100a can determine, based on permission information, whether the non-volatile memory device 100a is authorized to send peer commands to each of the non-volatile memory devices 100b-100n or at least one of their buffers 112 for the type of operation or task specified in the host command (e.g., write or copy). For example, the controller 110 of the non-volatile memory device 100a can use authentication credentials to authenticate each of the non-volatile memory devices 100b-100n or at least one of their buffers.

[0049] For example, the controller 110 of non-volatile memory device 100a can determine the priority level for sending peer commands to each of the non-volatile memory devices 100b-100n based on priority information. The controller 110 of non-volatile memory device 100a processes data or host commands associated with a higher priority level before processing data or host commands associated with a lower priority level. For example, a peer command may include a priority level for processing data associated with both the peer command and the host command. Each of the non-volatile memory devices 100b-100n may process data or peer commands associated with a higher priority level before processing data or peer commands associated with a lower priority level. The priority level contained in the peer command may correspond to the priority level contained in the content information of the host command.

[0050] The peer command includes another write command (e.g., a second write command). In some examples where a group of non-volatile memory devices (e.g., RAID-1 devices, such as non-volatile memory devices 100a~100n) comprises n devices, the controller 110 of non-volatile memory device 100a issues n-1 peer commands to n-1 non-volatile memory devices 100b~100n. In some examples, the peer command includes the address of data from which a second non-volatile memory device (e.g., non-volatile memory devices 100b~100n) can transfer data. The address of the data includes the address of the buffer 112 of the non-volatile memory device 100a, so the second non-volatile memory device (e.g., non-volatile memory devices 100b~100n) can use the address of the non-volatile memory device 100a to transfer data from the buffer 112 of the non-volatile memory device 100a, for example, at 260.

[0051] At 250, the controller 110 of non-volatile memory device 100a participates in transferring data from buffer 112 of non-volatile memory device 100a to buffer 112 of a second non-volatile memory device via bus 106. For example, the controller 110 of non-volatile memory device 100a exposes its buffer 112 (e.g., buffer address) to bus 106 (e.g., to any device such as non-volatile memory devices 100b-100n coupled to or connected to bus 106). The controller 110 of the second non-volatile memory device (e.g., each of non-volatile memory devices 100b-100n) performs DMA transfer by issuing a DMA request to the controller 110 using the address of buffer 112 of non-volatile memory device 100a. In response, the controller 110 of each of the non-volatile memory devices 100b to 100n transmits the data stored in the buffer 112 of the non-volatile memory device 100a to the buffer 112 of each of the non-volatile memory devices 100b to 100n.

[0052] At 260, the controller 110 of the second non-volatile memory device (e.g., each of non-volatile memory devices 100b-100n) writes data from the buffer 112 of the second non-volatile memory device into the non-volatile memory (e.g., memory array 120) of the second non-volatile memory device. For example, the controller 110 of the second non-volatile memory device issues an internal command to write the buffer 112 into its own flash medium (e.g., memory array 120). In some examples, the non-volatile memory devices 100b-100n simultaneously write data into their respective memory arrays 120 in parallel. In some examples, in response to the controller 110 of non-volatile memory device 100a receiving a write completion acknowledgment from each of the non-volatile memory devices 100b-100n, the controller 110 of non-volatile memory device 100a sends a write completion message to the processor 104 via interface 140.

[0053] Therefore, after steps 210 and 220, in order to copy and transfer data to non-volatile memory devices 100b-100n, no requests, commands, or data pass through root compound 105 to reach the processor 104 and memory 102 of host 101. Because root compound 105 does not involve providing an interface to bus 106 for processor 104 and memory 102, the routing of requests and data between non-volatile memory devices 100 can be performed via bus 106. Method 200 can reduce the workload of processor 104 and memory 102 by delegating repetitive operations between a group of devices 100. The amount of data transferred through root compound 105 can be reduced. The load on system memory 102 can also be reduced.

[0054] In some examples, the non-volatile memory device 100 may be an SSD. In some examples, at least one or two or more of the non-volatile memory devices 100 may be NVMeoF devices, which are attached to bus 106 via communication networks such as NVMeoF, iSCSI, FC, NFS, SMB, etc. NVMeoF devices can have peer-to-peer data transfer capabilities on the NVMeoF target side without needing to route communication through root complex 105.

[0055] Figures 3A, 3B, 3C, 3D, 3E, and 3F are schematic diagrams illustrating example propagation methods 300 according to various configurations. Figure 3G is a flowchart illustrating propagation methods 300 according to various configurations. Figures 3A-3G show the components of the non-volatile memory device 100 and the host 101 in Figure 1; some components are omitted for clarity. In some examples, method 300 involves a propagation operation where multiple non-volatile memory devices 100a-100n are part of a RAID-5 protection scheme and are part of a stripe spanning between non-volatile memory devices 100a-100n. A write update updates a portion of the stripe entirely located within the memory array 120 of the non-volatile memory device 100a. Method 300 shown in Figure 3G can be performed by the non-volatile memory device 100a.

[0056] At 310, the controller 110 of the first non-volatile memory device (e.g., non-volatile memory device 100a) receives a host command from the host 101 via an interface 140 including a root complex 105 and a bus 106. In some examples, the host command is a write command (e.g., a first write command) addressing the non-volatile memory device 100a (e.g., its buffer 112). The root complex 105 can route the host command to the controller 110 of the non-volatile memory device 100a on the bus 106 using the address of the non-volatile memory device 100a (e.g., its buffer 112). The host command includes the address of memory 102. The address of memory 102 of host 101 includes a buffer address, address descriptor, identifier, index, or another suitable indicator identifying memory 102 of host 101.

[0057] In some examples, the host command includes device content information. In some examples, the device content information includes the address of at least one other non-volatile memory device in the RAID group (e.g., the address of a parity check device, the address of non-volatile memory device 100n). The address of the non-volatile memory device may be a CMB address, SLM address, PMR address, address descriptor, identifier, index, or another suitable indicator identifying the buffer 112 of the non-volatile memory device, as described. In some examples, the host command includes the logical address of the data to be updated. In some examples, the device content information may include permission information for a second non-volatile memory device or at least one of its buffers 112. In some examples, the device content information may include priority information, which includes at least one of a priority level for processing data associated with the host command by the second non-volatile memory device or a priority level for communication between the first and second non-volatile memory devices regarding data related to the host command.

[0058] At 320, the controller 110 of the non-volatile memory device 100a transfers new data corresponding to a host command from the host to the buffer 112 of the non-volatile memory device 100a via interface 140. For example, the controller 110 of the non-volatile memory device 100a can perform a DMA transfer by issuing a DMA request to the host 101 to transfer new data stored in memory 102 (e.g., a host buffer) of the host 101 to the buffer 112, wherein the DMA request includes the address of memory 102. The PCI root complex 105 can use the address of memory 102 to route the DMA request to memory 102 on bus 106.

[0059] At 330, controller 110 reads the old data corresponding to the logical address contained in the host command from the non-volatile memory (e.g., memory array 120) of non-volatile memory device 100a into buffer 112. For example, controller 110 may determine the physical address mapped to the logical address through a suitable logical-to-physical (L2P) dialogue, and read the physical address at the non-volatile memory to obtain the old data and transfer it to buffer 112. At 340, controller 110 calculates the resulting data based on at least one of the new data and the old data. For example, the resulting data may be a bitwise XOR of the old data and the new data. Controller 110 stores the resulting data in buffer 112 of non-volatile memory device 100a.

[0060] This result data is sent to a member device (e.g., non-volatile memory device 100n) used to store the parity check of the stripe. At 350, the controller 110 of non-volatile memory device 100a sends a peer command to a second non-volatile memory device (e.g., non-volatile memory device 100n) via bus 106 based on device content information. The second non-volatile memory device may be a parity check device storing parity bits. For example, the controller 110 of non-volatile memory device 100a sends a peer command to the controller 110 of non-volatile memory device 100n based on (e.g., in) the address of non-volatile memory device 100n contained in device content information received from host 101. Bus 106 may route the peer command to the controller 110 of non-volatile memory device 100n using the address of non-volatile memory device 100n (e.g., its buffer 112).

[0061] For example, the controller 110 of the non-volatile memory device 100a can determine, based on permission information, whether the non-volatile memory device 100a is authorized to send peer commands to the non-volatile memory device 100n or its buffer 112 for the operation or task type (e.g., write or propagate) specified in the host command. For example, the controller 110 of the non-volatile memory device 100a can use authentication credentials to authenticate at least one of the non-volatile memory device 100n or its buffer.

[0062] For example, the controller 110 of the non-volatile memory device 100a can determine the priority level for sending peer commands to the non-volatile memory device 100n based on priority information. The controller 110 of the non-volatile memory device 100a processes data or host commands associated with a higher priority level before processing data or host commands associated with a lower priority level. For example, a peer command may include a priority level for processing data associated with both the peer command and the host command. The non-volatile memory device 100n may process data or peer commands associated with a higher priority level before processing data or peer commands associated with a lower priority level. The priority level contained in the peer command may correspond to the priority level contained in the content information of the host command.

[0063] The peer command includes another write command (e.g., a second write command). In some examples, the peer command includes the address of the resulting data, from which a second non-volatile memory device (e.g., non-volatile memory device 100n) can transfer the resulting data. The address of the data includes the address of buffer 112 of non-volatile memory device 100a, so the second non-volatile memory device can use its address to transfer the resulting data from buffer 112 of non-volatile memory device 100a, for example, at 360°.

[0064] At 360, the controller 110 of non-volatile memory device 100a participates in transferring data via bus 106 from buffer 112 of non-volatile memory device 100a to buffer 112 of a second non-volatile memory device. For example, the controller 110 of non-volatile memory device 100a exposes its buffer 112 (e.g., buffer address) to bus 106 (e.g., to any device such as non-volatile memory device 100n coupled to or connected to bus 106). The controller 110 of the second non-volatile memory device (e.g., non-volatile memory device 100n) can perform DMA transfers by issuing a DMA request to the controller 110 of non-volatile memory device 100a using the address of buffer 112 of non-volatile memory device 100a. In response, the controller 110 of the non-volatile memory device 100n transmits the data stored in the buffer 112 of the non-volatile memory device 100a to the buffer 112 of the non-volatile memory device 100n.

[0065] Parity is updated by bitwise mutually exclusive OR of the resulting data and the old parity data. For example, at 370, the controller 110 of the non-volatile memory device 100n reads the old parity data from the non-volatile memory (e.g., memory array 120) of the non-volatile memory device 100n into the buffer 112 of the non-volatile memory device 100n. In some examples, the second write command includes a logical address associated with the new data, and the controller 110 of the non-volatile memory device 100n determines the physical address mapped to the logical address through a suitable L2P dialog, reads the physical address located in the non-volatile memory to obtain the old parity data and transfers it to the buffer 112 of the non-volatile memory device 100n.

[0066] At 380, controller 110 calculates new parity data based on at least one of the result data and the old parity data. For example, the new parity data may be a bitwise mutually exclusive OR of the old parity data and the result data. Controller 110 of non-volatile memory device 100n stores the result data in buffer 112 of non-volatile memory device 100n.

[0067] At 390, the controller 110 of the second non-volatile memory device (e.g., non-volatile memory device 100n) writes new parity data from the buffer 112 of the second non-volatile memory device into the non-volatile memory (e.g., memory array 120) of the second non-volatile memory device. For example, the controller 110 of the second non-volatile memory device issues an internal command to write the buffer 112 into its own flash medium (e.g., memory array 120).

[0068] At 395, the controller 110 of the non-volatile memory device 100a writes new data from the buffer 112 of the non-volatile memory device 100a into the non-volatile memory (e.g., memory array 120) of the non-volatile memory device 100a. For example, the controller 110 of the non-volatile memory device 100a issues an internal command to write the buffer 112 into its own flash memory medium (e.g., memory array 120).

[0069] In some examples, the non-volatile memory device 100a writes new data to the memory array 120 at 395, which is parallel to one or more of 360, 370, 380, or 390. In some examples, 395 may occur immediately in response to 320, simultaneously with 330, after 330, simultaneously with 340, or after 340. In some examples, 395 may occur at any time before the new data is deleted or erased from the buffer 112 of the non-volatile memory device 100a.

[0070] In some examples, method 300 can be repeated for at least one additional device, such as a third non-volatile memory device, which can be one of non-volatile memory devices 100b to 100n-1. In this respect, the second non-volatile memory device becomes the first non-volatile memory device, and the third non-volatile memory device becomes the second non-volatile memory device. Method 300 is repeated at 330, where new data is calculated at 380. Method 300 can be repeated when the last device in the group (e.g., a RAID group) is reached.

[0071] In some examples, in response to the controller 110 of non-volatile memory device 100a receiving a write completion acknowledgment from the last non-volatile memory device in the propagation chain, the controller 110 of non-volatile memory device 100a sends a write completion message to the processor 104 via interface 140. Therefore, after 310 and 320, for the transmission of data in propagation, no requests, commands, or data pass through root compound 105 to reach the processor 104 and memory 102 of host 101. Because root compound 105 does not involve providing an interface to bus 106 for processor 104 and memory 102, the routing of requests and data between non-volatile memory devices 100 can be performed via bus 106. Method 300 can reduce the workload of processor 104 and memory 102 through offloading aggregation operations between a group of devices 100. The amount of data transferred through root compound 105 can be reduced. The load on system memory 102 can also be reduced.

[0072] Figures 4A, 4B, and 4C are schematic diagrams illustrating example partitioning methods 400 according to various configurations. Figure 4D is a flowchart illustrating partitioning method 400 according to various configurations. Figures 4A-4D show the components of the non-volatile memory device 100 and host 101 in Figure 1; some components are omitted for clarity. In some examples, method 400 involves a partitioning mechanism in which a non-volatile memory device (e.g., non-volatile memory device 100a) among multiple non-volatile memory devices 100a-100n partitions host data among the multiple non-volatile memory devices 100a-100n. The multiple non-volatile memory devices 100a-100n are part of a RAID-5 protection scheme and are part of a stripe spanning the non-volatile memory devices 100a-100n. Non-volatile memory device 100n is a parity check device that stores parity bits for data in stripes stored in non-volatile memory devices 100a-100n-1. The method 400 shown in FIG4D can be performed by non-volatile memory device 100a. This operation includes transferring host data from buffers 401a, 401b, ..., 401n-1 of memory 102 to non-volatile memory devices 100a-100n-1. Method 400 can reduce the load on processor 104 by allowing one execution partitioning task of non-volatile memory devices 100a-100n-1.

[0073] At 410, the controller 110 of the first non-volatile memory device (e.g., non-volatile memory device 100a) receives a host command from the host 101 via an interface 140 including a root complex 105 and a bus 106. In some examples, the host command is a write command addressing the non-volatile memory device 100a (e.g., its buffer 112). The root complex 105 can route the host command to the controller 110 of the non-volatile memory device 100a on the bus 106 using the address of the non-volatile memory device 100a. The host command includes the addresses of buffers 401a, 401b, ..., 401n-1 of memory 102, which correspond to the work to be performed.

[0074] In some examples, the host command includes device content information for a plurality of non-volatile memory devices. In some examples, the device content information includes the addresses of non-volatile memory devices 100a to 100n-1 in the RAID group. The addresses of the non-volatile memory devices may be, as described, CMB addresses, SLM addresses, PMR addresses, address descriptors, identifiers, indicators, or another suitable indicator identifying the buffer 112 of the non-volatile memory device. In some examples, the host command includes the logical address of the data to be updated, which includes a logical address corresponding to each of the buffers 401a to 401n-1 (e.g., a buffer address). In some examples, the device content information may include permission information for a plurality of non-volatile memory devices or at least one of their buffers 112. In some examples, the device content information may include priority information, which includes at least one of a priority level for processing data associated with a host command by each of a plurality of non-volatile memory devices or a priority level for communication between a first non-volatile memory device and each of the plurality of non-volatile memory devices regarding data related to a host command.

[0075] At 420, the controller 110 of the first non-volatile memory device (e.g., non-volatile memory device 100a) partitions the host data in the non-volatile memory device and issues a transfer request to transfer the host data to the buffer 112 of the non-volatile memory device. The non-volatile memory device may include non-parity non-volatile memory devices 100a~100n-1 in a RAID group.

[0076] In some examples, the controller 110 of the first non-volatile memory device maps each of the addresses of buffers 401a, 401b, ..., 401n-1 of memory 102 to one of the non-parity non-volatile memory devices 100a to 100n-1. In examples where the number of buffers 401a, 401b, ..., 401n-1 is the same as the number of non-parity non-volatile memory devices 100a to 100n-1, the controller 110 of the first non-volatile memory device maps the addresses of buffers 401a, 401b, ..., 401n-1 of memory 102 to the non-parity non-volatile memory devices 100a to 100n-1 according to a one-to-one relationship. For example, the addresses of buffers 401a, 401b, ..., 401n-1 of memory 102 can be mapped to non-parity non-volatile memory devices 100a to 100n-1 according to the increasing order of buffer addresses or IDs and the order of non-parity non-volatile memory devices 100a to 100n-1 in the RAID group. For example, buffer 401a is mapped to non-volatile memory device 100a, buffer 401b is mapped to non-volatile memory device 100b, ..., buffer 401n-1 is mapped to non-volatile memory device 100n-1, and so on. In some examples, the addresses of buffers 401a, 401b, ..., 401n-1 of memory 102 can be mapped to non-parity non-volatile memory devices 100a to 100n-1 according to the descending order of buffer addresses or IDs and the order of non-parity non-volatile memory devices 100a to 100n-1 in the RAID group. For example, buffer 401a is mapped to non-volatile memory device 100n-1, buffer 401b is mapped to non-volatile memory device 100n-2, ..., buffer 401n-1 is mapped to non-volatile memory device 100a, and so on.

[0077] In an example where the number of buffers 401a, 401b, ..., 401n-1 is less than the number of non-parity non-volatile memory devices 100a~100n-1, the controller 110 of the first non-volatile memory device maps the address of the buffer of memory 102 to the non-parity non-volatile memory device based on the incrementing or decrementing order of the buffer address or ID and the order of the non-parity non-volatile memory devices 100a~100n-1 in the RAID group. However, not all of the non-parity non-volatile memory devices 100a~100n-1 in the RAID group are mapped to one of the buffers 401a, 401b, ..., 401n-1.

[0078] In an example where the number of buffers 401a, 401b, ..., 401n-1 is greater than the number of non-parity non-volatile memory devices 100a~100n-1, the controller 110 of the first non-volatile memory device maps the address of the buffer of memory 102 to the non-parity non-volatile memory device based on the incrementing or decrementing order of the buffer address or ID and the order of the non-parity non-volatile memory devices 100a~100n-1 in the RAID group. One or more of the non-parity non-volatile memory devices 100a~100n-1 in the RAID group are mapped to two or more of the buffers 401a, 401b, ..., 401n-1. This mapping can be performed cyclically. For example, after mapping each of the non-volatile memory devices 100a to 100n-1 in the RAID group to a buffer, the next unmapped buffer is mapped to non-volatile memory device 100a, and so on to 100b, 100c.

[0079] The controller 110 of the non-volatile memory device 100a issues a transmission request based on partitioning to transmit data corresponding to host data to the buffer 112 of the non-parity non-volatile memory device. For example, non-volatile memory device 100a may issue a first DMA request to host 101 (e.g., to memory 102) to transfer data stored in buffer 401a to buffer 112 of the mapped non-volatile memory device (e.g., non-volatile memory device 100a), issue a second DMA request to host 101 (e.g., to memory 102) to transfer data stored in buffer 401b to buffer 112 of the mapped non-volatile memory device (e.g., non-volatile memory device 100b), ..., issue a (n-1)th DMA request to host 101 (e.g., to memory 102) to transfer data stored in buffer 401n-1 to buffer 112 of the mapped non-volatile memory device (e.g., non-volatile memory device 100n-1). Each transfer request includes the address of one of buffers 401a to 401n-1 and the address of the corresponding non-volatile memory device buffer 112. Data from buffers 401a to 401n-1 can be transferred to buffer 112 accordingly based on the DMA request.

[0080] At 430, the controller 110 of non-volatile memory device 100a sends peer commands via bus 106 to a plurality of non-volatile memory devices (e.g., each of non-volatile memory devices 100b to 100n-1 other than non-volatile memory device 100a) based on device content information. For example, the controller 110 of non-volatile memory device 100a sends peer commands to the controller 110 of each of non-volatile memory devices 100b to 100n-1 according to (e.g., in) the address of each of non-volatile memory devices 100b to 100n-1 contained in the device content information received from host 101. Bus 106 can use the address of each of the non-volatile memory devices 100b to 100n-1 (e.g., their buffers 112) to route peer commands to the controller 110 of each of the non-volatile memory devices 100b to 100n-1.

[0081] For example, the controller 110 of the non-volatile memory device 100a can determine, based on permission information, whether the non-volatile memory device 100a is authorized to send peer commands to each of the non-volatile memory devices 100b to 100n-1 or their buffers 112 for the operation or task type (e.g., write) specified in the host command. For example, the controller 110 of the non-volatile memory device 100a can authenticate each of the non-volatile memory devices 100b to 100n-1 or their buffers using authentication credentials.

[0082] For example, the controller 110 of the non-volatile memory device 100a can determine the priority level for sending peer commands to each of the non-volatile memory devices 100b to 100n-1 based on priority information. The controller 110 of the non-volatile memory device 100a processes data or host commands associated with a higher priority level before processing data or host commands associated with a lower priority level. For example, a peer command may include a priority level for processing data associated with both the peer command and the host command. Each of the non-volatile memory devices 100b to 100n-1 may process data or peer commands associated with a higher priority level before processing data or peer commands associated with a lower priority level. The priority level contained in the peer command may correspond to the priority level contained in the content information of the host command.

[0083] The peer command includes another write command (e.g., a second write command) instructing each of the non-volatile memory devices 100b-100n-1 to write data stored in buffer 112 corresponding to the corresponding one of buffers 401a-401n-1 to the memory array 120. The peer command may include the address of buffer 112 of the corresponding one of the non-volatile memory devices 100b-100n-1.

[0084] At 440, the controllers 110 of a plurality of non-volatile memory devices 100a to 100n-1 write corresponding data stored in buffers 112 to memory array 120. For example, the controller 110 of each of the non-volatile memory devices 100a to 100n-1 issues an internal command to write its buffer 112 to its own flash medium (e.g., memory array 120). In some examples, the non-volatile memory devices 100a to 100n-1 write data to memory array 120 in parallel at 440.

[0085] In some examples, a plurality of non-volatile memory devices can be grouped together, and a boot device can be designated. In some examples, the grouped plurality of non-volatile memory devices correspond to a group of non-volatile memory devices 100a~100n that are part of the same RAID group. In some examples, the grouped plurality of non-volatile memory devices correspond to a group of non-parity non-volatile memory devices 100a~100n-1 that are part of the same RAID group. The boot device is non-volatile memory device 100a in method 400, as shown in Figures 4A~4D.

[0086] In some examples, the grouping of a plurality of non-volatile memory devices can be determined according to any other method described herein. Figure 5A is a flowchart illustrating an example method 500 for group-guided computation according to various configurations. Figure 5B is a diagram illustrating the grouping of non-volatile memory devices according to various configurations. In Figure 5B, each non-volatile memory device 501a, 501b, 501c, 501d, 501e, 501f, 501g, and 501h can be a non-volatile memory device such as one of non-volatile memory devices 100a, 100b, and 100n. For example, each of the non-volatile memory devices 501a, 501b, 501c, 501d, 501e, 501f, 501g, and 501h may include a controller 110, a buffer 112, and a memory array 120. Method 500 can be performed by a controller 110, processor 104, or another suitable entity with processing capability of one or more of the non-volatile memory devices 501a, 501b, 501c, 501d, 501e, 501f, 501g, and 501h.

[0087] At 510, a plurality of non-volatile memory devices are grouped into at least one group (e.g., at least one first group). For example, n non-volatile memory devices, which may be a portion of a RAID group or a RAID group of non-parity non-volatile memory devices, can be grouped according to an identifier (e.g., 1 to n-1) assigned to each non-volatile memory device. For example, non-volatile memory device 501a is assigned identifier 1, non-volatile memory device 501b is assigned identifier 2, non-volatile memory device 501c is assigned identifier 3, non-volatile memory device 501d is assigned identifier 4, non-volatile memory device 501e is assigned identifier 5, non-volatile memory device 501f is assigned identifier 6, non-volatile memory device 501g is assigned identifier 7, and non-volatile memory device 501h is assigned identifier 8. Starting with identifier 1, a predetermined number (e.g., 2) of adjacent identifiers are grouped into the same group. For example, non-volatile memory devices 501a and 501b are grouped into group 502a. Non-volatile memory devices 501c and 501d are grouped into group 502b. Non-volatile memory devices 501e and 501f are grouped into group 502c. Non-volatile memory devices 501g and 501h are grouped into group 502d. These groups may include any other number (e.g., 3 or more) of non-volatile memory devices.

[0088] At 520, it is determined whether the number of total groups (e.g., at least one first group) is a non-zero number greater than 1. As shown in Figure 5B, the number (4) of groups 502a, 502b, 502c, and 502d is greater than 1. At 530, in response to determining that the number of total groups is greater than 1 (520: yes), a guide device (e.g., a first guide device) (e.g., a first non-volatile memory device from each of at least one first group) is selected for each group. In some examples, a device with an odd identifier (e.g., 1, 3, 5, and 7) in each of groups 502a, 502b, 502c, and 502d is selected as the guide. Therefore, non-volatile memory devices 501a, 501c, 501e, and 501g are selected for groups 502a, 502b, 502c, and 502d, respectively. In some examples, a device with an even identifier in each group can be selected as the guide. In some examples, a device with the smallest identifier in each group can be selected as the guide. In some examples, the device with the largest identifier in each group can be selected as the leader.

[0089] At 540, the boot device determines result data by performing at least one operation on the data in each group. For example, the controller 110 of the first boot device for each of at least one first group determines the first result data by performing an operation based on the first data from each of the non-volatile memory devices in each of at least one first group.

[0090] For example, the buffer of the first boot device (e.g., buffer 112) receives first data from at least one of the first non-volatile memory devices in each of at least one of the first groups. The controller 110 of the first boot device 501a determines the first result data by calculating the first result data based on the data stored in the buffer 112 of the first boot device and the first data from at least one of the first groups of non-volatile memory devices.

[0091] For example, the controller 110 of the first boot device 501a in group 502a performs a DMA transfer by issuing a DMA request to the controller 110 of the first non-volatile memory device 501b in group 502a using the address of the buffer 112 of the first non-volatile memory device 501b. Data at the address of the buffer 112 of the first non-volatile memory device 501b can be read from the memory array 120 of the first non-volatile memory device 501b into the buffer 112 of the first non-volatile memory device 501b, or can be received from the memory 102 by the buffer 112 of the first non-volatile memory device 501b through interface 140. In response, the controller 110 of the first boot device 501a transfers data stored in the buffer 112 of the first non-volatile memory device 501b to the buffer 112 of the first boot device 501a. The controller 110 of the first boot device 501a can read data stored in the non-volatile memory (e.g., memory array 120) of the first boot device 501a into the buffer 112 of the first boot device 501a. The controller 110 of the first boot device 501a can use the data from the first boot device 501a and first data from the first non-volatile memory device 501b to perform an operation to determine first result data. This operation includes a mutually exclusive OR (XOR) operation, wherein the data from the first boot device 501a and the first data from the first non-volatile memory device 501b are mutually exclusive ORed bit by bit. The first result data is stored in the buffer 112 of the first boot device 501a. The result data of groups 502b, 502c, and 502d can similarly be determined by boot devices 501c, 501e, and 501g, respectively.

[0092] In an example where there are three or more non-volatile memory devices in the same group, the first result data can be mutually exclusive ORed with data from another non-volatile memory device in the same group to determine the updated result data, and so on, which can then be mutually exclusive ORed with data from another non-volatile memory device in the same group, if any.

[0093] At 550, the guiding devices from at least one group are further grouped. For example, the first guiding devices 501a, 501c, 501e, and 501g of the first group 502a, 502b, 502c, and 502d can be divided into at least one second group 504a and 504b. As shown, the second group 504a includes devices 501a and 501c, and the second group 504b includes devices 501e and 501g. The grouping can be performed in the manner described with respect to 510. New identifiers can be assigned to devices 501a, 501c, 501e, and 501g. For example, non-volatile memory device 501a is assigned identifier 1, non-volatile memory device 501c is assigned identifier 2, non-volatile memory device 501e is assigned identifier 3, and non-volatile memory device 501g is assigned identifier 4. Grouping can be performed based on the updated identifier or using another suitable method as described.

[0094] In response to grouping the guiding devices at 550 (e.g., in response to grouping the first guiding devices into at least one second group), method 500 returns to 520 to determine whether the number of groups (e.g., at least one second group) is greater than one. In response to determining that the number of groups is greater than one (520: yes), method 500 continues to 530.

[0095] For example, at 530, a boot device (e.g., a second boot device) is selected for each group (e.g., a second non-volatile memory device from each of at least one second group). In some examples, a device with an odd-numbered identifier (e.g., 1 and 3) in each of the second groups 504a and 504b is selected as the boot device. Therefore, non-volatile memory devices 501a and 501e are selected for groups 504a and 504b, respectively. In some examples, a device with an even-numbered identifier in each group can be selected as the boot device. In some examples, a device with the smallest identifier in each group can be selected as the boot device. In some examples, a device with the largest identifier in each group can be selected as the boot device.

[0096] At 540, the boot device determines the result data by performing at least one operation on the data in each group. For example, the controller 110 of the second boot device in at least one second group determines the second result data by performing an operation based on second data from at least one of the second non-volatile memory devices in at least one of the at least one second group.

[0097] For example, the buffer (e.g., buffer 112) of the second boot device receives second data from at least one of the second non-volatile memory devices in each of at least one of the second groups. The controller 110 of the second boot device 501a determines the second result data by calculating the second result data based on the data stored in the buffer 112 of the second boot device 501a (which may be first result data determined by the boot device 501a) and the second data from each of the second non-volatile memory devices in each of at least one of the second groups. Therefore, the first result may be an intermediate result.

[0098] For example, the controller 110 of the second boot device 501a in group 504a performs a DMA transfer by issuing a DMA request to the controller 110 of the second non-volatile memory device 501c in group 504a using the address of the buffer 112 of the second non-volatile memory device 501c. Data at the address of the buffer 112 of the second non-volatile memory device 501c can be read from the memory array 120 of the second non-volatile memory device 501c to the buffer 112 of the second non-volatile memory device 501b, can be received from the memory 102 by the buffer 112 of the second non-volatile memory device 501c through interface 140, or can be the result data of a previous iteration of method 500 (e.g., first result data). In response, the controller 110 of the second boot device 501a transfers data stored in the buffer 112 of the second non-volatile memory device 501c to the buffer 112 of the second boot device 501a. The controller 110 of the second boot device 501a can read data stored in the non-volatile memory (e.g., memory array 120) of the second boot device 501a into the buffer 112 of the second boot device 501a, or the data can be the result data from a previous iteration of method 500 (e.g., first result data). The controller 110 of the second boot device 501a can use the data of the second boot device 501a and the second data from the second non-volatile memory device 501c to perform an operation to determine the second result data. This operation includes an XOR operation, wherein the data of the second boot device 501a and the second data from the second non-volatile memory device 501c are bitwise mutually exclusive. The second result data is stored in the buffer 112 of the second boot device 501a. The result data of group 504b can be similarly determined by the boot device 501e.

[0099] At 550, the second guiding devices 501a and 501e of the second groups 502a, 502b, 502c, and 502d can be divided into at least one third group 506. As shown, the second group 506 includes devices 501a and 501e. Grouping can be performed as described with respect to 510. New identifiers can be assigned to devices 501a and 501e. For example, non-volatile memory device 501a is assigned identifier 1, and non-volatile memory device 501e is assigned identifier 2. Grouping can be performed based on updated identifiers or using another suitable method described.

[0100] In response to grouping the guiding devices at 550 (e.g., in response to grouping the second guiding devices into a third group), method 500 continues to 560, wherein guiding devices are selected for group 506 according to any of the methods disclosed herein with respect to 530. At 570, as described with respect to 540, the guiding devices determine final result data by performing operations based on data in the remaining groups. The result data determined by the guiding devices in the final remaining groups is considered the final result data.

[0101] For example, the controller 110 of the third boot device 501a in group 506 performs a DMA transfer by issuing a DMA request to the controller 110 of the third non-volatile memory device 501e in group 506 using the address of the buffer 112 of the third non-volatile memory device 501e. Data at the address of the buffer 112 of the third non-volatile memory device 501e can be read from the memory array 120 of the third non-volatile memory device 501e into the buffer 112 of the third non-volatile memory device 501e, can be received from the memory 102 by the buffer 112 of the third non-volatile memory device 501e through interface 140, or can be the result data from a previous iteration of method 500 (e.g., second result data). In response, the controller 110 of the third boot device 501a transfers data stored in the buffer 112 of the third non-volatile memory device 501e to the buffer 112 of the third boot device 501a. The controller 110 of the third boot device 501a can read data stored in the non-volatile memory (e.g., memory array 120) of the third boot device 501a into the buffer 112 of the third boot device 501a, or the data can be the result data of a previous iteration from method 500 (e.g., second result data). The controller 110 of the third boot device 501a can use the data of the third boot device 501a and the third data from the third non-volatile memory device 501e to perform an operation to determine a third and final result data. This operation includes an XOR operation, wherein the data of the third boot device 501a and the third data from the third non-volatile memory device 501e are bitwise mutually exclusive. The third and final result data is stored in the buffer 112 of the third boot device 501a and can subsequently be stored in the non-volatile memory (e.g., memory array 120) or transferred to memory 102 via interface 140.

[0102] In some configurations, 510, 520, and 530 may be performed by one or more of a plurality of non-volatile memory devices. For example, at least one of the plurality of non-volatile memory devices manages a non-volatile memory device, or at least one non-volatile memory device other than a plurality of non-volatile memory devices manages a non-volatile memory device. The management of non-volatile memory devices can receive device content information from each of a plurality of non-volatile memory devices from the host. In some examples, the device content information includes addresses or identifiers of the plurality of non-volatile memory devices in a RAID group. The address of the non-volatile memory device can be a CMB address, SLM address, PMR address, address descriptor, identifier, index, or another suitable indicator of the buffer 112 of the non-volatile memory device itself, as described. In some examples, host commands include logical addresses of data subjects related to operations, including logical addresses of the host for the data (e.g., buffer addresses). In some examples, the device content information may include permission information for the plurality of non-volatile memory devices or at least one of their buffers 112. In some examples, the device content information may include priority information, which includes at least one of a priority level for processing data (e.g., group-based computation) by each of the plurality of non-volatile memory devices or a priority level for communication between the management non-volatile memory device or the boot device and each of the plurality of non-volatile memory devices regarding data related to group-based computation. The management non-volatile memory device may use address and authorization information in the device content information to communicate with one or more of the plurality of non-volatile memory devices. The management non-volatile memory device may transfer this group (e.g., identifier assignment) to all of the plurality of non-volatile memory devices or to the plurality of non-volatile memory devices in the same group as the management non-volatile memory device. At 520, the management non-volatile memory device may determine whether the number of groups is greater than 1 and notify the remaining boot devices whether additional operations need to be performed. In 530, a management non-volatile memory device can select a leader for all groups, or different management non-volatile memory devices can select a leader for each group. Various implementations involve grouping a plurality of non-volatile memory devices into at least one first group, determining that the number of at least one first group is greater than 1, selecting a first leader from the first non-volatile memory devices in each of the at least one first group, and determining first result data by performing operations based on first data from at least one of the first non-volatile memory devices in each of the at least one first group.

[0103] Method 500 can reduce the workload of processor 104 by dividing the work of processor 104 among device controllers 110. Since data transfers occur within the PCIe subsystem (e.g., in bus 106), method 500 can reduce the amount of data transferred through root compound 105. Method 500 can reduce the load on system memory 102 by using buffer 112 to recover intermediate results.

[0104] The preceding description is provided to enable any person skilled in the art to practice the various forms described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other forms. Therefore, the claims are not intended to limit the forms shown herein, but rather to encompass the full range consistent with the terminology used in the claims, wherein references to singular elements are not intended to mean "one and only one," unless specifically stated otherwise, but rather "one or more." Unless expressly stated otherwise, the term "some" means one or more. All structural and functional equivalents of elements of the various forms described throughout the foregoing description that are known or will be known by one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether or not such disclosure is expressly stated in the claims. No claim element should be construed as a means plus a function unless the term "means" is used to expressly describe the element.

[0105] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of illustrative method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged while remaining within the scope of the previously described. The attached method request presents the elements of various steps in an illustrative order and is not intended to limit us to the specific order or hierarchy presented.

[0106] The prior description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosed object. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the prior description. Therefore, the foregoing description is not intended to limit itself to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0107] The various examples illustrated and described are provided merely as examples to illustrate the various features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the associated example and may be used or combined with other examples shown and described. Furthermore, the claims are not intended to be limited to any single example.

[0108] The foregoing method descriptions and flowcharts are provided as illustrative examples only and are not intended to require or imply that the steps in the various examples must be performed in the order presented. As those skilled in the art will understand, the steps in the foregoing examples can be performed in any order. Words such as "afterwards," "then," and "next" are not intended to restrict the order of steps; these words are merely used to guide the reader through the description of the method. Furthermore, any reference to a request element in the singular form, such as the use of the articles "a," "an," or "the," should not be construed as limiting that element to the singular.

[0109] The various illustrative logic blocks, modules, circuits, and algorithm steps described in relation to the examples disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application; however, such implementation choices should not be construed as departing from the scope of this disclosure.

[0110] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits described in connection with the examples disclosed herein may be implemented or executed using a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware component, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry specific to a given function.

[0111] In some exemplary examples, the described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or program code on a nontransitory computer-readable storage medium or a nontransitory processor-readable storage medium. The steps of the methods or algorithms disclosed herein can be embodied in a processor-executable software module that can reside on a nontransitory computer-readable or processor-readable storage medium. A nontransitory computer-readable or processor-readable storage medium can be any storage medium accessible by a computer or processor. By way of example and not limitation, such nontransitory computer-readable or processor-readable storage media can include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical drive storage, magnetic drive storage or other magnetic storage, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. The drives and optical discs used in this document include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy drives, and Blu-ray discs. Drives typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. The above combinations also fall within the scope of non-transitory computer-readable and processor-readable media. Furthermore, the operation of a method or algorithm can reside as one or any combination or set of program code and / or instructions on non-transitory processor-readable and / or computer-readable storage media, and can be incorporated into a computer program product.

[0112] The foregoing description of the disclosed examples is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these examples will be apparent to those skilled in the art, and the general principles defined herein may be applied to some examples without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples shown herein, but is consistent with the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

[0113] 100: Non-volatile memory devices 101: Host 102: Memory 104: Processor / Host Processor 105: Root Complex 106: Busbar 140: Interface 100a: Non-volatile memory device 100b: Non-volatile memory device 100n: Non-volatile memory device 110: Controller 112: Buffer 114: Write buffer 116: Read buffer 120: Memory Array 130a: NAND flash memory device 130b: NAND flash memory device 130n: NAND flash memory device 100a: Non-volatile memory device 100b: Non-volatile memory device 100n: Non-volatile memory device 100n-1: Non-volatile memory device 401a: Buffer 401b: Buffer 401n-1: Buffer 501a: Non-volatile memory device / First boot device / Second boot device / Third boot device 501b: Non-volatile memory devices 501c: Non-volatile memory device / first boot device 501d: Non-volatile memory device 501e: Non-volatile memory device / First boot device / Second boot device 501f: Non-volatile memory device 501g: Non-volatile memory device / first boot device 501h: Non-volatile memory device 502a: Group / First Group 502b: Group / First Group 502c: Group / First Group 502d: Group / First Group 504a: Group / Second Group 504b: Group / Second Group 506: Group / Second Group / Third Group

Claims

1. A method for performing data processing using a non-volatile storage device that offloads host tasks, comprising: A plurality of non-volatile memory devices are divided into at least one first group and at least one other group, wherein host data is divided among each of the plurality of non-volatile memory devices, and the at least one first group and the at least one other group are configured to independently perform operations on the divided host data; a first boot device is selected from each of the first non-volatile memory devices in the at least one first group, wherein each of the plurality of non-volatile memory devices is assigned a numerical identifier, and wherein the selection is performed based on a numerical comparison between the numerical identifiers of each of the plurality of non-volatile memory devices in the at least one first group; and a controller of the first boot device determines first result data by obtaining data from the first boot device and first data from at least one of the first non-volatile memory devices in the at least one first group, and performs the operation of forming the first result data by combining the data obtained from the first boot device with the first data obtained from the at least one of the first non-volatile memory devices.

2. The method as described in claim 1, further comprising: The buffer of the first guiding device receives the first data from at least one of the first non-volatile memory devices in each of the at least one first group, wherein determining the first result data includes calculating the first result data based on the data from the first guiding device and the first data received from at least one of the first non-volatile memory devices in each of the at least one first group.

3. The method as described in claim 2, further comprising: The first guiding device is grouped from each of the at least one first group into at least one second group; And in response to grouping the first guiding device into the at least one second group, determining whether a second number of the at least one second group is greater than 1.

4. The method as described in claim 3, further comprising: Determine that the second quantity of the at least one second group is greater than 1; A second guiding device is selected from each of the second non-volatile memory devices in at least one of the second groups; The second controller of the second guiding device determines the second result data by performing operations based on second data from at least one of the second non-volatile memory devices in each of the at least one second group.

5. The method as described in claim 3, further comprising: The second quantity of the at least one second group is determined to be no greater than 1; The first result data is determined to be the final result.

6. The method as described in claim 1, wherein, The operation includes a mutually exclusive OR (XOR) operation.

7. The method as described in Request 1, wherein: At least one management non-volatile memory device groups the plurality of non-volatile memory devices into the at least one first group based on device content information; Furthermore, the at least one non-volatile memory management device determines, based on the device content information, that the quantity of the at least one first group is greater than 1.

8. The method as described in claim 7, wherein, The device content information includes the addresses of the plurality of non-volatile memory devices or the addresses of the buffers of the plurality of non-volatile memory devices.

9. The method as described in claim 7, wherein, The device content information includes permission information for the plurality of non-volatile memory devices or permission information for the buffers of the plurality of non-volatile memory devices.

10. The method as described in claim 7, wherein, The device content information includes priority information corresponding to host commands.

11. A system for performing data processing using a non-volatile storage device that offloads host tasks, comprising: At least one processor is configured to: group a plurality of non-volatile memory devices into at least one first group and at least one other group, wherein host data is divided among each of the plurality of non-volatile memory devices, and the at least one first group and the at least one other group are configured to independently perform operations on the divided host data; select a first boot device from a first non-volatile memory device in each of the at least one first group, wherein each of the plurality of non-volatile memory devices is assigned a numerical identifier, and wherein the selection is performed based on a numerical comparison between the numerical identifiers of each of the plurality of non-volatile memory devices in each of the at least one first group; determine first result data by obtaining data from the first boot device and first data from at least one of the first non-volatile memory devices in each of the at least one first group, and perform the operation of forming the first result data by combining the data obtained from the first boot device with the first data obtained from the at least one of the first non-volatile memory devices.

12. The system as described in claim 11, wherein, The buffer of the first guiding device further receives the first data from at least one of the first non-volatile memory devices in each of the at least one first group, wherein determining the first result data includes calculating the first result data based on the data from the first guiding device and the first data received from at least one of the non-volatile memory devices in each of the at least one first group.

13. The system as described in claim 12, wherein, The at least one processor is configured to: group the first guiding device from each of the at least one first group to at least one second group; And in response to grouping the first guiding device into the at least one second group, determining whether a second number of the at least one second group is greater than 1.

14. The system as described in claim 13, wherein, The at least one processor is configured to: determine that the second quantity of the at least one second group is greater than 1; select a second boot device from the second non-volatile memory devices in each of the at least one second group; and determine second result data by a second controller of the second boot device through performing operations based on second data from at least one of the second non-volatile memory devices in each of the at least one second group.

15. The system as described in claim 13, wherein, The at least one processor is configured to: determine that the second quantity of the at least one second group is not greater than 1; and determine that the first result data is the final result.

16. The system as described in claim 15, wherein, The operation includes a mutually exclusive OR (XOR) operation.

17. A non-transitory processor-readable medium comprising processor-readable instructions such that, when the instructions are executed, at least one processor: groups a plurality of non-volatile memory devices into at least one first group and at least one other group, wherein, Host data is divided among each of the plurality of non-volatile memory devices, and the at least one first group and the at least one other group are configured to independently perform operations on the divided host data; a first boot device is selected from each of the first non-volatile memory devices in the at least one first group, wherein each of the plurality of non-volatile memory devices is assigned a numerical identifier, and wherein the selection is performed based on a numerical comparison between the numerical identifiers of each of the plurality of non-volatile memory devices in the at least one first group; and a first result data is determined by obtaining data from the first boot device and first data from at least one of the first non-volatile memory devices in the at least one first group, and the operation of forming the first result data is performed by combining the data obtained from the first boot device with the first data obtained from at least one of the first non-volatile memory devices.

18. A non-transitory processor-readable medium as described in claim 17, wherein, The buffer of the first guiding device receives the first data from at least one of the first non-volatile memory devices in each of the at least one first group, wherein determining the first result data includes calculating the first result data based on the data from the first guiding device and the first data received from at least one of the first non-volatile memory devices in each of the at least one first group.

19. A non-transitory processor-readable medium as described in claim 18, wherein, The at least one processor is caused to: group the first guiding device from each of the at least one first group to at least one second group; And in response to grouping the first guiding device into the at least one second group, determining whether a second number of the at least one second group is greater than 1.

20. A non-transitory processor-readable medium as described in claim 19, wherein, The at least one processor is caused to: determine that the second quantity of the at least one second group is greater than 1; select a second boot device from the second non-volatile memory devices in each of the at least one second group; and determine second result data by a second controller of the second boot device through performing operations based on second data from at least one of the second non-volatile memory devices in each of the at least one second group.

Images