Method and system for improving big data analytics throughput in NAND-based read source storage

Abstract

One embodiment facilitates data access in a storage device. During operation, a system obtains a file from an original physical medium separated from the storage device through a storage device, wherein the file includes compressed data that has been previously encoded based on an error correction code (ECC). The system stores the obtained file as a read-only copy on the physical medium of the storage device. In response to receiving a request to read the file, the system decodes the copy based on the ECC through the storage device to obtain ECC-decoded data, wherein the ECC-decoded data is subsequently decompressed by a computing device associated with the storage device and returned as the requested file.

Description

Technical Field

[0001] The present disclosure relates generally to the field of data storage. More specifically, the present disclosure relates to methods and systems for improving the throughput of big data analytics in NAND-based read source storage. Background Art

[0002] The proliferation of the Internet and e-commerce continues to create a large amount of digital content. Various distributed storage systems have been created to access and store this digital content. In some read-intensive scenarios, the storage system may store the original version of the data, where the stored original data is frequently read or accessed, but not necessarily updated. An example of a read-intensive scenario is "big data" analysis, which involves examining a large number of different data sets (i.e., big data) to identify relevant information, such as hidden patterns, market trends, unknown correlations, and any information that may be relevant to the user. Big data analysis requires frequent reading of large amounts of data, while the source data remains unchanged. In order to speed up the data analysis processing performed by multiple data analysis servers, intermediate results are usually stored in storage-style media. However, this may result in considerable overhead in transmitting large amounts of data over long distances over Ethernet. To avoid this overhead, the read source is usually replicated to form a local copy at each site, so low cost and high throughput are both practical and basic requirements for storing copies of the source data.

[0003] In current big data analytics, conventional distributed storage systems can be used to provide the required storage for multiple storage copies of the original data source. However, using conventional distributed storage systems may lead to several challenges and inefficiencies, including: expensive deployment of the entire distributed storage system consisting of dozens of computing servers and storage servers (including networks); expensive rack space in data centers; high electricity costs; and high latency in accessing copies stored in storage servers.

[0004] As big data analytics continue to grow, the inefficiencies and challenges of conventional distributed storage systems will also continue to grow. Summary of the invention

[0005] One embodiment facilitates data access in a storage device. During operation, a system obtains a file from an original physical medium separated from the storage device through a storage device, wherein the file includes compressed error-free data that has been previously encoded based on an error correction code (ECC). The system stores the obtained file as a read-only copy on the physical medium of the storage device. In response to receiving a request to read the file, the system decodes the copy based on the ECC by the storage device to obtain ECC-decoded data, wherein the ECC-decoded data is subsequently decompressed by a computing device associated with the storage device and returned as the requested file.

[0006] In some embodiments, a request to read a file is received from a requesting entity based on a first protocol. After the computing device decompresses the ECC decoded data, the system returns the decompressed data as the requested file to the requesting entity without performing any ECC encoding.

[0007] In some embodiments, decompressing the ECC decoded data is performed by a parallel decompression engine of a computing device.

[0008] In some embodiments, a request to read a file is received by a computing device and sent by the computing device to a storage device through a system comprising at least one of: a first Ethernet switch and a second Ethernet switch; a first intelligent network interface card (NIC) and a second intelligent network interface card (NIC); a plurality of peripheral high-speed interconnect PCIe switches; wherein the first intelligent NIC and the second intelligent NIC each comprise a simple storage node comprising: an uplink to the first Ethernet switch and the second Ethernet switch; and a downlink of a plurality of PCIe switches via a plurality of PCIe channels, wherein the plurality of PCIe channels are used to connect to the storage device and a plurality of other storage devices.

[0009] In some embodiments, the request to read the file is further received by a second computing device and is also transmitted by the second computing device to the storage device via the system; the second computing device is a backup server or a high availability server of the computing device.

[0010] In some embodiments, the system stores, via a computing device, a mapping of file names to current physical block addresses associated with the storage device.

[0011] In some embodiments, in response to detecting a condition to move a file from the current physical block address to a new physical block address associated with a storage device, the system updates a mapping of the file name to the new physical block address.

[0012] In some embodiments, the format of the file stored in the storage device includes one or more of: a pre-amble indicating a starting position associated with the file; a unique file identifier of the file; the content of the file; a post-amble indicating an ending position associated with the file; and a cyclic redundancy check (CRC) signature for verifying the consistency of the file content.

[0013] In some embodiments, the first storage device does not include: a module or unit that performs ECC encoding; a dynamic random access memory (DRAM) interface and a DRAM module accessed through the DRAM interface; a processor that performs flash translation layer (FTL) functions, including mapping logical block addresses to physical block addresses.

[0014] Another embodiment provides a distributed storage system for facilitating one or more big data analysis applications through one or more clients. The distributed storage system includes:

[0015] A front-end head server, a switch, and multiple NAND cards connected to the switch; the front-end head server is configured to receive a request to read a file; the NAND card is configured to: receive a request to read a file stored on the NAND card through the switch; and the NAND card decodes the file based on an error correction code (ECC) to obtain ECC-decoded data; the front-end head server is configured to decompress the ECC-decoded data and return the decompressed data as the requested file.

[0016] In some embodiments, the NAND card is further configured to obtain a file from an original physical medium separated from the NAND card, wherein the file includes compressed data previously encoded based on an error correction code (ECC). The NAND card is further configured to return ECC-decoded data to the front-end head server without performing any ECC encoding. The NAND card does not include: a module or unit dynamic random access memory (DRAM) interface that performs ECC encoding and a DRAM module accessed through the DRAM interface; a processor that performs a flash translation layer (FTL) function, the function including mapping a logical block address to a physical block address; thereby facilitating big data analysis applications to efficiently analyze data by reducing the number of ECC encodings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 illustrates an exemplary environment for facilitating data access in a storage device according to the prior art.

[0018] Figure 2 An exemplary environment for facilitating data access in a storage device according to an embodiment of the present application is shown.

[0019] Figure 3 An exemplary environment including a data I / O path for data placement in an original data source and data placement copied to a read source replica according to an embodiment of the present application is shown.

[0020] Figure 4 An example diagram of a storage device including a NAND card with a simplified architecture according to an embodiment of the present application is shown.

[0021] Figure 5 An exemplary mapping of files to physical block addresses according to an embodiment of the present application is shown.

[0022] Figure 6 An example diagram of a file format and a NAND block layout according to an embodiment of the present application is shown.

[0023] Fig. 7A A flow chart illustrating a method for facilitating data access in a storage device according to an embodiment of the present application is presented.

[0024] Figure 7B A flow chart illustrating a method for facilitating data access in a storage device according to an embodiment of the present application is presented.

[0025] Figure 8 An exemplary computer system and storage system are shown that facilitate data access in a storage device according to an embodiment of the present application.

[0026] Fig. 9 An exemplary apparatus for facilitating data access in a storage device according to an embodiment of the present application is shown.

[0027] In the drawings, like reference numerals refer to like drawing elements. DETAILED DESCRIPTION

[0028] The following description is provided to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the embodiments described herein are not limited to the embodiments shown, but should be given the widest scope consistent with the principles and features disclosed herein.

[0029] Overview

[0030] The embodiments described herein address the inefficiency of big data analytics in current distributed storage systems by providing a novel distributed storage system with simplified "NAND cards" as storage devices. These storage devices offload ECC encoding to the original storage devices, eliminate DRAM controllers and modules in the storage devices, and move FTL functions to the computing devices at the front end of the system.

[0031] As mentioned above, big data analysis is an example of a read-intensive scenario that requires frequent reading of very large amounts of data, but the source data remains unchanged. In order to speed up the data analysis processing performed by multiple data analysis servers, intermediate results are usually stored in storage-style media. However, this may result in a large amount of overhead for transmitting large amounts of data over long distances over Ethernet. To avoid this overhead, the read source is usually replicated to form a local copy at each site. Therefore, both low cost and high throughput are practical and basic requirements for storing copies of source data.

[0032] In current big data analytics, conventional distributed storage systems can be used, for example, to provide the required storage for multiple storage copies of the original data source as needed and accessed by multiple data analytics servers. However, using conventional distributed storage systems can result in several challenges and inefficiencies, including: expensive deployment of the entire distributed storage system consisting of dozens of compute servers and storage servers (including networking); expensive rack space in data centers; high electricity costs; and high latency in accessing copies stored in storage servers (as described below with respect to Figure 1). As big data analytics continues to grow, the inefficiencies and challenges of conventional distributed storage systems will also continue to grow.

[0033] The embodiments described herein address these challenges and inefficiencies by providing a distributed storage system with high-capacity storage and high throughput. The system allows multiple data analysis servers to access data with low latency and low cost. The system includes a front-end head server connected to an Ethernet switch, which is connected to an intelligent network interface card (NIC) system on chip (SOC). The intelligent NIC SOC is connected to multiple peripheral high-speed interconnect (PCIe) switches (via multiple PCIe channels), wherein each PCIe switch is connected to multiple "NAND cards". Each NAND card can be a simplified storage device, including an ECC decoder, but does not include certain modules, such as: ECC encoder, FTL operations or functions performed by a processor, and DRAM interface / storage.

[0034] Instead, the functions performed by these modules can be offloaded in various ways. For example, the ECC encoding can be performed by the original data source storage device, and the ECC encoded data (which also contains error-free data because it has been previously ECC decoded) can be retrieved from the original data source storage device on the outbound data I / O path, or provided to a NAND card, such as Figure 3 In addition, for example, by processing a request from a data analysis server (e.g., a request to access a file stored as a read source data or a copy), the operation of mapping LBA to PBA associated with the FTL and the operation of maintaining the mapping can be offloaded to a computing device that acts as a front-end head of the distributed storage system. In addition, the NAND card can use a simple cache or cache to perform any necessary ECC decoding, thereby removing the DRAM interface and DRAM from the storage device.

[0035] Therefore, by simplifying the NAND cards in the manner described herein, embodiments of the present system can generate cost and power savings, and can also provide a more efficient distributed storage system for big data analytics and other similar applications that require frequent access to read-only data, where the distributed storage servers can provide the low cost and high throughput required to satisfy data access requests from multiple data analytics servers.

[0036] “Storage drive” refers to a device or drive with non-volatile memory that can provide persistent storage of data, such as a solid-state drive (SSD) or a hard disk drive (HDD).

[0037] A "storage server" refers to a computing device that may include multiple storage drives. A distributed storage system may include multiple storage servers.

[0038] NAND flash can store a certain number of bits per cell. For example, a "single-level cell" or "SLC" storage element can store one bit of information per cell; a "multi-level cell" or "MLC" storage element can store two bits of information per cell; a "triple-level cell" or "TLC" storage element can store three bits of information per cell; and a "quadruple-level cell" or "QLC" storage element can store four bits of information per cell.

[0039] "NAND card" refers to a storage device with a simplified architecture, for example, without a processor for FTL functionality, an ECC encoder module, or a DRAM interface / DRAM, as described below Figure 2 and Figure 4 discussed.

[0040] Exemplary environment of data access in the prior art

[0041] FIG1 shows an exemplary environment for facilitating data access in a storage device according to the prior art. In this conventional environment 100, a network attached storage (NAS) protocol can be implemented through a NAS protocol server, and data can be stored in a distributed storage cluster. For example: client 110 may include data analysis servers 112, 114, and 116; computing server 130 may include network attached storage (NAS) protocol servers 132, 134, and 136; storage server 150 may include a distributed storage cluster 151, and may include storage servers 152, 154, and 156. In environment 100, data can be transferred from data analysis servers 112-116 to NAS protocol servers 132-136 through load balancing module 120. Then, the data can be stored to or stored on storage servers 152-156 through data center Ethernet 140.

[0042] One advantage of the conventional distributed storage system of environment 100 is its ability to provide sufficient capacity to accommodate the growing amount of data that the data analysis server needs to access (i.e., the site as a replica of the data source). However, this conventional system has several disadvantages or shortcomings. First, the entire distributed storage system requires a large number of computing servers and storage servers (e.g., dozens of orders of magnitude) and the necessary network connections, which may be cumbersome for maintenance and expansion. The network costs associated with various border gateway protocols (BGP) may be large. Second, rack space in the data center may be expensive, and the rack space cost will inevitably increase in order to expand such a conventional distributed storage system. Third, the power cost required to store, access and maintain this replica data in multiple locations will be high, and the power cost may also increase during maintenance or expansion. Fourth, the delay involved in reading or accessing data from such a replica site may be high or long. Therefore, given the high cost of data center rack space and power budgets, the need to increase the throughput of replica data sources, and the high cost of relying on conventional distributed storage systems to frequently access read-only data in this way, these shortcomings may cause the system to be less efficient in terms of cost, performance and scalability. Example environment using simplified "NAND card" storage device to facilitate data access

[0043] Figure 2 An exemplary environment 200 for facilitating data access in a storage device according to an embodiment of the present application is shown. The environment 200 may describe a distributed storage system and may include several entities, including a pair of front-end heads, a pair of Ethernet switches, a pair of smart NIC system-on-chips (SOCs), multiple PCIe switches, and multiple "NAND cards". As described above, the term "NAND card" refers to a storage device with a simplified architecture, for example, without a processor, an ECC encoder module, or a DRAM interface / DRAM, as shown below Figure 4 In a pair of entities, one entity may be a primary (or active) entity and the other entity may be a standby (or non-active entity) that provides high availability when the primary entity fails or is otherwise unresponsive.

[0044] For example, the environment 200 may include: a front-end head A 222 with a parallel decompression engine (ASIC) 224; and a front-end head B 226 with a parallel decompression engine (ASIC) 228. Each front-end head can be configured to handle a specific protocol, for example, a network attached storage (NAS) protocol to handle incoming requests from multiple big data analysis servers. Each front-end head can be regarded as a simplified client node without a storage drive or with very limited capacity. Therefore, each front-end head is responsible for processing requests input through a certain protocol and is responsible for decompressing the data based on its respective parallel decompression engine.

[0045] Each front-end head can be connected to an Ethernet switch A 232 and an Ethernet switch B 234. Each Ethernet switch can be connected to an intelligent NIC SOC 242 and an intelligent NIC SOC 244. Each intelligent NIC SOC can be connected to multiple PCIe switches (through multiple PCIe channels), and each intelligent NIC SOC can further determine which PCIe channel to use. Each intelligent NIC SOC can act as a simple storage node and can have a simplified microprocessor, such as a microprocessor without an internal interlocked pipeline stage (MIPS) or an advanced RISC computer (ARM) processor. The intelligent NIC SOC 242 can be connected to PCIe switches 251, 252 and 253, and the intelligent NIC SOC 244 can be connected to PCIe switches 254, 255 and 256. That is, each intelligent NIC SOC has an uplink that is connected to at least one Ethernet switch via Ethernet, and each intelligent NIC SOC also has a downlink that includes a PCIe channel for connecting to a PCIe NAND card.

[0046] Each PCIe switch can be connected to multiple NAND cards, and these NAND cards can provide the storage capacity and high throughput required for frequent data access (e.g., big data analysis by data analysis servers 212, 214, and 110). Figure 4 An exemplary NAND card (eg, NAND card 261) is described in . To increase the capacity of the system described in environment 200, the system can add more entities at any of the described levels, including more Ethernet switches, more Smart NIC SOCs, more PCIe switches, and more NAND cards.

[0047] Therefore, in the embodiments described herein, the system provides a distributed storage system with high-capacity storage and high throughput by using multiple NAND cards in an environment such as environment 200. By placing protocol processing and compression functions only in the front-end header, the distributed storage system allows the NAND card to be used only as a data pool, which allows the entire distributed storage system to be easily moved and constructed, such as in a small data center, occupying or using less overall rack space, thereby reducing the total operating cost (TCO).

[0048] Exemplary Data I / O Paths for Data Access

[0049] Figure 3An exemplary environment for data placement in an original data source and data placement copied to a read-only source copy according to an embodiment of the present application, including a data I / O path, is shown. The original data can be stored in the original data source 310 via the data I / O path, which includes: receiving data to be stored (via communication 352); receiving and compressing the received data through the compression module 312; sending the compressed data to the ECC encoder module 314 (via communication 354); receiving and compressing the compressed data through the ECC encoder module 314; sending the ECC-encoded data ("original stored data" or "original version") to be stored in the NAND 316 (via communication 356). After receiving a request to read data, the data originally stored in NAND 316 can be accessed or read via a data I / O path, which includes: receiving ECC-encoded data by the ECC decoder module 318 (via communication 358); decoding the ECC-encoded data by the ECC decoder module 318; sending the ECC-decoded data to the decompression module 320; receiving and decompressing the ECC-decoded data by the decompression module 320; and returning the decompressed data (via communication 362).

[0050] During the data I / O path to access the data, after the data is decoded by the ECC decoder 318, the originally stored data has been successfully ECC decoded into "error-free data", and the ECC decoder 318 also has the entire ECC codeword including the ECC parity bits obtained from the NAND 316. At this point, the read source copy 330 can obtain or retrieve (or the original source 310 can transmit or send) the error-free data of the entire ECC codeword (including the ECC parity bits) from the ECC decoder module 318 (via communication 370). The read source copy 330 can store the obtained ECC codeword in the NAND 332 (as a read source or read-only copy).

[0051] When access to data is requested (e.g. Figure 2As shown, accessed by the data analysis server), retrieves the stored read-only copy (i.e., the ECC codeword) from NAND 332 and sends it to the ECC decoder 334 (via communication 372). The ECC decoder 334 can receive and decode the ECC codeword and send the ECC decoded data to a module such as multiplexer 336 (via communication 374). Multiplexer 336 can determine whether the data must be refreshed, needs to be moved, or has recently been moved to a new physical location. If the data needs to be refreshed or moved, the system can send the data back to NAND 332 (via communication 378). If the system does not need to be refreshed or moved, the system can send the data forward to the parity module 338 (via communication 376). The parity module 338 can discard the ECC parity bits to obtain compressed data. The parity module 338 can send the compressed data to the front-end head for decompression (via communication 380). The decompression module 340 may receive and decompress the compressed data (as a front-end operation 342 ) to obtain the data requested to be accessed or read, and may send back the requested data (via communication 382 ).

[0052] Thus, in the embodiments described herein, the system provides a high-density, high-capacity data source for high-intensity, frequent read access from multiple data analysis servers. By using these "simplified" NAND cards, the system can reduce total operating costs (TCO) and can eliminate the need for dozens or hundreds of servers in traditional distributed storage systems. Exemplary storage devices (NAND cards) for data access

[0053] Figure 4 An example diagram of a storage device including a NAND card 400 having a simplified architecture according to an embodiment of the present application is shown. The NAND card 400 may correspond to Figure 24. Any of the NAND cards 261-272 described in the specification, such as NAND card 261. NAND card 400 may include: a PCIe interface 412 for communication between NAND card 400 and a PCIe switch (via communication 442); a CRC module 418 for providing verification of data consistency based on a CRC signature; a buffer / cache 420 for storing data; an ECC decoder 422 for performing ECC decoding; and a NAND flash memory interface 430 for communicating between the NAND card and a physical storage medium associated with NAND card 400. For example, NAND card 400 may include multiple channels through which data may be transferred or communicated through NAND flash memory interface 430, including: NAND die 432 (accessed via channel 431); NAND die 434 (accessed via channel 433); NAND die 436 (accessed via channel 435); NAND die 438 (accessed via channel 437).

[0054] NAND card 400 does not include some modules or components previously included in conventional SSDs or conventional storage devices. These modules or components are indicated by a right slash. NAND card 400 does not include: processor (FTL) 414; ECC encoder 416; and DRAM interface 424 with associated DRAMs 426 and 428.

[0055] It should be noted that NAND flash memory is described as an exemplary non-volatile memory in this disclosure because it can provide higher reliability and performance in read / write operations compared to other types of non-volatile memory. However, in some embodiments, any type of memory may be used, including but not limited to hard disk drives, phase change memories, and other non-volatile memories.

[0056] Example Mapping of Files to Physical Block Addresses: Example File Format

[0057] In conventional file systems of distributed storage systems, a mapping is typically maintained between logical block addresses (LBAs) and physical block addresses (PBAs). In such conventional systems, LBAs are used to perform random writes or updates. In contrast, in the embodiments described herein ... Figure 2 In a file system coordinated by a front-end header A 222), the front-end header receives requests to read data at file granularity, rather than at block or page granularity, so there is no need to maintain a finer LBA to PBA mapping. In the embodiments described herein, although there may be no incoming write requirements, some data may need to be refreshed or moved periodically. However, because the data is based on a coarser granularity of files, the system can map file names to PBAs and can also use a log structure in a single block by appending new data at the end of the block.

[0058] Figure 5 An exemplary mapping of a file to a physical block address according to an embodiment of the present application is shown. The mapped file 510 may correspond to a logical block address 520 and a physical block address 540. The logical size may be the same as the physical block size. In the embodiments described herein, the system may offload traditional FTL functions (mapping LBA to PBA) from a processor inside a storage device controller (e.g., an SSD controller) to a computing device (e.g., a front-end computing device) associated with the storage device. The front-end computing device (or other computing devices or entities other than the storage device controller) may store a mapping of the file name 560 to the PBA 540.

[0059] For example, data file A 512 may correspond to multiple LBAs, each corresponding to a PBA. The front-end head computing device may store a mapping of the file name (e.g., "A") of data file A 512 to one or more PBAs associated with the LBAs corresponding to data file A 512. For example, data file A 512 (file name "A") may be associated with LBA 1 522, LBA 2 524, and LBA 3 526 corresponding to PBA k 542, PBAh 544, and PBA i 546, respectively. The front-end head (or other computing device separate from the storage device controller) may store a mapping of the file name "A" 562 to PBA k 542, PBA h 544, and PBA i 546. It should be noted that a file name may be mapped to multiple PBAs, just as a single PBA may be mapped to multiple file names (e.g., both file name "A" 562 and file name "B" 564 are mapped to PBA i 546). The system can determine the exact location within a PBA from which the requested file can be read, as shown below Figure 6 described.

[0060] Figure 6 An example diagram 600 of a file format and NAND block layout consistent with an embodiment of the present application is shown. Diagram 600 may include a file format corresponding to a file written to a non-volatile memory such as NAND. The file format may include: a preamble 612, which indicates the beginning of a given file; and a unique file identifier 614, which identifies the given file and does not conflict with any other file identifier; file content 616 corresponding to the given file; a postamble 618, which indicates the end of the given file; and a cyclic redundancy check (CRC) signature 620, which provides verification of the consistency of the given file content. Block 630 describes how the formatted file is stored in block 630.

[0061] During operation, when the mapping of file names to PBAs indicates that the requested file is stored at a certain PBA, the system can read out the page at the certain PBA to determine the starting point of the requested file (e.g., indicated by the preamble 612). Once the starting point is determined, the system can check whether the unique file identifier 614 corresponds to the requested file. If so, the system can read one physical block at a time until the system determines the end point of the requested file (e.g., the postamble 618). In block 630, the exemplary requested file includes file content 616 stored only in block 630. The file content of the requested file can also be stored across multiple blocks. When determining the end point of the requested file, the system can verify the consistency of the data based on checking the CRC signature 620.

[0062] Exemplary Methods for Facilitating Data Access

[0063] Fig. 7A A flowchart 700 is provided showing a method for facilitating data access in a storage device according to an embodiment of the present application. During operation, a system obtains a file from an original physical medium separated from the storage device by a storage device, wherein the file includes compressed error-free data that has been previously encoded based on an error correction code (ECC) (operation 702). The system stores the obtained file as a read-only copy on the physical medium of the storage device (operation 704). The system receives a request to read the file (operation 706). In response to receiving the request to read the file, the system decodes the copy based on the ECC by the storage device to obtain ECC-decoded data, wherein the ECC-decoded data is then decompressed by a computing device associated with the storage and returned as the requested file (operation 708), and the operation returns.

[0064] Figure 7B Flowchart 720 of a method for facilitating data access in a storage device consistent with an embodiment of the present application is shown. During operation, the system receives a request to read a file through a computing device associated with the storage device, wherein the file includes compressed error-free data that has been previously encoded based on an error correction code (ECC) (operation 722). The previous ECC encoding may be performed by, for example, the above Figure 3The system sends the request to the storage device by means of a computing device via a system including at least one of: first and second Ethernet switches; first and second intelligent network interface cards (NICs); and a plurality of peripheral interconnect express (PCIe) switches (operation 724). The system stores, by means of a computing device, a mapping of the file name to the current physical block address (PBA) associated with the storage device (operation 726). If the system detects a condition to move the file from the current PBA to the new PBA (decision 728), the system updates the mapping of the file name to the new physical block address (operation 730). If the system does not detect a condition to move the file from the current PBA to the new PBA (decision 728), the operation returns.

[0065] Exemplary Computer Systems and Devices

[0066] Figure 8 An exemplary computer system 800 and a storage system 820 for facilitating access to data in a storage device according to an embodiment of the present application are shown. The computer system 800 includes a processor 802, a volatile memory 804, and a storage device 806. The computer system 800 may correspond to Figure 2 The front end 222 of the computer system 800 may include a parallel decompression engine (e.g., in an ASIC). The volatile memory 806 may include, for example, a random access memory (RAM) that is used as management memory and may be used to store one or more memory pools. The storage device 806 may include persistent storage. In addition, the computer system 800 may be connected to peripheral input / output (I / O) user devices 850, such as a display device 852, a keyboard 854, and a pointing device 856. The storage device 806 may store an operating system 810, a content processing system 812, and data (not shown).

[0067] The content processing system 812 may include instructions that, when executed by the computer system 800, cause the computer system 800 to perform the methods and / or processes described in the present disclosure. Specifically, the content processing system 812 may include instructions for receiving and sending data packets including data to be read or written, input / output (I / O) requests (e.g., read requests or write requests), and data associated with the read requests, write requests, or I / O requests (communication module 814). The content processing system 812 may also include instructions for mapping, maintaining, and managing file names and / or LBAs to PBAs (FTL management module 816), such as storing a mapping of a file name to a current PBA associated with a storage device.

[0068] Computer system 800 may communicate with storage system 820 via a distributed storage system, and storage system 820 may include: processor / controller 822, volatile memory 824, and storage device 826. Storage system 820 may correspond to a storage server, and storage device 826 may correspond to Figure 2 NAND card 261. The processor / controller 822 or storage device 826 may include instructions for obtaining files from the original physical media separated from the storage system 820 (communication module 830). The processor / controller 822 or storage device 826 may include instructions for storing the obtained file as a read-only copy on the physical media of the storage device (data write module 832). The processor / controller 822 or storage device 826 may include instructions for decoding the copy based on ECC by the storage device to obtain ECC decoded data (ECC decoding module 838). The processor / controller 822 or storage device 826 may include instructions for responding to the detection of a condition for moving the file (data refresh module 836), and updating the mapping of the file name to the new physical block address (data refresh module 836).

[0069] Data 840 may include any data required as input or generated as output by the methods and / or processes described in the present disclosure. Specifically, data 840 may store at least: requests; read requests; write requests; input / output (I / O) requests; data associated with read requests, write requests, or I / O requests; error correction codes (ECC); codewords; parity bits; ECC parity bits; encoded data; ECC encoded / decoded data; compressed data; decompressed data; error-free data; read-only data; copies; files; file formats; indicators of Ethernet switches, NICs, PCIe switches or channels, uplinks, or downlinks; file names; preambles; file identifiers; file contents; postambles; CRC signatures; verification information; LBAs; PBAs; indicators of the original data source; and indicators of move or refresh data conditions.

[0070] Fig. 9 An exemplary apparatus 900 for facilitating data access in a storage device according to an embodiment of the present application is shown. Apparatus 900 may include multiple units or devices that may communicate with each other via wired, wireless, quantum optical, or electrical communication channels. Apparatus 900 may be implemented using one or more integrated circuits and may include, for example, Fig. 9 In addition, the apparatus 900 may be integrated into a computer system, or implemented as a separate device capable of communicating with other computer systems and / or devices. Specifically, the apparatus 900 may include units 902-912, which perform operations similar to those of FIG. Figure 8The functions or operations of modules 830-836 of the storage system 800 include: a communication unit 902; a data write unit 904; a data read unit 906; a data refresh unit 908; an ECC decoding unit 910; and a storage management unit 912 (which can process detection conditions to refresh data and move files from a current PBA to a new PBA).

[0071] The data structures and codes described in this detailed description are typically stored on a computer-readable storage medium, which can be any device or medium that can store code and / or data for use by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices (such as disk drives, magnetic tapes, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media now known or later developed that are capable of storing computer-readable media).

[0072] The methods and processes described in the detailed description section may be embodied as code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and code and stored in the computer-readable storage medium.

[0073] In addition, the above methods and processes may be included in a hardware module. For example, a hardware module may include, but is not limited to, an application specific integrated circuit (ASIC) chip, a field programmable gate array (FPGA), and other programmable logic devices now known or later developed. When the hardware module is activated, the hardware module will execute the methods and processes contained in the hardware module in hardware.

[0074] The foregoing embodiments described herein are provided for purposes of illustration and description only. They are not intended to be exhaustive or to limit the embodiments described herein to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. In addition, the above disclosure is not intended to limit the embodiments described herein. The scope of the embodiments described herein is defined by the appended claims.

Claims

1. A computer-implemented method for facilitating data access in a storage device, the method comprising: Receiving, by a storage device, a request to read a file stored on a physical medium of the storage device; and The storage device decodes the file according to an error correction code (ECC) to obtain ECC decoded data; wherein the ECC decoded data is subsequently decompressed by a computing device associated with the storage device and returned as the requested file; The method further comprises: Retrieving the file from an original physical medium separated from the storage device through the storage device; wherein the file includes compressed data previously encoded based on an error correction code (ECC); the storage device does not include a module or unit for performing ECC encoding; The retrieved file is stored as a read-only copy on a physical medium of the storage device.

2. The method according to claim 1, wherein: A request to read a file is received from a requesting entity based on a first protocol, wherein the ECC decoded data is decoded at the computing device. After compression, the method further comprises: The decompressed data is returned to the requesting entity as the requested file without performing any ECC encoding.

3. The method of claim 1, wherein decompressing the ECC decoded data is performed by a parallel decompression engine of the computing device.

4. The method according to claim 1, characterized in that The request to read the file is received by the computing device and sent by the computing device to the storage device via a system, the system comprising at least one of the following: a first Ethernet switch and a second Ethernet switch; a first intelligent network interface card (NIC) and a second intelligent network interface card (NIC); and Multiple Peripheral Interconnect Express (PCIe) switches; Each of the first intelligent NIC and the second intelligent NIC comprises a simple storage node, including: an uplink to a first Ethernet switch and a second Ethernet; Downlinks to a plurality of PCIe switches via a plurality of PCIe lanes, the plurality of PCIe lanes being used to connect to the storage device and a plurality of other storage devices.

5. The method according to claim 4, characterized in that The request to read the file is also received by a second computing device, and is also transmitted by the second computing device to the storage device via the system; The second computing device is a backup or high availability server for the computing device.

6. The method according to claim 1, further comprising: storing, by the computing device, a mapping from a file name of the file to a current physical block address associated with the storage device; In response to detecting a condition to move the file from the current physical block address to a new physical block address associated with the storage device, a mapping of the file name of the file to the new physical block address is updated.

7. The method according to claim 1, wherein: The file formats stored in the storage device include one or more of the following: a preamble indicating a starting position associated with the file; A unique file identifier for the file; the content of the document; a postamble indicating the end position associated with the file; and A cyclic redundancy check (CRC) signature used to verify the consistency of file contents.

8. The method of claim 1, wherein the storage device further comprises: A dynamic random access memory (DRAM) interface and a DRAM module accessed through the DRAM interface; and A processor that performs flash translation layer (FTL) functions, including mapping logical block addresses to physical block addresses.

9. A computer system for facilitating data access in a system, the system comprising: processor; and a memory connected to the processor and storing instructions, which when executed by the processor cause the processor to perform a method, wherein the computer system includes a storage device, the method comprising: Receiving, by a storage device, a request to read a file stored on a physical medium of the storage device; and The storage device decodes the file according to an error correction code (ECC) to obtain ECC decoded data, wherein the ECC decoded data is subsequently decompressed by a computing device associated with the storage device and returned as the requested file; The method further comprises: Retrieving the file from an original physical medium separated from the storage device through the storage device; wherein the file includes compressed data previously encoded based on an error correction code (ECC); and the storage device does not include a module or unit for performing ECC encoding; The retrieved file is stored as a read-only copy on a physical medium of the storage device.

10. The computer system according to claim 9, wherein: A request to read a file is received from a requesting entity based on a first protocol, wherein after the computing device decompresses the ECC decoded data, the method further comprises: The decompressed data is returned to the requesting entity as the requested file without performing any ECC encoding.

11. The computer system of claim 9, wherein decompressing the ECC decoded data is performed by a parallel decompression engine of the computing device.

12. The computer system according to claim 9, wherein: The request to read the file is received by the computing device and sent by the computing device to the storage device via a system, the system comprising at least one of the following: a first Ethernet switch and a second Ethernet switch; a first intelligent network interface card (NIC) and a second intelligent network interface card (NIC); and Multiple Peripheral Interconnect Express (PCIe) switches, Each of the first intelligent NIC and the second intelligent NIC comprises a simple storage node, including: an uplink to a first Ethernet switch and a second Ethernet; Downlinks to a plurality of PCIe switches via a plurality of PCIe lanes, the plurality of PCIe lanes being used to connect to the storage device and a plurality of other storage devices.

13. The computer system according to claim 12, wherein: The request to read the file is also received by a second computing device, and is also transmitted by the second computing device to the storage device via the system; The second computing device is a backup or high availability server for the computing device.

14. The computer system according to claim 9, wherein: The method further comprises: storing, by the computing device, a mapping from a file name of the file to a current physical block address associated with the storage device; In response to detecting a condition to move the file from the current physical block address to a new physical block address associated with the storage device, a mapping of the file name of the file to the new physical block address is updated.

15. The computer system according to claim 9, wherein: The file formats stored in the storage device include one or more of the following: a preamble indicating a starting position associated with the file; A unique file identifier for the file; the content of the document; a postamble indicating the end position associated with the file; and A cyclic redundancy check (CRC) signature used to verify the consistency of file contents.

16. The computer system according to claim 9, wherein: The storage devices mentioned above do not include: A dynamic random access memory (DRAM) interface and a DRAM module accessed through the DRAM interface; and A processor that performs flash translation layer (FTL) functions, including mapping logical block addresses to physical block addresses.

17. A distributed storage system for facilitating one or more big data analysis applications through one or more clients, the distributed storage system comprising: Front-end server; switch; and A plurality of NAND cards connected to the switch; wherein the NAND cards do not include a module or unit for performing ECC encoding; Wherein, a NAND card is configured as: receiving, by the switch, a request to read a file stored on the NAND card; and The NAND card decodes the file based on the error correction code (ECC) to obtain ECC decoded data, and The front-end header server is configured to decompress the decoded data and return the decompressed data as the requested file.

18. The distributed storage system according to claim 17, in, The front-end header server is configured to receive a request to read a file, wherein the NAND card is further configured to: Retrieving the file from an original physical medium separate from the NAND card, wherein the file includes compressed data that was previously encoded based on an error correction code (ECC); and returning said ECC decoded data to the front-end header server without performing any ECC encoding; and Wherein, the NAND card does not include: A module or unit that performs ECC encoding; Dynamic random access memory (DRAM) interface and DRAM accessed through the DRAM interface modules; and a processor that performs flash translation layer (FTL) functions, including mapping logical block addresses to physical block addresses, This reduces the number of ECC codes, making it easier for big data analytics applications to analyze data efficiently.

Images