Storage system, data processing method, storage component, apparatus, and program product

Abstract

This application provides a storage system, a data processing method, a storage component, an apparatus, and a program product. The method of this application achieves data transmission between the host application and the storage component through a storage engine, eliminating the need for file system and block I / O layer operations. The storage component can interface with the upper-layer host application through the driver layer and storage engine, and perform read and write operations by manipulating the physical space. The storage medium in the storage component is divided into multiple superpages, each superpage containing at least one medium particle. The storage capacity of a single medium particle is much larger than the traditional minimum write unit. Therefore, the storage component writes data to the storage medium in superpage sizes, significantly improving write speed. Addressing within the storage medium space is achieved through offsets and lengths. The storage component directly allocates the offset and length in the storage medium for the data to be stored, eliminating the need for mapping from logical address (LBA) to physical address (PBA), reducing mapping overhead and improving read and write speeds.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a storage system, data processing method, storage component, device, and program product. Background Technology

[0002] In intelligent computing model training, intermediate results generated during training need to be stored in the storage component using key-value (KV) storage. During intelligent computing model inference, state data needs to be cached using key-value storage to accelerate the inference process.

[0003] Currently, key-value stores use a block storage approach, implemented through a file system and block storage components (such as solid-state drives). Block storage components perform read and write access using logical block addresses (LBAs) and response-aligned data.

[0004] Block storage requires converting key-value data into block-mode storage, resulting in a mapping from key-value storage to block storage, and a mapping from logical block address (LBA) to physical block address (PBA), generating complex mapping data. Especially in model training / inference scenarios, the size of data units for reading and writing is often several hundred KB to tens of MB. Dividing data units into traditional block storage sizes (e.g., 4KB) will generate hundreds or thousands of blocks, resulting in complex LBA to PBA mappings, which incur significant mapping overhead and impact read and write speeds. Summary of the Invention

[0005] This application provides a storage system, data processing method, storage component, device, and program product to solve the problem that traditional key-value storage schemes using block storage have high mapping overhead, which affects read and write speed.

[0006] In a first aspect, this application provides a storage system, including: a storage component and a storage engine.

[0007] The storage component includes: a controller, a cache, and a storage medium, wherein the storage medium comprises a plurality of superpages, each superpage comprising at least one media particle, and the plurality of superpages comprising the same number of media particles;

[0008] The storage engine is used to: transfer data to be stored in the host application to the controller of the storage component;

[0009] The controller of the storage component is configured to: receive the data to be stored, allocate an offset and length in the storage medium for the data to be stored, store the data to be stored in the cache, aggregate the data in the cache into superpage data according to the superpage size, and write the superpage data into the storage medium.

[0010] Secondly, this application provides a data processing method applied to a controller of a storage component, the storage component including the controller, a cache, and a storage medium, the storage medium comprising a plurality of superpages, each superpage comprising at least one media particle, and the plurality of superpages comprising the same number of media particles; the method includes:

[0011] Receive data to be stored from the storage engine on the host machine;

[0012] The data to be stored is allocated an offset and length in the storage medium, the data to be stored is stored in the cache, and the data in the cache is aggregated into superpage data according to the size of the superpage.

[0013] Write the superpage data to the storage medium.

[0014] Thirdly, this application provides a data processing method applied to a storage engine running on a host computer, the method comprising:

[0015] Receive data to be stored from the host application running on the host;

[0016] The data to be stored is transmitted to the controller of the storage component, so that the controller of the storage component allocates an offset and length in the storage medium for the data to be stored, stores the data to be stored in a cache, aggregates the data in the cache into superpage data according to the size of the superpage, and writes the superpage data into the storage medium, wherein the superpage contains at least one medium particle of the storage medium.

[0017] Fourthly, this application provides a storage component, including: a controller, a cache, and a storage medium.

[0018] The storage medium includes multiple superpages, each superpage containing at least one media particle, and the multiple superpages contain the same number of media particles.

[0019] The controller is used to: receive data to be stored transmitted from the storage engine on the host machine;

[0020] The data to be stored is stored in the cache, and the data in the cache is aggregated into superpage data according to the size of the superpage.

[0021] Write the superpage data to the storage medium.

[0022] Fifthly, this application provides a host device running a storage engine and a host application, including:

[0023] At least one processor; and

[0024] A memory that is communicatively connected to the at least one processor;

[0025] The memory stores instructions that can be executed by the at least one processor to cause the host device to perform the method provided in the third aspect above.

[0026] Sixthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0027] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0028] The storage system, data processing method, storage component, device, and program products provided in this application achieve data transmission between the host application and the storage component through a storage engine, eliminating the need for file system and block I / O layer operations. The storage component can interface with the upper-layer host application through the driver layer and storage engine, and perform read and write operations by manipulating the physical space. The storage medium in the storage component is divided into multiple superpages, each containing at least one media particle. The storage capacity of a single media particle is much larger than the traditional minimum write unit (e.g., the minimum read and write unit of NAND flash memory is a "page," typically 4KB in size). Therefore, the storage component writes data to the storage medium at the superpage size, significantly improving write speed. Furthermore, addressing within the storage medium space is achieved through offset and length. The storage component directly allocates the offset and length in the storage medium for the data to be stored, eliminating the need for mapping from logical address (LBA) to physical address (PBA), reducing mapping overhead and improving read and write speeds. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] Figure 1 A schematic diagram of the key-value store architecture provided in this application;

[0031] Figure 2 A schematic diagram illustrating the implementation of key-value storage using block storage in this application;

[0032] Figure 3 An architecture diagram of a storage system provided for an exemplary embodiment of this application;

[0033] Figure 4 A schematic diagram of a key-value storage writing process provided for an exemplary embodiment of this application;

[0034] Figure 5 A schematic diagram illustrating the execution process of a write operation on a storage component provided in an exemplary embodiment of this application;

[0035] Figure 6 A schematic diagram illustrating the insertion of new hash information into a hash library as provided in an exemplary embodiment of this application;

[0036] Figure 7 A schematic diagram illustrating the execution process of a read operation of a storage component provided in an exemplary embodiment of this application;

[0037] Figure 8 A schematic diagram illustrating a serial comparison positioning and reading position provided for an exemplary embodiment of this application;

[0038] Figure 9 A schematic diagram illustrating the deletion and updating of data in a hash library provided as an exemplary embodiment of this application;

[0039] Figure 10 A schematic diagram of a controller for a storage component provided in an exemplary embodiment of this application;

[0040] Figure 11 A flowchart of a data processing method provided in an exemplary embodiment of this application;

[0041] Figure 12 A flowchart of a data processing method provided as another exemplary embodiment of this application;

[0042] Figure 13 An interactive flowchart of data processing provided for an exemplary embodiment of this application.

[0043] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0046] First, let me explain the terms used in this application:

[0047] Storage media particles, also known as storage media chips, are the core components in storage devices (such as solid-state drives and memory modules) used for actual data storage. They achieve data writing, reading, and saving through specific technologies (such as flash memory technology and dynamic random access memory technology).

[0048] A block is a logical partition or data unit within a storage medium particle, especially in non-volatile storage media such as NAND Flash, and usually refers to the smallest unit that can be erased.

[0049] Cache line: The basic unit (or smallest data block) in the processor's cache, used to store contiguous memory areas of data.

[0050] Write amplification (WAF) refers to the ratio between the amount of data actually written to a storage component (such as a hard drive) and the amount of data that needs to be written. Write amplification is typically a value greater than 1, meaning that more data is written to the hard drive than the amount of data needs to be written.

[0051] Block I / O layer: Primarily responsible for transforming, merging, sorting, and scheduling I / O requests from the file system, and ultimately sending the requests to the block device driver layer. It provides a unified I / O interface for the upper-layer file system by abstracting the details of the underlying hardware devices.

[0052] Data consistency unit: In data transmission, this typically refers to a block of data that is treated as a whole during transmission. Taking 4KB as an example, this means that each transmission or processing operation treats 4KB of data as an independent unit to ensure consistency and integrity.

[0053] Intelligent computing models primarily refer to models constructed using intelligent computing technologies and methods. These models are capable of handling complex, high-dimensional, dynamic, and unstructured data and problems. Intelligent computing models are an important component of the field of artificial intelligence, integrating knowledge from multiple disciplines such as computer science, mathematics, physics, and statistics. They optimize and enhance the functionality and performance of technical systems through intelligent technological means.

[0054] Intelligent computing models can encompass various types, such as neural network models, deep learning models, and machine learning models. These models, through different algorithms and architectures, achieve functions such as data analysis, processing, learning, and prediction. Intelligent computing models have wide applications in multiple fields, including big data analytics, intelligent robots, autonomous driving, smart homes, smart cities, smart healthcare, and fintech. These fields all require handling massive amounts of data and complex scenarios, and intelligent computing models can provide powerful computing capabilities and intelligent solutions, offering strong support for the development of these fields.

[0055] In intelligent computing model training, batch training data needs to be continuously loaded. This batch training data is typically in units of hundreds of KB to tens of MB. The loading of batch data is often done at a granularity much larger than the file system partition format (e.g., 4KB) to read, transfer, and import it into the model training. In addition, during the training process, state data is saved (called checkpointing), which is the sequential writing of the current results in batches. This can be seen as storing the required data in a key-value storage manner in a persistent storage component in sequence.

[0056] In intelligent inference, key-value caching is used to temporarily store state data to accelerate the inference process. Currently, key-value caching uses the graphics processing unit's (GPU) built-in video memory or mapped host memory (DIMM) capacity to store data. However, as the model size increases, the required cache capacity becomes extremely large, making it difficult to meet the key-value caching needs using expensive video memory or host memory.

[0057] Figure 1 The diagram shows the architecture of a key-value store. There is a one-to-one correspondence between keys and values; the key is used to query and save the corresponding value (i.e., data). Figure 1The diagram shows N sets of key-value data, where each set includes a corresponding key and value. For example... Figure 1 As shown, Key1 corresponds to Value1, Key2 corresponds to Value2, ..., Key N-1 corresponds to Value N-1, and Key N corresponds to Value N. The basic process of key-value storage is as follows: a hash function with an extremely low collision probability is selected, and the hash value corresponding to the Key is calculated using this hash function and stored in the hash database for subsequent comparison and query operations. This hash value is used as an index to record the record position of the corresponding data (i.e., Value) in the data bucket, thereby enabling subsequent read, delete, and update access. To improve write performance, the data bucket typically uses a log append write mode, and the storage address of the Value in the data bucket is usually recorded using a starting address (Offset) and data length (length).

[0058] Currently, key-value stores are implemented using block storage, through a file system and block storage components (such as solid-state drives, SSDs). The block storage physical components perform read and write access using Logical Block Addresses (LBAs) and response-aligned data, as follows: Figure 2 As shown. First, the KV storage application calls the file system through the system call layer, combining the metadata generated, stored, and updated by the file system to map the KV storage to the file system block storage. This process includes: aligning the Value in the KV data with the beginning of the file system block, splitting the Value according to the block size (e.g., 4KB), padding the split blocks, establishing the association mapping between the Key and the corresponding LBA, and recording and maintaining it. Figure 2 The image shows the KV to block mapping metadata. Further, equal-length data segments aligned with the LBA, along with the LBA allocated by the Block IO layer, are sent to the storage unit via a driver. The storage unit internally performs a mapping from logical address LBA to physical address PBA, thereby determining the physical address where the data is written and read. This enables KV storage in current standard block storage units (such as solid-state drives, SSDs).

[0059] Traditional key-value storage solutions using block storage have the following technical drawbacks:

[0060] 1) Existing block storage components (such as SSDs) only support block storage interfaces; key-value storage must be converted to block-mode storage to enable read and write access. Therefore, this has led to… Figure 2 The two layers of metadata shown, namely the KV storage to block storage mapping metadata and the LBA to PBA mapping metadata, generate a large amount of mapping data.

[0061] 2) In intelligent computing scenarios, the unit size (i.e., the capacity of the value) of batch data read and write is mostly from several hundred KB to tens of MB. However, dividing a value into traditional block storage sizes (e.g., 4KB) will generate hundreds or thousands of blocks and produce complex LBA to PBA mapping data. The mismatch between the block size and the value scale increases the mapping overhead and affects the read and write speed.

[0062] 3) The alignment, splitting, and padding (e.g., padding zeros) of values ​​during conversion to block mode result in wasted capacity and increased write amplification.

[0063] This application provides a storage system including a storage component and a storage engine. The storage component includes a controller, a cache, and a storage medium. The storage medium contains multiple superpages, each superpage containing at least one media particle, and the number of media particles contained in each superpage is the same. During data storage, the storage engine transmits data to be stored from the host application to the controller of the storage component. The controller of the storage component receives the data to be stored, stores it in the cache, and aggregates the data in the cache into superpage data according to the superpage size, so that the data to be stored is aggregated into superpage data of the superpage size in the cache, and then writes the superpage data to the storage medium.

[0064] The storage system provided in this application enables data transfer between host applications and storage components through a storage engine, eliminating the need for file system and block I / O layer operations. The storage components can interface with the upper-layer host application through the driver layer and storage engine, and perform read and write operations by manipulating the physical space. The storage medium within the storage components is divided into multiple superpages, each containing at least one media particle. The storage capacity of a single media particle is significantly larger than the traditional minimum write unit (e.g., the minimum read and write unit for NAND flash memory is a "page," typically 4KB in size). Therefore, the storage components write data to the storage medium at superpage sizes, greatly improving write speed. Furthermore, addressing within the storage medium space is achieved through offset and length. The storage components directly allocate the offset and length in the storage medium for the data to be stored, eliminating the need for mapping from logical address (LBA) to physical address (PBA), reducing mapping overhead and improving read and write speeds.

[0065] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0066] Figure 3This is an architecture diagram of the storage system provided in an embodiment of this application. Figure 3 As shown, the storage system includes a storage component 31 and a storage engine 32. The storage engine 32 is a program running on the host machine, capable of interacting with host applications running on the host. The storage engine 32 can transfer data with the storage component via a driver on the host machine.

[0067] The storage component 31 includes a controller 311, a cache 312, and a storage medium 313. The storage medium 313 contains multiple superpages, each superpage containing at least one media particle, and the multiple superpages contain the same number of media particles.

[0068] In practical applications, a storage medium (such as a NAND flash memory chip) contains multiple blocks. A block is the smallest unit of an erase operation, and a block usually contains multiple pages. A page is the smallest unit of a read and write operation. Figure 3 The example provided illustrates this by describing a superpage containing N media particles, and a media particle containing M blocks. Figure 3 As shown, medium particle 1 includes the following M blocks: Block 1, Block N+1, ..., Block (M-1)N+1. Medium particle N includes the following M blocks: Block N, Block 2N, ..., Block MN. Where N and M are both positive integers.

[0069] In this embodiment, the controller of the storage component manages N media particles, and all N media particles are activated simultaneously to increase concurrency. The physical space of each media particle in each channel that can simultaneously read and write data is called a superpage, meaning a superpage contains N media particles. Based on the writing order of physical addresses in the storage medium, the physical space of the storage medium can be viewed as a one-dimensional array of multiple superpages connected end-to-end. This one-dimensional array is used to store data, such as key-value data. The addressing of a data object in the one-dimensional array can be represented by offset and length. The offset represents the starting address of the data object in the physical space of the storage medium, and the length represents the length of the stored data object.

[0070] In this embodiment, the unit of data storage can be set according to the length requirements of the data to be stored. When the storage capacity of the key-value storage component is given, a smaller unit (e.g., 256B or 64B) results in a longer bit length representing a set of offsets and lengths. In practical applications, the unit of representation can be designed by comprehensively considering factors such as the storage capacity of the storage component, the amount of metadata information, and the overhead of DRAM (Dynamic Random Access Memory). For example, the unit of data storage can be set to the unit used for block storage (e.g., 4KB).

[0071] For example, the representation unit of data storage can be set to be smaller than the representation unit used by block storage (such as 4KB). For example, the representation unit can be set to 64B, 256B, etc., to reduce the capacity overhead of fill alignment.

[0072] For example, assuming the storage capacity of a storage unit is 32TB, if expressed in units of 64B, then 32TB contains 2 39 The 32TB / 64B representation unit requires 39 bits for both offset and length. Adding a bit for checksum verification (such as single parity), the offset and length each require 40 bits, or 5B. The 64B representation unit aligns with the cache line capacity, satisfying the key-value storage requirements of variables in memory. However, considering that the data to be stored may not be an integer multiple of 64B, when the data length is not an integer multiple of 64B, up to 64B of padding data is used to align it to an integer multiple of 64B. Compared to the traditional block storage's maximum 4KB padding alignment, this reduces the capacity overhead of padding alignment.

[0073] In this embodiment, the storage engine is used to transfer data to be stored in the host application to the controller of the storage component. Specifically, the storage engine converts the data to be stored into a format that can be transmitted between the host and the storage component, and, while ensuring data consistency, implements data transmission between the storage component and the host via a driver.

[0074] The driver can be a general-purpose driver on the host, such as an NVMe (Non-Volatile Memoryexpress, host controller interface specification) driver, a CXL (Compute Express Link, compute acceleration link) driver, or a SATA (Serial Advanced Technology Attachment) driver.

[0075] The controller of the storage unit is used to implement data storage. Specifically, the controller receives the data to be stored, allocates an offset and length for the data in the storage medium, and stores the data in the cache. Based on the superpage size, the data in the cache is aggregated into superpage data, so that the data to be stored is aggregated into superpage data of the superpage size in the cache. The controller writes the superpage data to the storage medium, realizing the writing of data to the storage medium at superpage sizes.

[0076] It should be noted that for value data whose length is less than the page size, multiple value data are concatenated end-to-end into a single page. For value data whose length is greater than the page size, the value data is split into multiple page data, which can also be described as concatenating the value data into multiple page data that are concatenated end-to-end.

[0077] In one embodiment, the controller of the storage component may include a host interface. The host interface enables data transfer between the host and the storage component controller. When the storage engine transmits data to the storage component, it transmits the data via a driver to the host interface of the storage component controller. The storage component controller receives data from the host side through the host interface.

[0078] The controller for the storage component may include a cache interface (such as a DRAM interface) and a storage media interface. The cache interface enables data transfer between the controller and the cache. The storage media interface enables data transfer between the controller and the storage media.

[0079] This embodiment provides a novel storage system comprising a storage component and a storage engine. Data transfer between the host application and the storage component is achieved through the storage engine, eliminating the need for file system and block I / O layer operations. The storage component can interface with the upper-layer host application through the driver layer and storage engine, and perform read and write operations by manipulating the physical space. The storage medium within the storage component is divided into multiple superpages, each containing at least one media particle. The storage capacity of a single media particle is significantly larger than the traditional minimum write unit (e.g., the minimum read and write unit for NAND flash memory is a "page," typically 4KB in size). Therefore, the storage component writes data to the storage medium at superpage sizes, greatly improving write speed. Furthermore, addressing within the storage medium space is achieved through offset and length. The storage component directly allocates the offset and length in the storage medium for the data to be stored, eliminating the need for logical address (LBA) to physical address (PBA) mapping, reducing mapping overhead and improving read and write speeds.

[0080] Figure 4This is a schematic diagram of a key-value storage write process provided in an exemplary embodiment of this application. Based on the storage system provided in the foregoing embodiment, when implementing key-value data storage, the storage engine obtains the key-value data to be stored in the host application. The key-value data includes key data and value data. Based on the host transmission protocol, the value data is padded and aligned to obtain first padded and aligned data, which includes value data and first padded data. The key data and the first padded and aligned data are then transmitted to the controller.

[0081] The controller of the storage unit receives key data and first padding alignment data from the storage engine. The controller extracts value data from the first padding alignment data, assigns an offset and length to the value data, and stores the value data in a cache. Based on the superpage size, the controller aggregates the value data in the cache into superpage data, ensuring that the value data in the cache is aggregated into superpage data of the superpage size. Furthermore, the controller writes the superpage data to the storage medium, thus writing data to the storage medium at superpage sizes.

[0082] Specifically, such as Figure 4 As shown, when a host application needs to store key-value data, it sends the key-value data to be stored to the storage engine. The key-value data to be stored includes key data and value data. For the value data within the key-value data, the storage engine pads and aligns the value based on the host transport protocol to obtain first padding-aligned data, ensuring data transmission consistency. The first padding-aligned data includes the value and the first padding data, as shown... Figure 4 The {value, first padding data} shown is shown.

[0083] Furthermore, the storage engine transmits the first padding and alignment data to the storage unit's controller via the driver. For the key data within the key-value data, the storage engine transmits the key data to the storage unit's controller via the driver.

[0084] For example, when padding and aligning the value, the value is padded and aligned according to the host transport protocol's data consistency unit (e.g., 4KB), ensuring that the resulting first padded and aligned data is an integer multiple of the data consistency unit. For instance, with a data consistency unit of 4KB, if a value has a length of 421KB, the storage engine will pad the value to 424KB. The value padding and alignment can be implemented using any existing data padding method based on the host transport protocol's data consistency unit; no specific limitation is made here.

[0085] The storage unit's controller receives the key data (key) and corresponding first padding alignment data from the storage engine. For the first padding alignment data, the controller discards the first padding data itself, extracts the original value data (value) from it, and then determines the value based on its length (length x) and the current write pointer position (e.g., ...). Figure 4 As shown in the diagram, Offset x) allocates a contiguous space of length x in the physical space of the storage medium to store the value data. The current write pointer position Offset x is used as the offset of the value data, and the length of the value data is length x, thus determining the offset and length of the value data.

[0086] For example, if the data length of the value is 421KB, and the length of the first padding and alignment data obtained after padding and alignment of the value is 424KB, the controller discards 3KB of padding data and obtains the original value with a length of 421KB.

[0087] Furthermore, the controller stores the value data in a cache. One or more values ​​are aggregated in the cache into a superpage, and the controller writes the cached value data to the storage medium in superpage size. Data is written to the storage medium using an append-only method.

[0088] For the received key data, the controller uses a hash function to perform a hash calculation on the key data to obtain a unique hash value for the key, such as... Figure 4 The Hash x shown is an example. The controller associates the hash value of the key with the offset and length of the value to generate hash information. This hash information includes the hash value of the key, the offset of the value, and its length. Figure 4 As shown, the hash value (Hash x) of the key, the offset (Offset x) of the value, and the length (Length x) constitute a hash record, which provides the addressing metadata of the value in the physical storage space. The uniqueness of the key's hash value maps a segment of physical storage space to key-value data and stores the value data. The value data can be accessed through its corresponding key's converted hash value, enabling value retrieval. The controller can store the hash information in a hash database for easy querying, updating, and deleting. The hash database can be stored in the storage component's cache.

[0089] In one optional implementation, after obtaining the first padding alignment data, the storage engine can generate a Cyclic Redundancy Check (CRC) code corresponding to the first padding alignment data and append the CRC to the first padding alignment data before transmitting it to the controller of the storage component. Upon receiving the first padding alignment data, the controller verifies the CRC appended to the first padding alignment data. If the CRC verification passes, the controller extracts the value data from the first padding alignment data and performs subsequent steps. For example, the CRC corresponding to the first padding alignment data can be appended to the end of the first padding alignment data.

[0090] This embodiment, based on the aforementioned storage system, implements a key-value data writing process. The storage engine obtains the key-value data to be stored from the host application, and pads and aligns the value data according to the host transmission protocol to obtain first padding-aligned data. The key data and the first padding-aligned data are then transmitted to the controller. The storage engine converts the data to be stored into a format that can be transmitted between the host and the storage component. While ensuring data consistency, the driver enables data transmission between the storage component and the host.

[0091] The controller of the storage unit receives key data and first padding alignment data; it extracts value data from the first padding alignment data, allocates offset and length to the value data, and stores the value data in a cache. Based on the superpage size, it aggregates the value data in the cache into superpage data, ensuring that the value data in the cache is aggregated into superpage data of the superpage size; it then writes the superpage data to the storage medium. Writing data to the storage medium in superpage size significantly improves write speed. Furthermore, addressing within the storage medium space is achieved through offset and length, eliminating the need for mapping from logical address (LBA) to physical address (PBA), reducing mapping overhead, and improving read and write speeds.

[0092] In one optional embodiment, the hash information stored in the hash library is sorted according to the size of the hash values ​​it contains. When storing hash information in the hash library, the controller compares the hash value of the key data in the hash information to be stored with the hash values ​​of the hash information already stored in the hash library to determine the insertion position of the hash information to be stored in the hash library; based on the insertion position, the hash information to be stored is inserted into the hash library to improve read and write speed.

[0093] Figure 5 This is a schematic diagram illustrating the execution process of a write operation on a storage component provided in an exemplary embodiment of this application. For example... Figure 5 As shown, when writing key-value data, the controller of the storage unit, on the one hand, determines the key-value data based on a hash function (such as...). Figure 5The hash function shown here performs a hash calculation on the key in the key-value data to obtain the hash value corresponding to the key (e.g., ...). Figure 5 (The Hash shown). On the other hand, a contiguous storage space is allocated for the value in the key-value data in the storage medium. The value is written sequentially into the contiguous storage space allocated for it in the storage medium, and the offset of the value is determined (e.g., ...). Figure 5 Offset (as shown) and length (as shown) Figure 5 (as shown in the length). Further, the controller generates hash information based on the addressing metadata of the value, including the hash value of the key, the offset of the value, and its length, such as... Figure 5 The [hash,{pffset,length}] shown is stored in the hash library.

[0094] Optionally, considering that hash values ​​are distributed in a sparse space, to facilitate querying, the hash information in the hash database is sorted in ascending order of hash value after comparison. Each time key-value data is written, new hash information is generated based on the key-value data to be written. The hash value in the new hash information is compared with the hash values ​​of existing hash information in the hash database, and the new hash information is inserted into the corresponding insertion position in the hash database according to the size relationship, so that the hash information in the hash database remains sorted in ascending order of hash value.

[0095] Figure 6 This is a schematic diagram illustrating the insertion of new hash information into a hash library provided as an exemplary embodiment of this application. For example... Figure 6 As shown, the new hash information obtained includes the hash value of the key, and the offset and length of the corresponding value, which can be denoted as hash information = {hash value, offset, and length}. To determine the insertion position of the new hash information in the hash database, the hash values ​​of existing hash information in the hash database need to be read multiple times and compared with the hash value of the new hash information. Each time, the position of the existing hash information to be read next is determined based on the current comparison result. For example, suppose the hash database currently stores 7 hash information items, and these 7 hash information items are arranged in ascending order. Then, the hash value at the 4th position (i.e., the middle position) in the hash database is read first, and the size of this read hash value is compared with the new hash value. If the new hash value is larger, the hash value at the middle position (i.e., the 6th position) from the portion after the 4th position is read, and the size of this read hash value is compared with the new hash value. If the new hash value is still larger, the hash value at the 7th position (i.e., one of the middle positions) in the hash database is read, and the size of the new hash value is compared with the new hash value. If the new hash value is larger, it is determined that the new hash value should be sorted into the 8th position. In the above process, similar reasoning can be used to deduce the process of selecting and comparing subsequent read positions when the new hash value is less than the read hash value.

[0096] Furthermore, after determining the insertion position of the new hash information in the hash database, such as Figure 6 As shown, inserting new hash information into the hash database according to the insertion position can be achieved in the following ways: 1) Create a new hash database (i.e., hash table); 2) Insert the new hash information into the new hash database at the position corresponding to the insertion position; 3) Copy the hash information in the old hash database before the insertion position to the new hash database before the insertion position, and copy the hash information in the old hash database after the insertion position to the new hash database after the insertion position; 4) Release the storage space of the old hash database.

[0097] In this embodiment, considering the frequent creation, copying, and deletion of the hash library, in order to reduce the amount of data transmitted and copied, it is not necessary to use a separate storage area to store offset and length information. Instead, the offset and length are associated with the hash value as a hash information. The offset and length move along with the hash value, which reduces system overhead and also lowers the probability of errors.

[0098] In an optional embodiment, the storage system provided in the foregoing embodiments can realize key-value data reading. In this embodiment, the storage engine is further configured to: receive target key data sent by the host application when data needs to be read, and transmit the target key data to the controller.

[0099] The controller is also used to: receive target key data, perform hash calculation on the target key data to obtain the target hash value of the target key data; look up the offset and length associated with the target hash value in the hash database according to the target hash value; read data from the storage medium according to the offset and length associated with the target hash value, and determine the target value data according to the data reading result; and transfer the target key data and target value data to the storage engine.

[0100] The storage engine is also used to transfer target key data and target value data to the host application.

[0101] Figure 7 This is a schematic diagram illustrating the execution process of a read operation on a storage component provided in an exemplary embodiment of this application. For example... Figure 7 As shown, when reading data, the host application sends the key data to the storage engine, and the storage engine transmits the key data to the controller of the storage unit. The controller of the storage unit then uses a hash function (such as...) to... Figure 7 The hash function shown in the image performs a hash calculation on the key to obtain the hash value corresponding to the key (e.g., ...). Figure 7(See the Hash example shown). The controller can first check if the hash value corresponding to the key exists in the hash database. If it does not exist, it returns "No such data". If it exists, it searches for the hash information containing the hash value corresponding to the key in the hash database, extracts the offset and length of the corresponding value from the hash information, and reads the data from the physical space of the storage medium according to the offset and length of the corresponding value to obtain the data read result.

[0102] Optionally, a Bloom filter can be used to check if the hash value corresponding to the key exists in the hash database, enabling fast hash value lookup. Alternatively, other similar filters or algorithms can be used to check if the hash value corresponding to the key exists in the hash database; this embodiment does not impose specific limitations on this.

[0103] Finding hash information containing the hash value corresponding to a key in a hash database can be broken down into multiple sequential comparisons. The search range in the hash database is adjusted based on the comparison results, and finally the location of the hash value corresponding to the key is found in the hash database.

[0104] For example, Figure 8 This is a schematic diagram illustrating the serial comparison and positioning of the read location provided in this embodiment. (See diagram below.) Figure 8 As shown, assume the current hash library already stores 7 hashes, and these 7 hashes are arranged in ascending order. In the key-value data reading process, according to the hash function (such as...) Figure 8 The hash function shown calculates the hash value of the target key data. When searching for the location in the hash database containing the hash value of the target key data, a loop comparison is used. The hash values ​​of existing hash information in the hash database are read sequentially and compared with the hash value of the target key data. Each time, the position of the next existing hash information to be read is determined based on the current comparison result, thus determining the target position of the target key data's hash value in the sorted hash database, and subsequently finding its corresponding offset and length.

[0105] like Figure 8 As shown, the fourth position in the hash library is read for the first time (e.g., Figure 8 The hash value at the position indicated by 1st in the diagram is compared with the hash value of the target key data. If they match, the current read position is determined to be the target position of the target key data's hash value in the sorted hash database. If they do not match, and the target key data's hash value is greater than the read hash value, the current read position (e.g., the position pointed to by 1st in the diagram) is reset. Figure 8 The 6th position after the position pointed to by 1st in the diagram (as shown) Figure 8The position pointed to by 2nd B in the diagram is used as the second read position; if the hash value of the target key data key is less than the hash value read, then the current read position (e.g., the position pointed to by 2nd B in the diagram) ... Figure 8 The second position before the position pointed to by 1st in the diagram (as shown) Figure 8 The position pointed to by 2nd A in the diagram is used as the second reading position.

[0106] If the second read position is the position pointed to by 2nd A, then the hash value read from the hash database at the position pointed to by 2nd A is compared with the hash value of the target key data. If they match, the target position of the target key data's hash value in the sorted hash database is determined to be the position pointed to by 2nd A. If the hash value read from the position pointed to by 2nd A does not match the hash value of the target key data, and the hash value of the target key data is greater than the read hash value, then the position pointed to by 3rd B is used as the third read position. If the hash value read from the position pointed to by 2nd A does not match the hash value of the target key data, and the hash value of the target key data is less than the read hash value, then the position pointed to by 3rd A is used as the third read position.

[0107] If the second read position is the location pointed to by 2nd B, the hash value read from the hash database at the location pointed to by 2nd B is compared with the hash value of the target key data. If they match, the target position of the target key data's hash value in the sorted hash database is determined to be the location pointed to by 2nd B. If the hash value read from the location pointed to by 2nd B does not match the hash value of the target key data, and the hash value of the target key data is greater than the read hash value, then the location pointed to by 3rd D is used as the third read position. If the hash value read from the location pointed to by 2nd B does not match the hash value of the target key data, and the hash value of the target key data is less than the read hash value, then the location pointed to by 3rd C is used as the third read position.

[0108] If the hash value read on the third read matches the hash value of the target key, the position of the third read is taken as the target position of the target key's hash value in the sorted hash database. If the hash value read on the third read still does not match the hash value of the target key, a message indicating that the corresponding data cannot be found is returned.

[0109] In an optional embodiment, the storage engine is further configured to: receive a key-value data deletion instruction issued by the host application, and transmit the key-value data deletion instruction to the controller, wherein the key-value data deletion instruction includes specified key data, and the specified key data is the key data in the key-value data to be deleted.

[0110] The controller of the storage unit is also used to: receive key-value data deletion instructions, perform hash calculation on the specified key data to obtain the specified hash value corresponding to the specified key data; search for hash information containing the specified hash value in the hash database according to the specified hash value, and use it as the target hash information; delete the target hash information from the hash database, and release the storage space in the storage medium corresponding to the offset and length of the target hash information.

[0111] Figure 9 This is a schematic diagram illustrating the deletion and updating of data in the hash library provided in this embodiment. For example... Figure 9 As shown, when deleting key-value data, a specified hash value is calculated based on the hash function corresponding to the specified key data to be deleted (denoted as the specified key). The hash information containing the specified hash value in the hash database (such as...) Figure 9 The deletion of [h1,{o1,l1}] indicates that the hash value is released, and the space occupied by the offset and length in the hash information in the hash library (as shown in the image) is freed. Figure 9 The space pointed to by a1 in the diagram is also released. Since the offsets and lengths of each key-value pair are in the same format, the space in the released hash library can be used for the next key-value pair writing. The controller deletes the value data from the physical storage space of the storage medium according to the offset and length corresponding to the specified hash value. The physical storage space occupied by the value data is released and asynchronously reclaimed in the background.

[0112] When new key-value data is written, a new hash value, denoted as h2, is calculated based on the new key data. The hash value h2 can be bound to position a1 in the recently released hash database, and the space corresponding to position a1 stores the hash value h2, as well as the offset and length of the new value in the physical space of the storage medium (e.g., ...). Figure 9 The [h2,{o2,l2}] shown (is determined by sequentially allocating the corresponding physical space after writing the pointer).

[0113] The process of updating key-value data can be viewed as a combination of deleting and writing key-value data. After the key in the key-value data is updated, a new hash value is generated (such as h2), and after the value is updated, it is written to a new physical address (such as o2, l2). Background reclamation of the physical space of the storage medium involves performing operations such as copying, migrating, and erasing of actual data, depending on the specific conditions of the storage medium.

[0114] In one optional embodiment, when the controller stores value data in a cache and aggregates the value data in the cache into superpage data according to the superpage size, so that the value data in the cache is aggregated into superpage data of the superpage size, the controller compresses the value data to obtain compressed data; determines the offset and length corresponding to the value data based on the length of the compressed data; aligns the compressed data with padding according to a preset representation unit to obtain padded value data; stores the padded value data in the cache, and aggregates the padded data in the cache into superpage data according to the superpage size, so that the padded value data in the cache is aggregated into superpage data of the superpage size. By compressing the value data to be stored before storage, storage space can be saved, storage efficiency can be improved, I / O operations can be reduced, and data access speed can be accelerated.

[0115] Here, the preset representation unit refers to the representation unit of data storage. In this embodiment, the preset representation unit can be smaller than the representation unit used by block storage (such as 4KB). For example, the representation unit can be set to 64B, 256B, etc., to reduce the capacity overhead of padding alignment.

[0116] Optionally, when implementing the storage of fill value data into the cache, the controller can add an error correction code (ECC) to the fill value data and store the fill value data with the added error correction code into the cache.

[0117] During the key-value data reading process, when determining the target value data based on the data reading result, the controller first verifies the error correction code in the data reading result. After the error correction code in the data reading result is verified, the fill value data contained in the data reading result is obtained.

[0118] Furthermore, the controller removes the padding data from the padding data to obtain the compressed data; the compressed data is then decompressed to obtain the target data.

[0119] Furthermore, the controller performs padding alignment on the target value data based on the data consistency unit (e.g., 4KB) in the host transport protocol to obtain second padding aligned data. The second padding aligned data includes the target value data and the second padding data. The controller adds the actual data length of the target value data and cyclic redundancy code to the second padding aligned data to obtain the target data to be transmitted; then, it transmits the target value data and the target data to the storage engine.

[0120] Furthermore, after the storage engine verifies the cyclic redundancy check in the target data, it extracts the target value data from the second padding and alignment data based on the actual data length of the target value data.

[0121] For example, Figure 10This is a schematic diagram of the controller for the storage component provided in this embodiment. In this embodiment, as... Figure 10 As shown, the controller for the storage component includes a host interface, a storage media interface, and a DRAM interface. The host interface is used for data transfer between the controller and the host. The storage media interface is used for data transfer between the controller and the storage medium. The DRAM interface is used for data transfer between the controller and the DRAM cache, enabling the storage of values ​​in the DRAM cache or the management of the hash library within the DRAM cache. Figure 10 As shown, the controller can protect the firmware used for storage medium space allocation and association, and includes the following functional modules: CRC check, length adjustment, compression, ECC encoding, ECC decoding, decompression, Hash function, and comparator.

[0122] based on Figure 10 The key-value data writing process implemented by the controller shown is as follows: First padding-aligned data is received through the host interface. This first padding-aligned data is obtained by the storage engine through padding alignment according to the host transport protocol's data consistency unit (e.g., 4KB). The size of the first padding-aligned data is an integer multiple of the data consistency unit (e.g., 4KB). The first padding-aligned data includes the value data and the first padding data, where the first padding data contains the actual data length of the value data. For example, the length of the transmitted first padding-aligned data is 424KB, and the actual data length of the value data is 421KB. The storage engine adds 3KB of padding data after the 421KB value, and inserts the actual length information of 421KB at a specific position (e.g., at the end) within the 3KB padding data to obtain the first padding-aligned data. The CRC of the first padding-aligned data is calculated and appended to the end of the first padding-aligned data before being transmitted to the controller of the storage unit.

[0123] The controller first performs a CRC check on the received first padding and alignment data to confirm that the received data was sent by the host driver. Specifically, after receiving 424KB of data, the controller calculates the CRC and compares the calculated CRC with the received CRC; if they are equal, the transmission is confirmed to be error-free; otherwise, an error is reported to the driver.

[0124] Furthermore, the controller adjusts the length of the first padding alignment data according to the actual data length of the value, discarding the padding data portion to restore the original value of the host application. The controller compresses the original value to obtain compressed data. Through the firmware's space allocation module, a contiguous physical space of corresponding length is allocated at the write pointer in the physical space of the storage medium according to the length of the compressed data to store the compressed data (i.e., the compressed value), and the offset and length of the value are determined.

[0125] The controller calculates the hash value corresponding to the key in the key-value data based on the hash function, associates the offset and length of the value with the hash value corresponding to the key, generates corresponding hash information, and stores the hash information in the hash library (which can be stored in the DRAM cache of the storage unit).

[0126] The hash information in the hash database is arranged in ascending order of hash value. When storing new hash information into the hash database, a comparator compares the new hash value with existing hash values ​​in the hash database to determine the insertion position of the new hash information, and then inserts the new hash information into the corresponding insertion position in the hash database.

[0127] When storing compressed data into the storage medium, the controller aligns the compressed data with padding according to a set representation unit (such as 64B, 256B, or other values) to obtain padding values. Then, the padding values ​​are ECC encoded, and error correction codes (ECC) are added to them. The ECC-enhanced padding values ​​are stored in a cache to achieve ECC protection. Further, after aggregating superpage data of the same size, the controller writes the padding values ​​and ECC codes for each value in the superpage data into the physical space of the storage medium, based on the offset and length of the padding values ​​for each value.

[0128] based on Figure 10The controller implementation for reading key-value data is as follows: It receives target key data (denoted as target key) from the host application via the storage engine through the host interface. The controller calculates the hash value corresponding to the target key using a hash function. A comparator compares the hash value of the target key with existing hash values ​​in the hash database, determining the target position of the target key's hash information in the hash database based on the comparison result. Based on the target position, it reads the hash information containing the target key from the hash database, extracting the corresponding offset and length. The controller directly reads the physical space of the storage medium based on the corresponding offset and length to obtain the data read result. The controller performs ECC decoding on the data read result to eliminate noise. Then, the controller obtains the padding data contained in the data read result, deletes the padding data to obtain compressed data, and decompresses the compressed data to obtain the target value data. Further, the controller performs padding alignment on the target value data according to the host transmission protocol's data consistency unit (e.g., 4KB) to obtain second-aligned padding data. The controller then adds the actual data length and CRC information of the target value data to the second padding data in the second-aligned padding data to obtain the target data. The length of the target data is an integer multiple of the data consistency unit (e.g., 4KB). The controller transmits the target data to the storage engine on the host via the host interface. After the storage engine performs a CRC check on the target data and it passes, it extracts the target value data from the target data according to the actual data length of the target value data. The storage engine then transmits the target value data and the target key data to the host application.

[0129] The storage system provided in this embodiment ensures and verifies the consistency of data transmission between the host and storage components. It incorporates data compression and solutions for addressing metadata creation, updating, and maintenance. The physical storage space of the storage medium is integrated into a one-dimensional array of storage space according to a set write mode (writing at superpage size). Space is allocated on this array with offsets and lengths for reading and writing value data. A hash function converts each key into a unique hash value, and the hash value of the key, along with the corresponding offset and length of the value, forms hash information, which is stored in a hash database for easy lookup and location in subsequent accesses. During read and write access, the corresponding offset and length are retrieved from the hash database based on the calculated hash value of the key. Based on the corresponding offset and length, the system directly performs write or read operations on contiguous physical addresses, improving read and write efficiency.

[0130] Figure 11This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. The execution entity in this embodiment is the controller of the storage component in the aforementioned storage system. Figure 11 As shown, the specific steps of this method are as follows:

[0131] Step S1101: Receive the data to be stored transmitted from the storage engine on the host machine.

[0132] Step S1102: Allocate the offset and length in the storage medium for the data to be stored, store the data to be stored in the cache, and aggregate the data in the cache into superpage data according to the size of the superpage.

[0133] Step S1103: Write the overpage data to the storage medium.

[0134] In one optional embodiment, the data to be stored is key-value data, which includes key data and value data. The controller receives the data to be stored transmitted from the storage engine on the host machine, specifically implemented in the following manner:

[0135] The process involves receiving key data and first padding alignment data. The first padding alignment data is obtained by padding and aligning the value data based on the host transport protocol. The first padding alignment data includes the value data and the first padding data. The data to be stored is stored in a cache. The data in the cache is aggregated into superpage data according to the superpage size, so that the data to be stored is aggregated into superpage data of the superpage size in the cache. This includes: extracting value data from the first padding alignment data and determining the offset and length of the value data; storing the value data in a cache; and aggregating the value data in the cache into superpage data according to the superpage size, so that the value data is aggregated into superpage data of the superpage size in the cache.

[0136] In an optional embodiment, the controller is also responsible for executing the following processing flow:

[0137] Perform a hash calculation on the key data to obtain the hash value of the key data; generate hash information based on the hash value of the key data and the offset and length of the value data. The hash information is used to associate the hash value of the key data with the offset and length of the corresponding value data; store the hash information in a hash library, where the hash information stored in the hash library is sorted according to the size of the hash values ​​it contains.

[0138] In one optional embodiment, an offset and length are allocated in the storage medium for the data to be stored, the value data is stored in a cache, and the value data in the cache is aggregated into superpage data according to the superpage size, so that the data to be stored is aggregated into superpage data of the superpage size in the cache. Specifically, this can be achieved in the following way:

[0139] The value data is compressed to obtain compressed data; an offset and length are allocated to the value data according to the length of the compressed data; the compressed data is padded and aligned according to the preset representation unit to obtain padded value data; the padded value data is stored in the cache; and the padded value data in the cache is aggregated into superpage data according to the superpage size, so that the padded value data in the cache is aggregated into superpage data of the superpage size.

[0140] In an optional embodiment, the controller is also responsible for executing the following processing flow:

[0141] Receive the target key data transmitted from the storage engine, perform hash calculation on the target key data to obtain the target hash value of the target key data; based on the target hash value, look up the offset and length associated with the target hash value in the hash database; based on the offset and length associated with the target hash value, read the data from the storage medium, and determine the target value data based on the data reading result; transmit the target key data and target value data to the storage engine.

[0142] Furthermore, based on the data reading results, the target value data is determined, and the target key data and target value data are transferred to the storage engine. This can be achieved in the following way:

[0143] After the error correction code in the data read result is verified, the padding data contained in the data read result is obtained; the padding data in the padding data is deleted to obtain the compressed data; the compressed data is decompressed to obtain the target data; based on the host transmission protocol, the target data is padded and aligned to obtain the second padded and aligned data, which includes the target data and the second padding data; the actual data length of the target data and the cyclic redundancy code are added to the second padded and aligned data to obtain the target data to be transmitted; the target data and the target data are transmitted to the storage engine.

[0144] In this embodiment, the implementation principle and technical effects of the storage engine and storage components in the storage system are described in the relevant content of the foregoing embodiments, and will not be repeated here.

[0145] Figure 12 This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. The execution entity in this embodiment is the storage engine in the aforementioned storage system. Figure 12 As shown, the specific steps of this method are as follows:

[0146] Step S1201: Receive the data to be stored sent by the host application running on the host.

[0147] Step S1202: The data to be stored is transferred to the controller of the storage unit, so that the controller of the storage unit allocates the offset and length of the data to be stored in the storage medium, stores the data to be stored in the cache, aggregates the data in the cache into superpage data according to the size of the superpage, and writes the superpage data into the storage medium. The superpage contains at least one medium particle of the storage medium.

[0148] In one optional embodiment, the data to be stored is key-value data, which includes key data and value data. The data to be stored is transmitted to the controller of the storage component, which can be achieved in the following manner:

[0149] Based on the host transmission protocol, the value data is padded and aligned to obtain the first padded and aligned data, which includes the value data and the first padded data; the key data and the first padded and aligned data are then transmitted to the controller.

[0150] In an optional embodiment, the storage engine is also used to perform the following processing:

[0151] Receive target key data sent by the host application when it needs to read data, and transmit the target key data to the controller; receive target key data and target data returned by the controller, the target data including target value data, the actual data length of the target value data and cyclic redundancy code; after the cyclic redundancy code in the target data passes the verification, extract the target value data from the target data according to the actual data length of the target value data; transmit the target key data and target value data to the host application.

[0152] In this embodiment, the implementation principle and technical effects of the storage engine and storage components in the storage system are described in the relevant content of the foregoing embodiments, and will not be repeated here.

[0153] Figure 13 This is an interactive flowchart of data processing provided for an exemplary embodiment of this application. Figure 13 As shown, in the actual data storage process, the interaction flow between the storage engine and storage components in the storage system is as follows:

[0154] Step S1301: The storage engine receives the data to be stored from the host application running on the host.

[0155] Step S1302: The storage engine transmits the data to be stored to the controller of the storage component based on the host transmission protocol.

[0156] Step S1303: The controller of the storage component receives the data to be stored transmitted from the storage engine on the host machine.

[0157] Step S1304: The controller of the storage unit allocates the offset and length of the data to be stored in the storage medium, stores the data to be stored in the cache, and aggregates the data in the cache into superpage data according to the size of the superpage.

[0158] Step S1305: The controller of the storage unit writes the overpage data to the storage medium.

[0159] In this embodiment, the implementation principle and technical effects of the storage engine and storage components in the storage system are described in the relevant content of the foregoing embodiments, and will not be repeated here.

[0160] This application provides a storage component, including a controller, a cache, and a storage medium. The storage medium includes multiple superpages, each superpage containing at least one media particle, and the multiple superpages contain the same number of media particles.

[0161] The controller is used to: receive data to be stored transmitted from the storage engine on the host; store the data to be stored in the cache; aggregate the data in the cache into superpage data according to the superpage size, so that the data to be stored is aggregated into superpage data of the superpage size in the cache; and write the superpage data to the storage medium.

[0162] The implementation principle and technical effects of the controller provided in this embodiment are detailed in the relevant descriptions of the controller of the storage component in the foregoing embodiments, and will not be repeated here.

[0163] This embodiment provides a host device on which a storage engine and a host application run. The host device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the host device to perform the processes performed by the host application, storage engine, and driver in the foregoing embodiments.

[0164] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0165] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solution provided in any of the above method embodiments. The specific functions and the technical effects that can be achieved are not described here.

[0166] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0167] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a GPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules from at least one processor.

[0168] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc. The aforementioned memory can be object storage service (OSS). The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk, or optical disc.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] The order of the embodiments described above is merely for illustrative purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0172] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0173] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A storage system, characterized in that, include: Storage components and storage engines The storage component includes: a controller, a cache, and a storage medium, wherein the storage medium comprises a plurality of superpages, each superpage comprising at least one media particle, and the plurality of superpages comprising the same number of media particles; The storage engine is used to: transfer data to be stored in the host application to the controller of the storage component; The controller of the storage component is configured to: receive the data to be stored, allocate an offset and length in the storage medium for the data to be stored, store the data to be stored in the cache, aggregate the data in the cache into superpage data according to the superpage size, and write the superpage data into the storage medium.

2. The storage system according to claim 1, characterized in that, The storage engine transmits data to be stored in the host application to the controller of the storage component, including: The storage engine obtains key-value data to be stored in the host application, wherein the key-value data includes key data and value data; Based on the host transmission protocol, the value data is padded and aligned to obtain first padded and aligned data, which includes the value data and the first padded data. The key data and the first padding alignment data are transmitted to the controller.

3. The storage system according to claim 2, characterized in that, The controller of the storage component receives the data to be stored, allocates an offset and length in the storage medium for the data to be stored, stores the data to be stored in the cache, aggregates the data in the cache into superpage data according to the superpage size, and writes the superpage data into the storage medium, including: The controller receives the key data and the first padding alignment data; Extract the value data from the first padding and alignment data, assign an offset and length to the value data, and store the value data in the cache; Based on the size of the superpage, the value data in the cache is aggregated into superpage data; Write the superpage data to the storage medium.

4. The storage system according to claim 3, characterized in that, The controller is also used for: Perform a hash calculation on the key data to obtain the hash value of the key data; Hash information is generated based on the hash value of the key data and the offset and length of the value data. The hash information is used to associate the hash value of the key data with the corresponding offset and length of the value data. The hash information is stored in a hash library.

5. The storage system according to claim 4, characterized in that, The hash information stored in the hash database is sorted according to the size of the hash values ​​it contains. The controller stores the hash information in the hash database, including: The controller compares the hash value of the key data in the hash information to be stored with the hash value in the hash information already stored in the hash library, and determines the insertion position of the hash information to be stored in the hash library based on the comparison result. According to the insertion position, the hash information to be stored is inserted into the hash library.

6. The storage system according to claim 3, characterized in that, The storage engine is also used to add a cyclic redundancy check code to the first padding alignment data; The controller is further configured to verify the cyclic redundancy check code on the first padding alignment data, and after the cyclic redundancy check code is verified, execute the step of extracting the value data from the first padding alignment data and subsequent steps.

7. The storage system according to claim 3, characterized in that, The controller stores the value data in the cache, and aggregates the value data in the cache into superpage data according to the superpage size, including: The controller compresses the value data to obtain compressed data; Based on the length of the compressed data, determine the offset and length corresponding to the value data; The compressed data is padded and aligned according to a preset representation unit to obtain padded value data; The fill value data is stored in the cache, and the fill value data in the cache is aggregated into superpage data according to the size of the superpage.

8. The storage system according to claim 7, characterized in that, The controller stores the fill value data into the cache, including: The controller adds an error correction code to the fill value data and stores the fill value data with the added error correction code into the cache.

9. The storage system according to any one of claims 1-8, characterized in that, The storage engine is also used to: receive target key data sent by the host application when it needs to read data, and transmit the target key data to the controller; The controller is further configured to: receive the target key data, perform a hash calculation on the target key data, and obtain the target hash value of the target key data; Based on the target hash value, find the offset and length associated with the target hash value in the hash database; Data is read from the storage medium based on the offset and length associated with the target hash value, and the target value data is determined based on the data reading results; The target key data and the target value data are transferred to the storage engine; The storage engine is also used to transmit the target key data and the target value data to the host application.

10. The storage system according to claim 9, characterized in that, The controller determines the target value data based on the data reading results, including: After the controller verifies the error correction code in the data reading result, it obtains the fill value data contained in the data reading result. Delete the padding data from the padding value data to obtain the compressed value data; The compressed value data is decompressed to obtain the target value data.

11. The storage system according to claim 9, characterized in that, The controller transmits the target key data and the target value data to the storage engine, including: The controller performs padding and alignment on the target value data based on the host transmission protocol to obtain second padding and alignment data, the second padding and alignment data including the target value data and the second padding data; In the second padding data of the second padding alignment data, the actual data length of the target value data and the cyclic redundancy code are added to obtain the target data to be transmitted; The target value data and the target data are transmitted to the storage engine.

12. The storage system according to claim 11, characterized in that, The storage engine is also used to extract the target value data from the target data based on the actual data length of the target value data after the cyclic redundancy check in the target data passes.

13. The storage system according to any one of claims 1-8, characterized in that, The storage engine is also configured to: receive a key-value data deletion instruction issued by the host application, transmit the key-value data deletion instruction to the controller, wherein the key-value data deletion instruction contains specified key data, and the specified key data is the key data in the key-value data to be deleted; The controller is also configured to: receive the key-value data deletion instruction, perform hash calculation on the specified key data, and obtain the specified hash value corresponding to the specified key data; Based on the specified hash value, search the hash database for hash information containing the specified hash value, and use it as the target hash information; Delete the target hash information from the hash library and release the storage space in the storage medium corresponding to the offset and length in the target hash information.

14. A data processing method, characterized in that, A controller for a storage component, the storage component including the controller, a cache, and a storage medium, the storage medium comprising a plurality of superpages, each superpage comprising at least one media particle, and the plurality of superpages comprising the same number of media particles; The method includes: Receive data to be stored from the storage engine on the host machine; The data to be stored is allocated an offset and length in the storage medium, the data to be stored is stored in the cache, and the data in the cache is aggregated into superpage data according to the size of the superpage. Write the superpage data to the storage medium.

15. The method according to claim 14, characterized in that, The data to be stored is key-value data, which includes key data and value data. The data to be stored transmitted by the storage engine on the receiving host includes: Receive the key data and first padding alignment data, wherein the first padding alignment data is obtained by padding and aligning the value data based on the host transport protocol, and the first padding alignment data includes the value data and the first padding data; The process of allocating an offset and length in the storage medium for the data to be stored, storing the data in the cache, and aggregating the data in the cache into superpage data according to the superpage size includes: Extract the value data from the first padding alignment data, and assign an offset and length to the value data; The value data is stored in the cache, and the value data in the cache is aggregated into superpage data according to the size of the superpage.

16. The method according to claim 15, characterized in that, Also includes: Perform a hash calculation on the key data to obtain the hash value of the key data; Hash information is generated based on the hash value of the key data and the offset and length of the value data. The hash information is used to associate the hash value of the key data with the corresponding offset and length of the value data. The hash information is stored in a hash library, wherein the hash information stored in the hash library is sorted according to the size of the hash values ​​it contains.

17. The method according to claim 15, characterized in that, The step of storing the value data in the cache and, based on the size of the superpage, aggregating the value data in the cache into superpage data includes: The value data is compressed to obtain compressed data; Based on the length of the compressed data, determine the offset and length corresponding to the value data; The compressed data is padded and aligned according to a preset representation unit to obtain padded value data; The fill value data is stored in the cache, and the fill value data in the cache is aggregated into superpage data according to the size of the superpage.

18. The method according to any one of claims 14-17, characterized in that, Also includes: Receive the target key data transmitted by the storage engine, perform a hash calculation on the target key data, and obtain the target hash value of the target key data; Based on the target hash value, find the offset and length associated with the target hash value in the hash database; Data is read from the storage medium based on the offset and length associated with the target hash value, and the target value data is determined based on the data reading results; The target key data and the target value data are transferred to the storage engine.

19. The method according to claim 18, characterized in that, The step of determining the target value data based on the data reading result and transmitting the target key data and the target value data to the storage engine includes: After the error correction code in the data reading result is verified, the fill value data contained in the data reading result is obtained; Delete the padding data from the padding value data to obtain the compressed value data; Decompress the compressed value data to obtain the target value data; Based on the host transport protocol, the target value data is padded and aligned to obtain second padded and aligned data, which includes the target value data and the second padded data. Add the actual data length of the target value data and the cyclic redundancy code to the second padding alignment data to obtain the target data to be transmitted; The target value data and the target data are transmitted to the storage engine.

20. A data processing method, characterized in that, The method, applied to a storage engine running on a host, includes: Receive data to be stored from the host application running on the host; The data to be stored is transmitted to the controller of the storage component, so that the controller of the storage component allocates an offset and length in the storage medium for the data to be stored, stores the data to be stored in a cache, aggregates the data in the cache into superpage data according to the superpage size, and writes the superpage data into the storage medium, wherein the superpage contains at least one medium particle of the storage medium.

21. The method according to claim 20, characterized in that, The data to be stored is key-value data, which includes key data and value data. The controller that transmits the data to be stored to the storage component includes: Based on the host transmission protocol, the value data is padded and aligned to obtain first padded and aligned data, which includes the value data and the first padded data. The key data and the first padding alignment data are transmitted to the controller.

22. The method according to claim 20 or 21, characterized in that, Also includes: Receive target key data sent by the host application when it needs to read data, and transmit the target key data to the controller; Receive target key data and target data returned by the controller, wherein the target data includes target value data, the actual data length of the target value data, and a cyclic redundancy code; After the cyclic redundancy check in the target data passes, the target value data is extracted from the target data according to the actual data length of the target value data; The target key data and the target value data are transmitted to the host application.

23. A storage component, characterized in that, include: Controller, cache, and storage media, The storage medium includes multiple superpages, each superpage containing at least one media particle, and the multiple superpages contain the same number of media particles. The controller is used to: receive data to be stored transmitted from the storage engine on the host machine; The data to be stored is stored in the cache, and the data in the cache is aggregated into superpage data according to the size of the superpage. Write the superpage data to the storage medium.

24. A host device, characterized in that, It runs storage engines and host applications, including: At least one processor; and A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the host device to perform the method of any one of claims 20-22.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 14-19 or 20-22.

26. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method as described in any one of claims 14-19 or 20-22.

Images