Metadata management method and device based on online deduplication function

Abstract

The present invention relates to the field of computer technology and provides a metadata management method and device based on an online deduplication function. The method comprises: receiving a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; when the metadata I / O request is a write request, the metadata I / O request includes the identifier key and a first logical address key in the metadata; based on the identifier key and the first logical address key, determining a key-value pair to be written, writing the key-value pair to be written into a non-volatile memory of a key-value disk; and recording the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk. The present invention reduces the complexity of metadata management process design on the storage controller and reduces resource consumption in the storage device.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a metadata management method and device based on an online deduplication function. Background Art

[0002] Inline deduplication runs the deduplication process while the target data block is being stored on the storage device. If the storage device already has a data block identical to the target data block (i.e., a duplicate), the target data block will not be written to disk but will be deleted before it is written. This ensures that only one copy of the same data is stored on the storage device, saving storage space.

[0003] To locate and determine whether data blocks are identical, each data block maintains a set of metadata. This metadata includes a mapping between the data block's fingerprint (i.e., a unique identifier for the data block, such as a hash value) and the physical block address (PBA) storing the data block, as well as a mapping between the data block's logical block address (LBA) and the physical block address. Currently, metadata management is implemented by the storage device's (multiple) controllers. For fast access, metadata is cached in the storage device's memory. However, with the rapid increase in user data volume, some metadata must also be stored on large-capacity disks. Therefore, each online deduplication operation requires metadata mirroring (metadata mirroring between multiple controllers), power-loss protection, and cache data organization structure conversion. This series of operations complicates metadata management logic and consumes significant CPU and memory resources on the storage device. Summary of the Invention

[0004] The present invention provides a metadata management method and device based on online deduplication function, which are used to solve the problem that the controller in the existing storage device has complex metadata management logic and consumes more CPU and memory resources of the storage device.

[0005] The present invention provides a metadata management method based on online deduplication function.

[0006] Receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value;

[0007] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into the non-volatile memory of the key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0008] According to a metadata management method based on online deduplication function provided by the present invention, based on the identification key and the first logical address key, the key-value pair to be written is determined, and the key-value pair to be written is written into the non-volatile memory of the key-value disk, including the following steps.

[0009] The identification key is searched in a record in the volatile memory of the key-value disk.

[0010] When the identification key is found, the value corresponding to the identification key is determined to be the target value, the key-value pair to be written is formed with the first logical address key and the target value, and the key-value pair to be written is written into the non-volatile memory of the key-value disk.

[0011] If the identification key is not found, the physical address block of the data block written by the storage controller is obtained as the target value, a first key-value pair is formed with the identification key and the target value, and a second key-value pair is formed with the first logical address key and the target value, and it is determined that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and the key-value pairs to be written are written into the non-volatile memory of the key-value disk.

[0012] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes the following steps.

[0013] The address index corresponding to the second logical address key is searched in the records of the volatile memory of the key-value disk.

[0014] A target storage address of the key-value pair to be read is obtained based on the address index corresponding to the second logical address key.

[0015] A physical address block corresponding to a second logical address key in a nonvolatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

[0016] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes the following steps.

[0017] An address index corresponding to the third logical address key is searched in the records of the volatile memory of the key-value disk.

[0018] The target storage address of the key-value pair to be deleted is obtained based on the address index corresponding to the third logical address key.

[0019] The key-value pair to be deleted is deleted from the non-volatile memory based on the target storage address.

[0020] The third logical address key and the corresponding address index are deleted from the record in the volatile memory of the key-value disk.

[0021] According to a metadata management method based on online deduplication function provided by the present invention, the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; writing the key-value pairs to be written into the non-volatile memory of the key-value disk includes: writing the key-value pairs to be written into the non-volatile memory, and when the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0022] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes the following steps.

[0023] The address index corresponding to the second logical address key is searched in the records of the volatile memory of the key-value disk.

[0024] A target storage address of the key-value pair to be read is obtained based on the address index corresponding to the second logical address key.

[0025] In a case where the target storage address is the address of the non-volatile memory, a physical address block corresponding to the second logical address key in the non-volatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

[0026] When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory, and then the physical address block corresponding to the second logical address key is read from the target storage block, and the physical address block is returned to the storage controller.

[0027] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes the following steps.

[0028] An address index corresponding to the third logical address key is searched in the records of the volatile memory of the key-value disk.

[0029] The target storage address of the key-value pair to be deleted is obtained based on the address index corresponding to the third logical address key.

[0030] In the case where the target storage address is the address of the non-volatile memory, the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory is deleted, and the third logical address key and the corresponding address index are deleted from the record of the volatile memory of the key-value disk.

[0031] In the case where the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory, the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0032] According to a metadata management method based on online deduplication function provided by the present invention, it also includes: when the key-value disk is started after power failure, traversing all key-value pairs in the non-volatile memory, rebuilding the address indexes of all key-value pairs, and storing the reconstructed address indexes and corresponding keys in the volatile memory.

[0033] The present invention also provides a metadata management device based on online deduplication function, which includes the following modules.

[0034] An I / O request receiving module is used to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0035] A metadata writing module is used to, when the metadata I / O request is a write request and the metadata I / O request includes an identification key and a first logical address key in the metadata, determine a key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into the non-volatile memory of the key-value disk, and record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0036] The present invention also provides a key-value disk, comprising: an in-disk processor, a volatile memory and a non-volatile memory, wherein the in-disk processor is connected to the volatile memory and the non-volatile memory.

[0037] The non-volatile memory is used to store key-value pairs related to metadata, and the key-value pairs include: a key-value pair with the identifier of the data block as the identification key and the physical address block of the data block as the value, and a key-value pair with the logical address block of the data block as the logical address key and the physical address block of the data block as the value.

[0038] The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory.

[0039] The processor in the disk stores a computer program, which is used to implement any of the above-mentioned metadata management methods based on the online deduplication function when executing the computer program.

[0040] The present invention also provides a storage device, comprising: a storage controller and the above-mentioned key-value disk, wherein the storage controller is connected to the on-disk processor and is used to send metadata I / O requests to the on-disk processor.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the metadata management method based on the online deduplication function as described above is implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for managing metadata based on online deduplication function as described above is implemented.

[0043] The metadata management method and device based on the online deduplication function provided by the present invention receive a metadata I / O request sent by a storage controller in a storage device, wherein in the metadata, the identifier of the data block is an identification key, the logical address block of the data block is a logical address key, and the physical address block of the data block corresponding to each of the identifier and the logical address block is a value; when the metadata I / O request is a write request, the metadata I / O request includes the identification key and the first logical address key in the metadata, and based on the identification key and the first logical address key, a key-value pair to be written is determined, the key-value pair to be written is written into a non-volatile memory of a key-value disk, and the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory are recorded in the volatile memory of the key-value disk. Since the metadata key-value pairs and corresponding address indexes are stored in the key-value disk during metadata write operations, when the storage controller performs the online deduplication function, it sends a metadata I / O request to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is one of the flow charts of the metadata management method based on the online deduplication function provided by the present invention.

[0046] Figure 2 The present invention provides a hardware structure diagram for implementing the metadata management method based on the online deduplication function.

[0047] Figure 3 This is the second flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0048] Figure 4 This is the third flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0049] Figure 5 This is the fourth flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0050] Figure 6 It is a structural diagram of a metadata management device based on online deduplication function provided by the present invention.

[0051] Figure 7 This is a flow chart of the storage device provided by the present invention implementing the online deduplication function when writing data.

[0052] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0054] The metadata management method based on online deduplication function of the embodiment of the present invention is as follows Figure 1 As shown, the following steps are included.

[0055] Step S110: Receive a metadata I / O request sent by a storage controller in a storage device, wherein in the metadata, the identifier of the data block is the identifier key, the logical address block of the data block is the logical address key, and the physical address block of the data block corresponding to the identifier and the logical address block is the value, that is, according to the mapping of the identifier H and the physical address block P in the metadata, a key-value pair HP (key-value) from the identifier H to the physical address block P is formed, and according to the mapping of the logical address block L and the physical address block P in the metadata, a key-value pair LP from the logical address block L to the physical address block P is formed.

[0056] It should be noted that the metadata management method based on online deduplication in this embodiment is no longer executed by the storage controller in the storage device, but rather by a key-value disk (e.g., a key-value solid-state drive (KV SSD), specifically, the on-disk processor in the key-value disk. Specifically, when a host performs an I / O operation (read, write, or delete) on a data block on the storage device, it sends an I / O request for the data block to the storage device. Since the storage device needs to perform corresponding operations on the metadata when performing operations on the data block, the storage controller initiates a metadata I / O request to the on-disk processor. For example, if a host needs to write a data block, the I / O write request for the data block sent by the host includes a logical address block L and the data block D to be written. Due to the online deduplication function, upon receiving the I / O write request for the data block, the storage controller calculates the identifier H (i.e., the hash value) of the data block D and sends the metadata I / O write request, including the identifier H and the logical address block L, to the on-disk processor in the key-value disk. Specifically, the key-value disk's Store operation interface can be invoked to send a metadata I / O write request. The interface details must be compatible with the NVM-Express-Key-Value-Command-Set-Specification in the NVMe 2.0 standard, which defines the Store, Exist, and Delete operation interfaces. In this embodiment, the Store operation interface can be implemented as follows: Steps S121 to S122.

[0057] Step S120: When the metadata I / O request is a write request, perform a write operation on the metadata. The metadata I / O request includes an identification key and a first logical address key in the metadata. Specifically, the metadata write operation includes the following steps.

[0058] Step S121: Based on the identification key and the first logical address key, determine the key-value pair to be written, and write the key-value pair to be written into the non-volatile memory of the key-value disk. The key-value pair to be written is LP, or LP and HP. It is necessary to determine whether it is LP, or LP and HP based on the query result of the online deduplication function. The non-volatile memory capacity of the key-value disk is large and can store metadata corresponding to massive data blocks. The key-value disk is preferably a KV SSD. KV SSD has a high read and write speed, which ensures efficient operation of metadata. Figure 2 As shown in FIG, a KV SSD is set in the storage device to store and manage metadata, and the remaining SSD disks are used to store data blocks.

[0059] Step S122: Record the key of the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk. Specifically, this is recorded using a hash table structure, where each key and the address index of the corresponding key-value pair are linked as a record in the hash table to facilitate fast query. Volatile memory has higher read and write speeds. Recording the relationship between the key and the address index of the corresponding key-value pair in the volatile memory of the key-value disk allows for rapid retrieval of the corresponding key-value pair address based on the key, thereby improving metadata query efficiency and the efficiency of subsequent read and delete operations.

[0060] In the metadata management method based on the online deduplication function of this embodiment, since the metadata key-value pairs and corresponding address indexes are all stored in the key-value disk during the metadata write operation, when the storage controller performs the online deduplication function, it sends a metadata I / O request to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device.

[0061] In some embodiments, as Figure 3 As shown, step S121 specifically includes the following steps.

[0062] Step S310: searching for the identification key in the records of the volatile memory of the key-value disk, that is, searching for the identification key H, to confirm whether the data block identified as H to be written has been stored in the SSD disk of the storage device.

[0063] Step S320: When the identification key is found, the value corresponding to the identification key is determined to be the target value, the key-value pair to be written is formed with the first logical address key and the target value, and the key-value pair to be written is written into the non-volatile memory of the key-value disk. Specifically, the identification key is found in the record of the volatile memory, indicating that the data block identified as H has been stored in the SSD disk of the storage device, and the HP key-value pair has been stored in the non-volatile memory of the key-value disk. The P in the HP key-value pair is the physical address block of the data block identified as H on the SSD disk of the storage device. The address index of the key-value pair corresponding to the identification key in the record of the volatile memory is obtained, the location of the key-value pair stored in the non-volatile memory of the key-value disk is found according to the address index, the key-value pair is read, and the physical address block where the data block identified as H was previously stored is obtained, that is, the above-mentioned target value. The first logical address key and the target value are combined to form a key-value pair to be written, and the pair is stored in the non-volatile memory of the key-value disk. Specifically, the first logical address specified by the host is associated with the physical address block previously stored in the data block identified as H, and the data block identified as H is no longer stored repeatedly. When the identification key is found in the record in the volatile memory, the on-disk processor of the key-value disk returns information indicating the identification key has been found to the storage controller via the Store operation interface. Upon receiving this information, the storage controller no longer stores the data block identified as H, thereby avoiding duplicate storage of the data block.

[0064] Step S330: If the identification key is not found, the physical address block of the data block to be written by the storage controller is obtained as the target value, a first key-value pair is formed with the identification key and the target value, and a second key-value pair is formed with the first logical address key and the target value, and the first key-value pair and the second key-value pair are determined to be the key-value pairs to be written, and the key-value pairs to be written are written into the non-volatile memory of the key-value disk. Specifically, if the identification key is not found, it indicates that the data block identified as H is not stored in the storage device. The on-disk processor of the key-value disk returns a storage command to the storage controller via the Store operation interface to store the data block identified as H. After receiving the command, the storage controller stores the data block identified as H and obtains the corresponding physical address block. The physical address block is returned via the Store operation interface, and the physical address block is used as the target value. A first key-value pair of the identification key and the target value is established, and a second key-value pair of the first logical address key and the target value is established. The first key-value pair and the second key-value pair are both written as the key-value pairs to be written into the non-volatile memory of the key-value disk.

[0065] It should be noted that to prevent hash conflicts and ensure identical data blocks when the identifiers are the same, the metadata I / O write request also includes the data block to be written. If the identifier key is found, the data blocks are compared. If they are identical, the identifier key is determined. Otherwise, an identifier conflict message is returned to the storage controller, and the next metadata I / O request from the storage controller is awaited. Upon receiving the identifier conflict message, the storage controller recalculates the identifier and issues the metadata I / O write request again.

[0066] In some embodiments, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: step S130: metadata read operation, the second logical address key corresponds to the logical address block of the data block to be read by the host, such as Figure 1 As shown, step S130 specifically includes the following steps.

[0067] Step S131: searching for the address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk.

[0068] Step S132: Obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key.

[0069] Step S133: Read the physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller so that the storage controller reads the data block to be read according to the physical address block and returns it to the host.

[0070] Specifically, the Exist operation interface of the key-value disk is called to send a metadata I / O read request. In this embodiment, the Exist operation interface can be implemented according to steps S131 to S133 described above. Because the corresponding metadata read operation is performed in the non-volatile memory of the key-value disk when reading a data block, the process of converting the cache data organization structure when a cache miss occurs in existing read methods is avoided, thereby improving the overall data block reading efficiency.

[0071] In some embodiments, when the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: step S140: metadata delete operation, the third logical address key corresponds to the logical address block of the data block to be deleted by the host, such as Figure 1 As shown, step S140 specifically includes the following steps.

[0072] Step S141: searching for the address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk.

[0073] Step S142: Obtain the target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key.

[0074] Step S143: deleting the key-value pair to be deleted from the non-volatile memory based on the target storage address.

[0075] Step S144: deleting the third logical address key and the corresponding address index from the record in the volatile memory of the key-value disk.

[0076] Specifically, the Delete operation interface of the key-value disk is called to send a metadata I / O delete request. In this embodiment, the Delete operation interface can be implemented according to steps S141 to S144 described above. Because metadata access is required to locate the data block to be deleted when deleting a data block, both the access and deletion operations for the corresponding metadata are performed in the non-volatile memory of the key-value disk. This avoids the process of converting the cache data organization structure when a cache miss occurs in existing access methods, thereby improving the overall efficiency of data block deletion.

[0077] In some embodiments, the non-volatile memory includes: non-volatile memory and non-volatile flash memory (Flash). Based on this, in a metadata write operation, the step of writing the key-value pairs to be written into the non-volatile memory of the key-value disk specifically includes: writing the key-value pairs to be written into the non-volatile memory; when the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, transferring all key-value pairs corresponding to the storage block size in the non-volatile memory to the non-volatile flash memory, and modifying the address index of each transferred key-value pair in the record in the volatile memory. Non-volatile memory has faster read and write speeds, but higher hardware costs. Non-volatile flash memory has slower read and write speeds than non-volatile memory, but faster read speeds than ordinary SSDs, and lower hardware costs. Therefore, in this embodiment, non-volatile memory is used as a cache and non-volatile flash memory as a large-capacity metadata storage space, ensuring fast access to metadata while reducing the hardware cost of the key-value disk.

[0078] Since non-volatile flash memory is stored based on a certain size of storage block as the minimum access unit, preferably, in order to facilitate caching metadata in the non-volatile memory and storing the storage block read from the non-volatile flash memory in subsequent metadata read or delete operations, the capacity of the non-volatile memory is designed to be greater than twice the size of the storage block.

[0079] Furthermore, the capacity of the non-volatile memory is designed to be N times the storage block size, where N is an integer greater than or equal to 3. The step of writing the key-value pairs to be written into the non-volatile memory of the key-value disk specifically includes: writing the key-value pairs to be written into the non-volatile memory; when the amount of key-value pairs in the non-volatile memory reaches N-1 times the storage block size, transferring all key-value pairs corresponding to n storage block sizes to the non-volatile flash memory according to a preset transfer strategy. The preset transfer strategy may be nK (K key-value pairs are equal to one storage block size) key-value pairs with the least recent access times. n=1, 2, ..., N-1. The value of n is set based on actual conditions. For example, if n=1, the non-volatile memory always has N-2 storage block sizes of key-value pairs, which increases the hit probability of key-value pairs in the non-volatile memory and the access speed of key-value pairs, but increases the number of key-value pairs transferred from the non-volatile memory to the non-volatile flash memory. In this embodiment, n can be set to ensure that there are more key-value pairs in the non-volatile memory, improve the access speed of the key-value pairs, and minimize the number of times the key-value pairs are transferred from the non-volatile memory to the non-volatile flash memory, thereby reducing additional overhead. n is selected according to actual business conditions to achieve a balance between the access speed of the key-value pairs and the reduction of the overhead of the number of transfers. For example: in a scenario with more metadata writing business, n is set to a larger value, even n=N-1; in a scenario with more metadata reading business, n is set to a smaller value, even n=1.

[0080] For a structure in which the non-volatile memory includes a non-volatile memory and a non-volatile flash memory (Flash), in some embodiments, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, such as Figure 4 The metadata reading operation includes the following steps.

[0081] Step S410: searching for an address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk.

[0082] Step S420: Obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key.

[0083] Step S430: when the target storage address is the address of the non-volatile memory, read the physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller.

[0084] Step S440: When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory, and then the physical address block corresponding to the second logical address key is read from the target storage block, and the physical address block is returned to the storage controller. Specifically, the target storage block is read from the non-volatile flash memory to the non-volatile memory, and after returning the physical address block to the storage controller, the space of the target storage block in the non-volatile memory is released; or, the space of the target storage block in the non-volatile flash memory is released, and the transfer is performed according to the above-mentioned preset transfer strategy, and the address index of each transferred key-value pair is modified in the record of the volatile memory (here, transfer refers to transfer from the non-volatile flash memory to the non-volatile memory, and transfer from the non-volatile memory to the non-volatile flash memory).

[0085] For a non-volatile memory including a non-volatile memory and a non-volatile flash memory (Flash), in some embodiments, when the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, such as Figure 5 The steps of the metadata deletion operation include the following steps.

[0086] Step S510: searching for an address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk.

[0087] Step S520: Obtain the target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key.

[0088] Step S530: When the target storage address is the address of the non-volatile memory, delete the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and delete the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk.

[0089] Step S540: When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory into the non-volatile memory (i.e., the data of the target storage block is read into the non-volatile memory), the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted from the record in the volatile memory of the key-value disk. If the data size of the key-value pairs currently in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory (or transferred according to the above-mentioned preset transfer strategy), and the address index of each transferred key-value pair is modified in the record in the volatile memory (here, transfer refers to transfer from the non-volatile flash memory to the non-volatile memory, and transfer from the non-volatile memory to the non-volatile flash memory).

[0090] In some embodiments, the metadata management method based on the online deduplication function further includes: when the key-value disk is powered off and started up, traversing all key-value pairs in the non-volatile memory, rebuilding the address indexes of all key-value pairs, and storing the reconstructed address indexes and corresponding keys in the volatile memory. In a structure where the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory (Flash), both the non-volatile memory and the non-volatile flash memory need to be traversed. In this embodiment, by traversing the key-value pairs in the non-volatile memory of the key-value disk, the address indexes of the key-value pairs and the records of the corresponding keys can be quickly reconstructed.

[0091] The metadata management device based on the online deduplication function provided by the present invention is described below. The metadata management device based on the online deduplication function described below and the metadata management method based on the online deduplication function described above can refer to each other.

[0092] The metadata management device based on online deduplication function of the embodiment of the present invention is as follows Figure 6 As shown, it includes the following modules.

[0093] The I / O request receiving module 610 is used to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0094] The metadata writing module 620 is used to, when the metadata I / O request is a write request, determine the key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into the non-volatile memory of the key-value disk, and record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0095] In the metadata management device based on the online deduplication function of this embodiment, since the metadata key-value pairs and corresponding address indexes are stored in the key-value disk during the metadata write operation, when the storage controller performs the online deduplication function, the metadata I / O request is sent to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device.

[0096] Optionally, the metadata writing module 620 includes the following modules.

[0097] The identification key search module is used to search for the identification key in the records of the volatile memory of the key-value disk.

[0098] A key-value pair writing module is used to, when the identification key is found, determine that the value corresponding to the identification key is the target value, form the key-value pair to be written with the first logical address key and the target value, and write the key-value pair to be written into the non-volatile memory of the key-value disk; when the identification key is not found, obtain the physical address block of the data block written by the storage controller as the target value, form a first key-value pair with the identification key and the target value, form a second key-value pair with the first logical address key and the target value, determine that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and write the key-value pairs to be written into the non-volatile memory of the key-value disk.

[0099] Optionally, in the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata reading module 630, used to search for the address index corresponding to the second logical address key in the record of the volatile memory of the key-value disk; obtain the target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; read the physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller.

[0100] Optionally, in the case where the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata deletion module 640, which is used to search for the address index corresponding to the third logical address key in the record of the volatile memory of the key-value disk; obtain the target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; delete the key-value pair to be deleted from the non-volatile memory based on the target storage address; and delete the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk.

[0101] Optionally, the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; the metadata writing module 620 is specifically used to write the key-value pairs to be written into the non-volatile memory, and when the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0102] Optionally, in the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata reading module 630, used to search for the address index corresponding to the second logical address key in the record of the volatile memory of the key-value disk; obtain the target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; in the case where the target storage address is the address of the non-volatile memory, read the physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller; in the case where the target storage address is the address of the non-volatile flash memory, read the target storage block where the target storage address is located from the non-volatile flash memory, and then read the physical address block corresponding to the second logical address key from the target storage block, and return the physical address block to the storage controller.

[0103] Optionally, in the case where the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the metadata management device based on the online deduplication function further includes: a metadata deletion module 640, which is used to search for an address index corresponding to the third logical address key in the record of the volatile memory of the key-value disk; obtain a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; and in the case where the target storage address is the address of the non-volatile memory, delete the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and delete the third logical address key and the key-value pair in the record of the volatile memory of the key-value disk. corresponding address index; in the case where the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory, the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0104] Optionally, the metadata management device based on the online deduplication function also includes: an address index reconstruction module, which is used to traverse all key-value pairs in the non-volatile memory when the key-value disk is started after power failure, rebuild the address indexes of all key-value pairs, and store the reconstructed address indexes and corresponding keys in the volatile memory.

[0105] The present invention also provides a key-value disk, comprising: an in-disk processor, a volatile memory and a non-volatile memory, wherein the in-disk processor is connected to the volatile memory and the non-volatile memory.

[0106] The non-volatile memory is used to store key-value pairs related to metadata, and the key-value pairs include: a key-value pair with the identifier of the data block as the identification key and the physical address block of the data block as the value, and a key-value pair with the logical address block of the data block as the logical address key and the physical address block of the data block as the value.

[0107] The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory.

[0108] The processor in the disk stores a computer program for implementing the metadata management method based on the online deduplication function as described in the above embodiments when executing the computer program.

[0109] Furthermore, the non-volatile memory includes: non-volatile memory and non-volatile flash memory.

[0110] The key-value disk is used to implement the key-value disk of the metadata management method based on the online deduplication function, and the key-value disk also has corresponding technical effects.

[0111] The present invention also provides a storage device, comprising: a storage controller and the above-mentioned key-value disk, wherein the storage controller is connected to the on-disk processor and is used to send metadata I / O requests to the on-disk processor, and the key-value disk is a KV SSD. Figure 2 , deploy KV SSD in the storage device, and use KV SSD to implement the metadata management method based on the online deduplication function.

[0112] The storage device includes the key-value disk for implementing the metadata management method based on the online deduplication function, and the storage device also has the corresponding technical effect.

[0113] based on Figure 2 The process of implementing online deduplication when writing data on a storage device is as follows: Figure 7 As shown, the following steps are included.

[0114] Step S710: The storage controller of the storage device receives the data block to be written and the logical address block L. The data block to be written and the logical address block L are sent by the host through an I / O write request.

[0115] Step S720: The storage controller calculates the identifier H (hash value) of the data block to be written as the identifier of the data block to be written.

[0116] Step S730: The storage controller calls the interface (Store operation interface) of the KV SSD disk to perform fingerprint search.

[0117] Step S740: The KV SSD's on-disk processor searches for a corresponding identification key based on the identifier H. If a corresponding identification key is found, it indicates that the data block to be written has been stored in the storage device and the HP key-value pair has been stored in the non-volatile memory of the KV SSD disk. Then, step S770 is executed; otherwise, step S750 is executed.

[0118] Step S750: The data block to be written is written to the disk. When the identification key is not found, it means that the data block to be written identified as H is not stored in the storage device. The on-disk processor of the key-value disk will return a storage command to the storage controller through the Store operation interface to store the data block to be written identified as H. After receiving the command, the storage controller will store the data block to be written identified as H and obtain the corresponding physical address block P.

[0119] Step S760: Obtain the physical address block P, and write the HP key-value pair into the KV SSD disk. Specifically, write the HP key-value pair into the non-volatile memory of the KV SSD disk.

[0120] Step S770: Write the LP key-value pair to the KV SSD disk. Specifically, obtain the address index of the key-value pair corresponding to the identification key from the record in the volatile memory of the KV SSD disk, find the location where the key-value pair is stored in the non-volatile memory of the key-value disk according to the address index, read the key-value pair, and thus obtain the physical address block P stored before the data block to be written marked as H, and write the LP key-value pair to the non-volatile memory of the KV SSD disk.

[0121] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may invoke logic instructions in the memory 830 to execute a metadata management method based on an online deduplication function, which includes the following steps.

[0122] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0123] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into the non-volatile memory of the key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0124] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the metadata management method based on the online deduplication function provided by the above methods, and the method includes the following steps.

[0126] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0127] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into the non-volatile memory of the key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0128] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the metadata management method based on the online deduplication function provided by the above methods, and the method includes the following steps.

[0129] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0130] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into the non-volatile memory of the key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0132] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A metadata management method based on online deduplication function, characterized in that: include: Receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; In a case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determining a key-value pair to be written based on the identification key and the first logical address key, writing the key-value pair to be written into a non-volatile memory of the key-value disk, and recording the key in the key-value pair to be written and an address index of a storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk; Determining a key-value pair to be written based on the identification key and the first logical address key, and writing the key-value pair to be written into a non-volatile memory of the key-value disk, comprising: Searching for the identification key in the records in the volatile memory of the key-value disk; In the case where the identification key is found, determining that the value corresponding to the identification key is a target value, forming the key-value pair to be written with the first logical address key and the target value, and writing the key-value pair to be written into the non-volatile memory of the key-value disk; If the identification key is not found, the physical address block of the data block written by the storage controller is obtained as the target value, a first key-value pair is formed with the identification key and the target value, and a second key-value pair is formed with the first logical address key and the target value, and it is determined that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and the key-value pairs to be written are written into the non-volatile memory of the key-value disk.

2. The metadata management method based on online deduplication function according to claim 1, characterized in that: In a case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk; Obtaining a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; A physical address block corresponding to a second logical address key in a nonvolatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

3. The metadata management method based on online deduplication function according to claim 1, characterized in that: In a case where the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk; Obtaining a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; Deleting the key-value pair to be deleted from the non-volatile memory based on the target storage address; The third logical address key and the corresponding address index are deleted from the record in the volatile memory of the key-value disk.

4. The metadata management method based on online deduplication function according to claim 1, characterized in that: The non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; writing the key-value pair to be written into the non-volatile memory of the key-value disk includes: The key-value pairs to be written are written into the non-volatile memory. When the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

5. The metadata management method based on online deduplication function according to claim 4, characterized in that: In a case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk; Obtaining a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; In a case where the target storage address is an address of the non-volatile memory, reading a physical address block corresponding to a second logical address key in the non-volatile memory based on the target storage address, and returning the physical address block to the storage controller; When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory, and then the physical address block corresponding to the second logical address key is read from the target storage block, and the physical address block is returned to the storage controller.

6. The metadata management method based on online deduplication function according to claim 4, characterized in that: In a case where the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk; Obtaining a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; In a case where the target storage address is an address of the non-volatile memory, deleting the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and deleting the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk; In the case where the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory, the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

7. The metadata management method based on online deduplication function according to any one of claims 1 to 6, characterized in that: Also includes: When the key-value disk is started after power failure, all key-value pairs in the non-volatile memory are traversed, address indexes of all key-value pairs are rebuilt, and the rebuilt address indexes and corresponding keys are stored in the volatile memory.

8. A metadata management device based on online deduplication function, characterized in that: include: An I / O request receiving module is configured to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; a metadata writing module configured to, when the metadata I / O request is a write request and the metadata I / O request includes an identification key and a first logical address key in the metadata, determine a key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into a non-volatile memory of the key-value disk, and record in the volatile memory of the key-value disk the key in the key-value pair to be written and an address index of the storage address of the key-value pair to be written in the non-volatile memory; The metadata writing module includes: an identification key search module, configured to search for the identification key in the records of the volatile memory of the key-value disk; A key-value pair writing module is used to, when the identification key is found, determine that the value corresponding to the identification key is the target value, form the key-value pair to be written with the first logical address key and the target value, and write the key-value pair to be written into the non-volatile memory of the key-value disk; when the identification key is not found, obtain the physical address block of the data block written by the storage controller as the target value, form a first key-value pair with the identification key and the target value, form a second key-value pair with the first logical address key and the target value, determine that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and write the key-value pairs to be written into the non-volatile memory of the key-value disk.

9. A key-value disk, characterized in that: include: An on-disk processor, a volatile memory, and a non-volatile memory, wherein the on-disk processor is connected to the volatile memory and the non-volatile memory; The non-volatile memory is used to store key-value pairs related to metadata, the key-value pairs including: a key-value pair with an identifier of a data block as an identifier key and a physical address block of the data block as a value, and a key-value pair with a logical address block of the data block as a logical address key and a physical address block of the data block as a value; The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory; The processor in the disk stores a computer program, which is used to implement the metadata management method based on the online deduplication function according to any one of claims 1 to 7 when executing the computer program.

10. A storage device, characterized in that: include: A storage controller and the key-value disk according to claim 9, wherein the storage controller is connected to the on-disk processor and is used to send metadata I / O requests to the on-disk processor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the metadata management method based on the online deduplication function is implemented as described in any one of claims 1 to 7.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the metadata management method based on the online deduplication function is implemented according to any one of claims 1 to 7.

Images