Data storage method and device and storage system

Abstract

The invention provides a data storage method and device and a storage system, relates to the technical field of storage, and aims to adaptively select a proper data writing mode so as to reduce the overhead of a data writing process. The method is applied to a storage controller in a storage system, the storage system comprises a plurality of storages, and the method comprises the following steps: obtaining target data; determining a write-in mode when the target data is written into the storage pool from a plurality of write-in modes supported by the storage system; the storage pool is a storage space mapped by a plurality of memories in the storage system, the write-in mode of the data comprises an overwrite mode or an additional write mode, and the storage pool supports the overwrite mode and the additional write mode and supports switching of the write-in mode; and writing the target data into the storage pool according to the determined writing mode.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data storage method, device and storage system. Background Art

[0002] In the field of storage technology, mapping the storage space of a hard disk into a storage pool helps manage the storage space.

[0003] During the process of writing data to the storage pool, after the storage device receives the data write request, it maps the logical address in the write request (the logical address indicates the identifier, offset address and length of the logical unit) to the address in the storage space corresponding to the storage pool (which can be simply referred to as the storage space address), and then writes the data to the storage space corresponding to the storage space address.

[0004] Currently, in the prior art, there is a problem of additional overhead due to read-write amplification in the process of writing data to a storage pool. Summary of the invention

[0005] The present application provides a data storage method, device and storage system, which can adaptively select a suitable data writing mode, thereby reducing the overhead of the data writing process.

[0006] This application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a data storage method, which is applied to a storage controller in a storage system, the storage system including multiple memories, the method including: obtaining target data; and determining a write mode when writing the target data into a storage pool from multiple write modes supported by the storage system; the storage pool is a storage space mapped by multiple memories in the storage system, the data write mode includes an overwrite write mode or an append write mode, the storage pool supports the overwrite write mode and the append write mode, and supports switching the write mode; and writing the target data into the storage pool according to the determined write mode.

[0008] The data storage method provided in the present application can support both the overwrite write mode and the append write mode, and can be compatible with the advantages of the overwrite write mode and the append write mode. It can adaptively and flexibly select a data write mode with better performance for the target data, which can reduce read-write amplification and thus reduce the overhead of the data writing process.

[0009] In one possible implementation, the above-mentioned determination of the write mode when writing the target data to the storage pool includes: when the overwrite write amplification degree of the target data is less than the append write amplification degree, determining the write mode to be the overwrite write mode; when the overwrite write amplification degree of the target data is greater than the append write amplification degree, determining the write mode to be the append write mode, wherein the overwrite write amplification degree is the size of the read-write amplification generated when the overwrite write mode is used to write the target data, and the append write amplification degree is the size of the read-write amplification generated when the append write mode is used to write the target data.

[0010] In the present application, the greater the read / write amplification (i.e., the greater the amplification degree) generated in the process of writing data, the greater the overhead required for writing data. Therefore, in the process of determining which write mode to use to write the target data into the storage pool, the write mode of the target data can be determined according to the overwrite write amplification degree and the append write amplification degree of the target data. Specifically, the write mode with the smallest amplification degree between the overwrite write amplification degree and the append write amplification degree is determined as the write mode of the target data, which can reduce the additional overhead caused by the process of writing the target data.

[0011] In a possible implementation, the above-mentioned determination of the write mode when writing the target data into the storage pool includes: when the redundancy level of the storage pool is a redundancy level without verification information, determining that the write mode when writing the target data into the storage pool is an overwrite write mode.

[0012] In this application, when the redundancy level is a redundancy level without verification information, it indicates that the target data has no verification information (no need to calculate the verification information). In this case, when the target data is written in the append write mode, a new storage space needs to be applied for. After the target data is written in the storage space, the address index table (the correspondence between the logical address and the storage space address) needs to be updated, and the original storage space needs to be recovered. When the target data is written in the overwrite write mode, there is no need to apply for a new storage space, and the data can be written to the original storage space. It can be seen that compared with the overwrite write mode, the append write mode needs to update the address index table and recycle garbage, which will generate read-write amplification and additional overhead, so the overwrite write mode is selected.

[0013] Since the target data has no verification information, when the target data is stored in an overwrite mode, there is no need to calculate the verification information even if full stripe writing is not possible. This can reduce read-write amplification and the additional overhead of the data writing process to a certain extent.

[0014] In a possible implementation, the above-mentioned determination of the write mode when writing the target data into the storage pool includes: when the size of the target data is equal to the stripe size of an integer number of storage pools, determining that the write mode when writing the target data into the storage pool is an overwrite write mode.

[0015] In this application, when the size of the target data is equal to the integer stripe size of the storage pool, when the target data is written in the append write mode, it is necessary to apply for a new storage space. After writing the target data and the calculated verification information (if there is verification information) in the storage space, it is also necessary to update the address index table and recycle the original storage space; when the target data is written in the overwrite write mode, there is no need to apply for a new storage space, and the target data and the calculated verification information can be written into the original storage space. It can be seen that compared with the overwrite write mode, the append write mode needs to update the address index table and recycle garbage, which will generate read-write amplification and additional overhead, so the overwrite write mode is selected.

[0016] Since the size of the target data is equal to an integer number of stripe sizes, full stripe writing can be achieved without sending a large number of write requests in batches from the cache multiple times. Read-write amplification and additional overhead (caused by reading other data and backing up data and / or verification information) caused by solving the write hole problem can also be avoided. In this way, read-write amplification can be reduced to a certain extent, and the additional overhead of the data writing process can be reduced.

[0017] In a possible implementation, the redundancy level indicating that the target data has no verification information includes at least one of the following: RAID0, RAID1, RAID10, or multiple copies.

[0018] In a possible implementation, when the write mode is the overwrite write mode, writing the target data into the storage pool according to the determined write mode specifically includes: mapping the logical address of the target data into the first storage space address based on the address index table in the storage system; and writing the target data into the first storage space. The first storage space address indicates the first storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

[0019] In a possible implementation, when the write mode is the append write mode, writing the target data into the storage pool according to the determined write mode and the logical address of the target data specifically includes: writing the target data into the second storage space in the storage pool; and updating the first storage space address corresponding to the logical address of the target data in the address index table in the storage system to the second storage space address. The second storage space address indicates the second storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

[0020] In a second aspect, the present application provides a storage device, including an acquisition module, a determination module, and a write data module. The acquisition module is used to acquire target data, the determination module is used to determine the write mode when writing the target data into a storage pool from a plurality of write modes supported by the storage system, the storage pool is a storage space mapped by a plurality of memories in the storage system, the write mode includes an overwrite write mode or an append write mode, the storage pool supports the overwrite write mode and the append write mode, and supports switching the write mode; the write data module is used to write the target data into the storage pool according to the determined write mode.

[0021] In a possible implementation, the determination module is specifically used to determine that the write mode is the overwrite write mode when the overwrite write amplification degree of the target data is less than the append write amplification degree; and to determine that the write mode is the append write mode when the overwrite write amplification degree of the target data is greater than the append write amplification degree. The overwrite write amplification degree is the magnitude of the read-write amplification generated when the overwrite write mode is used to write the target data, and the append write amplification degree is the magnitude of the read-write amplification generated when the append write mode is used to write the target data.

[0022] In a possible implementation, the determination module is specifically configured to determine that the write mode when writing the target data into the storage pool is an overwrite write mode when the redundancy level of the storage pool is a redundancy level without verification information.

[0023] In a possible implementation, the determination module is specifically configured to determine that the write mode when writing the target data into the storage pool is the overwrite write mode when the size of the target data is equal to the stripe size of an integer number of storage pools.

[0024] In a possible implementation, the redundancy level without check information includes at least one of the following: RAID0, RAID1, RAID10, or multiple copies.

[0025] In a possible implementation, when the write mode is the overwrite write mode, the data writing module is specifically used to map the logical address of the target data to the first storage space address based on the address index table in the storage system, and write the target data into the first storage space. The first storage space address indicates the first storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

[0026] In a possible implementation, when the write mode is the append write mode, the data write module is specifically used to write the target data into the second storage space in the storage pool, and update the first storage space address corresponding to the logical address of the target data in the address index table in the storage system to the second storage space address. The second storage space address indicates the second storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

[0027] In a third aspect, the present application provides a storage device comprising a memory and at least one processor connected to the memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the computer instructions are executed by at least one processor, the storage device executes the method of the first aspect and any one of its possible implementations.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium storing computer instructions, which, when executed on a computer, execute the method of the first aspect and any one of its possible implementations.

[0029] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are run on a computer, the method of the first aspect and any one of its possible implementations is executed.

[0030] In a sixth aspect, the present application provides a chip system, comprising: a processor, configured to call and run a computer program from a memory, so that a storage device equipped with the chip system executes the method of the first aspect and any one of its possible implementations.

[0031] In a seventh aspect, the present application provides a storage system comprising a storage controller and multiple memories; the storage controller is used to execute the method of the first aspect and any one of its possible implementations to write target data into a storage pool mapped by multiple memories.

[0032] It should be understood that the beneficial effects achieved by the technical solutions of the second to seventh aspects of the present application and the corresponding possible implementation methods can be referred to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 One of the architectural diagrams of a storage system provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the relationship between the storage space of a storage pool and the physical space of a hard disk provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a logical unit in a storage pool provided in an embodiment of the present application;

[0036] Figure 4 A second schematic diagram of the architecture of a storage system provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a redundancy mechanism in a storage system provided in an embodiment of the present application;

[0038] Figure 6 One of the schematic diagrams of the data storage method provided in the embodiment of the present application;

[0039] Figure 7 A second schematic diagram of a data storage method provided in an embodiment of the present application;

[0040] Figure 8 The third schematic diagram of the data storage method provided in the embodiment of the present application;

[0041] Fig. 9 A schematic diagram of a process of writing data in an overwrite write mode provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of a process of writing data in an overwrite write mode provided in an embodiment of the present application;

[0043] Fig.11 One of the schematic diagrams of a data writing process provided in an embodiment of the present application;

[0044] Fig.12 A second schematic diagram of a data writing process provided in an embodiment of the present application;

[0045] Fig.13 A complete flowchart of the data storage method provided in the embodiment of the present application;

[0046] Fig.14 A schematic diagram of a process for determining a data writing mode provided in an embodiment of the present application;

[0047] Fig.15 One of the structural schematic diagrams of a storage device provided in an embodiment of the present application;

[0048] Fig.16 The second structural diagram of a storage device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0050] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first storage space and a second storage space are used to distinguish different storage spaces rather than to describe a specific order of storage spaces.

[0051] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two, and “plurality” may also be described as “at least two”.

[0053] First, some concepts and technical knowledge involved in a data storage method, device, and storage system provided in an embodiment of the present application are explained.

[0054] Combination Figure 1 The storage system architecture diagram shown in FIG. 1 is a storage system provided in an embodiment of the present application. The storage system includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 110 ( Figure 1 Two computing nodes 110 are shown in FIG. 1 , but are not limited to two computing nodes 110 . Each computing node 110 can communicate with each other. The storage node cluster includes multiple storage nodes 120 ( Figure 1 Two storage nodes 120 are shown in the figure, but are not limited to two storage nodes 120). A storage node may also be referred to as a storage device.

[0055] The computing node 110 is a computing device, such as a server, a desktop computer, or a storage array controller. Figure 1 As shown, the computing node 110 at least includes a processor 1101, a memory 1102, and a network card 1103. The processor 1101 is a central processing unit (CPU) for processing data access requests from outside the computing node 110, or requests generated inside the computing node 110. Exemplarily, when the processor 1101 receives a data write request sent by a user device, the data in the data write request can be temporarily stored in the memory 1102. When the total amount of data in the memory 1102 reaches a certain threshold, the processor 1101 sends the data stored in the memory 1102 to the storage node 120 for persistent storage.

[0056] In addition, the processor 1101 is also used to calculate or process data, such as metadata management, duplicate data deletion, data compression, virtualized storage space, and address conversion. Figure 1 Only one CPU is shown in the figure. In actual applications, the computing device 110 may include multiple CPUs, wherein one CPU may have one or more CPU cores. The embodiment of the present application does not limit the number of CPUs and the number of CPU cores.

[0057] The memory 1102 of the computing node 110 refers to an internal memory that directly exchanges data with the processor 1101. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. The memory 1102 may include at least two types of memories, for example, the memory 1102 may be either a random access memory (RAM) or a read only memory (ROM).

[0058] For example, RAM is dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a semiconductor memory, and like most random access memories, it is a volatile memory device. SCM is a composite storage technology that combines traditional storage devices with memory characteristics. SCM can provide faster read and write speeds than hard disks, but the access speed is slower than DRAM and the cost is cheaper than DRAM. Optionally, DRAM and SCM are only exemplary descriptions in the embodiments of the present application, and memory 1102 may also include other random access memories, such as static random access memory (SRAM).

[0059] For example, the ROM may be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc.

[0060] In addition, the memory 1102 may also be a dual in-line memory module (DIMM), that is, a module composed of a dynamic random access memory (DRAM), and the memory 1102 may also be a solid state disk (SSD).

[0061] Optionally, in actual applications, multiple memories 1102 and different types of memories 1102 may be configured in the computing node 110. The embodiment of the present application does not limit the number and type of memory 1102. In addition, the memory 1102 may be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 1102 will not be lost. A memory with a power-saving function is called a non-volatile random access memory (NVRAM), a non-volatile dual in-line memory module (NVDIMM). For example, the NVRAM may be a double data rate synchronous dynamic random access memory (DDR SDRAM).

[0062] The network card 1103 of the computing device 110 is used to communicate with the storage node 120. For example, when the total amount of data in the memory 1102 reaches a certain threshold, the computing node 110 can send a request to the storage node 120 through the network card 1103 to store the data persistently. In addition, the computing node 110 can also include a bus for communication between the components inside the computing node 110. Figure 1 The main function of the computing node 110 in the system is computing services. When storing data, remote storage can be used to achieve persistent storage. Therefore, it has less local storage than a conventional server, thereby saving costs and space. However, this does not mean that the computing node 110 cannot have local storage. In actual implementation, the computing node 110 can also have a small number of hard disks built in or externally connected.

[0063] Combination Figure 1 Any computing node 110 can access any storage node 120 in the storage node cluster through the network. The storage node can be, for example, a server, a desktop computer, or a controller or a hard disk enclosure of a storage array.

[0064] A storage node 120 includes one or more controllers (or control units) 1201, a network card 1202, and multiple hard disks 1203. The network card 1202 is used to communicate with the computing node 110. The hard disk 1203 is used to store data and can be a disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The controller 1201 is used to write data to the hard disk 105 or read data from the hard disk 1203 according to the data read / write request sent by the computing node 110. In the process of reading and writing data, the controller 1201 needs to convert the address carried in the data read / write request into an address that the hard disk can recognize. It can be seen that the controller 1201 also has some simple computing functions.

[0065] In practical applications, the controller 1201 may have a variety of forms. In one case, the controller 101 includes a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 1203, or read from the hard disk 1203 to be sent to the computing node 110. In some cases, the controller 1201 is a programmable electronic component, such as a data processing unit (DPU). The DPU has the versatility and programmability of the CPU, but is more specialized and can efficiently run network data packets, storage requests or analysis requests. Unlike the CPU, the DPU has a greater degree of parallelism (needs to process a large number of requests). Optionally, the DPU can also be replaced with processing chips such as graphics processing units (GPUs) and embedded neural-network processing units (NPUs).

[0066] Generally, the number of controllers 1201 may be one, two or more. When the storage node 120 includes at least two controllers 1201, the hard disk 1203 and the controller 1201 may have an ownership relationship. When the hard disk 1203 and the controller 1201 have an ownership relationship, each controller can only access the hard disk that belongs to it. If the storage space is insufficient, when adding a new hard disk 1203 in the storage node 120, the ownership relationship between the hard disk 1203 and the controller 1201 needs to be re-bound.

[0067] In another embodiment, the functions of the controller 1201 can be offloaded to the network card 1202. In other words, the storage node 120 does not have a controller 1201 inside, but the network card 1202 completes data reading and writing, address conversion and other computing functions. In this case, the network card 1202 is an intelligent network card, which can include a CPU and a memory. The CPU is used to perform operations such as address conversion and reading and writing data, and the memory is used to temporarily store data to be written to the hard disk 1203, or read from the hard disk 1203 to be sent to the computing node 110. In this case, there is no ownership relationship between the network card 1202 and the hard disk 1203 in the storage node 120, and the network card 1202 can access any hard disk 1203 in the storage node 120, so when the storage space is insufficient, the hard disk can be expanded more conveniently.

[0068] for Figure 1 The storage system shown in the figure can create a storage pool based on the hard disks 1203 of at least two storage nodes in the storage node cluster, that is, the storage space provided by each storage node 120 is centralized as (mapped to) a storage pool for unified management. The storage pool is a virtual storage space, and the physical space corresponding to the storage pool comes from the various hard disks included in each storage node 120. In this case, the storage node cluster also includes a management node ( Figure 1 The management node is used to create and manage the storage pool. The management node can be deployed as an independent node in the storage node cluster, or it can be deployed in conjunction with the storage node 120, that is, one storage node can be selected from each storage node 120 as the management node, that is, one storage node 120 is selected to assume the function of the management node. The management node can communicate with any storage node 120. Optionally, there can be one management node, or multiple management nodes can be deployed.

[0069] It can be understood that the storage pool spans at least two storage nodes 120, and the storage space of the storage pool is composed of one or more types of hard disks in at least two storage nodes 120. The storage pool is used to store data persistently, especially data with low access frequency. Specifically, when the amount of data stored in the memory reaches a set threshold, part of the data in the memory is written to the storage pool for storage.

[0070] In the embodiment of the present application, the process of creating a storage pool includes: first, each storage node 120 regularly reports the status information of the hard disk to the management node through the heartbeat channel. The status information of the storage includes but is not limited to: the types of various hard disks contained in the storage node, the health status, the total capacity of each hard disk, and the available capacity, etc. Secondly, the management node creates a storage pool based on the collected information. However, in some scenarios, the storage node 120 can selectively provide hard disks to the storage pool according to its own situation, such as the health status of the hard disk. In other words, it is possible that some hard disks in some storage nodes are not part of the storage pool. Further, the management node needs to uniformly address the storage space included in the storage pool. After uniform addressing, each storage space of the storage pool (each space can be a "page") has a unique address (this address is referred to as a storage space address in the embodiment of the present application), and the space indicated by the storage space address is unique in the storage pool. After physical space is allocated to a section of the storage pool, the address of the space (i.e., the storage space address) has its corresponding physical address, which indicates on which hard disk of which storage node 120 the storage space represented by the storage space address is actually located, as well as the offset in the hard disk, i.e., the location of the physical space.

[0071] like Figure 2 As shown, in some embodiments, the relationship between the storage space of the storage pool and the physical space of the hard disk is: the physical space of the hard disk provided to the storage pool is divided into multiple physical blocks (chunks), one physical block is mapped to a logical block, and multiple logical blocks constitute a storage pool. Among them, one or more logical blocks in the storage pool can form a logical block group. It can be understood that the multiple logical blocks constituting a logical block group can be logical blocks with continuous or discontinuous addresses in the storage pool. One or more logical block groups can constitute a logical unit (logical unit, LU), each logical unit has a unique logical unit number (logical unit number, LUN), and the computing node 110 can directly perceive the logical unit number. Those skilled in the art usually directly use LUN to refer to the logical unit, and each LUN has a LUN ID for identifying a LUN. In summary, the storage pool can be divided into multiple logical units.

[0072] refer to Figure 3 , which illustrates the relationship between logical units, logical block groups, stripes, and logical blocks in a storage pool. Figure 3 In the example, a logical unit of the storage pool includes 2 logical block groups, each logical block group includes 3 stripes, and each stripe includes 5 logical blocks.

[0073] Combination Figure 2 and Figure 3The specific location of data in a logical unit can be determined by the starting address (i.e., the offset address in the logical unit) and the length of the data. The starting address is usually referred to as a logical block address (LBA) by those skilled in the art. It can be understood that the three factors of LUN ID, LBA, and length identify a certain address segment, which corresponds to a storage space in the storage pool, and a storage space in the storage pool corresponds to a storage space in the physical storage space (one or more hard disks).

[0074] In the embodiment of the present application, from the perspective of the logical unit, the storage location LUN ID, LBA and length of the data in the logical unit can be called the logical address of the data, and the logical address has a mapping relationship with the storage space address, and the storage space address has a mapping relationship with the physical address. The storage node stores a mapping table of logical addresses and storage space addresses (which can be called a first address index table) and a mapping table of storage space addresses and physical addresses (which can be called a second address index table).

[0075] like Figure 4 As shown, storage nodes (such as Figure 3 The software part of the storage node 120) includes the operating system of the storage node, and the operating system includes software components of a block / file system, a cache, a storage pool, and the like.

[0076] Among them, the block / file system is the software in the operating system that is responsible for managing and storing file information. For example, the file system can organize and allocate the space of the storage device. It is a system that is responsible for file storage and protects and retrieves the stored files. Specifically, the file system is responsible for creating files for users, storing, reading, modifying, and dumping files, controlling file access, and revoking files when the user no longer uses them.

[0077] The cache is used to temporarily store data. The cache may be a non-volatile medium (such as NVRAM, NVDIMM, etc.). In an embodiment of the present application, the cache also includes a cache table, in which the logical address corresponding to each write request is recorded.

[0078] The storage pool can provide large-capacity hard disk space. The software components of the storage pool provide a manageable virtualized storage pool and provide unified space management and indexing for the file system or block business.

[0079] The software components of the storage pool may include the WAP module (append write module), GC module (garbage collection module), WIP module (overwrite write module), cost-based optimizer (CBO), Write Hole module, redundant version, index table and other software modules. Among them, the WAP module is used to support the append write process, the GC module is used for garbage collection, the WIP module is used to support the overwrite write process, the CBO is used to calculate the read-write amplification generated during the data writing process, and the WriteHole module is used to handle the write hole problem.

[0080] Combination Figure 1 and Figure 4 The architecture of the storage system shown in the figure describes the data writing process.

[0081] Step 1: A computing node in the storage system receives a data access request.

[0082] The data access request is a data write request, which usually carries the target data (i.e., the data to be written into the storage pool, and the target data in the following embodiments refers to the data to be written into the storage pool) and the logical address (LUN ID, LBA and length). The logical address carried in the data access request is the write address of the target data.

[0083] Step 2: The computing node sends a data access request to the storage node.

[0084] The computing node sends the data access request to the storage node through the network card of the computing node, and the storage node receives the data access request through the network card of the storage node.

[0085] Step 3: The storage node writes the target data into the cache (or memory) of the storage node, and stores the logical address in the access request into the cache table.

[0086] In an embodiment of the present application, after the storage node writes the target data into the cache, the storage node can return a write success response to the computing node. Subsequently, the background of the storage node stores the data in the cache persistently in the storage pool, that is, the background performs disk flushing to complete the persistent storage of the data. It can be understood that the data access requests corresponding to the multiple logical addresses stored in the cache are arranged in order to form a disk flushing queue, and the disk flushing queue is actually a queue of disk flushing requests.

[0087] Step 4: The storage node writes the data in the cache to the storage pool.

[0088] It should be noted that, in the embodiment of the present application, since the storage space address of the storage pool corresponds one-to-one to the physical address of the hard disk, storing data in the storage pool is equivalent to storing the data in the hard disk.

[0089] Specifically, the storage node reads the logical address corresponding to the disk flushing request from the cache table according to the disk flushing queue, and reads the target data from the cache, and then converts the logical address into the storage space address of the storage pool according to the address index table, and writes the target data into the storage space indicated by the storage space address (for example, Figure 4 D0, D1, D2 and D3 in the storage space address constitute the target data, and P0 is the verification information of the target data). It should be understood that writing the target data into the storage space indicated by the storage space address specifically includes writing the target data into the physical storage space indicated by the physical address corresponding to the storage space address (i.e. storing the target data into the hard disk).

[0090] Correspondingly, in the scenario of reading data from the storage pool, the data access request received by the computing node is a read data request, which includes the logical address of the data to be read. The computing node then sends the data access request to the storage node, and the storage node converts the logical address in the data access request into a storage space address according to the address index table. The storage node then reads the data from the storage space corresponding to the storage space address and returns the read data to the computing node.

[0091] In combination with the above content, when writing data to the storage pool, when the data in the memory reaches a certain size, the data can be divided into multiple data slices (or data blocks), and these data slices are stored in logical units. These data slices can constitute a stripe (a stripe includes multiple logical blocks). In some cases, it is also necessary to calculate the check slices (or check blocks) based on multiple data slices, and multiple data slices and check slices constitute a stripe. For example, the size of a stripe is 32K, and the size of a logical block is 8K (that is, a stripe includes 4 logical blocks). When the amount of data in the memory reaches 32K, the data can be divided into 4 data slices of 8K in size, and then the 4 data slices are written to the 4 logical blocks respectively to achieve full stripe writing.

[0092] Optionally, the size of a stripe of the storage pool can be configured according to actual needs, for example, the size of a stripe can be 64K, 128K, 256K, etc. It should be noted that the size of the stripe described in the embodiment of the present application refers to the size of the portion of the stripe storing data, that is, the capacity of the storage space used to store the data portion, excluding the size of the portion storing the verification information of the data. For example, if the size of a stripe is 64K, it means that the storage space corresponding to the stripe can be used to store 64K of data.

[0093] Generally, storage systems can use redundancy technology to ensure data reliability and prevent data loss. Redundancy technology includes a variety of different redundancy mechanisms (also known as redundancy modes). Redundancy mechanisms can include multiple copy mechanisms, redundant array of independent disks (RAID) mechanisms, or erasure coding (EC) verification mechanisms. Redundancy mechanisms can include different redundancy levels. According to the feature of whether the data has verification information, the redundancy level can be divided into a redundancy level with verification information and a redundancy level without verification information. A redundancy level with verification information means that when this redundancy level is used for data storage, it is necessary to calculate the verification information of the data and store the verification information of the data. The verification information of the data is used to recover the data. A redundancy level without verification information means that when this redundancy level is used for data storage, it is not necessary to calculate the verification information of the data, that is, it is not necessary to store the verification information of the data.

[0094] The multi-copy mechanism means that at least two identical copies of data are stored in one stripe, and at least two copies of data are stored in two different physical addresses, that is, the physical blocks storing the two copies of data are located on different hard disks or on different storage nodes. When one copy of the data is lost, the other copies of the data can be used to restore the data. Each redundancy level of the multi-copy mechanism is a redundancy level without verification information.

[0095] The RAID mechanism combines multiple independent disks (such as hard disks) into a disk group with huge capacity, divides the data into many segments, and stores them on each hard disk.

[0096] The RAID mechanism includes multiple redundancy levels, such as RAID0, RAID1, RAID2, RAID3, RAID4, RAID5, RAID6, RAID7, RAID10, etc. Among them, RAID0, RAID1 and RAID10 are redundancy levels without verification information, and other redundancy levels in the RAID mechanism are redundancy levels with verification information.

[0097] The following is a brief introduction to several redundancy levels in the RAID mechanism.

[0098] RAID0 divides data into multiple copies and writes them to multiple disks in a distributed manner. Multiple disks support parallel reading and writing. For example, refer to Figure 5In (a), taking two hard disks as an example, the target data is divided into 6 data blocks (or data slices), namely A1, A2, A3, A4, A5, and A6. These 6 data blocks are scattered and stored in two different hard disks, and there is no backup storage, nor is there any storage of the data block check blocks (or check slices). Figure 5 As shown in (a), in the data blocks of the target data, A1, A3 and A5 are stored in hard disk 1, and A2, A4 and A6 are stored in hard disk 2. RAID0 has a faster read and write speed, but RAID0 does not provide a data protection mechanism. The entire storage system is very unreliable. If a failure occurs, the data cannot be recovered.

[0099] RAID1 is to mirror the data of one disk to another disk (which can be called a mirror disk). That is to say, when data is written to one disk, a mirror file will be generated on the other disk. In this way, in any pair of disks in the storage system, when one disk fails, data can be read and written from the other mirror disk, which has good disk redundancy. For example, refer to Figure 5 In (b), the six data blocks of the target data are stored in two hard disks respectively. Six data blocks are stored in hard disk 1 and six data blocks are stored in hard disk 2. The two hard disks back up each other. There is no check block for storing data blocks in RAID 1.

[0100] RAID10 can be seen as a combination of RAID0 and RAID1, where multiple data blocks of the target data are dispersedly stored in multiple pairs of mutually mirrored disks (using RADI1 to store data). For example, refer to Figure 5 In (c), data blocks A1, A3 and A5 of the target data are stored in hard disk 1 and hard disk 2 in RAID1 redundant mode, and A2, A4 and A6 are stored in hard disk 3 and hard disk 4 in RAID1 redundant mode, and the data blocks (A1, A3, A5) in hard disk 1 and hard disk 2 and the data blocks (A2, A4, A6) in hard disk 3 and hard disk 4 are equivalent to being stored in RAID0 redundant mode.

[0101] RAID2, RAID3, RAID4, RAID5, RAID6 and RAID7 are redundancy levels with checksum information. In the redundancy level with checksum information, when the data in the memory reaches a certain size, it can be divided into multiple data blocks according to the set RAID level (or type), and the check blocks are calculated based on the multiple data blocks, and the data blocks and check blocks are sent to multiple different hard disks or different storage nodes.

[0102] In the embodiment of the present application, the size of a stripe of the storage pool (i.e., the number of logical blocks contained in a stripe) depends on which redundancy mechanism is used to ensure data reliability. Taking RAID5 as an example, if the target data is divided into 4 data blocks and 1 check block is calculated, then 4 data blocks and 1 check block constitute a stripe, and these 5 data blocks are located in 5 different hard disks, wherein the check blocks are randomly distributed on a hard disk. Taking the target data being divided into 4 data blocks (respectively denoted as B0, B1, B2, and B3) as an example, data blocks B0, B1, B2, and B3 and check block P0 (P0 is calculated based on B0, B1, B2, and B3) are stored in a stripe of a logical unit, and one logical block in the stripe corresponds to one data block.

[0103] Assuming that the size of a logical block is 8KB, for RAID5 shown in Figure 5, the size of a stripe is equal to the size of 4 data blocks, that is, the size of a stripe is 32KB. When the data stored in the memory reaches 8KB*4=32KB, the data is divided into 4 data blocks, each of which is 8KB in size, and then a check block is calculated, and the size of the check block is also 8KB.

[0104] According to this example, when the size of the target data is equal to the size of the stripe, after writing the target data into a stripe, the stripe can be filled, that is, full stripe writing can be achieved. Usually, for a redundancy level with checksum information, for a stripe, the stripe needs to be filled in order to calculate the checksum block. Optionally, when the size of the target data is smaller than the stripe size, after writing the target data into the stripe, other data needs to be written to fill the stripe, for example, other data can be read and written or 0 can be written to fill the stripe.

[0105] The EC verification mechanism refers to dividing the data to be stored into at least two data shards, calculating the verification shards of at least two data shards according to a certain verification algorithm, and when one of the data shards is lost, the data can be restored using the other data shard and the verification shard. That is, at least two data shards and one verification shard are stored in one stripe, and the physical blocks storing at least two shards and one verification shard are located on different hard disks or different storage nodes.

[0106] Based on the above description, it should be noted that the number and capacity of logical units in the above storage pool, the size of the logical block group, the size of the stripe, the redundancy mode and the redundancy level are all configuration information of the storage pool, and all support pre-configuration, for example, by the user when creating the storage pool.

[0107] At present, when writing data to a storage pool, one data writing mode is the write append only (WAP) mode. The WAP mode means retaining the data in the original storage space. The original storage space is the storage space indicated by the storage space address corresponding to the logical address in the data write request. That is, data is not written to the original storage space, but a new storage space is re-applied and the data is written to the new storage space, and then the storage space address corresponding to the logical address is modified to the address of the new storage space. After that, the original storage space is recycled through the background garbage collection (GC) program.

[0108] It should be noted that garbage collection is performed on a logical block group basis (a logical block group may be a stripe or a logical unit). If only part of the data in a logical block group needs to be deleted, the other valid data in the logical block group needs to be read out and stored in the reserved space, and then the entire logical block group can be recycled.

[0109] The advantage of using the WAP mode to write data is that it supports writing small data, and a large amount of small data can be aggregated into large data and then written to the hard disk sequentially.

[0110] The disadvantages of using WAP mode to write data include:

[0111] 1. The storage space address corresponding to the logical address needs to be updated, that is, the new storage space address needs to be written to the index table (the index table belongs to metadata), which will cause metadata write amplification and consume the CPU of the storage node, that is, it will generate additional overhead.

[0112] It is understandable that write amplification can be simply understood as exceeding the number of write operations and / or the amount of data. For example, the amount of data to be written in a write operation is 4KB, but due to various reasons, the actual amount of data written is 1024KB, resulting in write amplification.

[0113] 2. When performing garbage collection, other valid data in a logical block group needs to be read out and written to other reserved spaces, which will cause write amplification and additional read operations, bringing additional overhead.

[0114] It is understandable that the purpose of writing data is to write target data, and there is no need to read data. However, in fact, additional read operations may be introduced in the process of writing data. For example, when writing data in the append write mode, additional read operations are generated in the garbage collection process, and additional read operations also bring additional overhead. For the convenience of description, in the embodiment of the present application, the number of read operations and / or the amount of data generated in the process of writing data is referred to as read amplification.

[0115] 3. To implement garbage collection, over-provisioning (OP) space is required. The over-provisioning space is used to write other valid data in the current logical unit or logical block group. The over-provisioning space is only used for garbage collection and cannot be used for other purposes.

[0116] 4. In order to meet full-strip write requirements, a large number of write requests need to be sent in batches from the cache, which will result in additional write amplification.

[0117] When writing data to the storage pool, another data writing mode is write in place (WIP). The WIP mode overwrites the data in the storage space indicated by the storage space address corresponding to the logical address in the data write request, that is, the data in the original storage space is not retained, and there is no need to modify the storage space address corresponding to the logical address.

[0118] The advantages of using the WIP mode to write data are: it can support large data overwriting, without applying for additional storage space, without garbage collection, and without modifying the address index table.

[0119] The disadvantages of writing data in WIP mode include:

[0120] 1. If the data to be written is small and cannot be fully striped, the EC checksum mechanism and the partial redundancy type of the RAID mechanism need to recalculate the checksum information, which will generate additional write amplification and additional overhead.

[0121] For example, if the size of a stripe is 64K and the target data is 32K, then you may need to write another 32K of data to fill the stripe, which will cause write amplification. In addition, you need to read the original 32K data in the stripe to recalculate the checksum information, so there are additional read operations, which brings additional overhead.

[0122] In the process of writing data in the overwrite write mode, when full stripe writing cannot be achieved, the number of read operations and / or the amount of data generated by reading other data to fill the stripe is also called read amplification.

[0123] 2. For partial redundancy types of EC verification mechanism and RAID mechanism, if the written data is small data, there may be a write hole problem. To solve the write hole problem, additional overhead will be generated.

[0124] The write hole problem means that for a stripe, when writing data and verification information to different disks, it may not be possible for all of them to be written successfully, which may cause the data in the original stripe to be destroyed.

[0125] To solve the write hole problem, data and / or verification information may be backed up so that data can be restored based on the backed-up information when data writing fails.

[0126] When the target data is small and cannot be written in full stripes, the checksum information needs to be recalculated and written based on the target data and other data in the stripe. This process will generate read amplification (read amplification generated by reading other data to fill the stripe). In addition, write holes may occur during the data writing process. To solve the write hole problem, the target data and / or checksum information are backed up to the cache or hard disk, but backing up the data and / or checksum information requires a certain amount of overhead.

[0127] When the target data can be written in full stripes, the checksum information is calculated and the data and checksum information are written to the stripes. Similarly, write holes may also occur during data writing. However, when full stripes can be written, since the cache stores all the data written to the stripe, there is no need to back up the data and / or checksum information. It can be seen that full stripe writing can avoid the additional read-write amplification and overhead caused by solving the write hole problem.

[0128] In summary, no matter the WAP mode or the WIP mode, there is a problem of additional overhead due to write amplification and / or read amplification.

[0129] In response to the above problems, an embodiment of the present application provides a data storage method, which can be applied to a storage controller in a storage system. The storage system includes multiple memories. The storage space mapped to one or more memories in the storage system is called a storage pool. The storage pool supports overwrite write mode and append write mode, and supports switching write mode. The method can adaptively select a suitable data write mode, thereby reducing the overhead of the data writing process.

[0130] The data storage method provided in the embodiment of the present application can be applied to the above Figure 1 The storage system shown in the figure is composed of a storage controller (eg, Figure 1 The controller 1201 in the embodiment executes the method. For more descriptions of the storage system and the storage node, please refer to the above Figure 1 And related content will not be repeated here.

[0131] Based on the contents in the above embodiments, the data storage method provided in the embodiments of the present application is described in detail below.

[0132] like Figure 6 As shown, the data storage method provided in the embodiment of the present application includes S601-S603.

[0133] S601, obtaining target data.

[0134] The target data is the data to be written into the storage pool.

[0135] S602: Determine a write mode for writing target data into a storage pool from among multiple write modes supported by the storage system.

[0136] Among them, the data writing mode includes an overwrite writing mode or an append writing mode. The storage pool is a storage space mapped by multiple memories (such as hard disks) in the storage system. For relevant content about the storage pool, please refer to the content of the above embodiment, which will not be repeated here.

[0137] The storage pool in the embodiment of the present application supports an overwrite write mode and an append write mode, and supports switching write modes, that is, when writing data in the same storage pool, some data is written in an overwrite write mode and some data is written in an append write mode.

[0138] Combination Figure 6 ,like Figure 7 As shown, in one implementation, the above S602 can be specifically implemented through S6021.

[0139] S6021. When the overwrite write amplification degree of the target data is less than the append write amplification degree, determine the write mode to be the overwrite write mode; when the overwrite write amplification degree of the target data is greater than the append write amplification degree, determine the write mode to be the append write mode.

[0140] The overwrite write amplification degree is the magnitude of the read / write amplification generated when the target data is written in the overwrite write mode, and the append write amplification degree is the magnitude of the read / write amplification generated when the target data is written in the append write mode.

[0141] It should be noted that the read-write amplification generated in the above-mentioned data writing process may include read amplification and / or write amplification, that is, in the process of writing target data, write amplification may occur in some cases, read amplification may occur in some cases, and read amplification and write amplification may occur in some cases.

[0142] In the embodiment of the present application, the greater the read / write amplification generated in the process of writing data (i.e., the greater the amplification degree caused by writing data), the greater the overhead required for writing data. Therefore, in the process of determining which write mode to use to write the target data into the storage pool, the write mode of the target data can be determined according to the overwrite write amplification degree and the append write amplification degree of the target data. Specifically, the write mode with the smallest amplification degree between the overwrite write amplification degree and the append write amplification degree is determined as the write mode of the target data, which can reduce the additional overhead caused by the process of writing the target data.

[0143] Optionally, the sizes of read / write amplification corresponding to the two write modes can be calculated according to the stripe size, redundancy level and target data size of the storage pool, that is, the size of the read / write amplification generated when writing data in the overwrite write mode is calculated according to the stripe size, redundancy level and target data size of the storage pool, and the size of the read / write amplification generated when writing data in the append write mode is calculated according to the stripe size, redundancy level and target data size of the storage pool. The specific calculation process of the overwrite write amplification size and the append write amplification size will be described in detail by way of example in the following embodiments.

[0144] Optionally, in the embodiment of the present application, the method for determining the magnitude of the overwrite write amplification degree and the append write amplification degree of the target data may also include the following method 1 or method 2:

[0145] In method 1, when the redundancy level of the storage pool is a redundancy level without verification information, the overwrite write amplification degree of the target data is smaller than the append write amplification degree.

[0146] According to the contents of the above embodiments, the redundancy mechanism of the storage pool includes EC verification mode, multi-copy mode and RAID mode. Each redundancy mode may include different redundancy levels. The redundancy level may also be called redundancy type. The redundancy level is a kind of configuration information of the storage pool and supports pre-configuration. For example, in EC verification mode, different redundancy levels may be defined according to the number of data shards; for example, in multi-copy mode, different redundancy levels may be defined according to the number of copies; in RAID mode, redundancy levels may be defined according to different redundancy methods, for example, the redundancy levels of RAID mode include RADI0, RAID1, RAID2, RAID3, RAID4, RAID5, RAID6, RAID7, RAID10, etc.

[0147] It is understandable that the redundancy level of the storage pool may indicate whether the target data requires verification information. In the embodiment of the present application, the redundancy level without verification information may include at least one of the following: RAID0, RAID1, RAID10, or multiple copies. Optionally, among the various redundancy levels in the above examples, the redundancy level other than RAID0, RAID1, RAID10, and multiple copies may indicate that the target data does not require verification information.

[0148] In the embodiment of the present application, when the target data has no verification information (no need to calculate the verification information), when the target data is written in the append write mode, it is necessary to apply for a new storage space. After the target data is written in the storage space, it is also necessary to update the address index table (the correspondence between the logical address and the storage space address), and it is also necessary to recycle the original storage space; when the target data is written in the overwrite write mode, it is not necessary to apply for a new storage space, and the data can be written to the original storage space. It can be seen that compared with the overwrite write mode, the append write mode needs to update the address index table and recycle garbage, which will generate read-write amplification and additional overhead, so the overwrite write mode is selected.

[0149] Since the target data has no verification information, when the target data is stored in an overwrite mode, there is no need to calculate the verification information even if full stripe writing is not possible. This can reduce read-write amplification and the additional overhead of the data writing process to a certain extent.

[0150] According to method 1, a method for determining a write mode when writing target data into a storage pool is: when the redundancy level of the storage pool is a redundancy level without verification information, determining that the write mode when writing target data into the storage pool is an overwrite write mode.

[0151] Method 2: When the size of the target data is equal to an integer number of storage pool stripe sizes, the overwrite write amplification degree of the target data is less than the append write amplification degree.

[0152] It can be understood that the stripe size of the storage pool is also a kind of configuration information of the storage pool. The stripe size can be configured according to actual needs. For example, the size of a stripe can be 64K, 128K, 256K, etc.

[0153] In the embodiment of the present application, when the size of the target data is equal to the stripe size of the storage pool, when the target data is written in the append write mode, it is necessary to apply for a new storage space. After writing the target data and the calculated verification information (if there is verification information) in the storage space, it is also necessary to update the address index table and recycle the original storage space; when the target data is written in the overwrite write mode, it is not necessary to apply for a new storage space, and the target data and the calculated verification information can be written into the original storage space. It can be seen that compared with the overwrite write mode, the append write mode needs to update the address index table and recycle garbage, which will generate read-write amplification and additional overhead. It can be seen that the read-write amplification degree of the overwrite write mode is less than that of the append write mode, so the overwrite write mode is selected.

[0154] When the size of the target data is equal to an integer number of stripe sizes, full stripe writing can be achieved without sending a large number of write requests in batches from the cache multiple times. Read-write amplification and additional overhead (caused by reading other data and backing up data and / or verification information) caused by solving the write hole problem can also be avoided. In this way, read-write amplification can be reduced to a certain extent, and the additional overhead of the data writing process can be reduced.

[0155] According to method 2, a method for determining the write mode when writing target data into a storage pool is: when the size of the target data is equal to the stripe size of an integer number of storage pools, determining the write mode when writing the target data into the storage pool is an overwrite write mode.

[0156] S603: Write the target data into the storage pool according to the determined writing mode.

[0157] Combination Figure 7 ,like Figure 8 As shown, in one implementation, when the write mode is the overwrite write mode, the above S603 can be implemented through S6031a-S6031b.

[0158] S6031a. Based on an address index table in the storage system, map the logical address of the target data to a first storage space address, where the first storage space address indicates a first storage space in the storage pool.

[0159] According to the relevant description in the above embodiments, the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses. The address index table is used for address conversion, that is, for converting the logical address in the access request into the storage space address, so as to access the data in the storage pool (for example, read data or write data) based on the storage space address.

[0160] S6031b. Write the target data into the first storage space.

[0161] The following describes the process of writing data in overwrite mode with an example. Assume that the stripe size is 64K, the index granularity of the logical block address LBA (i.e., the starting address or address offset) is 64K, that is, the logical block addresses LBA are 0K, 64K, 128K, 192K, 256K, ... respectively.

[0162] like Fig. 9As shown in the figure, assume that before writing data to the storage pool, in a logical unit, the data stored at the positions of LBA 0, 64K, 128K, and 256K are "D0", "D1", "D2", and "D3", respectively, and the length of each data is 64K. The data storage process includes: applying for 4 stripes (each stripe size is 64K) in the storage pool and writing "D0", "D1", "D2", and "D3". Fig. 9 In the example, the data in stripe 1 is "D0", the data in stripe 2 is "D1", the data in stripe 3 is "D2", and the data in stripe 4 is "D3". After the data is written, when the user needs to read the data at the LBA 64K position in the logical unit, the storage space address corresponding to the logical unit identifier (LUN ID) and LBA (64K) is searched based on the address index table, and then the corresponding data "D1" can be read from the storage pool. Subsequently, data can continue to be written to the storage pool for new data write requests.

[0163] Continue to refer Fig. 9 , if the storage node receives a new data write request, the data write request includes the target data "D4" and the logical address of the target data, and the logical unit indicated by the logical unit identifier in the logical address is Fig. 9 In the logical unit shown, the LBA is 256K and the data length is 64K. When writing the data in the overwrite write mode, after converting the logical address into a storage space address, "D4" is written in an overwrite manner in the storage space indicated by the storage space address, such as Fig. 9 As shown, "D3" is overwritten in the storage space that originally stored "D3" and "D4" is written. The overwrite write mode does not require modifying the address index table.

[0164] like Figure 8 As shown, in another implementation, when the write mode is the append write mode, the above S603 can be implemented through S6032a-S6032b.

[0165] S6032a. Write the target data into the second storage space in the storage pool.

[0166] In the embodiment of the present application, when the append write mode is adopted, storage space needs to be re-applied from the storage pool, and the second storage space mentioned above is the storage space re-applied from the storage pool according to the size of the target data. For example, if the target data is 64K, the second storage space is 64K.

[0167] S6032b. Update the first storage space address corresponding to the logical address of the target data in the address index table of the storage system to a second storage space address, where the second storage space address indicates the second storage space in the storage pool.

[0168] The process of writing data in the append write mode is still described by taking an example.

[0169] Continuing with the example in S6031b, Fig.10 As shown, if the storage node receives a new data write request, the data write request includes the target data "D4" and the logical address of the target data, and the logical unit indicated by the logical unit identifier in the logical address is Fig.10 In the logical unit shown, the LBA is 256K and the data length is 64K. When writing the data in the append write mode, a new storage space of 64K is requested (for example Fig.10 Then, the address index table is modified, and the storage space address corresponding to the logical address in the data write request (the first storage space address, i.e., the address of the storage space corresponding to stripe 4) is modified to the second storage space address (the address of the storage space corresponding to stripe 5).

[0170] Furthermore, after the data is written to the second storage space, the original storage space (i.e., the storage space corresponding to "D3") is marked as a recycling state, and garbage collection is then performed. When garbage collection is performed, the entire logical block group where "D3" is located needs to be recycled. Therefore, "D2" needs to be read out and then written to the reserved space, and then the data in the entire logical block group is deleted to release the logical block group. In this way, the storage space in the logical block group can be reallocated to other data write requests later.

[0171] It should be noted that the above Fig. 9 and Fig.10 The data writing process shown is described from the perspective of the target data, and the check information is not described. It should be understood that when the redundancy level indicates that the target data has check information, in both writing modes, the check information needs to be calculated and written to the storage space together with the target data.

[0172] In summary, in the data storage method provided in the embodiment of the present application, a suitable data writing mode, such as an append writing mode or an overwrite writing mode, can be determined for the target data, and the target data can be written into the storage pool according to the determined writing mode. In other words, the storage system using this method can support both the overwrite writing mode and the append writing mode, and can be compatible with the advantages of the overwrite writing mode and the advantages of the append writing mode, and can adaptively and flexibly select a data writing mode with better performance for the target data, which can reduce read-write amplification, thereby reducing the overhead of the data writing process.

[0173] In combination with all contents of the above embodiments and the accompanying drawings, the complete process of the data storage method is described below from the perspective of a storage device.

[0174] like Fig.11 As shown, the data storage method provided in the embodiment of the present application includes the following steps.

[0175] S1101. Receive a data write request.

[0176] S1102: Write the target data in the data write request into the cache.

[0177] After the storage device writes the data in the data write request into the cache, it may terminate the write request, that is, return a write completion response to the computing node, and record the logical address in the data write request into the cache table.

[0178] S1103, adding the data write request to the disk flushing queue (ie, the disk flushing queue aggregation).

[0179] It can be understood that in the embodiment of the present application, the storage device receives multiple data write requests in sequence, writes the data into the cache, and stores the logical address in the data write request into the cache table. When the data in the cache reaches a certain threshold, the data is written from the cache to the storage pool. This process can be understood as a disk flushing process, that is, the process of flushing data from the cache to the hard disk. In the cache, multiple data write requests form a disk flushing queue in order. In the disk flushing queue, a data write request can also be called a disk flushing request.

[0180] S1104, read the disk flushing request from the disk flushing queue.

[0181] It should be understood that the steps after S1104 are all actions performed by the storage device in the background. The controller of the storage device asynchronously reads the disk flushing request from the disk flushing queue, reads the target data corresponding to the disk flushing request from the cache, and reads the logical address corresponding to the disk flushing request from the cache table (hereinafter, the logical address corresponding to the disk flushing request read this time is referred to as the first logical address).

[0182] S1105: Determine whether the first logical address exists in the address index table.

[0183] It should be noted that if the first logical address does not exist in the address index table, the following S1106-S1108 are executed to write the target data to the new storage space in the storage pool. If the first logical address exists in the address index table, S1109 and subsequent steps are executed to write the target data to the storage pool in an overwrite or append write mode.

[0184] S1106. Apply for storage space (which may be called third storage space).

[0185] S1107. Write the target data into the third storage space.

[0186] S1108. Insert (ie, write) the first logical address and the storage space address storing the target data in the third storage space (referred to as the third storage space address) into the address index table according to the corresponding relationship.

[0187] S1109: Determine a write mode for target data.

[0188] like Fig.12 As shown, the process of determining the writing mode of the target data (S1109) includes the following steps.

[0189] S1201. Determine whether the redundancy level of the storage pool is a redundancy level without verification information (RAID0, RAID1, RAID10, or multiple copies).

[0190] If the redundancy level of the storage pool is any one of RAID0, RAID1, RAID10 or multiple copies, determine that the write mode is the overwrite write mode; otherwise, execute S1202.

[0191] S1202: Determine whether the size of the target data completely covers one or more stripes, that is, whether the size of the target data is equal to an integer number of stripe sizes.

[0192] That is, when the redundancy level of the storage pool is with verification information, the write mode of the target data is determined according to the stripe size of the storage pool and the size of the target data. If the size of the target data is equal to an integer number of stripe sizes, that is, the target data fills the current stripe, the write mode is determined to be the overwrite write mode; otherwise (the size of the target data is smaller than the stripe size, that is, the target data is small and cannot fill the current stripe), execute S1203-S1204.

[0193] S1203: Determine the overwrite write amplification degree and the append write amplification degree of the target data.

[0194] S1204: Determine whether the overwrite write amplification degree is less than the append write amplification degree.

[0195] That is, when the size of the target data is smaller than an integer number of stripe sizes of the storage pool, the overwrite write amplification degree and the append write amplification degree of the target data are determined. If the overwrite write amplification degree is smaller than the append write amplification degree, the write mode is determined to be the overwrite write mode; otherwise (the overwrite write amplification degree is greater than the append write amplification degree), the write mode is determined to be the append write mode.

[0196] Optionally, the read-write amplification generated when writing data in overwrite mode can be calculated based on the stripe size, redundancy level and target data size of the storage pool to obtain the overwrite write amplification degree, and the read-write amplification generated when writing data in append write mode can be calculated based on the stripe size, redundancy level and target data size of the storage pool to obtain the append write amplification degree.

[0197] Exemplary, reference Fig.13 , if the stripe size is 64K (referring to the capacity of the storage space used to store the data part is 64K), the redundancy level is RAID5, the stripe includes 4 data blocks and 1 check block, and the size of each data block is 16K. The data stored in the storage space corresponding to the logical address in the data write request (which can be called the original storage space) includes data blocks D0 and D1, and D0, D1, D2, D3 and check block P0 belong to the same logical block group. The size of each data block and check block is 16K. If the size of the target data contained in the data write request is 32K, the target data can be divided into 2 data blocks, namely D4 and D5.

[0198] When using the append write mode, you need to reapply for storage space and write D4 and D5 in the storage space, and other data (such as Fig.13 After the data D2', D3' in the data are gathered into a full strip, the check block P2 is calculated, and the data D4, D5, D2', D3' and the check block P1 are written. The read-write amplification generated in the process of writing the target data in the append write mode includes: the read-write amplification generated by garbage collection, the read-write amplification generated by updating the address index table and the garbage mark, among which the read-write amplification generated by garbage collection includes read amplification and write amplification, and the read-write amplification generated by updating the address index table and the garbage mark includes write amplification.

[0199] Among them, the size of the read-write amplification caused by garbage collection is 16K. The specific calculation process is as follows: the read-write amplification caused by garbage collection is related to the amount of garbage in the logical block group. For example, for an 8M logical block group, when the amount of garbage in the logical block group reaches 90% of the capacity of the logical block group, the logical block group begins to be recycled. Specifically, other data in the logical block group (non-garbage data, accounting for 10%) is read out and written to other storage spaces. Therefore, the size of the read-write amplification caused by garbage collection of the entire logical block group is 8M*10%*3=2.4M. Further, the size of the read-write amplification allocated to the data to be written (D4 and D5 are 32K in total) is 2.4M*(32K / 8M)=9.6K. Then, 9.6K is rounded up. The rounded value should be an integer multiple of the size of a data block in the hard disk (the minimum is 8K). Therefore, 9.6K is rounded up to 16K.

[0200] The read / write amplification caused by updating the address index table and garbage mark is 8K, which is an estimated value.

[0201] In summary, the size of the read / write amplification generated in the process of writing the above target data in the append write mode is 24K (16K+8K).

[0202] Continue to refer Fig.13 When the overwrite write mode is adopted, the storage space corresponding to the logical address in the data write request is written to D4 and D5 in an overwrite manner, and a new check block P1 is calculated based on D4, D5, D2, and D3, and D4, D5, and P1 are written. It should be noted that in this example, the check block P1 is backed up to the cache, not to the hard disk, so the read-write amplification caused by the backup check block P1 to the hard disk is not considered. The read-write amplification generated in the process of writing the target data in the overwrite write mode includes: the read amplification generated by reading the existing data D2 and D3 in the stripe when calculating the check block P1, that is, 32K.

[0203] according to Fig.13 In the example of , the overwrite write amplification degree is 32K, and the append write amplification degree is 24K. Since the overwrite write amplification degree is greater than the append write amplification degree, it is determined that the write mode of the target data is the append write mode.

[0204] Exemplary, reference Fig.14 , if the stripe size is 64K (referring to the capacity of the storage space used to store the data part is 64K), the redundancy level is RAID5, the stripe includes 4 data blocks and 1 check block, and the size of each data block is 16K. The data stored in the storage space corresponding to the logical address in the data write request (which can be called the original storage space) includes data blocks D0 and D1, and D0, D1, D2, D3 and check block P0 belong to the same logical block group. The size of each data block and check block is 16K. If the size of the target data contained in the data write request is 48K, the target data can be divided into 3 data blocks, namely D4, D5, and D6.

[0205] When using the append write mode, you need to reapply for storage space and write D4, D5 and D6 in the storage space, and other data (such as Fig.14 After the data D3' in the append write mode are gathered into a full strip, the check block P4 is calculated, and the data D4, D5, D6, D3' and the check block P4 are written. The read-write amplification generated in the process of writing the target data in the append write mode includes: the read-write amplification generated by garbage collection, the update address index table and the read-write amplification generated by garbage marking.

[0206] The size of the read-write amplification caused by garbage collection is 16K. Similarly, the calculation process is as follows: the read-write amplification caused by garbage collection is related to the amount of garbage in the logical block group. For example, for an 8M logical block group, when the amount of garbage in the logical block group reaches 90% of the capacity of the logical block group, the logical block group begins to be recycled. Specifically, other data in the logical block group (non-garbage data, accounting for 10%) is read out and written to other storage spaces. Therefore, the size of the read-write amplification caused by garbage collection of the entire logical block group is 8M*10%*3=2.4M. Further, the size of the read-write amplification allocated to the data to be written (D4, D5 and D6, a total of 48K) is 2.4M*(48K / 8M)=14.4K. Then, 14.4K is rounded up. The rounded value should be an integer multiple of the size of a data block in the hard disk (the minimum is 8K). Therefore, 14.4K is rounded up to 16K.

[0207] The read / write amplification caused by updating the address index table and garbage mark is 8K, which is an estimated value.

[0208] In summary, the size of the read / write amplification generated in the process of writing the above target data in the append write mode is 24K (16K+8K).

[0209] Continue to refer Fig.14 When the overwrite write mode is adopted, the storage space corresponding to the logical address in the data write request is written to D4, D5 and D6 in an overwrite manner, and a new check block P3 is calculated based on D4, D5, D6 and D3, and D4, D5, D6 and P3 are written. It should be noted that in this example, the check block P3 is backed up to the cache, not to the hard disk, so the read-write amplification caused by the backup check block P3 to the hard disk is not considered. The read-write amplification generated in the process of writing the target data in the overwrite write mode includes: the read amplification generated by reading the existing data D3 in the stripe when calculating the check block P3, that is, 16K.

[0210] according to Fig.14 In the example, the overwrite write amplification degree is 16K, the append write amplification degree is 24K, the overwrite write amplification degree is smaller than the append write amplification degree, and therefore it is determined that the write mode of the target data is the overwrite write mode.

[0211] In an embodiment of the present application, according to the overwrite write amplification degree and the append write amplification degree, a write mode with a smaller read / write amplification degree is selected to write target data, which can reduce the read / write amplification and thus reduce the overhead of writing data.

[0212] Optionally, in the embodiment of the present application, when the overwrite write magnification degree is equal to the append write magnification degree, one of the overwrite write mode and the append write mode may be selected; or a user-configured write mode may be used. It is understandable that the user may pre-configure a preferred write mode, for example, the user's configuration content is: when the overwrite write magnification degree is equal to the append write magnification degree, the overwrite write mode is selected first.

[0213] In the embodiment of the present application, if it is determined that the write mode of the target data is an overwrite write mode, S1110 is executed; if it is determined that the append write mode of the target data is determined, the following S1111 is executed.

[0214] S1110 , write target data into the storage pool in an overwrite write mode.

[0215] In the embodiment of the present application, when the overwrite write mode is adopted, in order to make full stripe write as much as possible, during the above disk flushing process, data is read in batches adjacent (or continuous) according to the logical address and written into the storage pool.

[0216] S1111. Write target data into the storage pool in append write mode.

[0217] For a detailed description of the above S1110, reference may be made to the contents of S6031a-S6031b in the above embodiment. For a detailed description of S1111, reference may be made to the contents of S6032a-S6032b in the above embodiment, which will not be repeated here.

[0218] It is understandable that the above-mentioned data storage method is executed by a storage device, and the storage device includes a hardware structure and / or software module corresponding to each function in order to realize the above-mentioned functions. It should be easily appreciated by those skilled in the art that, in combination with the method steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0219] The embodiment of the present application can divide the functional modules of the above-mentioned storage device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0220] In the case of dividing each functional module into corresponding functional modules, Fig.15 A possible structural diagram of the storage device involved in the above embodiment is shown. The storage device includes an acquisition module 1501, a determination module 1502 and a data writing module 1503.

[0221] Among them, the acquisition module 1501 is used to execute S601 in the above method embodiment; the determination module 1502 is used to execute S602 (S602 includes S6021) and S1109 (S1109 is implemented by S1201-S1204) in the above method embodiment; the write data module 1503 is used to execute S603 (S603 includes S6031a-S6031b or includes S6032a-S6032b), S1110, and S1111 in the above method embodiment.

[0222] Optionally, the storage device provided in the embodiment of the present application further includes a garbage collection module 1504 for reclaiming storage space of the storage system.

[0223] Each module of the above-mentioned storage device can also be used to execute other actions in the above-mentioned method embodiment. All relevant contents of each step involved in the above-mentioned method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0224] In the case of an integrated unit, Fig.16 FIG. 2 shows another possible structural diagram of the storage device involved in the above embodiment. Fig.16 As shown, the storage device provided in the embodiment of the present application may include: a processing module 1601 and a communication module 1602. The processing module 1601 may be used to control and manage the actions of the storage device. For example, the processing module 1601 may be used to support the above Fig.14 The acquisition module 1501, determination module 1502, and data writing module 1503 in the embodiment of the present invention perform corresponding steps, and / or other processes for the technology described herein. The communication module 1602 can be used to support the communication device with other network entities, for example, to support the storage device to communicate with a computing node. Fig.16 As shown, the communication device may also include a storage module 1603 for storing computer instructions and data.

[0225] The processing module 1601 may be a processor or a controller (for example, the processor 1601 may be Figure 1The controller 1201 in the storage node 120 in the embodiment of the present invention) may be a combination of processors 1601 that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The communication module 1602 may be a communication interface (for example, the communication module 1602 may be a Figure 1 The storage module 1603 may be a memory (for example, the storage module 1603 may be Figure 1 When the processing module 1601 is a processor, the communication module 1602 is a communication interface, and the storage module 1603 is a memory, the processor, the transceiver, and the memory may be connected via a bus.

[0226] For more details on how the modules included in the storage device implement the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.

[0227] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) mode or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0228] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk, and other media that can store program codes.

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data storage method, characterized in that: A storage controller applied to a storage system, wherein the storage system includes a plurality of memories, and the method includes: Get target data; Determine, from among a plurality of write modes supported by the storage system, a write mode when writing the target data into a storage pool; the storage pool is a storage space mapped by a plurality of memories in the storage system, the write mode includes an overwrite write mode or an append write mode, the storage pool supports the overwrite write mode and the append write mode, and supports switching the write mode; The target data is written into the storage pool according to the determined writing mode.

2. The method according to claim 1, characterized in that The determining of a write mode when writing the target data into a storage pool includes: When the overwrite write amplification degree of the target data is less than the append write amplification degree, determining the write mode to be the overwrite write mode; when the overwrite write amplification degree of the target data is greater than the append write amplification degree, determining the write mode to be the append write mode; The overwrite write amplification degree is the magnitude of the read / write amplification generated when the target data is written in the overwrite write mode, and the append write amplification degree is the magnitude of the read / write amplification generated when the target data is written in the append write mode.

3. The method according to claim 1 or 2, characterized in that: The determining of a write mode when writing the target data into a storage pool includes: When the redundancy level of the storage pool is a redundancy level without verification information, it is determined that the write mode when writing the target data into the storage pool is an overwrite write mode.

4. The method according to claim 1 or 2, characterized in that: The determining of a write mode when writing the target data into a storage pool includes: When the size of the target data is equal to an integer number of stripe sizes of the storage pool, it is determined that the write mode when writing the target data into the storage pool is an overwrite write mode.

5. The method according to any one of claims 1 to 4, characterized in that: The redundancy level without verification information includes at least one of the following: RAID0, RAID1, RAID10 or multiple copies.

6. The method according to any one of claims 1 to 5, characterized in that: When the write mode is an overwrite write mode, writing the target data into the storage pool according to the determined write mode includes: Based on the address index table of the storage system, mapping the logical address of the target data to a first storage space address; the first storage space address indicates a first storage space in the storage pool, and the address index table includes a plurality of logical addresses and storage space addresses corresponding to the plurality of logical addresses; The target data is written into the first storage space.

7. The method according to any one of claims 1 to 5, characterized in that: When the write mode is the append write mode, writing the target data into the storage pool according to the determined write mode includes: Writing the target data into a second storage space in the storage pool; In the address index table of the storage system, the first storage space address corresponding to the logical address of the target data is updated to a second storage space address; the second storage space address indicates the second storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

8. A storage device, characterized in that: Including acquisition module, determination module and data writing module; The acquisition module is used to acquire target data; The determination module is used to determine a write mode when writing the target data into a storage pool from a plurality of write modes supported by the storage system; the storage pool is a storage space mapped by a plurality of memories in the storage system, the write mode includes an overwrite write mode or an append write mode, the storage pool supports the overwrite write mode and the append write mode, and supports switching the write mode; The data writing module is used to write the target data into the storage pool according to the determined writing mode.

9. The storage device according to claim 8, characterized in that: The determination module is specifically used to determine that the write mode is an overwrite write mode when the overwrite write amplification degree of the target data is less than the append write amplification degree; and to determine that the write mode is an append write mode when the overwrite write amplification degree of the target data is greater than the append write amplification degree; wherein the overwrite write amplification degree is the size of the read-write amplification generated when the overwrite write mode is used to write the target data, and the append write amplification degree is the size of the read-write amplification generated when the append write mode is used to write the target data.

10. The storage device according to claim 8 or 9, characterized in that: The determination module is specifically configured to determine that a write mode when writing the target data into the storage pool is an overwrite write mode when the redundancy level of the storage pool is a redundancy level without verification information.

11. The storage device according to claim 8 or 9, characterized in that: The determining module is specifically configured to determine that the write mode when writing the target data into the storage pool is the overwrite write mode when the size of the target data is equal to an integer number of stripe sizes of the storage pool.

12. The storage device according to any one of claims 8 to 11, characterized in that: The redundancy level without verification information includes at least one of the following: RAID0, RAID1, RAID10 or multiple copies.

13. The storage device according to any one of claims 8 to 12, characterized in that: When the write mode is the overwrite write mode, The data write module is specifically used to map the logical address of the target data to a first storage space address based on an address index table in the storage system, and write the target data into the first storage space; the first storage space address indicates a first storage space in the storage pool, and the address index table includes multiple logical addresses and storage space addresses corresponding to the multiple logical addresses.

14. The storage device according to any one of claims 8 to 12, characterized in that: When the write mode is the append write mode, The data writing module is specifically used to write the target data into the second storage space in the storage pool; and update the first storage space address corresponding to the logical address of the target data in the address index table in the storage system to the second storage space address; The second storage space address indicates the second storage space in the storage pool, and the address index table includes a plurality of logical addresses and storage space addresses corresponding to the plurality of logical addresses.

15. A storage device, characterized in that: The method comprises a memory and at least one processor connected to the memory, wherein the memory is used to store computer program code, and the computer program code comprises computer instructions. When the computer instructions are executed by the at least one processor, the storage device executes the method according to any one of claims 1 to 7.

16. A storage system, characterized in that: It comprises a storage controller and a plurality of memories; the storage controller is used to execute the method according to any one of claims 1 to 7 to write target data into a storage pool mapped by the plurality of memories.

17. A computer-readable storage medium, characterized in that: Computer instructions are stored, and when the computer instructions are executed on a computer, the method according to any one of claims 1 to 7 is executed.

Images