Adaptive matching method, device and computer program product

Abstract

The application provides an adaptive matching method, device and computer program product, which are used for automatically matching a transaction file system block size and a client IO size. The method comprises creating a shared domain on a first file system, wherein the block size of the first file system is a first block size. The method further comprises determining a block size of the shared domain as a second block size, wherein the second block size is not equal to the first block size. If the block size of a second file system is the second block size, the shared domain is migrated from the first file system to the second file system. By implementing the application, user operation can be simplified, operation convenience is improved, and storage space fragmentation and indirect blocks can be reduced, and the performance of a storage system is further improved.

Description

Background Technology

[0001] Transactional file systems (TFS) grant administrators special privileges to select the underlying file system (FS) block size (BS) to optimally match the read / write / access (I / O) sizes of pre-defined applications (APPs) and their clients. The file system block size is the minimum guaranteed physical mapping within the file system; for traditional server message blocks (SMBs) and network file systems (NFS), the file system block size is typically set to 8KB. However, if the file system block size does not match the different client I / O sizes, it can lead to wasted storage space. Summary of the Invention

[0002] According to an example embodiment of this disclosure, an adaptive matching scheme is provided for automatically matching the transactional file system block size with the client I / O size.

[0003] In a first aspect of this disclosure, an adaptive matching method is provided, comprising: creating a shared domain on a first file system having a block size of a first block size; determining that the block size of the shared domain is a second block size, wherein the second block size is not equal to the first block size; and migrating the shared domain from the first file system to the second file system if the block size of the second file system is the second block size.

[0004] Implementing the first aspect simplifies file system configuration settings. Users no longer need to consider whether the transactional file system block size matches the application's I / O block size; the system can automatically match each client's I / O to a file system with an appropriate block size. Since users do not need to know the I / O pattern of a specific application when creating a transactional file system, implementing the first aspect simplifies user operations, improves operational convenience, and helps reduce storage fragmentation and indirect blocks, further improving storage system performance.

[0005] In conjunction with the first aspect, in some embodiments, the first file system and the second file system are included in a file system group. One or more file systems in the file system group may originate from a single storage system, a cluster, or even different storage systems in different clusters. This distributed nature helps to achieve load balancing because one or more I / O operations can actually be distributed across different file systems.

[0006] In conjunction with the first aspect, some embodiments further include: determining that a second file system with a block size of the second block size exists in the file system group before migrating the shared domain from the first file system to the second file system.

[0007] In conjunction with the first aspect, some embodiments further include: before migrating the shared domain from the first file system to the second file system, determining that there is no file system in the file system group with a block size of the second block size, and then creating the second file system in the file system group. Creating the second file system facilitates the subsequent migration steps.

[0008] In conjunction with the first aspect, in some embodiments, the block size of the shared domain is determined in several ways, including determining the block size of the shared domain based on the client's access behavior, which helps to improve storage performance and make full use of storage space.

[0009] In conjunction with the first aspect, in some embodiments, in response to a shared domain being accessed by one or more clients, the block size of the shared domain is determined as a second block size based on the read / write (IO) block size of one or more clients, the second block size being the IO block size in the shared domain that accounts for a proportion greater than a first threshold.

[0010] In conjunction with the first aspect, in some embodiments, in response to a shared domain being accessed by one or more clients, a second block size is determined based on the read / write (IO) block size of one or more clients. The second block size is a default block size, wherein the proportion of the IO block size of one or more clients in the shared domain is less than a first threshold.

[0011] In conjunction with the first aspect, in some embodiments, in response to the shared domain not being accessed by any client, the block size of the shared domain is determined to be a second block size, which is the default block size.

[0012] In conjunction with the first aspect, in some embodiments, when a shared domain is created on a first file system, there is a first jump link between the shared domain and the first file system.

[0013] In conjunction with the first aspect, some embodiments further include: deleting the first redirect link after migrating the shared domain from the first file system to the second file system.

[0014] In conjunction with the first aspect, some embodiments further include: after migrating the shared domain from the first file system to the second file system, generating a second jump link between the shared domain and the second file system.

[0015] In a second aspect of this disclosure, an apparatus is provided, comprising: a processor, and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the apparatus to perform actions when executed by the processor, the actions including: creating a shared domain, the shared domain being created on a first file system having a block size of a first block size; determining that the block size of the shared domain is a second block size, the second block size being not equal to the first block size; and if the block size of the second file system is the second block size, migrating the shared domain from the first file system to the second file system.

[0016] Implementing the second aspect simplifies file system configuration settings. Users no longer need to consider whether the transactional file system block size matches the application's I / O block size; the system can automatically match each client's I / O to a file system with an appropriate block size. Since users do not need to know the I / O patterns of a specific application when creating a transactional file system, implementing the first aspect simplifies user operations, improves ease of use, and helps reduce storage fragmentation and indirect blocks, further improving storage system performance.

[0017] In conjunction with the second aspect, in some embodiments, the first file system and the second file system are included in a file system group. One or more file systems in the file system group may originate from a single storage system, a cluster, or even different storage systems in different clusters. This distributed nature helps to achieve load balancing because one or more I / O operations can actually be distributed across different file systems.

[0018] In conjunction with the second aspect, in some embodiments, the action further includes: determining that a second file system with a block size of the second block size exists in the file system group before migrating the shared domain from the first file system to the second file system.

[0019] In conjunction with the second aspect, in some embodiments, before migrating the shared domain from the first file system to the second file system, it is determined that there is no file system in the file system group with a block size of the second block size, and then the second file system is created in the file system group. The creation of the second file system is for the purpose of facilitating the migration.

[0020] In conjunction with the second aspect, in some embodiments, in response to a shared domain being accessed by one or more clients, the block size of the shared domain is determined as a second block size based on the read / write (IO) block size of one or more clients, the second block size being the IO block size in the shared domain that accounts for a proportion greater than a first threshold.

[0021] In conjunction with the second aspect, in some embodiments, the block size of the shared domain is determined in several ways, including determining the block size of the shared domain based on the client's access behavior, which helps to improve storage performance and make full use of storage space.

[0022] In conjunction with the second aspect, in some embodiments, in response to a shared domain being accessed by one or more clients, a second block size is determined based on the read / write (IO) block size of one or more clients. The second block size is a default block size, wherein the proportion of the IO block size of one or more clients in the shared domain is less than a first threshold.

[0023] In conjunction with the second aspect, in some embodiments, in response to the shared domain not being accessed by any client, the block size of the shared domain is determined to be a second block size, which is the default block size.

[0024] In conjunction with the second aspect, in some embodiments, when a shared domain is created on a first file system, there is a first jump link between the shared domain and the first file system.

[0025] In conjunction with the second aspect, in some embodiments, the action further includes: deleting the first redirect link after migrating the shared domain from the first file system to the second file system.

[0026] In conjunction with the second aspect, some embodiments further include: after migrating the shared domain from the first file system to the second file system, generating a second jump link between the shared domain and the second file system.

[0027] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method according to a first aspect of this disclosure.

[0028] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a device, causes the device to perform the method described in accordance with a first aspect of this disclosure.

[0029] As can be seen from the above description, the solutions according to the various embodiments of this disclosure can automatically match the transactional file system block size and the client IO size for the file storage system, simplifying user operations, improving operational convenience, and helping to reduce storage space fragmentation and indirect blocks, thereby further improving the performance of the storage system.

[0030] It should be understood that the summary section is provided for the purpose of presenting the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0031] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0032] Figure 1 A flowchart is shown that can be used to implement an adaptive matching method according to some embodiments of the present disclosure;

[0033] Figure 2 A schematic diagram is shown of a structure that can be used to implement some embodiments of the file system according to this disclosure;

[0034] Figure 3A A schematic diagram is shown that can be used to implement the creation of a file system according to some embodiments of the present disclosure;

[0035] Figure 3B A flowchart illustrating creation check logic that can be used to implement some embodiments of this disclosure is shown;

[0036] Figure 4 A flowchart of a client I / O analysis proxy algorithm that can be used to implement some embodiments of this disclosure is shown;

[0037] Figure 5 A flowchart illustrating migration check logic that can be used to implement some embodiments of this disclosure is shown;

[0038] Figure 6 The diagram illustrates an example scenario related to adaptive matching that can be used to implement some embodiments of this disclosure;

[0039] Figure 7 The diagram illustrates an example scenario related to adaptive matching that can be used to implement some embodiments of this disclosure;

[0040] Figure 8 The diagram illustrates an example scenario related to adaptive matching that can be used to implement some embodiments of this disclosure;

[0041] Figure 9 The diagram illustrates an example scenario related to adaptive matching that can be used to implement some embodiments of this disclosure;

[0042] Figure 10 The diagram illustrates an example scenario related to adaptive matching that can be used to implement some embodiments of this disclosure;

[0043] Figure 11 A schematic structural diagram of an example device that can be used to implement some embodiments of the present disclosure is shown. Detailed Implementation

[0044] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0045] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0046] Since transactional file systems are typically used to support the workloads of specific applications, some products offer the ability to set the file system block size to 8KB, 16KB, 32KB, or 64KB during file system configuration to best suit the client I / O size typically used by a particular application.

[0047] In some embodiments, administrators or users may not always know the client I / O sizes of different applications when creating or configuring a File System (FS). If an administrator or user sets a fixed file system block size, the FS block size may not match the client I / O sizes of some applications. This can increase fragmentation and / or indirect blocks or indexes, leading to unnecessary waste of storage space resources and metadata. If administrators manually query the client I / O sizes of each application and manually set the file system block size accordingly, it is cumbersome and inefficient.

[0048] Automatically setting the file system (FS) block size based on the client I / O size of different applications is a problem that urgently needs to be solved. Matching the appropriate file system block size to different client I / O sizes helps reduce space fragmentation and / or indirect blocks, thereby improving storage space utilization.

[0049] When creating or configuring a file system file (FS), users may not always know the client I / O size for each application. For common applications, a mapping table can be stored to correspond to these applications and their client I / O sizes, thus replacing the need for users to manually set the client I / O sizes for these applications. The system then uses this mapping table to set the file system block size to match the client I / O sizes of these applications.

[0050] The method for setting up the mapping table still has room for improvement, for example, as follows:

[0051] First, typically, only one block size can be set when creating a file system. Therefore, when a user wants to run multiple applications, they need to create multiple file systems with different block sizes to ensure that applications with different client I / O sizes correspond to file systems with appropriate block sizes. Alternatively, only one file system can be created for multiple applications, but this might result in a mismatch between the file system block size and the client I / O size of some applications.

[0052] Second, the number of available applications is limited. If an application is not recorded in the mapping table, the most suitable block size for that application cannot be determined.

[0053] Third, when creating a file system, users should know which specific applications will run on that file system. However, customers often do not want to manually select which application corresponds to which file system, and the applications used or run may change as user needs evolve. This approach forces users to worry about which application should run on which file system.

[0054] To address this issue, this application proposes an adaptive matching method based on analytical cognitive algorithms to automatically match the transactional file system block size with the application's client I / O size, thus resolving the problem of the aforementioned inability to automatically match the transactional file system block size with the application's client I / O size. Compared to techniques that require users to manually match the transactional file system block size with the application's client I / O size, this method makes the storage system more intelligent. Such an intelligent storage system provides the ability to adaptively match the application's client I / O size with the file system block size, eliminating the need for users to manually manage which application should run on which file system.

[0055] Preferably, this application proposes an adaptive matching method for automatically creating the block size of the distributed transactional thin-provision file system (DTTFS) to match the client IO size of different applications.

[0056] The "DTTFS" proposed in this application is a software-defined distributed file system that, in most cases, can ensure that the client I / O block size matches the transactional file system block size by using a client I / O analysis agent (i.e., a host I / O analysis agent). DTTFS provides functionalities including: adaptive grouped file system and adaptive matching migration, adaptive routing of client I / O to FSs of appropriate block size, distributed storage load balancing, etc.

[0057] In the embodiments of this disclosure, a distributed file system (DFS) refers to a file system whose managed physical storage resources are not necessarily directly connected to local nodes, but are connected to nodes through a computer network, or are a complete hierarchical file system formed by combining several different logical disk partitions or volume labels. DFS provides a logical tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files distributed across the network. DFS can extend a file system fixed at a certain location to any number of locations / file systems, with numerous nodes forming a file system network. Each node can be distributed across different locations, communicating and transmitting data between nodes through the network. When using a distributed file system, users do not need to worry about which node the data is stored on or retrieved from; they only need to manage and store the data in the file system as if using a local file system. DFS distributes large amounts of data across different nodes, greatly reducing the risk of data loss. DFS has redundancy; the failure of some nodes does not affect the overall normal operation, and even if the data stored on the failed computer is corrupted, the corrupted data can be recovered by other nodes. Therefore, security is the most important feature of a distributed file system.

[0058] Thin provisioning is an on-demand storage technology that provides storage space to applications from a regular storage pool as needed, essentially eliminating the waste of allocated but unused space. The main advantage of thin provisioning is improved overall storage utilization. It automates storage allocation, significantly reducing the workload of storage administrators. It can reduce the amount of storage required to serve applications and also reduces the number of disks needed, resulting in a substantial reduction in energy consumption.

[0059] Transactional storage technology is a concurrency control mechanism in which applications accessing the same storage are classified into several transactions and run independently. Ultimately, all these transactions are committed or fail to avoid conflicts.

[0060] This application proposes a novel concept called a shared domain, also known as a mount domain, which replaces the entire file system as the unit for corresponding client I / O access. I / O analysis based on each shared domain, as well as data migration and configuration based on each shared domain, makes this method more flexible and lightweight to implement.

[0061] The term "mount" in this application's embodiments refers to the mechanism in a file system that distinguishes different logical paths. In a computer system, files are stored under different paths. The operating system makes files and directories on computer storage devices accessible to users through the computer's file system. The purpose of mounting is to determine the storage path corresponding to files and directories, establish the association between files and directories and the file system, and make the file system the access point for those files and directories. For example, mounting a shared domain to a client can mean establishing an association between the client and the shared domain, allowing the client to read and write through the shared domain's paths.

[0062] The client I / O analysis agent proposed in this application, as a service, can be used to determine the optimal block size, also known as the target block size, of a shared domain, i.e., the best block size for the shared domain to match the majority of client I / O. Furthermore, its analysis results can also indicate how to allocate the shared domain for a given F. Here, F can refer to a set of file systems that can serve various clients as a group.

[0063] The following describes a method embodiment of an adaptive matching method provided in this application. The method can be implemented by a device 1100 that carries out the embodiments of this disclosure. (Reference) Figure 1 :

[0064] Box 101, Create a shared domain. This shared domain is created on the first file system, and the block size of the first file system is the first block size.

[0065] In this embodiment, a shared domain, also known as a mounted domain, is located on the file system and serves as a unit for access by one or more clients. A shared domain can be accessed by zero or more clients; in other words, a shared domain can be mounted to one or more clients.

[0066] The first file system is a file system within a file system group. The file system group, referred to as F above, can refer to a set of file systems, including one or more file systems, which can serve various clients as a group. For example, in the example scenario 1 described later, the file system group is equivalent to serving file system 701, the first file system can be FS1 (702), and the second file system can be FS2 (703).

[0067] In one example, the shared domain can be shared domain 3 (706) in the example scenario 1 described later, the first file system can be file system FS1 (702) in the example scenario 1 described later, and the first block size can be the block size 8KB of file system FS1 (702) in the example scenario 1 described later.

[0068] In some embodiments, when creating a file system within a file system group, the block size of that file system can be specified. If the block size of the file system is not specified, it is created with the default block size, such as 8KB.

[0069] When a shared domain is created on the first file system, there exists a first jump link corresponding to the logical relationship between the shared domain and the first file system. For example, the jump link between shared domain 3 (706) and FS1 (702) in Example Scenario 1 described later.

[0070] Box 102, determines the target block size of the shared domain as the second block size, the second block size is not equal to the first block size.

[0071] The action of determining the target block size of a shared domain can be performed by a first analysis agent service, which can be the client IO analysis agent described later. As a service, it can be used to determine the target block size, or optimal block size, for each shared domain—that is, the best block size for the shared domain to match the vast majority of client IO. The client IO analysis agent continuously monitors the IO block size of one or more clients accessing each shared domain. Furthermore, the analysis results of the target block size for each shared domain by the client IO analysis agent can also indicate how to allocate the shared domain in F. The logic of the client IO analysis agent's operation can be found in the embodiments described later, and will not be repeated here.

[0072] The shared domain is used to support client access, and the second block size is determined based on the client access scenario. In one example, the second block size could be the IO target block size of shared domain 3 (706) in example scenario 1 described later, which is 32KB.

[0073] The determination of the target block size includes the following situations.

[0074] In some embodiments, a shared domain is accessed by one or more clients. The target block size can be determined based on the read / write / access (IO) block sizes of one or more clients. If the size of an IO block in the shared domain is greater than a first threshold, then that IO block size is determined to be the target block size. In one example, the first threshold could be 90%. For example, in Example Scenario 1 described later, the primary block size on shared domain 2 (705) is 8KB, therefore the target IO block size for shared domain 2 (705) is determined to be 8KB.

[0075] In some embodiments, a shared domain is accessed by one or more clients. The target block size can be determined based on the read / write / access (IO) block size of one or more clients. If the proportion of each IO block size in the shared domain is less than a first threshold, then the target block size is determined to be the default block size. In one example, the default block size is 8KB. For example, in Example Scenario 1 described later, shared domain 1 (704) is shared by multiple applications with different block sizes. The proportions of its block sizes of 32KB and 64KB do not meet the criteria for a target block (e.g., the required proportion is greater than 90%), and it is not suitable for migration. Therefore, the IO target block size of shared domain 1 (704) is determined to be the default block size of 8KB.

[0076] In some embodiments, if no client accesses the shared domain, the target block size of the shared domain is determined to be the default block size. For example, in Example Scenario 1 described later, no application is running on shared domain 4 (707), so the IO target block size of shared domain 4 (707) is determined to be the default block size of 8KB.

[0077] Box 103: If the block size of the second file system is the second block size, then the shared domain is migrated from the first file system to the second file system.

[0078] The second file system is one of the file systems in the file system group. In one example, the second file system could be file system FS2(703) as described in Example Scenario 1 below.

[0079] In some embodiments, before migrating a shared domain from a first file system to a second file system, it is confirmed that a second file system with a block size of the second block already exists in the file system group, and then the migration can be initiated. For example, in Example Scenario 1 described later, a shared domain 5 (708) with a target block size of 32KB is migrated from FS1 (702) with a block size of 8KB to an established FS2 (703) with a block size of 32KB.

[0080] In some embodiments, if a file system with a block size of the second block size is not detected in the file system group before migrating a shared domain from the first file system to the second file system, then a second file system with a block size of the second block size needs to be created in the file system group before performing the migration step. For example, in Example Scenario 1 described later, if a file system with a block size of 32KB is not detected in the service file system group 701, then a file system FS2 (703) with a block size of 32KB is created first, and then the shared domain 3 (706) with a target block size of 32KB is migrated from FS1 (702) with a block size of 8KB to FS2 (703) with a block size of 32KB.

[0081] While migrating the shared domain from the first file system to the second file system, the aforementioned first jump link is deleted, and a second jump link corresponding to the logical relationship between the shared domain and the second file system is generated. For example, in Example Scenario 1 described later, when migrating shared domain 3 (706) from file system FS1 (702) to file system FS2 (703), the jump link between shared domain 3 (706) and FS1 (702) is deleted, and a jump link between shared domain 3 (706) and FS2 (703) is generated.

[0082] For details regarding the specific implementation of the file system creation and inspection logic, the client IO analysis agent service mechanism, and the shared domain migration inspection logic, please refer to the descriptions in the embodiments described later; they will not be repeated here.

[0083] Another possible implementation is to create a second file system with a block size of the second block size before creating the shared domain. The user can then directly create the shared domain onto the second file system, thus performing the matching manually without requiring an automatic migration process. For example, in Example Scenario 1, when creating a new shared domain 5 (708) for a user, the system can provide one or more available options for the client's IO block size. If the user selects a 32KB block size when creating shared domain 5 (708), and a FS2 (703) with a block size of 32KB already exists, then shared domain 5 (708) can be directly created onto FS2 (703). In this case, since the 32KB block size matches the client IO analysis output, there is no need to migrate the shared domain.

[0084] Further exemplary embodiments of this application will now be described in conjunction with the accompanying drawings.

[0085] About the creation of DTTFS.

[0086] Creating a DTTFS is similar to creating a common transactional file system, but two rules need to be noted: First, a thinly configured transactional file system is being created; second, the user does not need to select which IO block size will run on this file system. The default value for the block size used to initialize E is 8KB. If the user sets a new value when creating the file system, the new value will replace the default value.

[0087] Regarding the structure of DTTFS.

[0088] refer to Figure 2 The DTTFS structure is shown.

[0089] In this context, Client 1 (H1), Client 2 (H2), Client 3 (H3), Client 4 (H4), Client 5 (H5), etc., shown in box 201 represent multiple different client I / Os.

[0090] Client I / O Analysis Agent 202 is a service that can be used to determine the optimal block size for each shared domain based on the client I / O size of each accessing client, and its analysis results can also indicate how to allocate the shared domain for F.

[0091] The distributor (203) is used to match shared domains with file systems. DTTFS includes one or more shared domains and one or more file systems. Different shared domains can be created on different file systems depending on the block size. The distributor can be used to migrate shared domains to suitable file systems, and also for allocating and managing storage space.

[0092] Matcher 203 includes functional modules such as data router 204 and space manager 205.

[0093] Data router 204 is used to provide cross-connection path entry points for different shared domains, associating different shared domains with appropriate file systems with corresponding block sizes.

[0094] Space Manager 205 is used to provide space query and / or space management functions when F works as a group, where the available space is the sum of the individual file systems in the F group.

[0095] A serving file system (FS) 206 refers to a group of file systems F that provide storage services, serving one or more clients as a group. It can contain multiple file systems, such as file system 1 (FS1), file system 2 (FS2), file system 3 (FS3), etc. Different file systems can have different block sizes and can be used to create one or more shared domains, providing appropriately sized storage space for applications with varying client I / O requirements.

[0096] The symbols used in the DTTFS creation algorithm are as follows.

[0097] F: A set of file systems that can serve as a group of service clients. F can contain one or more elements f, each f referring to a general transactional, streamlined configuration file system.

[0098] B: The existing supported block size in F is the set of block sizes used by one or more existing file systems f. When a new file system f is added to F, a block size is added at the same time.

[0099] Shared domain: Also known as a shared domain or mount domain, it allows you to designate a portion of the file system as a shared domain, which can be accessed by one or more clients.

[0100] E: The analysis result of the client IO analysis agent. It is a list used to record one or more client IO target block sizes in "KB" on the shared domain. Where e[n] is the target block size of shared domain n, representing the block size of most IOs in shared domain n. The initialized E is the default block size, for example, when the default block size is 8KB, E = [8].

[0101] M: M is the merged element in E, that is, filtering out duplicate values ​​and selecting unique values. If E = [8, 8, 16, 8, 16], then M = merge(E) = [8, 16].

[0102] The creation of a file system in DTTFS can be divided into two stages.

[0103] Phase 1:

[0104] refer to Figure 3A As shown, when there is only one default value in M, a generic transactional thin configuration file system FS1 will be created to serve one or more client I / Os. It's important to note that the client I / O analysis agent is not triggered during the first phase of initialization; it is only triggered when any shared domain is created, as shown below. Figure 3A As shown, when one or more shared domains are created to serve one or more client I / Os, the client I / O analysis agent continuously monitors the size of client I / O blocks accessing the shared domain.

[0105] Phase Two:

[0106] When the client IO analysis agent has new analysis output, a workflow is triggered to check whether a new file system should be created.

[0107] refer to Figure 3B , Figure 3B A flowchart of the creation and inspection logic of FS in the second stage is shown.

[0108] Box 301: The client IO analysis agent monitors the IO target block size of one or more shared domains and then outputs the analysis result E.

[0109] Box 302 calculates N = merge(E). M is the element to be merged in E, that is, filtering out duplicate values ​​and selecting unique values. For example, if E = [8, 8, 16, 8, 16], then M = merge(E) = [8, 16].

[0110] Box 303 determines whether the target block size s of the new shared domain, which is not in B, is in M. If not, that is, if there is no value s in M, then execute box 304; if yes, that is, if there is a value s in M, then exit this process and execute the subsequent process.

[0111] Box 304: Create a new file system f with block size s, add s to B, and add f to F.

[0112] The following describes the service mechanism of the client-side I / O analysis proxy.

[0113] The Client I / O Analysis Agent is a background service. It is triggered when any shared domain is created, and then continuously monitors the client I / O block size accessing the shared domain until a stopping condition is met.

[0114] The following algorithm describes how the client-side I / O analysis agent works.

[0115] The symbols used in the client-side I / O analysis proxy algorithm are as follows.

[0116] E: A list of IO target block sizes in "KB", where e[n] is the IO target block size of share domain n, representing the block size of most IOs on share domain n.

[0117] C: The transactional file system supports candidate block sizes in "KB".

[0118] I: Number of candidate block sizes, length of C.

[0119] NH: Used to record the number of client IOs of different block sizes, where nh[i] tracks the number of IOs (one or more) of block size c[i].

[0120] total ios When the client IO analysis agent is tracking the IO status of shared domain n, it records the total number of write IOs accessing shared domain n.

[0121] selector_threshold: The threshold for the number of IO operations that triggers the client IO analysis agent to output analysis results.

[0122] T: Time interval, the period during which the client IO analysis agent checks whether the stop condition is met.

[0123] WM: Watermark for stopping conditions. Generally, it can be 90%.

[0124] The main steps of the client IO analysis proxy algorithm are as follows.

[0125] Box 401: When a new shared domain n is created, initialize a client I / O analysis agent for that shared domain n.

[0126] For example, initialize the target block size of the shared domain n: e[n] = 8, the candidate block size: C = [8, 16, 32, 64], nh[i] = 0, where i ∈ {0, 1, ..., M}, total ios =0.

[0127] Box 402: Monitor the block size of client I / O and update records for one or more clients accessing shared domain n.

[0128] For write I / O operations, update total. ios Furthermore, if the block size of the accessed IO is C[i]KB, then increase nh[i].

[0129] Box 403: Every time T, check if NH meets the stopping condition. If it does, execute box 404; otherwise, execute box 402.

[0130] Box 404: Update the analysis result as the new e[n].

[0131] The following is a partial display of virtual code:

[0132]

[0133] Normally, if the second phase of creation is not triggered, there is only one transactional thin profile system in F, and DTTFS works normally like a regular transactional thin profile system, without anything special. This situation is called F operating in "single mode".

[0134] If the second phase of creation is triggered, F's mode will change to "group mode" because it has multiple file systems. Based on E, the shared domain with the corresponding block size will be migrated to the newly added file system. The matcher is triggered to provide data router and space manager functionality. Users will not be aware of these changes. After the new shared domain is created on the first file system FS1, it is continuously monitored by the client IO analysis agent to determine if migration is necessary.

[0135] The migration check logic is explained below (see reference). Figure 5 .

[0136] Box 501: The client IO analysis agent monitors the IO target block size of one or more shared domains and then outputs the analysis result E.

[0137] Box 502, update e[n] of shared domain n.

[0138] Box 503: Determine if the block size e[n] is in B. If not, execute box 505; if yes, execute box 504.

[0139] Box 504: Migrate shared domain n to file system f with block size s, then proceed with the subsequent process.

[0140] Box 505 waits for B to update, then returns to execution box 503.

[0141] Combination Figure 6 The examples illustrate the creation of file systems and the migration of shared domains. For example... Figure 6 As shown, shared domain 1 (604), shared domain 2 (605), shared domain 3 (606), and shared domain 4 (607) are created in the service file system group 601. It is assumed that the target block size of shared domain 1 (604), shared domain 2 (605), and shared domain 4 (607) is 8KB, and the target block size of shared domain 3 (606) is 32KB. One possible implementation is that when a user creates shared domain 1 (604), shared domain 2 (605), shared domain 3 (606), and shared domain 4 (607), shared domain 1 (604), shared domain 2 (605), shared domain 3 (606), and shared domain 4 (607) (default) are created on FS1 (602) with a block size of 8KB. Then, based on the creation check logic and migration check logic in the aforementioned embodiment, if the target block size of shared domain 3 (606) is detected to be 32KB, a suitable block size of FS2 (603) with a block size of 32KB is created, and shared domain 3 (606) is migrated to FS2 (603). Another possible implementation is that before creating shared domain 1 (604), shared domain 2 (605), shared domain 3 (606), and shared domain 4 (607), FS1 (602) with a block size of 8KB and FS2 (606) with a block size of 32KB have already been created. Then, when the user creates shared domain 1 (604), shared domain 2 (605), shared domain 3 (606), and shared domain 4 (607), shared domain 1 (604), shared domain 2 (605), and shared domain 4 (607) are created directly on FS1 (602) with a block size of 8KB, and shared domain 3 (606) is created directly on FS2 (603) with a block size of 32KB. In this way, the user manually performs the matching, and there is no need to perform the migration process.

[0142] The following example scenario is provided to better help understand the technical solution of this application.

[0143] The workflow of Example Scenario 1 includes the following description.

[0144] Create a transactional thin provisioning service file system with a capacity of 1TB. (701)

[0145] Create four shared domains to serve one or more client I / O operations: shared domain 1 (704), shared domain 2 (705), shared domain 3 (706), and shared domain 4 (707).

[0146] Client 1 (H1) (711) runs an application (block size 64KB) on shared domain 1 (704).

[0147] Client 2 (H2) (712) runs the application (block size is mainly 8KB) on shared domain 2 (705).

[0148] Client 3 (H3) (713) runs the application (block size is mainly 8KB) on shared domain 2 (705).

[0149] Client 4 (H4) (714) runs the application (block size is mainly 32KB) on shared domain 3 (706).

[0150] Client 5 (H5) (715) runs an application on shared domain 1 (704) with block sizes of 32KB and 64KB, in a 1:1 ratio.

[0151] Shared domain 4 (707) was created, but no applications are running on it.

[0152] This step allows users to create one or more shared domains as needed, and associate each client with each shared domain, allowing each client to access the corresponding shared domain.

[0153] Client I / O analysis agent 710 acts as a background service. In this scenario, client I / O analysis agent 710 monitors the size of one or more client I / O blocks in each shared domain, determines the target I / O block size for each shared domain, and then updates the result E. The algorithm principle is described above, where the target block size represents the block size of most I / Os in a shared domain.

[0154] In this scenario, the client-side IO analysis proxy algorithm can determine the target block size based on the IO block size of each shared domain. Specifically, according to H1(711) and H5(715), shared domain 1(704) is shared by multiple applications with different block sizes. The proportions of its 32KB and 64KB blocks do not meet the standard for a target block (requiring a proportion greater than 90%), making it unsuitable for migration. Therefore, the IO target block size for shared domain 1(704) is determined to be the default block size of 8KB. According to H2(712) and H3(713), the primary block size on shared domain 2(705) is 8KB, thus the IO target block size for shared domain 2(705) is determined to be 8KB. According to H4(714), the primary block size on shared domain 3(706) is 32KB, thus the IO target block size for shared domain 3(706) is determined to be 32KB. No application is running on shared domain 4 (707), therefore the target block size for IO in shared domain 4 (707) is determined to be the default block size of 8KB. After determining the target block size for each of the above shared domains, the result E is updated to [8,8,32,8].

[0155] The client IO analysis agent 710's action of updating E will trigger the creation check logic described in the second phase of FS creation. In this example scenario, the first phase has already created a file system FS1 (702) with a block size of 8KB. According to E, a new transactional thin profile system FS2 (703) with a block size of 32KB is then created in F, which can have a logical capacity of 1TB.

[0156] Create a file system FS2(703) root node to route client access to the shared domain.

[0157] refer to Figure 7 As shown, H1(711) corresponds to accessing shared domain 1(704), H2(712) corresponds to accessing shared domain 2(705), H3(713) corresponds to accessing shared domain 2(705), H4(714) corresponds to accessing shared domain 3(706), H5(715) corresponds to accessing shared domain 1(704), and shared domain 4(707) is not accessed by any client. Shared domain 1(704), shared domain 2(705), shared domain 3(706), and shared domain 4(707) are created on FS1(702).

[0158] After FS2(703) is created, the shared domain 3(706) originally on FS1(702) will be migrated to FS2(703). After the migration is completed, the original redirect link between shared domain 3(706) and FS1(702) will be deleted immediately, and the domain will be redirected to FS2(703). A redirect link will be generated to link shared domain 3(706) to FS2(703).

[0159] refer to Figure 8 As shown, the shared domain 3 (706) originally on FS1 (701) was migrated to FS2 (703).

[0160] A new shared domain 5 (708) is created to serve client 6 (H6)(716), which runs an application with 32KB block-size I / O. This triggers client I / O analysis agent 710 to monitor shared domain 5 (708) through these I / Os.

[0161] refer to Figure 9 As shown, H6(716) corresponds to accessing shared domain 5(708), and when shared domain 5(708) is created, it is created on FS1(702) by default.

[0162] The client IO analysis agent 710 updates E = [8,8,32,8,32]. This update triggers the creation check logic. The target block size of shared domain 5 (708) is determined to be 32KB, so shared domain 5 (708) is migrated to FS2 (703). After the migration is complete, the original jump link between shared domain 5 (708) and FS1 (702) is immediately deleted, and redirected to FS2 (703). A new jump link is generated linking shared domain 5 (708) to FS2 (703).

[0163] refer to Figure 10 As shown, the shared domain 5 (708) with a target block size of 32KB is migrated to FS2 (703) with a block size of 32KB, and a jump link is generated that directs the shared domain 5 (708) to FS2 (703).

[0164] It's important to note that in some implementations, when creating a new shared domain, the new domain can be created on the default file system, and then the system automatically checks and confirms that the new shared domain needs to be migrated to another file system. In other implementations, when creating a new shared domain, the user can manually select the client I / O block size, and then the new shared domain is directly created on a file system with the appropriate block size. In one example, when creating a new shared domain 5 (708) for a user, the system can provide one or more available options for the client I / O block size. In step 607, if the user selects a 32KB block size when creating shared domain 5 (708), and FS2 (703) with a block size of 32KB already exists, then shared domain 5 (708) can be directly created on FS2 (703). In this case, since the 32KB block size matches the client I / O analysis output, there is no need to migrate the shared domain.

[0165] Based on the above embodiments and in conjunction with the embodiments, it can be understood that implementing the adaptive matching method, related devices and systems provided in this application may include the following beneficial effects.

[0166] First, this application provides some analytical ideas for selecting an appropriate FS block size, which helps to reduce storage space fragmentation and indirect blocks, and further improves the performance of the storage system.

[0167] Second, one or more file systems in F can come from a storage system, a cluster, or even different storage systems in different clusters. This distributed nature helps to achieve load balancing because one or more IOs are actually distributed across different file systems.

[0168] Third, this application simplifies file system configuration settings. Users do not need to consider whether the transactional file system block size matches the application's I / O block size; the system can automatically match each client's I / O to a file system with an appropriate block size. Since users do not need to know the I / O mode of a specific application when creating a transactional file system, this application simplifies user operations and improves ease of use.

[0169] Figure 11 A schematic block diagram of a device 1100 that can be used to implement embodiments of the present disclosure is shown. Device 1100 can be the device or apparatus described in the embodiments of the present disclosure, such as a first device that can implement an adaptive matching method, etc. Figure 11 As shown, device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1102 or loaded from storage unit 1108 into random access memory (RAM) 1103. Various programs and data required for the operation of device 1100 can also be stored in RAM 1103. CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104. Although not shown in... Figure 11 As shown, device 1100 may also include a coprocessor.

[0170] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The various methods or processes described above can be executed by processing unit 1101. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more steps or actions in the methods or processes described above can be performed.

[0172] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0173] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0174] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media within the respective computing / processing device.

[0175] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0176] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0177] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0179] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for creating a shared domain to mitigate storage inefficiency, the method comprising: The shared domain is created by a system including a processor, wherein the shared domain is created on a first file system, wherein a first block size is defined as the minimum physical mapping block size that can be guaranteed in the first file system, and wherein creating the shared domain includes storing the data object of the shared domain in the first file system according to the first block size; Determine a second block size as the minimum physical mapping that can be guaranteed in the shared domain, wherein the second block size is not equal to the first block size, wherein the determination includes: In response to the total number of requests for the shared domain satisfying a first defined criterion, and determining that the proportion of requests with a read / write (IO) block size out of a plurality of read / write (IO) block sizes among the requests satisfies a second defined criterion, the second block size is set as the read / write (IO) block size; and In response to determining that the block size defining the minimum physical mapping that can be guaranteed in the second file system is the second block size, the shared domain is migrated from the first file system to the second file system, wherein migrating the shared domain includes storing the data object of the shared domain in the second file system according to the second block size.

2. The method of claim 1, wherein the first file system and the second file system are included in a file system group.

3. The method according to claim 2, further comprising: Before migrating the shared domain from the first file system to the second file system, it is determined that there exists a second file system in the file system group with a block size of the second block size.

4. The method according to claim 2, further comprising: Before migrating the shared domain from the first file system to the second file system, it is determined that there is no file system in the file system group with a block size of the second block size; as well as Create the second file system in the file system group.

5. The method according to claim 1, wherein the first definition criterion includes the total number of requests being greater than a first definition threshold, and the second definition criterion includes the percentage being greater than a second definition threshold.

6. The method of claim 1, wherein determining the block size of the shared domain as the second block size further comprises: In response to the total number of requests to the shared domain satisfying the first defined criterion, and determining that no percentage of the requests satisfy the second defined criterion, the second block size is set to the default block size.

7. The method of claim 1, wherein determining the block size of the shared domain as the second block size further comprises: In response to determining that the shared domain has not been accessed by any client within the defined time period, the second block size is set to the default block size.

8. The method according to claim 1, further comprising: A first jump link is created between the shared domain and the first file system.

9. The method according to claim 8, further comprising: After migrating the shared domain from the first file system to the second file system, delete the first redirect link.

10. The method of claim 8, further comprising: After migrating the shared domain from the first file system to the second file system, a second jump link is generated between the shared domain and the second file system.

11. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: Create a shared domain, wherein the shared domain is created on a first file system, wherein a first block size is defined as the minimum physical mapping block size that can be guaranteed in the first file system, wherein creating the shared domain includes storing the data object of the shared domain in the first file system according to the first block size; Determine a second block size as the minimum physical mapping that can be guaranteed in the shared domain, wherein the second block size is not equal to the first block size, wherein the determination includes: In response to the fact that the total number of requests for the shared domain meets a first defined criterion, and that the proportion of requests with multiple read / write (IO) block sizes among the requests meets a second defined criterion, the second block size is set as the read / write (IO) block size; In response to determining that the block size defining the minimum physical mapping that can be guaranteed in the second file system is the second block size, the shared domain is migrated from the first file system to the second file system, wherein migrating the shared domain includes storing the data object of the shared domain in the second file system according to the second block size.

12. The electronic device of claim 11, wherein the first file system and the second file system are included in a file system group.

13. The electronic device according to claim 12, further comprising: Before migrating the shared domain from the first file system to the second file system, it is determined that there exists a second file system in the file system group with a block size of the second block size.

14. The electronic device according to claim 12, wherein the action further includes: Before migrating the shared domain from the first file system to the second file system, it is determined that there is no file system in the file system group with a block size of the second block size; as well as Create the second file system in the file system group.

15. The electronic device of claim 11, wherein the first definition criterion includes the total number of the requests being greater than a first definition threshold, and the second definition criterion includes the percentage being greater than a second definition threshold.

16. The electronic device of claim 11, wherein determining the block size of the shared domain as the second block size further comprises: In response to the total number of requests to the shared domain satisfying the first defined criterion, and determining that no percentage of the requests satisfy the second defined criterion, the second block size is set to the default block size.

17. The electronic device of claim 11, wherein determining the block size of the shared domain as the second block size further comprises: In response to determining that the shared domain has not been accessed by any client within the defined time period, the second block size is set to the default block size.

18. The electronic device of claim 11, further comprising: A first jump link is created between the shared domain and the first file system.

19. The electronic device according to claim 18, wherein the action further comprises: After migrating the shared domain from the first file system to the second file system, delete the first redirect link.

20. The electronic device according to claim 18, wherein the action further comprises: After migrating the shared domain from the first file system to the second file system, a second jump link is generated between the shared domain and the second file system.

21. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 10.

Images