A multi-core extensible file system construction method and device

Abstract

The application provides a multi-core extensible file system construction method and device, wherein the method comprises the following steps: obtaining the access right of the file system by modifying the counter of the local CPU core; modifying the index of the file combination and the file cache of each file after obtaining the access right; and writing the memory cache data into the independent partition storage device in parallel to complete the file system expansion. The method improves the file system throughput, improves the parallelism of the file system storage data, and improves the ability of the file system performance expansion with the increase of the CPU core.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file system, and in particular to a multi-core extensible file system construction method. BACKGROUND

[0002] A file system is a storage system that organizes data in a tree structure. The tree structure of a file system mainly includes directories and files; the directories store information about the files and subdirectories under their sub-trees, and the files store actual user data. The file system mainly stores three types of data, namely file system metadata, file metadata and user data. The file system metadata records state information describing the entire file system, such as the space usage of the storage device managed by the file system; the file metadata includes the permissions, times and user data indexes of directories and files; and the user data refers to the data written by the user into the file. The file system organizes the three types of data of the file system in the computer memory through a shared index structure, and manages the space of the entire storage device through a shared data structure. The file system provides users with operation interfaces such as reading, writing and persisting files; at the same time, the file system also regularly performs checkpoint operations to permanently store the data temporarily stored in the memory to the storage device.

[0003] Modern computers are generally multi-core CPU architectures, that is, a computer is equipped with multiple complete computing units. A multi-core computer can run multiple different tasks at the same time, and can access the shared data of the file system at the same time. In order to ensure the correct operation of the system, the file system uses concurrent control technology to handle the conflicting access of different tasks to the same shared data. However, the concurrent control technology of the existing file system is inefficient; this leads to the fact that as the number of CPU cores of the computer increases, the throughput provided by the file system to the upper layer application does not increase, and may even decrease, that is, the multi-core of the file system is not scalable or has poor multi-core scalability. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, a first object of the present application is to provide a multi-core extensible file system construction method to improve the throughput of the file system, improve the parallelism of the file system in storing data, and improve the ability of the file system to expand performance with the increase of CPU cores.

[0006] A second object of the present application is to provide a multi-core extensible file system construction device.

[0007] To achieve the above object, a method according to an embodiment of the first aspect of the present application is provided, which comprises:

[0008] S1, obtaining the permission to access the file system by modifying the counter of the local CPU core;

[0009] S2, modifying the index of the file combination and the file cache where each file is located after obtaining the permission;

[0010] S3, and writing the memory cache data into the independent partitioned storage device in parallel to complete the file system expansion.

[0011] The multi-core scalable file system construction method of the embodiment of the application obtains the permission to access the file system by modifying the counter of the local CPU core, modifies the index of the file combination and the file cache where each file is located after obtaining the permission, and writes the memory cache data into the independent partitioned storage device in parallel to complete the file system expansion. By modifying the counter of the local CPU core, the competition of the file modification operation on the permission to access the file system is avoided, by dividing the file system memory cache in the file granularity, the conflict of the file operation competition to access the memory cache is reduced, by dividing the storage device into independent partitions and parallelizing the process of storing the file metadata and data into the storage device, the file system throughput is improved, the parallelism of the file system storage data is improved, and the ability of the file system to expand the performance with the increase of the CPU core is improved.

[0012] To achieve the above purpose, the second aspect embodiment of the application provides a multi-core scalable file system construction device, comprising:

[0013] The permission obtaining module is configured to obtain the permission to access the file system by modifying the counter of the local CPU core;

[0014] The execution module is configured to modify the index of the file combination and the file cache where each file is located after obtaining the permission;

[0015] The writing module is configured to write the memory cache data into the independent partitioned storage device in parallel to complete the file system expansion.

[0016] The multi-core scalable file system construction device of the embodiment of the application obtains the access right of the file system by modifying the counter of the local CPU core, modifies the index of the file combination where each file is located and the file cache after obtaining the right, and writes the memory cache data into the independent partition storage device in parallel to complete the file system expansion. By modifying the counter of the local CPU core, the competition of the file modification operation on the access right of the file system is avoided, by cutting the file system memory cache in the file granularity, the conflict of the file operation competition access memory cache is reduced, by cutting the storage device into independent partitions, the process of storing the file metadata and data into the storage device is parallelized, the file system throughput is improved, the parallelism of the file system storage data is improved, and the ability of the file system performance expansion with the increase of the CPU core is improved.

[0017] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and / or additional aspects and advantages of the application will become apparent and be readily understood by considering the following detailed description, including the accompanying drawings, in which:

[0019] Figure 1 The flow chart of the multi-core scalable file system construction method according to one embodiment of the application.

[0020] Figure 2 The schematic diagram of the concurrency control technology of the multi-core scalable file system construction method according to one embodiment of the application.

[0021] Figure 3 The schematic diagram of the file system data cutting of the multi-core scalable file system construction method according to one embodiment of the application.

[0022] Figure 4 The schematic diagram of the storage device partition of the multi-core scalable file system construction method according to one embodiment of the application.

[0023] Figure 5 The schematic diagram of the file system crash recovery of the multi-core scalable file system construction method according to one embodiment of the application.

[0024] Figure 6 The structural schematic diagram of the multi-core scalable file system construction device according to one embodiment of the application. DETAILED DESCRIPTION

[0025] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout the drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0026] A multi-core scalable file system construction method and device according to an embodiment of the present application is described below with reference to the accompanying drawings.

[0027] Modern computers are generally multi-core CPU architecture, that is, a computer is equipped with multiple complete computing units. Multi-core computers can run multiple different tasks simultaneously and can simultaneously access shared data of a file system. In order to ensure correct operation of the system, the file system uses concurrent control technology to handle conflicting access of different tasks to the same shared data. However, the concurrent control technology of the existing file system is inefficient; this leads to the fact that as the number of CPU cores of a computer increases, the throughput provided by the file system to the upper layer application does not increase, and may even decrease, that is, the file system is not scalable or has poor scalability in multi-core.

[0028] The multi-core scalable file system construction method according to an embodiment of the present application avoids competition of file modification operations on access to file system permissions by modifying a counter of a local CPU core, reduces conflict of competing access to a memory cache of the file system by file granularity division of the memory cache of the file system, improves file system throughput by parallelizing processes of storing file metadata and data to a storage device by dividing the storage device into independent partitions, improves parallelism of storing data of the file system, and improves the ability of the file system to expand performance as the number of CPU cores increases.

[0029] Figure 1 A flowchart of a multi-core scalable file system construction method according to an embodiment of the present application is shown in FIG. 1.

[0030] As shown in FIG. 1, the multi-core scalable file system construction method includes the following steps. Figure 1

[0031] In step S1, a counter of a local CPU core is modified to obtain permission to access a file system.

[0032] Specifically, a file modification operation obtains permission to access a file system by modifying a counter of a local CPU (computer central processing unit) core; and a file system internal checkpoint operation depends on a CPU scheduler to check that counters of all CPU cores are zeroed to obtain permission to access the file system.

[0033] ​Further, in one embodiment of the present application, the file modification operation includes modifying a file, creating a file, deleting a file, creating a directory, deleting a directory, renaming a file, and renaming a directory.

[0034] The following will be described in detail in combination with Figure 2 The above-mentioned file modification operation will be described in detail. Figure 2 The schematic diagram of the concurrent control technology of the multi-core scalable file system construction method provided in the embodiments of the present application.

[0035] It can be understood that the operation of modifying the file system is divided into two categories, one is that the user issues a file modification operation, such as a write file operation, and the other is a checkpoint operation inside the file system. Figure 2 As shown in the figure, the file system maintains a counter on each core. When a write request arrives, the write operation needs to add 1 to the counter to obtain the permission to access the file system. Multiple write operations can simultaneously modify the counter on their own core in parallel. When a checkpoint operation is triggered, it needs to check the counter on all cores. Only when the counter on all cores is zero, the file system can grant the checkpoint operation permission. During the execution of the check process, the file system uses the CPU scheduler to check when scheduling user tasks to a specific core.

[0036] Further, the file modification operation refers to an operation that needs to update the file system metadata or user data.

[0037] Further, the counter of the local CPU core includes the number of unfinished file modification operations on the current CPU core.

[0038] Further, the permission to access the file system refers to the ability of exclusive reading and modifying a single or multiple files granted by the file system.

[0039] Further, the checkpoint operation inside the file system includes that the file system writes all updates to the file system to the storage device.

[0040] Step S2, after obtaining the permission, modify the index and file cache of each file combination where the file is located.

[0041] It can be understood that the counter of the local CPU core is added by one before the execution of the file modification operation, and the counter of the local CPU core is reduced by one after the completion of the file modification operation.

[0042] Specifically, the file system divides the memory cache into files, and combines the divided memory cache into multiple file combinations; each file combination is organized by a separate index; when executing the file modification operation, only the index and file cache of the file combination where the file is located are modified.

[0043] Further, the memory cache data includes a file system metadata, a file metadata and a user data temporary copy in the memory.

[0044] The following will be described in detail Figure 3 The memory cache mentioned above is cut into files. Figure 3 A file system data cutting schematic diagram of the multi-core scalable file system construction method provided by the embodiment of the present application.

[0045] It can be understood that one embodiment of the present application cuts and recombines the memory cache into files, as Figure 3 shown. Multiple files are combined into a file combination, and all files in the file system are divided into multiple file combinations. A file combination uses a tree index to organize all files in the combination. The leaf node of the tree structure stores a file cache, including a file system metadata, a file metadata and user data related to the file. For a file modification operation, the modification operation is first located to a specific file combination through a hash algorithm (such as a modulo operation of the unique identifier of the file and the total number of file combinations). Since the file modification operation does not change the tree index, it then reads a lock on the tree index. The file modification operation is finally located to a specific file cache through the tree index, and updates the file system metadata, the file metadata and the user data. In one embodiment of the present application, the steps of the file creation and deletion operation are similar to those of the file modification operation, except that a write lock is required on the tree index, because an additional leaf node is added or removed, which may change the tree index structure.

[0046] In step S3, the memory cache data is written into the independently partitioned storage device in parallel, and the file system expansion is completed.

[0047] Specifically, the file system divides the storage device into several independent partitions; the file modification operation selects an arbitrary idle independent partition, and writes data into the storage device in parallel.

[0048] It can be understood that the independent partition includes a part of storage space with non-intersecting address space; the idle independent partition includes an independent partition without data written at the current time.

[0049] The following will be described in detail Figure 4 The above-mentioned data writing into the storage device in parallel is described in detail. Figure 4 A storage device partitioning schematic diagram of the multi-core scalable file system construction method provided by the embodiment of the present application.

[0050] One embodiment of the present application cuts the storage device space into multiple subspaces, as Figure 4As shown in FIG. 1, the file system receives a write file 1 operation and allocates storage subspace 1 to the write operation. The metadata and data of file 1 are written into storage subspace 1. At the same time, the file system receives a create file 2 operation and allocates the remaining free storage subspace, such as subspace 2, to the create operation. The metadata and data of file 2, as well as the metadata and data of the parent directory of file 2, are written into subspace 2.

[0051] Further, when the file system crashes, the latest file metadata is obtained by comparing the version numbers of the file metadata in different independent partitions, and the file system is reconstructed.

[0052] It can be understood that, in an embodiment of the present application, the crash recovery includes sudden power-off of the computer or the file system running error and being unable to work; the version number refers to the sequence of file writing into the storage device; and the file with the larger version number is the latest file.

[0053] The following will be described in detail in combination with Figure 5 The system recovery mentioned above will be described in detail. Figure 5 The file system crash recovery of the multi-core scalable file system construction method provided by the embodiment of the present application is shown in FIG. 2.

[0054] It can be understood that, in an embodiment of the present application, the system crash is solved by comparing the version numbers of the files, and the data conflicts of the storage subspaces are solved, such as Figure 5 As shown in FIG. 1, the file system receives a write file 1 operation and allocates storage subspace 1 to the write operation. The metadata and data of file 1 are written into storage subspace 1. At the same time, the file system receives a create file 2 operation and allocates the remaining free storage subspace, such as subspace 2, to the create operation. The metadata and data of file 2, as well as the metadata and data of the parent directory of file 2, are written into subspace 2. Figure 5 , the version number 2 in subspace 2 is larger than the version number in subspace 1, and therefore the file in subspace 2 is the latest.

[0055] The multi-core scalable file system construction method proposed in this embodiment of the invention obtains access permissions to the file system by modifying the counter of the local CPU core. After obtaining permissions, it modifies the index and file cache of the file combination containing each file, and writes the memory cache data to the storage device of an independent partition in parallel, thus completing the file system expansion. By modifying the counter of the local CPU core, it avoids competition for file modification operations on file system access permissions. By dividing the file system memory cache at the file granularity, it reduces conflicts in file operation competition for access to the memory cache. By dividing the storage device into independent partitions and parallelizing the process of storing file metadata and data on the storage device, it improves the file system throughput, enhances the parallelism of file system data storage, and improves the ability of the file system to scale performance as the number of CPU cores increases.

[0056] Next, a multi-core scalable file system building apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0057] Figure 6 This is a schematic diagram of a multi-core scalable file system building apparatus according to an embodiment of the present invention.

[0058] like Figure 6 As shown, this multi-core scalable file system building apparatus includes:

[0059] Permission acquisition module 100, execution module 200, and writing module 300.

[0060] The permission acquisition module is used to obtain access to the file system by modifying the counter of the local CPU core;

[0061] The execution module is used to modify the index and file cache of the file group containing each file after obtaining permissions;

[0062] The write module is used to write memory cache data in parallel to the storage device of an independent partition, thereby completing the file system expansion.

[0063] Furthermore, this embodiment of the invention also includes: a system recovery module, used to obtain the latest file metadata and rebuild the file system by comparing the version numbers of each file's metadata in different independent partitions when the file system crashes and is being recovered.

[0064] It should be noted that the foregoing explanation of the embodiment of the multi-core scalable file system construction method also applies to the multi-core scalable file system construction apparatus of this embodiment, and will not be repeated here.

[0065] The multi-core scalable file system construction device provided by the embodiment of the present application obtains the access right of the file system by modifying the counter of the local CPU core, modifies the index of the file combination where each file is located and the file cache after obtaining the access right, and writes the memory cache data into the independent partition storage device in parallel to complete the file system expansion. In the present application, the competition of the file modification operation on the access right of the file system is avoided by modifying the counter of the local CPU core, the conflict of the file operation competition access memory cache is reduced by dividing the file system memory cache in the file granularity, the file system throughput is improved, the parallelism of the file system storage data is improved, and the ability of the file system performance expansion with the increase of the CPU core is improved by dividing the storage device into independent partitions and parallelizing the process of storing the file metadata and data into the storage device.

[0066] In addition, the terms "first", "second", "third", etc. are used herein only to describe various circumstances, and cannot be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0067] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the different embodiments or examples described in the present application and the features of the different embodiments or examples can be combined and modified by those skilled in the art without contradiction.

[0068] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-core scalable file system construction method, characterized by, The method comprises the following steps: Access permission of the file system is obtained by modifying a counter of a local CPU core; the operation of modifying the file system is divided into two categories, one is a file modification operation issued by a user, such as a file writing operation, and the other is a checkpoint operation in the file system; the file system maintains a counter on each core, when a write request arrives, the write operation needs to increase the counter by 1 to obtain access permission of the file system; multiple write operations modify the counter on their own cores simultaneously and in parallel; when a checkpoint operation is triggered, it needs to check the counters on all cores; only when the counters on all cores are zero, the file system can grant permission to the checkpoint operation; during the execution of the check, the file system uses the CPU scheduler to check when scheduling user tasks to specific cores; After obtaining the permission, the index and file cache of each file combination are modified; and Memory cache data is written into the storage device in the independent partition in parallel, and the file system expansion is completed; The modification of the index and file cache of each file combination comprises: The memory cache is cut into units of files, and the cut memory cache is combined into multiple file combinations; each file combination is organized by a separate index; The memory cache data comprises file system metadata, file metadata and user data temporarily stored in the memory.

2. The method of claim 1, wherein, Further comprising: During the crash recovery of the file system, the latest file metadata is obtained by comparing the version numbers of the file metadata in different independent partitions, and the file system is reconstructed.

3. The method of claim 1, wherein, The memory cache data is written into the storage device in parallel, comprising: The storage device is divided into several independent partitions, an arbitrary idle independent partition is selected, and the memory cache data is written into the storage device in parallel.

4. The method of claim 1, wherein, The counter of the local CPU core is increased by one before the file modification operation, and the counter of the local CPU core is decreased by one after the file modification operation.

5. The method of claim 4, wherein, The file modification operation refers to an operation that needs to update the file system metadata or user data.

6. The method of claim 1, wherein, The independent partition comprises a part of storage space with non-intersecting address space.

7. The method of claim 1, wherein, The counter of the local CPU core comprises the number of unfinished file modification operations on the current CPU core.

8. A multi-core scalable file system construction apparatus using the method of claim 1, wherein, Comprise: A permission obtaining module is configured to obtain access permission of the file system by modifying a counter of a local CPU core; An execution module is configured to modify the index and file cache of each file combination after obtaining the permission; A writing module is configured to write memory cache data into the storage device in the independent partition in parallel, and complete the file system expansion.

Images