File system, operating system, and computing device

Abstract

A file system, comprising: a plurality of first file systems, wherein a plurality of first file systems are mounted in different partitions of a same storage medium or are mounted on different storage media; and a second file system, mounted on the plurality of first file systems and used for carrying out fault-tolerant verification on consistency data in memory metadata of the plurality of first file systems across the plurality of first file systems. In this way, one second file system is overlaid on the original first file systems, so that fault-tolerant verification can be carried out on the consistency data in the memory metadata of the first file systems across the first file systems, without requiring redundant backup of these consistency data in the first file systems, thereby reducing the memory overhead, and handling a memory fault with low memory overhead.

Description

File system, operating system and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on February 7, 2024, with application number 202410175776.0 and application name “A file system, operating system and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of information technology (IT), and in particular to a file system, an operating system, and a computing device. Background Art

[0003] The software structure responsible for managing and storing file data in a computer operating system (OS) can be called a file system. The file system can define the structure of file data and directories and provide interfaces for accessing, modifying, and managing these file data. The reliability of the file system directly affects the reliability of the OS, which in turn directly affects the reliability of the devices equipped with the OS. Currently, in order to improve the reliability of the file system, fault tolerance is often provided for stored file data to solve the problem of file system failures caused by storage failures. Although this method can improve the reliability of the file system to a certain extent, it lacks the fault tolerance capability for the file system metadata stored in the memory managed by the OS. This results in serious consequences such as file system unavailability and file / directory loss when the file system metadata stored in the memory fails.

[0004] Summary of the Invention

[0005] The present application provides a file system, an operating system, a memory management method, a memory management device, a computing device, a computer storage medium, and a computer product, which can cover memory failures with low memory overhead and improve the reliability of the file system.

[0006] In a first aspect, the present application provides a file system comprising: multiple first file systems, the multiple first file systems being mounted on different partitions of the same storage medium or mounted on different storage media; a second file system being mounted on the multiple first file systems and being used to perform fault-tolerance verification on consistency data in memory metadata of the multiple first file systems between the multiple first file systems.

[0007] In this way, by superimposing a second file system on the original first file system, fault-tolerant verification of the consistency data in the memory metadata of the first file system can be achieved between the first file systems without the need for redundant backup of these consistency data within the first file system, thereby reducing memory overhead and achieving memory failure coverage with low memory overhead.

[0008] In one possible implementation, the second file system is further configured to perform fault-tolerance verification on the file data of multiple first file systems. This allows for simultaneous coverage of memory and storage failures with low memory overhead, thereby improving the reliability of the file system.

[0009] In one possible implementation, the second file system is further configured to perform fault-tolerance verification within each of the first file systems for inconsistent data in the memory metadata of the multiple first file systems. This allows for fault-tolerance verification of all data in the memory metadata, further improving the reliability of the file system.

[0010] In a possible implementation, the file system further includes: a virtual file system, wherein the virtual file system is mounted on the second file system.

[0011] In one possible implementation, consistent data in the memory metadata of a first file system is stored in memory in a single copy, while inconsistent data in the memory metadata of a first file system is stored in memory in multiple copies. This eliminates the need for redundant backup of all data in the memory metadata of the first file system, reducing memory overhead.

[0012] In one possible implementation, consistent data in the memory metadata of a first file system is stored in memory in a single copy; inconsistent data in the memory metadata of a first file system is stored in memory in a single copy, and each inconsistent data is erasure coded. This eliminates the need for redundant backup of the memory metadata of the first file system, reducing memory overhead.

[0013] In one possible implementation, consistent data in the memory metadata of a first file system is stored in memory in a single copy; first sub-data included in inconsistent data in the memory metadata of the first file system is stored in memory in a single copy, with erasure coding added to the first sub-data, and second sub-data included in the inconsistent data is stored in memory in multiple copies. This eliminates the need for redundant backup of all data in the memory metadata of the first file system, reducing memory overhead.

[0014] In a possible implementation, the consistency data in the memory metadata includes: one or more of: a file name, access rights, and a group identifier; and the inconsistency data in the memory metadata includes: a timestamp and / or a super block pointer.

[0015] In a possible implementation, the mounting relationship between the first file system and the second file system is specified by the user, thereby improving the flexibility of mounting and enhancing the user experience.

[0016] In a possible implementation, the number of first file systems is specified by the user, thereby increasing the flexibility of mounting and improving the user experience.

[0017] In one possible implementation, both the consistent data and the inconsistent data in the memory metadata of the first file system are specified by the user, thereby increasing the flexibility of memory metadata redundancy backup and improving user experience.

[0018] In one possible implementation, the second file system is further configured to: determine, based on file attributes and / or operation types included in the read / write request, multi-mode information of the target file targeted by the read / write request, where the multi-mode information indicates the number of backup copies and the locations of the backup files of the target file; if the multi-mode information indicates that the target file has multiple backup copies, initiate the read / write operation to the first file system to which each backup file location belongs; if the multi-mode information indicates that the target file has a single backup copy, initiate the read / write operation to the first file system to which the backup file location belongs. In this way, the data read / write operation is completed.

[0019] In a possible implementation, the file attribute includes one or more of a user identifier, an organization identifier, a process identifier, a file name, and a file type.

[0020] In a second aspect, the present application provides an operating system, on which a file system as described in the first aspect or any possible implementation of the first aspect can be mounted.

[0021] In a third aspect, the present application provides a computing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein a file system as described in the first aspect or any possible implementation of the first aspect is deployed on the processor, and the processor executes the computer program to implement the operation of the file system.

[0022] In a fourth aspect, the present application provides a memory management method, which is applied to an operating system with multiple file systems mounted thereon, wherein the multiple file systems are mounted on different partitions of the same storage medium or mounted on different storage media. The method comprises: reading the metadata of each file system from the storage medium or partition on which the multiple file systems are mounted; storing a copy of the first sub-data in the metadata of each file system in the memory, wherein the first sub-data is consistent data in the metadata of the multiple file systems; storing multiple copies of the second sub-data in the metadata of each file system in the memory, and / or adding erasure codes to the second sub-data and storing them in the memory, wherein the second sub-data is inconsistent data in the metadata of the multiple file systems.

[0023] In one possible implementation, the method further includes: in response to the acquired file read operation, reading the target file from at least two file systems among the multiple file systems, and reading the metadata of the at least two file systems from the memory; performing fault-tolerance check on the read target file through consistency arbitration; performing fault-tolerance check on the read first sub-data through consistency arbitration, and, for any one of the multiple file systems, performing fault-tolerance check on the read second sub-data of any one of the multiple file systems through consistency arbitration or erasure code error detection.

[0024] In a possible implementation, the first sub-data includes: one or more of: a file name, access rights, and a group identifier; and the second sub-data includes: a timestamp and / or a super block pointer.

[0025] In a fifth aspect, the present application provides a memory management device, which is deployed on an operating system with multiple file systems mounted thereon, wherein the multiple file systems are mounted on different partitions of the same storage medium or mounted on different storage media. The device includes: a reading module for reading the metadata of each file system from the storage medium or partition on which the multiple file systems are mounted; a processing module for storing a copy of the first sub-data in the metadata of each file system in the memory, wherein the first sub-data is consistent data in the metadata of the multiple file systems; the processing module is also used to store multiple copies of the second sub-data in the metadata of each file system in the memory, and / or, to add erasure codes to the second sub-data and store them in the memory, wherein the second sub-data is inconsistent data in the metadata of the multiple file systems.

[0026] In one possible implementation, the reading module is further configured to, in response to an acquired file read operation, read a target file from at least two of the multiple file systems, and to read metadata of the at least two file systems from a memory. The processing module is further configured to perform a fault-tolerance check on the read target file by means of consistency arbitration. The processing module is further configured to perform a fault-tolerance check on the read first sub-data by means of consistency arbitration, and to perform a fault-tolerance check on the read second sub-data of any one of the multiple file systems by means of consistency arbitration or erasure code error detection.

[0027] In a possible implementation, the first sub-data includes: one or more of: a file name, access rights, and a group identifier; and the second sub-data includes: a timestamp and / or a super block pointer.

[0028] In a sixth aspect, the present application provides a computing device comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in the fourth aspect or any possible implementation of the fourth aspect.

[0029] In a seventh aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the fourth aspect or any possible implementation of the fourth aspect. For example, the computing device cluster may include one or more computing devices.

[0030] In an eighth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the fourth aspect or any possible implementation of the fourth aspect. For example, the computing device cluster may include one or more computing devices.

[0031] It can be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of an operating environment of an OS provided in an embodiment of the present application;

[0033] FIG2 is a schematic diagram of data in different partitions and metadata of these data in memory provided by an embodiment of the present application;

[0034] FIG3 is a schematic diagram of a data writing process provided by an embodiment of the present application;

[0035] FIG4 is a schematic diagram of a data reading process provided by an embodiment of the present application;

[0036] FIG5 is a flow chart of a memory management method provided in an embodiment of the present application;

[0037] FIG6 is a schematic structural diagram of a memory management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0039] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0040] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0041] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0042] Exemplarily, FIG1 shows a schematic diagram of an operating environment of an OS provided in an embodiment of the present application. As shown in FIG1 (A), the operating environment of the OS includes: an application (APP), multiple file systems (FS), a high-reliability multi-mode file system (N modular redundancy file system, NMRFS) and a storage medium (such as a hard disk, partition, network storage, etc.). Among them, during the operation of the OS, multiple FSs can be mounted to different storage media / or different partitions of the same storage medium, and NMRFS can be mounted to multiple FSs. The number of FSs having a mounting relationship with NMRFS can be, but is not limited to, specified by the user. For example, the user can specify the number of FSs in the form of a configuration file or a command line. In addition, the mounting of FSs to storage media, and the mounting of NMRFS to multiple FSs, can be achieved by automatic mounting or manual mounting. For example, on a Linux system, you can use the mount command "mount –t yaffs / dev / mtdblock1 / tmp / yaffs1" to mount the yaffs1 file system onto the mtdblock1 partition, and use the mount command "mount –t yaffs / dev / mtdblock2 / tmp / yaffs2" to mount the yaffs2 file system onto the mtdblock2 partition. You can then use the mount command "mount –t nmrfs -odir1= / tmp / yaffs1,dir2= / tmp / yaffs2none / tmp / nmrfs" to mount NMRFS onto both the yaffs1 and yaffs2 file systems. After NMRFS is mounted, subsequent read and write operations only require access to / tmp / nmrfs, and NMRFS will automatically distribute the read and write operations to / tmp / yaffs1 and / tmp / yaffs2. When the data in the file a.txt is required to be read, the file path in the read request can be / tmp / nmrfs / a.txt. After that, NMRFS will forward the request to read a.txt to yaffs1 and yaffs2 respectively. Optionally, as shown in (B) of Figure 1, the operating environment of the OS can also include a virtual file system (VFS). The VFS can be in kernel state and located between the APP and the NMRFS, that is, the APP communicates with the NMRFS through the VFS. Among them, the VFS can be mounted on the NMRFS. In some embodiments, multiple FSs and NMRFSs can form a new FS, and the new FS can be mounted to the OS. Of course, multiple FSs, NMRFSs and VFSs can also form a new FS, and the FS can also be mounted to the OS.

[0043] In this embodiment, an APP is a collection of instructions and codes written by a user or other software entity that can perform specific tasks, provide specific functions, or solve specific problems. For example, an APP can initiate a read / write request to NMRFS. The read / write request can include a file path.

[0044] FS is primarily used to organize and manage files in a computer. It can provide functions such as creating files and directories, reading and writing files, deleting files and directories, searching for files, and protecting files. Exemplary FSs can be Yet Another Flash File System (YAFFS), Internet Flash File System (IFFS), Hybrid Flash File System (HFFS), or Temporary File System (TMPFS).

[0045] NMRFS can be used to determine the multi-mode information of the file targeted by the read / write request after the APP initiates the read / write request. The multi-mode information is used to indicate the number of backup copies of the file and the location of the backup file. After determining the multi-mode information of the file targeted by the read / write request, NMRFS can initiate a read / write operation to at least one FS based on the determined multi-mode information. Specifically, when the multi-mode information indicates that the number of backup copies of the file is a single copy, NMRFS can initiate a read / write operation to one FS; when the multi-mode information indicates that the number of backup copies of the file is multiple copies, NMRFS can initiate a read / write operation to multiple FSs. In some embodiments, the user can specify the data that needs to be redundantly backed up. Specifically, the user can specify the data that needs to be redundantly backed up in the form of a configuration file or a command line. For example, the user can specify that all data processed by a process is redundantly backed up, or specify that all data in a certain file is redundantly backed up, etc. In addition, the number of redundant backups of data can also be specified by the user.

[0046] In addition, NMRFS can also be used to perform fault-tolerance verification on file data in multiple FSs (e.g., consistency arbitration, etc.). NMRFS can also be used to perform fault-tolerance verification on consistency data in the memory metadata of multiple FSs between FSs (e.g., consistency arbitration, etc.). NMRFS can also be used to perform fault-tolerance verification on inconsistent data in the memory metadata of multiple FSs within each FS (e.g., consistency arbitration or erasure code error detection, etc.). In addition, NMRFS can also be used to perform fault-tolerance verification on two or three of the file data in multiple FSs, the consistency data in the memory metadata of multiple FSs, and the inconsistent data in the memory metadata of multiple FSs. This achieves fault tolerance for file data and / or FS memory metadata, thereby covering memory failures and storage failures. By performing fault-tolerance verification on the consistency data in the memory metadata of multiple FSs between FSs, the consistency data in the memory metadata of each FS only needs to be stored in memory in one copy, thereby reducing memory overhead. At the same time, erasure codes can be added to the inconsistent data in the FS's memory metadata. Because erasure codes occupy less space, using erasure codes to detect inconsistent data in the FS's memory metadata can also reduce memory overhead. When using consistency arbitration to detect inconsistent data in the FS's memory metadata, multiple copies of the inconsistent data in the FS's memory metadata can be stored in memory for each FS. This approach consumes higher memory overhead than erasure code error detection, but because only one copy of the consistent data in the FS's memory metadata needs to be stored in memory for each FS, this fault-tolerance verification approach in this embodiment can also reduce memory overhead compared to storing multiple copies of all memory metadata. Exemplarily, the consistent data in the memory metadata of multiple FSs may include one or more of the following: file name, access rights, group ID, etc. The inconsistent data in the memory metadata of multiple FSs may include timestamps and / or superblock pointers. In some embodiments, users can also specify the sub-data included in the memory metadata to be included in the fault-tolerance verification through a configuration file or command line.

[0047] To facilitate understanding, the following example is used to illustrate.

[0048] For example, as shown in Figure 2 (A), the memory metadata of FS1, FS2, and FS3 are backed up in memory separately. However, the memory metadata of different FSs are not aware of each other. When performing fault-tolerance verification on the memory metadata of FSs, consistency arbitration is only performed within the FS, and fault-tolerance verification cannot be performed across FSs. In Figure 2 (A), it is equivalent to directly superimposing 3-mode memory (i.e., storing three copies of the same data) on top of 3-mode storage (i.e., storing three copies of the metadata for each FS), resulting in a 3x3 memory overhead.

[0049] FIG2(B) shows that the inconsistent data (such as i_atime and i_sb) in the memory metadata of FS1, FS2 and FS3 are backed up in memory respectively, while the consistent data (such as i_mode, i_uqid and i_flags) in the memory metadata of FS1, FS2 and FS3 are no longer backed up. In FIG2(B), the consistent data in the memory metadata of FS1, FS2 and FS3 are mutually aware of each other. Therefore, NMRFS can perform fault-tolerance verification on the consistent data in the memory metadata of these FSs between different FSs through consistency arbitration. In FIG2(B), the inconsistent data in the memory metadata of FS1, FS2 and FS3 are not mutually aware of each other. Therefore, for any FS, NMRFS can perform fault-tolerance verification on the inconsistent data in the memory metadata of the FS within the FS through consistency arbitration.

[0050] FIG2(C) shows that erasure codes are added to the inconsistent data (such as i_atime and i_sb) in the memory metadata of FS1, FS2, and FS3 in memory, and these inconsistent data are no longer backed up. At the same time, the consistent data (such as i_mode, i_uqid, and i_flags) in the memory metadata of FS1, FS2, and FS3 are not backed up. In FIG2(C), the consistent data in the memory metadata of FS1, FS2, and FS3 are mutually aware of each other. Therefore, NMRFS can perform fault-tolerance verification on the consistent data in the memory metadata of these FSs between different FSs through consistency arbitration. In FIG2(C), the inconsistent data in the memory metadata of FS1, FS2, and FS3 are not mutually aware of each other. Therefore, for any FS, NMRFS can perform fault-tolerance verification on the inconsistent data in the memory metadata of the FS within the FS through erasure code error detection.

[0051] Comparing Figures 2(A), 2(B), and 2(C) shows that the fault-tolerant verification method for the in-memory FS metadata shown in Figure 2(A) requires a large memory overhead. The fault-tolerant verification method for the in-memory FS metadata shown in Figure 2(B) can reduce some memory overhead compared to Figure 2(A). The fault-tolerant verification method for the in-memory FS metadata shown in Figure 2(C) can further reduce some memory overhead compared to Figure 2(B).

[0052] The above is an introduction to the operating environment of the OS provided in the embodiment of the present application. The following describes the data reading / writing process in the operating environment of the OS.

[0053] For example, Figure 3 shows a schematic diagram of a data writing process provided by an embodiment of the present application. As shown in Figure 3, the data writing process may include the following steps:

[0054] S301: APP sends a write request to NMRFS. Exemplarily, the write request may include: file attributes and / or operation type of the file targeted by the write request. In some embodiments, APP may send the write request to NMRFS via VFS, but is not limited to VFS.

[0055] S302: NMRFS determines the multi-mode information of the file targeted by the write request based on the file attributes and / or operation type included in the write request. NMRFS may query a preconfigured redundancy range based on the file attributes and / or operation type included in the write request to determine whether the file targeted by the write request requires redundant backup. If redundant backup is required, NMRFS may query the NMRFS metadata and / or the FS metadata using, but not limited to, metadata of the file targeted by the write request to obtain the multi-mode information of the file. This multi-mode information may be used to indicate the number of backup copies and the location of the backup files. If redundant backup is not required, NMRFS may query the file path included in the write request to obtain the desired write location, i.e., the backup file location. At this point, the multi-mode information of the file is obtained, i.e., the number of backup copies indicated by the multi-mode information is one, and the backup file location is determined based on the file path. Exemplarily, file attributes may include one or more of: user identity (UID), group identity (GID), process identity (PID), file name, and file type. Exemplarily, the redundancy range may be, but not limited to, specified by the user. For example, a user can specify that the number of backups for a file named "foo" is three. In some embodiments, NMRFS can also parse the file path contained in the write request to obtain the directory path and file name. NMRFS can then search the mount table based on the starting point of the directory path to determine the FS to which the file corresponding to the file path belongs. If a file belongs to multiple FSs, it indicates that the file needs to be backed up multiple times, and the FS to which the file belongs is the backup file location.

[0056] S303: NMRFS determines whether the number of backups of the file indicated by the determined multi-mode information of the file is multiple. If not, that is, if it is a single copy, S304-S305 can be executed; if it is multiple copies, S306-S307 can be executed.

[0057] S304. When the number of backup copies of the file indicated by the multi-mode information is a single copy, NMRFS initiates a write operation to the FS associated with the backup file location in the multi-mode information. Exemplarily, the write operation initiated by NMRFS may include: the data content to be written and the data write location. Exemplarily, after NMRFS parses the file path included in the write request to obtain the directory path, it can check the mount table based on the starting point of the directory path to find out the FS to which the file corresponding to the file name included in the file path belongs. The mount table may record information about each FS mounted by NMRFS. For ease of description, the following description will be based on NMRFS initiating a write operation to FS1.

[0058] S305 , after receiving the write operation initiated by NMRFS, FS1 may write data to the mounted partition 1 and update its memory metadata in the memory.

[0059] S306: When the file indicated by the multi-mode information has multiple backup copies, NMRFS initiates write operations to multiple FSs. Alternatively, NMRFS may first perform erasure coding, encryption, or other processing on the data to be written before initiating the write operation. Furthermore, when initiating the write operation, NMRFS may optimize the data write process through asynchronous writes and load balancing. For ease of description, the following description assumes that NMRFS initiates write operations to FS1, FS2, and FS3, respectively.

[0060] S307. After receiving the write operation initiated by NMRFS, FS1 can write the data to be written to its mounted partition 1 according to the data write location in the write operation, and update its in-memory metadata 1. After receiving the write operation initiated by NMRFS, FS2 can write data to its mounted partition 2 and update its in-memory metadata 2. After receiving the write operation initiated by NMRFS, FS3 can write data to its mounted partition 3 and update its in-memory metadata 3. Exemplarily, when updating its in-memory metadata, each FS can store one copy of the consistent data in the memory metadata in memory, and multiple copies of the inconsistent data in the memory metadata in memory, or add erasure codes to the inconsistent data, or a combination of the two.

[0061] In this way, the data writing process is completed. It should be understood that FIG3 is an example of an OS running environment including three FSs. When the OS running environment includes other numbers of FSs, the data writing process can refer to the description in FIG3 and will not be repeated here.

[0062] For example, Figure 4 shows a schematic diagram of a data reading process provided by an embodiment of the present application. As shown in Figure 4, the data reading process may include the following steps:

[0063] S401: The APP sends a read request to the NMRFS. For example, the read request may include: file attributes and / or operation type of the file targeted by the read request. In some embodiments, the APP may send the read request to the NMRFS via, but is not limited to, the VFS.

[0064] S402: NMRFS determines the multi-mode information of the file targeted by the read request based on the file attributes and / or operation type included in the read request. This step can be referred to the description of S202 in FIG2 , and will not be described in detail here.

[0065] S403: NMRFS determines whether the number of backups of the file indicated by the determined multi-mode information of the file is multiple. If not, that is, if it is a single copy, S404-S407 can be executed; if it is multiple copies, S408-S412 can be executed.

[0066] S404. When the number of backup copies of the file indicated by the multi-mode information is a single copy, NMRFS initiates a read operation to one of the multiple FSs. Exemplarily, the read operation initiated by NMRFS may include: data reading location. Exemplarily, after NMRFS parses the file path included in the read request to obtain the directory path, it can check the mount table based on the starting point of the directory path to find out the FS to which the file corresponding to the file name included in the file path belongs. The mount table may record information about each FS mounted by NMRFS. For ease of description, the following description will be based on the case where NMRFS initiates a read operation to FS1.

[0067] S405 , after receiving the read operation initiated by NMRFS, FS1 can read the target data from the mounted partition 1 according to the data reading position in the read operation.

[0068] S406 , FS1 transmits the read target data to NMRFS.

[0069] S407. NMRFS returns the target data to the APP.

[0070] S408: When the number of backup copies of the file indicated by the multi-mode information is multiple, the NMRFS initiates a read operation to each of the multiple FSs. For ease of description, the following description assumes that the NMRFS initiates a read operation to FS1, FS2, and FS3, respectively.

[0071] S409. After FS1 receives the read operation initiated by NMRFS, it can read the target data 1 from the mounted partition 1 according to the data read position in the read operation, and update its memory metadata 1 in the memory. After FS2 receives the read operation initiated by NMRFS, it can read the target data 2 from the mounted partition 2 according to the data read position in the read operation, and update its memory metadata 2 in the memory. After FS3 receives the read operation initiated by NMRFS, it can read the target data 3 from the mounted partition 3 according to the data read position in the read operation, and update its memory metadata 3 in the memory. Exemplarily, when each FS updates its memory metadata in the memory, it can store one copy of the consistent data in the memory metadata in the memory, and multiple copies of the inconsistent data in the memory metadata in the memory, or add erasure codes to the inconsistent data, or a combination of the two.

[0072] S410, FS1 sends target data 1 and memory metadata 1 of target data 1 to NMRFS; FS2 sends target data 2 and memory metadata 2 of target data 2 to NMRFS; FS3 sends target data 3 and memory metadata 3 of target data 3 to NMRFS. Among them, for the consistent data in metadata 1, 2 and 3, each FS sends a copy to NMRFS. For the inconsistent data in metadata 1, 2 and 3, FS1 can send a copy to NMRFS and send an erasure code at the same time, FS2 can also send a copy to NMRFS and send an erasure code at the same time, and FS3 can also send a copy to NMRFS and send an erasure code at the same time. In addition, for the inconsistent data in metadata 1, 2 and 3, FS1 can send multiple copies to NMRFS, FS2 can also send multiple copies to NMRFS, and FS3 can also send multiple copies to NMRFS. In addition, for the inconsistent data in metadata 1, 2 and 3, FS1, 2 and 3 can also choose to send erasure codes or multiple copies of data according to actual conditions, which is not limited here.

[0073] S411: NMRFS performs fault-tolerance checks on the target data and metadata. NMRFS can perform consistency arbitration on target data 1, target data 2, and target data 3, as well as consistency arbitration on the consistent data in memory metadata 1, memory metadata 2, and memory metadata 3. Furthermore, NMRFS can perform erasure code error detection or consistency arbitration on inconsistent data in the memory metadata of each FS within each FS. For details, see (B) and (C) of Figure 2 above, which will not be repeated here.

[0074] S412 : When the fault-tolerance check indicates that the target data and the metadata are correct, the NMRFS returns any one of the target data 1 , the target data 2 , and the target data 3 to the APP.

[0075] In this way, the data reading process is completed.

[0076] As can be seen from the data writing and reading processes described above, in the OS operating environment shown in Figure 1, overlaying the aforementioned NMRFS on top of the existing FS can simultaneously achieve fault tolerance for both file data and the FS's memory metadata, thus covering both memory and storage failures without replacing or intrusively modifying the original FS. Furthermore, there's no need to redundantly back up all data in each FS's memory metadata in the OS-managed memory, which reduces memory overhead and achieves end-to-end file storage and memory reliability with minimal overhead.

[0077] It should be noted that before data is written or read, NMRFS needs to be mounted to FS. Among them, each FS can be mounted to different partitions of the same storage medium or to different storage media first, and then NMRFS can be mounted to each FS. Exemplarily, the mounting process of FS and / or NMRFS can be completed automatically by the system, or it can be completed by the user by entering the mount command in the form of a configuration file or command line. In addition, the user can also specify the path, number or target location of the FS required to be mounted by NMRFS in the form of a configuration file or command line. In this way, the user can configure the number and location of data redundancy backups by himself.

[0078] Furthermore, after completing the mounting of NMRFS and FS, users can also configure NMRFS, such as which data needs to be redundantly backed up, the number of backups of the data that needs redundant backup, which data needs fault-tolerance verification, etc. Among them, users can configure NMRFS through configuration files or command lines. In this way, users can flexibly configure different redundancy granularities and fault-tolerance behaviors of file data and FS memory metadata. Of course, users can also choose not to configure NMRFS and use the default configuration of NMRFS. The specific configuration can be determined according to actual conditions and is not limited here.

[0079] It is understandable that the FS memory metadata can be read into the memory when the FS is mounted on the OS, when the APP opens the file, during the file operation process, or when NMRFS parses the file path. The specific method depends on the actual situation and is not limited here.

[0080] The above is an introduction to the operating environment of the OS and the data read / write process provided by the embodiment of the present application. Next, based on the above content, a memory management method provided by the embodiment of the present application is introduced.

[0081] Exemplarily, FIG5 shows a flow chart of a memory management method provided by an embodiment of the present application. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Exemplarily, the memory management method can be applied to an operating system with multiple file systems mounted thereon, and the multiple file systems can be mounted on different partitions of the same storage medium or mounted on different storage media. As shown in FIG5 , the memory management method can include the following steps:

[0082] S501: Read metadata of each file system from storage media or partitions mounted with multiple file systems.

[0083] In this embodiment, when a file system is mounted to an operating system, an APP opens a file in a file system, or a file in a file system is operated, the metadata of the file system can be read from the storage medium or partition on which the file system is mounted.

[0084] S502: Store a copy of the first sub-data in the metadata of each file system in the memory, wherein the first sub-data is consistent data in the metadata of multiple file systems.

[0085] In this embodiment, after reading the metadata of a file system, a copy of the first sub-data in the metadata of that file system can be stored in memory. This first sub-data is consistent data in the metadata of multiple file systems, such as file names, access permissions, or group identifiers. In this way, a copy of the consistent data in the metadata of each file system is stored in memory.

[0086] S503: Storing multiple copies of the second sub-data in the metadata of each file system in the memory, and / or adding erasure codes to the second sub-data and storing them in the memory, wherein the second sub-data is inconsistent data in the metadata of multiple file systems.

[0087] In this embodiment, after reading metadata of a file system, multiple copies of the second sub-data in the metadata of the file system can be stored in memory, or a single copy can be stored in memory and an erasure code can be added to the second sub-data. The second sub-data is inconsistency data in the metadata of multiple file systems, such as a timestamp and / or a superblock pointer.

[0088] In this way, during the subsequent file reading process, the target file can be read from multiple file systems, and the metadata of these multiple file systems can be read from the memory respectively. Then, the read target file is checked for fault tolerance through consistency arbitration, and the read first sub-data is checked for fault tolerance through consistency arbitration. For any of the multiple file systems, the second sub-data of any file system is checked for fault tolerance through consistency arbitration or erasure code error detection. This achieves the simultaneous coverage of memory failures and storage failures, and reduces memory overhead.

[0089] It should be understood that the order of execution of the steps in the above embodiments does not necessarily imply a specific order of execution. The order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual circumstances, and the combined solutions are still within the scope of protection of this application.

[0090] Based on the contents in the above embodiments, an embodiment of the present application provides a memory management device.

[0091] Exemplarily, FIG6 shows a schematic structural diagram of a memory management device. The memory management device can be deployed in an operating system on which multiple file systems are mounted, and different file systems are mounted on different partitions of the same storage medium or mounted on different storage media, and different file systems back up each other. As shown in FIG6 , the memory management device 600 may include: a reading module 601 and a processing module 602. The reading module 601 is used to read the metadata of each file system from the storage medium or partition on which the multiple file systems are mounted. The processing module 602 is used to store a copy of the first sub-data in the metadata of each file system in the memory, wherein the first sub-data is the consistent data in the metadata of multiple file systems; and to store multiple copies of the second sub-data in the metadata of each file system in the memory, and / or to add erasure codes to the second sub-data and store them in the memory, wherein the second sub-data is the inconsistent data in the metadata of multiple file systems.

[0092] In some embodiments, the reading module 601 is further configured to, in response to the acquired file read operation, read the target file from each file system and read the metadata of each file system from the memory. The processing module 602 is further configured to perform a fault-tolerance check on the read target file through consistency arbitration; perform a fault-tolerance check on the read first sub-data through consistency arbitration; and perform a fault-tolerance check on the read second sub-data of any file system through consistency arbitration or erasure code error detection for any file system among the multiple file systems.

[0093] In some embodiments, the first sub-data includes: one or more of: a file name, an access permission, and a group identifier; the second sub-data includes: a timestamp and / or a super block pointer.

[0094] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0095] An embodiment of the present application also provides a computing device, which may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein a file system consisting of at least the FS and NMRFS described above is deployed on the processor, and the processor executes the computer program to implement the operation of the file system.

[0096] The present application also provides another computing device that may include one or more processors and a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of the above embodiment.

[0097] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive), etc.

[0098] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment.

[0099] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0100] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0102] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A file system, characterized in that: include: Multiple first file systems, where the multiple first file systems are mounted on different partitions of the same storage medium or mounted on different storage media; The second file system is mounted on the plurality of the first file systems and is used to perform fault-tolerance verification on consistency data in the memory metadata of the plurality of the first file systems among the plurality of the first file systems.

2. The file system according to claim 1, wherein: The second file system is further used to perform fault-tolerance verification on the file data of the plurality of first file systems.

3. The file system according to claim 1 or 2, characterized in that: The second file system is further configured to perform fault-tolerance verification on inconsistent data in the memory metadata of the plurality of first file systems within each of the first file systems.

4. The file system according to any one of claims 1 to 3, characterized in that: Also includes: A virtual file system, wherein the virtual file system is mounted on the second file system.

5. The file system according to any one of claims 1 to 4, characterized in that: A copy of the consistency data in the memory metadata of the first file system is stored in the memory; Inconsistent data in the memory metadata of the first file system is stored in multiple copies in the memory.

6. The file system according to any one of claims 1 to 5, characterized in that: A copy of the consistency data in the memory metadata of the first file system is stored in the memory; A copy of the inconsistency data in the memory metadata of the first file system is stored in the memory, and an erasure code is added to each of the inconsistency data.

7. The file system according to any one of claims 1 to 6, characterized in that: A copy of the consistency data in the memory metadata of the first file system is stored in the memory; A first sub-data included in the inconsistency data in the memory metadata of the first file system is stored in one copy in the memory, and an erasure code is added to the first sub-data, and a second sub-data included in the inconsistency data is stored in multiple copies in the memory.

8. The file system according to any one of claims 1 to 7, characterized in that: The consistency data in the memory metadata includes: one or more of the following: file name, access permission, and group identifier; The inconsistency data in the memory metadata includes: a timestamp and / or a super block pointer.

9. The file system according to any one of claims 1 to 8, characterized in that: The mounting relationship between the first file system and the second file system is specified by a user.

10. The file system according to any one of claims 1 to 9, characterized in that: The number of the first file systems is specified by a user.

11. The file system according to any one of claims 1 to 10, characterized in that: The consistency data and the inconsistency data in the memory metadata of the first file system are both specified by the user.

12. The file system according to any one of claims 1 to 11, characterized in that: The second file system is further used for: Determine, based on the file attributes and / or operation type included in the read / write request, the multi-mode information of the target file targeted by the read / write request, wherein the multi-mode information is used to indicate the number of backups and the location of the backup files of the target file; and search for the multi-mode information through the metadata of the file. In a case where the multi-mode information indicates that there are multiple backup copies of the target file, initiating a read / write operation to the first file system to which each backup file location belongs respectively; When the multi-mode information indicates that the number of backup copies of the target file is single, a read / write operation is initiated to the first file system to which the backup file location belongs.

13. The file system according to claim 12, wherein: The file attributes include: one or more of a user identifier, an organization identifier, a process identifier, a file name, and a file type.

14. An operating system, characterized in that: A file system according to any one of claims 1 to 13 is mounted.

15. A computing device, characterized in that The system comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein a file system according to any one of claims 1 to 13 is deployed on the processor, and the processor implements the operation of the file system when executing the computer program.

16. A memory management method, characterized in that: Applicable to an operating system with multiple file systems mounted on it, where the multiple file systems are mounted on different partitions of the same storage medium or on different storage media; The method comprises: Reading metadata of each of the file systems from the storage medium or partition to which the multiple file systems are mounted; storing a copy of the first sub-data in the metadata of each file system in a memory, wherein the first sub-data is consistent data in the metadata of the plurality of file systems; The second sub-data in the metadata of each of the file systems is stored in multiple copies in the memory, and / or an erasure code is added to the second sub-data and stored in the memory, wherein the second sub-data is inconsistent data in the metadata of multiple file systems.

17. The method according to claim 16, characterized in that The method further comprises: In response to the acquired file read operation, respectively read the target file from at least two file systems among the plurality of file systems, and respectively read metadata of the at least two file systems from the memory; Performing fault-tolerance verification on the read target file by means of consistency arbitration; The first sub-data read is checked for fault tolerance by means of consistency arbitration, and, for any one of the at least two file systems, the second sub-data read from the any one file system is checked for fault tolerance by means of consistency arbitration or erasure code error detection.

18. The method according to claim 16 or 17, characterized in that The first sub-data includes: one or more of: a file name, an access permission, and a group identifier; The second sub-data includes: a timestamp and / or a super block pointer.

19. A memory management device, characterized in that: Deployed on an operating system with multiple file systems mounted on it, where the multiple file systems are mounted on different partitions of the same storage medium or on different storage media; The device comprises: A reading module, configured to read metadata of each of the file systems from the storage medium or partition to which the multiple file systems are mounted; a processing module, configured to store a copy of the first sub-data in the metadata of each of the file systems in a memory, wherein the first sub-data is consistent data in the metadata of the plurality of file systems; The processing module is further configured to store multiple copies of the second sub-data in the metadata of each of the file systems in the memory, and / or to add erasure codes to the second sub-data and store them in the memory, wherein the second sub-data is inconsistent data in the metadata of the multiple file systems.

20. The device according to claim 19, characterized in that The reading module is further configured to, in response to the acquired file reading operation, read target files from at least two of the multiple file systems, and read metadata of the at least two file systems from the memory, respectively; The processing module is further configured to perform a fault-tolerance check on the read target file by means of consistency arbitration; and perform a fault-tolerance check on the read first sub-data by means of consistency arbitration, and perform a fault-tolerance check on the read second sub-data of any one of the multiple file systems by means of consistency arbitration or erasure code error detection.

21. The device according to claim 19 or 20, characterized in that The first sub-data includes: one or more of: a file name, an access permission, and a group identifier; The second sub-data includes: a timestamp and / or a super block pointer.

22. A computing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 16 to 18.

23. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 16 to 18.

24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 16 to 18.

Images