Metadata processing method, apparatus, device, and medium

Abstract

The application discloses a metadata processing method, device, equipment and medium, and is applied to a distributed file system. The method comprises the following steps: accepting a file operation request; in response to the file operation request, analyzing a metadata operation; and processing metadata according to the type of the metadata operation and the type of a file, wherein the metadata is stored based on a key-value pair, a key key comprises a unique number of a parent file and a file name of a child file, and a value value is file information of the child file. The method can improve the efficiency of metadata processing and accurately recover data in an abnormal situation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a metadata processing method, apparatus, device, and medium. Background Technology

[0002] In common distributed file system architectures, file data and metadata are stored separately. In distributed file system applications, users frequently perform numerous file read / write, file lookup, and directory lookup operations, generating a large amount of metadata access and modification records.

[0003] In related technologies, to perform repeated operations on the same file, a relatively fixed index is used to directly locate the file, thereby improving the efficiency of file information retrieval and modification. However, the operation process in these methods is relatively cumbersome, and data recovery is not easy in abnormal situations. Summary of the Invention

[0004] This application provides a metadata processing method, apparatus, device, and medium that can improve the efficiency of metadata processing and accurately recover data in abnormal situations.

[0005] The first aspect of this application discloses a metadata processing method applied to a distributed file system. The method includes: accepting a file operation request; responding to the file operation request by parsing the metadata operation; and processing the metadata according to the type of the metadata operation and the file type, wherein the metadata is stored based on key-value pairs, where the key includes a unique number of the parent file and the filename of the child file, and the value is the file information of the child file.

[0006] In one possible implementation of the first aspect above, when the sub-file is a folder, a second key is constructed based on the unique number of the sub-file; and a second value is constructed based on the unique number of the parent file and the filename of the sub-file.

[0007] In one possible implementation of the first aspect above, when the sub-file is a hard link file, the value includes a hard link marker; and a third key is constructed based on the unique number of the sub-file, and a third value is constructed based on the file information of the hard link file.

[0008] In one possible implementation of the first aspect above, when the sub-file is a symbolic link file, the value includes symbolic link path information.

[0009] In one possible implementation of the first aspect above, when the type of the metadata operation is renaming a file, the method includes: first changing the records of the sub-file to hard links; then inserting the renamed sub-file records; and finally deleting the sub-file.

[0010] In one possible implementation of the first aspect above, when the type of the metadata operation is creating a folder, the method includes constituting the key after constituting the value.

[0011] In one possible implementation of the first aspect above, when the type of the metadata operation is to create a hard link file, the method includes: first modifying the hard link records in the source file and the target file; then modifying the key-value records in the source file; and finally inserting the key-value records in the target file.

[0012] In one possible implementation of the first aspect above, the method includes writing data to a metadata cache and updating the data in the metadata cache to the key-value records according to the length of the file being written.

[0013] The second aspect of this application discloses a metadata management device for a distributed file system. The device includes: a file service module for receiving file operation requests from clients and parsing metadata operations; and a metadata service module for processing metadata according to the type of the metadata operation and the file type, wherein the metadata is stored based on key-value pairs, where the key includes a unique number of the parent file and the filename of the child file, and the value is the file information of the child file.

[0014] In one possible implementation of the second aspect above, the file service module further includes a metadata cache module for writing data to the metadata cache and updating the data in the metadata cache to the key-value record according to the length of the file being written.

[0015] A third aspect of this application discloses an electronic device, the device including a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device performs the method of the first aspect embodiment described above.

[0016] The fourth aspect of this application discloses a computer-readable storage medium storing one or more computer programs, which are executed by one or more processors to cause the processors to perform the methods described in the first aspect of the embodiment.

[0017] The metadata processing method, apparatus, device, and medium provided in this application, upon receiving a metadata operation request, perform metadata operations based on the type of operation and the type of the file to be processed. The metadata is stored in key-value (KV) format, and the file information of the child file is retrieved based on the unique identifier of the parent file and the filename of the child file. This single KV record improves metadata processing efficiency and saves storage space. Furthermore, by processing metadata differently based on the type of operation and the type of file, metadata information can be accurately recovered in abnormal situations. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of a distributed file system 100 according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a metadata processing method 200 in one embodiment of this application.

[0020] Figure 3 This is a schematic diagram of a metadata organization method in one embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a metadata organization method in one embodiment of this application;

[0022] Figures 5a-5d This is a flowchart illustrating a metadata processing method according to an embodiment of this application;

[0023] Figures 6a-6b This is a schematic diagram illustrating metadata cache updates according to one embodiment of this application;

[0024] Figure 7 This is a schematic diagram illustrating metadata cache recovery according to one embodiment of this application;

[0025] Figure 8 This is a schematic diagram of a metadata processing apparatus according to an embodiment of this application;

[0026] Figure 9 This is a block diagram of an electronic device 900 according to an embodiment of this application. Detailed Implementation

[0027] The present application will be further described below with reference to specific embodiments and accompanying drawings. It is understood that the illustrative embodiments of this disclosure include, but are not limited to, metadata processing methods, systems, devices, and media. The specific embodiments described herein are merely for explaining the present application and not for limiting it. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to the present application, and not all of the structures or processes.

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] Figure 1 A distributed file system 100 according to an embodiment of this application is shown. The distributed file system 100 includes a client 101, a file service module 102, a data service module 103, a metadata service module 104, and a storage service module 105. The client 101 sends a user request to the file service module 102 based on the user's file operation, and the file service module 102 returns the file operation result to the client 101.

[0030] In common distributed file system architectures, file data and metadata are stored separately to achieve data access separation. Metadata includes file information and directory information, managed by a separate metadata server. File information includes filename, file permissions, file size, creation time, modification time, etc., while directory information includes subfile information and tree-structured directory information. In the distributed file system 100, the file service module 102 splits file operations into data operations and metadata operations based on user requests. Data operations include reading and writing file content, while metadata operations include file queries, permission modifications, and other operations that do not involve file data. Data operations are processed by the data service module 103, which can return file data to the file service module 102. Metadata operations are processed by the metadata service module 104, which can return file metadata to the file service module 102. There may be some information update operations between the data service module 103 and the metadata service module 104. The storage service module 105 can store the data transferred between the data service module 103 and the metadata service module 104.

[0031] In large-scale distributed file systems, users frequently perform numerous file read / write, file query, and directory lookup operations, generating a large amount of metadata access and modification, making metadata management difficult. How to balance the performance and reliability of metadata services while ensuring rapid recovery in the event of system crashes or other anomalies has become a key research focus in metadata management.

[0032] In related technologies, to perform repeated operations on the same file, a relatively fixed index is used to directly locate the file for querying and modifying file information. In this scenario, key-value (kv) pairs are suitable for storing metadata. However, in existing distributed file system metadata management schemes based on kv storage, file query operations are relatively cumbersome.

[0033] For example, patent application CN102110146A first finds the bitmap index of the subfile using the parent directory index number and the subfile name, and then finds the subfile information using the parent directory index and the subfile bitmap index. In this scheme, each file corresponds to two key-value records, which is quite redundant. Given the parent directory index and filename, two searches are necessary to obtain the file information.

[0034] For example, patent CN106874481A first finds the file path based on the file index number, then finds the file information based on the file path, and thus reads the metadata. This method, given the parent directory index and filename, requires finding the parent directory path first, then finding the parent directory information based on the parent directory path, and finally finding the child file information based on the parent directory information, making the multiple searches rather cumbersome.

[0035] Additionally, comparing file CN114116613A and querying metadata based on the parent inode representation and the filename to be queried can reduce the number of queries and simplify the metadata query process. However, this technical solution is applicable to only one type of file and cannot recover data when the metadata service is down.

[0036] To address the aforementioned problems, one embodiment of this application provides a metadata processing method 200, applied to a distributed file system, see reference. Figure 2 .

[0037] S210, Receive metadata operation request.

[0038] S220, based on the metadata operation request, parse the metadata operation.

[0039] S230, Metadata is processed according to the type of metadata operation and the file type, wherein the metadata is stored based on key-value pairs, the key includes the unique number of the parent file and the file name of the child file, and the value is the file information of the child file.

[0040] The following is through Figure 3-4 Table 1-3 provides further explanation of metadata processing method 200.

[0041] Figure 3 This illustration shows a metadata organization method in one embodiment of this application. When querying files, a top-down search is typically performed using a tree-like directory structure, i.e., the file is searched based on an index formed by the parent file information and the filename of the file to be queried. To simplify the index format, each file can be assigned a unique number. For example, in... Figure 3 In the tree-like directory structure, the root directory file is numbered 1, the subfiles of the root directory file are numbered 2, and the files at the next level are numbered 3 and 4.

[0042] When creating key-value pairs, the index consisting of the parent file information and the filename of the file to be queried can be used to construct the key, and the file information of the file to be queried can be used to construct the value, as shown in Table 1.

[0043] Table 1 Figure 3 key-value pairs

[0044] Key Value ( / ) 1. ROOT,... (1, dir) 2. Folder,... (2, file1) 3. Files,... (2, file2) 4. Documents,...

[0045] In Table 1, the index of the root directory file is empty, the key is ( / ), and the value of the root directory file is file information, including one or more of the following: file number, file type, file size, and file creation information. For subfiles of the root directory file, the key is constructed using the index (1, dir) formed by the root directory file number and the subfile's filename dir, and the value is the subfile's file information. The file type of this subfile is a folder. The next level of the folder dir includes two files: file1 and file2. For file1, the key is constructed using the index (2, file1) formed by the parent file number 2 and the filename file1, and the value is the file information of file1, indicating that file1's file type is a file. For file2, the key is constructed using the index (2, file2) formed by the parent file number 2 and the filename file2, and the value is the file information of file2, indicating that file2's file type is a file.

[0046] exist Figure 3 In this metadata organization method, as long as the file index (Key) is identified through the file path, file information in the Value can be searched, modified, or deleted. For each file, the corresponding file information can be retrieved using only one key-value record and the index. This metadata organization method is simple and saves storage space.

[0047] The key-value records in Table 1 can satisfy the need to find child files based on parent files, but not the need to find parent files based on child files. Normally, when accessing the file system, the complete path can be constructed by adding the current file to the current path. However, in practice, due to metadata caching, the current path information may be lost when the service restarts or encounters an anomaly, leaving only the information of the current file. In this case, it is necessary to reverse-engineer the entire path from the current file; the key-value records in this case are shown in Table 2.

[0048] Table 2 Figure 3 Updated key-value pairs

[0049] Key Value ( / ) 1. ROOT,... (1, dir) 2. Folder,... (2，.) (1, dir) (2, file1) 3. Files,... (2, file2) 4. Files,...

[0050] Compared to Table 1, Table 2 adds a key-value record. The key includes the folder number ("folder 2") and a specific character, while the value is the folder's index (1, dir), representing the parent file number and the folder's filename. In this example, the specific character is ".". In other examples, other symbols can be used to represent the specific character. When current path information is lost, although the parent filename is unknown, the parent inode ID (i.e., number 2) is known. The added record's value allows the current path to be determined. Therefore, this added key-value record helps in tracing the entire path upwards.

[0051] Because the folder has added key-value records for reverse lookup, the key-value records in Table 2 can satisfy both the need to find child files based on parent files and the scenario of finding parent files based on child files. Even when an anomaly occurs, the metadata information can still be accurately queried.

[0052] Figure 3 The metadata organization method illustrates two types of metadata: folders and regular files. In practice, file systems may also include other file types, such as symbolic links and hard links. (See reference...) Figure 4 .

[0053] Figure 4 In the metadata organization. Files / dir, / dir / file1, and / dir / file2 are related to... Figure 3 Similarities to those in [the text] will not be elaborated upon here. [And / ] Figure 3 The difference is, Figure 4 It also includes the symbolic link file / dir / symlink_file, and the hard links / dir / file3 and / dir / hardlink_file. Among them, / dir / file3 and / dir / hardlink_file share the file information numbered 6.

[0054] Table 3 Figure 4 key-value pairs

[0055]

[0056]

[0057] Table 3 shows the corresponding... Figure 4 The metadata is organized into a key-value record table. The symbolic link file / dir / symlink_file is numbered 5. The key is similar to that of a regular file, including the parent file number and its own filename (2, symlink_file). The value includes the symbolic link path information.

[0058] Figure 4 The hard link operation in the table causes files / dir / file3 and / dir / hardlink_file to share file information with the number 6. When searching for a file using the key: index (2, hardlink_file) (first search), if the value contains a hard link marker, a second search is required. In this case, the hard link file information in the corresponding value can be found using the key (6, / ) in the key-value record in Table 3.

[0059] The key-value records in Table 3 are not only applicable to metadata management of ordinary files and folders, but can also be applied to metadata management of special files, including soft link files and hard link files.

[0060] For metadata operations in S220, when the operation is to create a file, a new key-value record is added based on the file type. Assume the parent folder number is X, and the file to be created is named A. When the file to be created is a regular file, a key-value record is created with the key (X, A) and the value being the file information. (X, A) can be stored in the metadata cache for later modification of the file information. When the file to be created is a folder, two key-value records are created. The first key-value record has the key (X, A) and the value being the file information. The second key-value record has the key being the file number and a specific character, and the value being (X, A). Similarly, (X, A) is also stored in the metadata cache. When the file to be created is a hard link, the value of the hard link source file must first be modified to mark the source file as a hard link. Then, a target file record and a hard link record are added, and the hard link index can be stored in the metadata cache.

[0061] For metadata operations in S220, when the operation is to query a file, the search is performed based on the file path. According to Table 3, when the file to be queried is a regular file, folder, or symbolic link, only one key-value record needs to be queried. When the file to be queried is a hard link, two queries are required: first, the file number is retrieved based on the first key-value record, and then the hard link information is retrieved based on the second key-value record.

[0062] When the operation involves modifying file attributes, such as modifying file permissions, the file permission information can be modified based on the key value.

[0063] When the operation is to delete a file, the corresponding record is deleted based on the key value. Specifically, if the file to be deleted is a folder, two records need to be deleted. If the number of hard links to a file in the key-value record is 1, deleting the hard link file requires deleting the hard link record.

[0064] The preceding text discussed various file operations under normal circumstances. However, in real-world situations, abnormal scenarios such as machine failures and network failures may occur. When adding, deleting, or modifying a file in the metadata service module, anomalies may arise. Typical key-value (KV) storage can only guarantee the success or failure of a single operation, not the success or failure of multiple operations simultaneously. Therefore, operations such as creating folders, creating hard links, and deleting files require adding or modifying more than one KV record. In some implementations, method 200 also includes a process for updating KV records to ensure that the metadata service can accurately recover metadata information after a crash; see [link to relevant documentation]. Figures 5a-5d .

[0065] exist Figure 5a In the process of creating a folder, the folder information record (such as "2, folder,..." in Table 3) (see S511) can be inserted into the key-value record first, followed by the folder index record (such as "(1, dir)" in Table 3) (see S512). If a service restart occurs during this period and a folder query finds a missing index record, it can be assumed that the folder information is incomplete and the folder does not yet exist, and the folder can be recreated. Conversely, if the folder index record is inserted first, followed by the folder information record, a service restart may occur during this period, and the corresponding folder information may not be found when querying the folder based on the folder index record.

[0066] exist Figure 5b In the process of creating a hard link, taking Table 3 as an example, you can first insert or modify the hard link record (i.e., key: (6, / ), value: 6, hard link file, ... in Table 3) (see S521). If the service restarts at this time, the source file record will not be affected, and the hard link can be retried. Next, you can modify the source file key-value record (i.e., the value in the key-value record of file / dir / file3) (see S522). If the service restarts at this time, the source file information can be found through the hard link, and the hard link retry will not be affected. Finally, insert the target file record (i.e., the key-value record of file / dir / hard_file) (see S523) so that the source and target can share the hard link record.

[0067] exist Figure 5c In this process, when renaming a file, the source file record can be converted into a hard link (see S531), that is, the file information in the value field of the source file is modified, and then the key-value record of the target file is inserted (see S532). If the service restarts at this time, the system will at most generate more files, but it will not affect the correctness of the file content. Finally, the key-value record of the source file is deleted (see S533), so that only the key-value record of the target file remains in the system.

[0068] exist Figure 5dIn this process, when deleting a file, if it contains hard links, the file record can be deleted first (see S541), such as the record in Table 3 (key: (2, file3), value: "6, hard link"). When the hard link list shows no hard link files (for example, the number of hard links in the value corresponding to key (6, / ) in Table 3 is 0), then the hard links are deleted, i.e., (key: (6, / ), value: "6, hard link file, number of hard links, ...") (see S542). This can prevent the number of hard links from being less than the actual number of files after a restart, which could cause problems with subsequent deletions.

[0069] Figures 5a-5d The process of updating multiple records has been optimized based on the file operation type, enabling the metadata service to accurately recover metadata information after a crash, and allowing users to retry operations that did not return successfully before the crash.

[0070] Two file operations are particularly expensive: repeated file lookups and file writes. Repeated file lookups involve accessing the same file information multiple times; file writes update the file length, resulting in a file attribute modification with each write operation.

[0071] To improve the performance of metadata services, one embodiment of this application provides a method for optimizing metadata caching. This method adds a metadata caching module 106 to the file service module 102, as shown in Figure 6. After a file operation, file information can be written to the metadata caching module 106, and then read directly from the metadata caching module 106 on the next access. The metadata cache in the metadata caching module 106 is shown in Table 4.

[0072] Table 4. Metadata Cache Data Illustration

[0073] Document Number Key File Information Loading time 1 ( / ) 1. ROOT,... ... 2 (1, dir) 2. Folder,... ... 3 (2, symlink_file) 5. Symbolic link files, symbolic link information,... ... 4 (6， / ) 6. Hard link files,... ...

[0074] Furthermore, the data in the metadata cache module 106 is not updated synchronously with the key-value data in the metadata service module 104. For files that a user is currently reading or writing, the metadata cache can store the file's information. However, using key-value storage is not suitable for updating I / O; therefore, the file length is updated in the metadata cache. When the file length increases to a certain size, a reasonable expiration time can be set to refresh the cached data to the metadata service.

[0075] Figures 6a-6b An example of metadata cache updates is shown. Figure 6aIn steps 601 and 602, the client writes I / O to the data service module through the file service module. If the update length is set to 100, meaning the metadata service is updated every 100 bytes added to the file length, then if the current file length is 50 bytes, and the file needs to be written with an additional 10 bytes, then the key-value store does not need to be updated; only the metadata cache length is updated (see step 603). In step 604, the file service module returns a successful I / O message to the client.

[0076] In another example, if the current file length is 99, and the file needs to be written by adding 10 characters from the end, then the key-value store needs to be updated before continuing the I / O, such as... Figure 6b As shown in Figure 611, the client sends an IO request to the file service module. In Figure 612, the file service module writes the updated data of length 109 to the metadata service module. In Figure 613, after the key-value data in the metadata service module is successfully updated, the file service module sends a write IO request to the data service module (614). In Figure 615, the file length is updated in the metadata cache. In Figure 616, the file service module returns a successful IO message to the client.

[0077] The above-mentioned method for optimizing metadata caching can reduce the pressure on the metadata server by adding a metadata caching module, and can greatly improve performance when writing file data.

[0078] In some implementations, when a file service failure requires a restart, resulting in the loss of metadata cache data, the metadata service module needs to access the data service to recover the file length. See [link to relevant documentation]. Figure 7 .

[0079] In step 701, the client sends a request to the file service module to query file information using the index of the file to be queried (parent file number, filename of the file to be queried). If the data in the metadata cache module of the file service module is empty at this time, the process proceeds to step 702, where the file service module sends a key-value data query request to the metadata service module. In step 703, the metadata service module sends a request to the data service module to request the file length based on the file number. In step 704, the data service module returns the recovered file length to the metadata service module. In step 705, the metadata service module uses the received file length to update the file information of the file to be queried in the metadata cache module. In step 706, the file service module returns the updated file information to the client.

[0080] In these implementations, when data in the metadata cache is lost, the data can be recovered from the data service module through the metadata service module.

[0081] One embodiment of this application provides a metadata management device 800 for a distributed file system, the device 800 including:

[0082] File service module 810 is used to receive file operation requests from clients and parse metadata operations; and metadata service module 820 is used to process metadata according to the type of metadata operation and file type, wherein the metadata is stored based on key-value pairs, where the key includes the unique number of the parent file and the file name of the child file, and the value is the file information of the child file.

[0083] In some implementations, the file service module 810 also includes a metadata cache module for writing data to the metadata cache and updating the data in the metadata cache to the key-value records according to the length of the file being written.

[0084] Now for reference Figure 9 The diagram shown is a block diagram of an electronic device 900 according to an embodiment of the present application. The electronic device 900 may include one or more processors 902, system control logic 908 connected to at least one of the processors 902, system memory 904 connected to the system control logic 908, non-volatile memory (NVM) 906 connected to the system control logic 908, and network interface 910 connected to the system control logic 908.

[0085] Processor 902 may include one or more single-core or multi-core processors. Processor 902 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments herein, processor 902 may be configured to perform operations according to... Figure 1-7 One or more embodiments of the various embodiments shown.

[0086] In some embodiments, system control logic 908 may include any suitable interface controller to provide any suitable interface to at least one of the processors 902 and / or any suitable device or component communicating with system control logic 908.

[0087] In some embodiments, system control logic 908 may include one or more memory controllers to provide an interface to system memory 904. System memory 904 may be used to load and store data and / or instructions. In some embodiments, memory 904 of device 900 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0088] NVM / Memory 906 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / Memory 906 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of HDD (Hard Disk Drive), CD (Compact Disc) drive, and DVD (Digital Versatile Disc) drive.

[0089] NVM / Storage 906 may include a portion of storage resources mounted on the device 900, or it may be accessible by the device but is not necessarily part of the device. For example, NVM / Storage 906 may be accessed over a network via network interface 910.

[0090] Specifically, system memory 904 and NVM / memory 906 may each include a temporary copy and a permanent copy of instruction 920. Instruction 920 may include, when executed by at least one of processors 902, causing device 900 to perform, as Figure 1-7 The instructions for the method shown. In some embodiments, the instructions 920, hardware, firmware and / or their software components may additionally / alternatively be located in the system control logic 908, the network interface 910 and / or the processor 902.

[0091] Network interface 910 may include a transceiver for providing a radio interface to device 900, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 910 may be integrated into other components of device 900. For example, network interface 910 may be integrated into at least one of processor 902, system memory 904, NVM / memory 906, and firmware device (not shown) with instructions that, when at least one of processor 902 executes the instructions, device 900 implements... Figure 1-7 One or more embodiments of the various embodiments shown.

[0092] The network interface 910 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 910 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0093] In one embodiment, at least one of the processors 902 may be packaged together with the logic of one or more controllers for system control logic 908 to form a system-in-package (SiP). In another embodiment, at least one of the processors 902 may be integrated on the same die with the logic of one or more controllers for system control logic 908 to form a system-on-a-chip (SoC).

[0094] Device 900 may further include: input / output (I / O) device 912. I / O device 912 may include a user interface that enables a user to interact with device 900; the design of the peripheral component interface enables peripheral components to also interact with device 900.

[0095] In some embodiments, the user interface may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0096] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0097] All methods and implementations of this application can be implemented in the form of software, magnetic files, firmware, etc.

[0098] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0099] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this paper are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0100] One or more aspects of at least one embodiment can be implemented by representational instructions stored on a computer-readable storage medium, the instructions representing various logics in a processor, which, when read by a machine, cause the machine to create logic for performing the techniques described herein. These representations, referred to as “IP cores,” can be stored on a tangible computer-readable storage medium and provided to multiple customers or production facilities for loading into manufacturing machines that actually manufacture the logic or processor.

[0101] One embodiment of this application discloses a computer-readable medium storing one or more programs, which can be executed by one or more processors to implement... Figure 1-8 One or more embodiments of the various embodiments shown.

[0102] The above description illustrates the implementation of this application through specific embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this invention are limited to this embodiment. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of this application. To provide a thorough understanding of this application, many specific details will be included in the following description. This application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0103] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0104] Unless the context otherwise specifies, the terms “contains,” “has,” and “includes” are synonyms. The phrase “A / B” means “A or B.” The phrase “A and / or B” means “(A and B) or (A or B).”

[0105] As used herein, the terms “module” or “unit” may refer to, be, or include: application-specific integrated circuits (ASICs), electronic circuits, (shared, dedicated, or group) processors and / or memories that execute one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0106] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism, but not limited to, for storing or transmitting information in a machine-readable (e.g., computer-readable) form, such as floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0107] In the accompanying drawings, certain structural or methodological features are shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. In some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0108] It should be understood that although terms such as "first," "second," etc., may be used herein to describe various units or data, these units or data should not be limited by these terms. These terms are used merely to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0109] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

Claims

1. A metadata processing method applied to a distributed file system, characterized in that, The method includes: Accept file operation requests; In response to the file operation request, parse the metadata operation; Metadata is processed according to the type of metadata operation and the file type. The metadata is stored based on key-value pairs, where the key includes the unique number of the parent file and the file name of the child file, and the value is the file information of the child file. After a file operation, the file information is written to the metadata cache module. For a file that is being read or written, the file length is updated in the metadata cache module. When the file length increases to a predetermined value, the data in the metadata cache module is periodically refreshed to the metadata service module.

2. The metadata processing method according to claim 1, characterized in that, When the sub-file is a folder The second key is constructed based on the unique number of the sub-file; The second value is constructed based on the unique number of the parent file and the filename of the child file.

3. The metadata processing method according to claim 2, characterized in that, When the sub-file is a hard link file The value includes a hard link marker; and a third key is constructed based on the unique number of the sub-file, and a third value is constructed based on the file information of the hard link file.

4. The metadata processing method according to claim 1, characterized in that, When the sub-file is a symbolic link file, the value includes symbolic link path information.

5. The metadata processing method according to claim 1, characterized in that, When the type of the metadata operation is renaming a file, the method includes first changing the records of the sub-file to hard links; Then insert the renamed sub-file record; finally, delete the sub-file.

6. The metadata processing method according to claim 2, characterized in that, When the type of the metadata operation is creating a folder, the method includes constructing the key after constructing the value.

7. The metadata processing method according to claim 3, characterized in that, When the type of the metadata operation is to create a hard link file, the method includes: first modifying the hard link records in the source file and the target file; then modifying the key-value records in the source file; and finally inserting the key-value records in the target file.

8. The metadata processing method according to claim 1, characterized in that, The method includes writing data to a metadata cache and updating the data in the metadata cache to the key-value records according to the length of the file being written.

9. A metadata management device for a distributed file system, characterized in that, The device includes: The file service module is used to receive file operation requests from clients and parse metadata operations. The metadata service module is used to process metadata according to the type of metadata operation and the file type. The metadata is stored based on key-value pairs, where the key includes the unique number of the parent file and the file name of the child file, and the value is the file information of the child file. After the file operation, the file information is written to the metadata cache module. For the file being read or written, the file length is updated in the metadata cache module. When the file length increases to a predetermined value, the data in the metadata cache module is periodically refreshed to the metadata service module.

10. The metadata management device according to claim 9, characterized in that, The file service module also includes a metadata caching module, which is used to write data into the metadata cache and update the data in the metadata cache to the key-value records according to the length of the file being written.

11. An electronic device, characterized in that, The device includes a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device performs the method according to any one of claims 1 to 8.

12. A computer-readable medium, characterized in that, The medium stores one or more programs, which can be executed by one or more processors to implement the method of any one of claims 1 to 8.

Images