File processing method, device and equipment and computer readable storage medium

Abstract

The invention discloses a file processing method, device and equipment and a computer readable storage medium, and is applied to the technical field of distributed storage, and the file processing method comprises the following steps: determining key metadata attribute information corresponding to each target file; wherein the key metadata attribute information comprises basic metadata information, metadata extension attributes and permission attributes; and constructing a target tree based on the key metadata attribute information of each target file, so as to query metadata corresponding to the target file based on the target tree. Compared with all the data information corresponding to the target file which is stored in sequence at present, the target tree corresponding to the key metadata attribute information is constructed by determining the key metadata attribute information corresponding to the target file, so that the key metadata attribute information can be directly retrieved subsequently based on the target tree, and the retrieval efficiency is improved. Therefore, the retrieval efficiency of the subsequent key metadata attribute information is improved, namely, the metadata access performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a file processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In the distributed storage system architecture, the metadata of the file system is responsible for managing information such as file namespace, file attribute permissions, and data block indexes. Currently, when managing metadata, all metadata is stored in a dispersed manner on various storage nodes. This is a fragmented storage method. This fragmented storage method requires traversing a large amount of content when searching for metadata, resulting in relatively low metadata retrieval efficiency.

[0003] It can be seen that how to improve the retrieval efficiency of metadata is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a file processing method, device, equipment and computer-readable storage medium, which solve the technical problem of low metadata retrieval efficiency in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a file processing method, comprising:

[0006] Determine key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes;

[0007] A target tree is constructed based on the key metadata attribute information of each target file, so as to query metadata corresponding to the target file based on the target tree.

[0008] In some embodiments, constructing a target tree based on the key metadata attribute information of each target file includes:

[0009] Determine a parent directory based on the target file, and a subdirectory corresponding to the parent directory;

[0010] When it is determined that the key metadata attribute information of the subdirectory is consistent with that of the parent directory, it is determined that the key metadata attribute information of the subdirectory inherits the key metadata attribute information of the parent directory;

[0011] The prefix of the key metadata attribute information in the subdirectory that is the same as that in the parent directory is deleted to construct a target tree of prefix tree structure.

[0012] In some embodiments, the basic metadata information includes at least one of the file's location information, file name, user group to which the file belongs, file size, file creation time, and file access time; the metadata extended attributes include at least one of file directory quota, file audit, and file metadata description; and the permission attributes include at least one of access rights and operation rights.

[0013] In some embodiments, after constructing the target tree based on the key metadata attribute information of each target file, the method further includes:

[0014] When the inserted information is a target permission, determining whether the path name of the inserted new directory permission already exists in the target tree;

[0015] If the path name exists, make sure to report an error;

[0016] When the path name does not exist, determine the current directory permission information of the parent directory corresponding to the inserted new directory permission, and determine whether the new directory permission is the same as the current directory permission information;

[0017] If they are the same, it is determined that the target tree is not updated;

[0018] If they are not the same, a new node is created under the child node of the current target tree to represent the inserted new directory permission;

[0019] When it is determined to delete the parent directory in the target tree, it is determined to delete the subdirectory having the same attribute as the parent directory;

[0020] When it is determined to update the permission attribute of the target tree, starting from the root directory or the specified target directory, recursively traverse the entire target tree and apply the same permission attribute setting to each node;

[0021] When performing a recursive permission update operation, an error handling mechanism and / or a rollback mechanism is used; wherein the error handling mechanism is a mechanism for recording an error state, and the rollback mechanism is a mechanism for rolling back to a previous permission attribute state.

[0022] In some embodiments, after building a target tree based on the key metadata attribute information of each target file to query metadata corresponding to the target file based on the target tree, the method further includes:

[0023] Slicing the target tree to obtain sub-directory slices, and constructing a directory subset based on the sub-directory slices;

[0024] The directory subset is mapped to a plurality of metadata services according to the total number of metadata services and the number of each subset in the directory subset.

[0025] In some embodiments, slicing the target tree to obtain sub-directory fragments, and constructing a directory subset based on the sub-directory fragments includes:

[0026] Determine the directory size based on the sub-directory fragments, and determine the maximum number of files carried by a single directory subset;

[0027] The number of constructed directory subsets is determined based on the directory size and the maximum number of files; wherein the target subset includes a unique identifier, a main subset and a subset number; the unique identifier is an identifier for distinguishing subsets of different directories, the main subset is a set of the total number of files under the storage directory, and the subset number is the number of the file.

[0028] In some embodiments, after constructing the target tree based on the key metadata attribute information of each target file, the method further includes:

[0029] When it is determined to update key metadata attribute information, the corresponding key metadata attribute information in the metadata storage service is first updated, and then the target tree is updated based on the key metadata attribute information; wherein the metadata storage service is a service for persisting all key metadata attribute information.

[0030] The embodiment of the present invention further provides a file processing device, comprising:

[0031] A key metadata attribute information determination module is used to determine the key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes information separated from the metadata attribute information and having a retrieval rate greater than a set retrieval rate threshold;

[0032] The target tree construction module is used to construct a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree.

[0033] The embodiment of the present invention further provides a file processing device, including:

[0034] Memory for storing computer programs;

[0035] A processor is used to execute the computer program to implement the steps of the above-mentioned file processing method.

[0036] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned file processing method are implemented.

[0037] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned file processing method when executed by a processor.

[0038] To solve the above technical problems, an embodiment of the present invention provides a file processing method, which may include: determining key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes; constructing a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree.

[0039] It can be seen from the above technical solution that the beneficial effect of the present invention is that, compared with the current fragmented storage of metadata in each storage node, the present invention determines the key metadata attribute information corresponding to the target file and constructs a target tree corresponding to the key metadata attribute information, thereby improving the efficiency of subsequent key metadata attribute information search and improving the performance of accessing metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a file processing method provided by an embodiment of the present invention;

[0042] Figure 2 An overall architecture diagram of a distributed storage system provided by an embodiment of the present invention;

[0043] Figure 3 A system architecture diagram of a file processing method based on a distributed storage system provided by an embodiment of the present invention;

[0044] Figure 4 A flowchart of a file processing method provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0049] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0050] Next, a file processing method provided by an embodiment of the present invention is described in detail. Figure 1 A flowchart of a file processing method provided by an embodiment of the present invention, the method may include:

[0051] S101, determining key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes.

[0052] The execution subject of this embodiment is a metadata separation management service module, and this embodiment does not limit the specific type of the metadata separation management service module. For example, the metadata separation management service module in this embodiment can be a node in a distributed storage system. Or the metadata separation management service module in this embodiment can be a device set in a distributed storage system. This embodiment does not limit specific key metadata attribute information. For example, the key metadata attribute information in this embodiment can be basic metadata information; or the key metadata attribute information in this embodiment can be metadata extension attributes and permission attributes. The basic metadata information in this embodiment includes at least one of the location information of the file, the file name, the user group to which the file belongs, the file size, the file creation time and the file access time; the metadata extension attributes in this embodiment include at least one of the file directory quota, file audit and file metadata description, and the permission attributes in this embodiment include at least one of the access rights and operation rights. The basic metadata information is the information that determines each metadata, the metadata extension attributes are the information that describes the metadata characteristics provided, and the permission attributes are the information that determines the authority to perform various processing on the metadata. In this embodiment, the basic metadata information refers to the location information, name, uig (user group to which the file belongs), gid (changes corresponding to the user group to which the file belongs), size, creation time, and access time of the file; the metadata extension attributes in this embodiment refer to the mechanism of storing additional metadata information attached to the directory. These additional metadata information are different from the basic attributes of the file (such as size, permissions, owner, etc.), but are attributes that can be customized by users or applications. They are used to store more information related to the file, such as file directory quotas, audits, and metadata descriptions. Some applications may need to attach specific information to files to support their functions, such as index information, data classification, etc.; the permission attributes in this embodiment refer to the access rights and operation rights for managing directory metadata. These permissions control the reading, writing, and execution operations of metadata, ensuring that the file and directory metadata in the system are properly protected and managed. During the initialization process of the distributed storage system, the basic metadata information, metadata extension attributes, and permission attributes are synchronously stored from the metadata service cluster in the metadata separation management service for metadata separation management. This method simplifies the client's query process for file quotas, permissions, etc., and reduces metadata processing latency. By storing basic metadata information, metadata extended attributes and permission attributes separately and centralizing them in the metadata separation management service, the metadata separation management service can efficiently retrieve metadata extended attributes or permission attributes by querying the root node in the directory tree.

[0053] S102: construct a target tree based on key metadata attribute information of each target file, so as to query metadata corresponding to the target file based on the target tree.

[0054] This embodiment does not limit the specific process of constructing the target tree based on the key metadata attribute information of each target file. For example, this embodiment can determine the parent directory and subdirectory based on the target file, and thus construct the target tree based on the parent directory and subdirectory; or this embodiment can construct a target tree with a prefix tree structure based on the key metadata attribute information based on the target file. This embodiment can clean all the key metadata attribute information before constructing the target tree to ensure the accuracy of the key metadata attribute information.

[0055] It should be further explained that, in order to improve the efficiency of building the target tree, the target tree is built based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree, which may include:

[0056] S1021, determine the data structure corresponding to the current target file, build a target tree based on the data structure and key metadata attribute information, and use a hash table for optimization during the construction process.

[0057] This embodiment can select a suitable data structure and algorithm. For example, using a balanced tree structure (such as an AVL tree or a red-black tree) can reduce the height of the tree, thereby speeding up the search, insertion, and deletion operations. This embodiment can use a hash table to store visited nodes during the construction process to avoid repeated calculations, thereby improving overall efficiency.

[0058] S1022, determining the type of the target file, and updating the target data by using an incremental construction and update method based on the type.

[0059] This embodiment adopts the incremental construction mode for data that needs to be updated frequently, and only rebuilds the changed parts instead of starting from the beginning every time. In some cases, the update operation can be delayed until a certain threshold or condition is reached before batch update to reduce the overhead caused by frequent operations.

[0060] S1023, during the construction process, analyzing the branches of the target tree to determine the branches to be pruned.

[0061] During the construction process, this embodiment terminates unnecessary calculations in advance according to certain conditions, such as pruning branches in the decision tree that are unlikely to produce useful results.

[0062] It should be further explained that, based on any of the above embodiments, in order to improve the efficiency of target tree construction, the above target tree construction based on the key metadata attribute information of each target file may include: determining the parent directory and the subdirectory corresponding to the parent directory based on the target file; when it is determined that the key metadata attribute information of the subdirectory is consistent with the key metadata attribute information of the parent directory, determining that the key metadata attribute information of the subdirectory inherits the key metadata attribute information of the parent directory; deleting the prefix of the key metadata attribute information in the subdirectory that is the same as the parent directory, and constructing a target tree with a prefix tree structure. This embodiment adopts the characteristic that the subdirectory permissions inherit the parent directory permissions in the file system, and merges the same metadata extended attributes and permission entries to improve the metadata retrieval efficiency. For a directory tree structure with the same metadata extended attributes and permission entries, the permission entries inherited by the subdirectory are merged, and the parent directory permissions and entries with unique permissions are retained. And for the same extended attributes or permissions, for example, directory A and directory B have the same metadata quota and permissions, in the metadata separation management service, the metadata information of directories A and B is merged, which can reduce the storage capacity of the metadata separation service. At the same time, the metadata extended attributes and permission entry path prefixes that are the same as the parent directory are deleted, and the path of each directory is shortened to a compact prefix tree structure so that the metadata can be retrieved more efficiently. In the above manner, metadata search, insertion, deletion and update operations can be efficiently processed. For metadata permission search for a specified path requested by a user, the present invention designs a path prefix matching method from the root node to the leaf node, starting from the root node, first matching in the metadata directory tree structure until the path does not match a leaf node. If no matching node is found in the directory tree, it is determined that the directory inherits the permissions of the parent directory.

[0063] It should be further explained that, based on any of the above embodiments, in order to improve the accuracy of updating the target data, after constructing the target tree based on the key metadata attribute information of each target file, the following steps may also be included:

[0064] S1: When the inserted information is a target permission, determine whether the path name of the inserted new directory permission already exists in the target tree.

[0065] S2: If the path name exists, confirm and report an error.

[0066] S3: When the path name does not exist, determine the current directory permission information of the parent directory corresponding to the inserted new directory permission, and judge whether the new directory permission is the same as the current directory permission information.

[0067] When inserting new directory permissions, this embodiment will return an error if a node with a matching path name is found, that is, the directory permissions already exist in the storage system; if the permissions of the current node are the same as the new permissions to be added, no update is required because the permissions of the directory will be inherited from its parent directory; if the permissions are different, a new node will be created under the child node of the current directory tree structure node to represent the directory permissions to be inserted.

[0068] S4: When they are the same, determine not to update the target tree.

[0069] S5: When they are not the same, create a new node under the child node of the current target tree to represent the inserted new directory authority.

[0070] S6: When it is determined to delete the parent directory in the target tree, it is determined to delete the subdirectories having the same attributes as the parent directory.

[0071] In the deletion operation of this embodiment, when a directory is deleted, the extended attribute and permission attribute directory tree structure will delete the subdirectories with the same attributes under the directory to ensure the consistency of metadata.

[0072] S7: When it is determined to update the permission attribute of the target tree, starting from the root directory or the specified target directory, the entire target tree is recursively traversed, and the same permission attribute setting is applied to each node.

[0073] S8: When performing a recursive permission update operation, an error handling mechanism and / or a rollback mechanism is used; wherein the error handling mechanism is a mechanism for recording an error state, and the rollback mechanism is a mechanism for rolling back to a previous permission attribute state.

[0074] In terms of update operations, this embodiment uses a recursive method to modify the permissions of the entire directory and all its subdirectories to the same value. First, determine the target permissions to be set for the entire directory tree (including the directory and all its subdirectories) (set according to requirements, permissions: weights of delete, read, and write), start from the root directory or the specified target directory, and recursively traverse the entire directory structure. For each directory node, including subdirectories, the same permission settings are applied during the recursive process. For each directory node, the target permission settings are applied. This includes operations such as setting ACL (Access Control List), modifying ownership, or applying special permissions to ensure that each element in the entire directory structure complies with the required permission configuration. When performing recursive permission update operations, an error handling and rollback mechanism is used. For example, if an error occurs when updating a subdirectory, it may be necessary to record the error status and consider rolling back to the previous permission status.

[0075] It should be further explained that, based on any of the above embodiments, in order to ensure the access efficiency of metadata, after constructing a target tree based on the key metadata attribute information of each target file to query the metadata corresponding to the target file based on the target tree, it can also include: slicing the target tree to obtain sub-directory fragments, constructing directory subsets based on the sub-directory fragments; mapping the directory subsets to multiple metadata services according to the total number of metadata services and the number of each subset in the directory subset. The metadata service in this embodiment is used for distributed metadata storage layout and efficient management. In the distributed storage system cluster, each node starts an optimized metadata service process. This embodiment maps the directory subsets to multiple metadata services according to the total number of metadata services and the number of each subset in the directory subset, so that the metadata can be managed in a multi-path manner based on dual-consistent hash mapping, ensuring the load balancing of metadata and reducing the migration cost of the metadata subtree. The first step in this embodiment is to construct a directory subset, and the second step is to map the directory subset to the metadata service process for efficient retrieval access and load balancing of metadata. In order to ensure the consistency and reliability of metadata, the present invention uses RocksDB database (RocksDB is a high-performance embedded key-value storage database system) to record metadata updates and complete local persistent storage.

[0076] It should be further explained that the above-mentioned slicing of the target tree to obtain sub-directory slices and constructing directory subsets based on the sub-directory slices can include: determining the directory scale based on the sub-directory slices, and determining the maximum number of files carried by a single directory subset; determining the number of constructed directory subsets based on the directory scale and the maximum number of files; wherein the target subset includes a unique identifier, a main subset and a subset number; the unique identifier is an identifier for distinguishing subsets of different directories, the main subset is a collection of the total number of files under the storage directory, and the subset number is the number of the file.

[0077] Managing metadata using metadata services consists of three main steps:

[0078] 1) First, slice and partition the directory tree into independent sub-directory shards.

[0079] 2) Construct directory subsets based on sub-directory fragmentation, further divide each directory into smaller directory subsets, and divide the subsets when there are hundreds of millions of files in a single directory to improve the efficiency of traversing and searching directory files. The present invention designs the first-level consistent Hash mapping method for constructing the number of directory subsets as follows:

[0080] First, determine the size of the catalog , that is, the sum of the number of files in a single directory and the number of subdirectories.

[0081] Then, the maximum number of files that a single directory subset can hold .

[0082] Finally, according to and Calculate the number of constructed directory subsets as , each directory subset has three factors:

[0083] First, each constructed directory subset is uniquely identified by an ID, which consists of two parts: the path name of the corresponding file system directory, which is used to distinguish subsets of different directories, and a numerical identifier that increases by 1 to distinguish subsets in the same directory.

[0084] Second, the total number of files under the directory is stored in the first primary subset, and each directory has only one primary subset.

[0085] Third, when a directory has multiple subsets, the files in it will be evenly distributed in these subsets according to the consistent hashing algorithm for their file names, and the subset number to which each file belongs is .

[0086] Finally, in order to ensure the load balancing of metadata, when the directory size changes or the subset size needs to be adjusted, the migration of metadata subtree shards is minimized. The present invention proposes a second-layer consistent hashing algorithm to map different subsets to the metadata service of the present invention, that is,

[0087] , where Mdsn is the total number of distributed cluster metadata services.

[0088] 3) Through the above method, the directory subset to which each file belongs is calculated, and then the directory subset is evenly distributed and stored in the metadata service. First, the subset number of the file is calculated through Hash, and then the metadata service is located and stored according to the subset number Hash mapping. The client can efficiently locate the location of the metadata according to the subset number, which improves the access efficiency of the metadata and reduces the file storage IO delay.

[0089] It should be further explained that, based on any of the above embodiments, in order to improve the consistency of metadata, after constructing the target tree based on the key metadata attribute information of each target file, it may also include: when it is determined to update the key metadata attribute information, first update the corresponding key metadata attribute information in the metadata storage service, and then update the target tree based on the key metadata attribute information; wherein the metadata storage service is a service for persisting all key metadata attribute information. This embodiment does not limit the specific metadata storage service. For example, the metadata storage service in this embodiment can be a metadata service node; or the metadata storage service in this embodiment can be a disk. The present invention adopts a direct write method to ensure the consistency of the metadata storage service and the metadata separation management service, that is, the metadata in the cache (the storage medium is a memory) is kept consistent with the metadata in the persistent storage medium (disk, etc.) during the direct write process. For example, when the extended attribute quota or permission is updated, first, the metadata service is updated, and then synchronized to the metadata separation management service. When the metadata separation management service fails, the client will rebuild the metadata extended attributes and permission results from the underlying persistent storage of the metadata service, thereby ensuring the high reliability of the metadata of the distributed storage cluster.

[0090] The embodiment of the present invention provides a file processing method, which may include: S101, determining key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes; S102, constructing a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree. Compared with the current storage of metadata in fragments in each storage node, the present invention determines the key metadata attribute information corresponding to the target file and constructs a target tree corresponding to the key metadata attribute information, thereby improving the efficiency of subsequent key metadata attribute information search and the performance of accessing metadata.

[0091] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2The overall architecture diagram of a distributed storage system provided by an embodiment of the present invention is based on scalability and fault tolerance, eliminating performance bottlenecks such as single-point metadata, adopting a distributed and decentralized design architecture, slicing data and distributing and storing them on multiple nodes in the form of objects, and ensuring that the load and capacity of each node are balanced, and using network connections to access and manage data, while providing a unified storage interface. Since the distributed storage system adopts a decentralized design architecture, it is highly scalable; the failure of a single storage server node will not affect the unavailability of the entire distributed storage cluster; it supports dynamic expansion to add or delete storage server nodes, and can automatically relocate data objects in the cluster to balance the storage space utilization between storage servers. The distributed storage system architecture includes multiple components, such as a unified, self-controlled, and scalable distributed storage consistency management system, a storage pool, a controllable, scalable, distributed data balancing placement algorithm, and a metadata and monitoring service cluster. In the storage pool, the client's data is stored by slicing, fixed by default as 4MB objects, and the objects are grouped. The group id is the modulo value of Hash (object x) and the number of groups, and finally stored in the underlying device OSD (Object-based Storage Device, object-based storage device).

[0092] The distributed storage system architecture includes the following key components:

[0093] 1) Provide file system services, through which distributed storage clients can perform distributed file storage IO operations.

[0094] 2) Unified, self-controlled, and scalable distributed storage consistency management system: This component is the core component of the distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects, and each object has a unique identifier and related data. It is responsible for storing objects in a distributed manner on each node of the storage cluster and provides functions such as data replication, recovery, and load balancing to ensure data reliability and high-performance access.

[0095] 3) Storage pool: A distributed storage cluster consists of multiple storage nodes, each of which can contain multiple hard disks or storage devices. The storage pool includes a file metadata pool and a data pool, which divides storage resources into different pools to meet different storage needs.

[0096] 4) A controllable, scalable, distributed data balancing placement algorithm, which evenly distributes data objects across the nodes of the storage cluster to avoid data hot spots and improve system performance.

[0097] 5) Metadata service cluster and monitoring service cluster: The metadata service cluster is only responsible for managing the metadata information of the file system, including the attributes and location of files and directories. The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster. Its technical principles include maintaining the status information, configuration information, health status, etc. of the cluster to ensure the consistency and availability of the cluster.

[0098] In order to make the present invention easier to understand, please refer to Figure 3 , Figure 3 A system architecture diagram of a file processing method based on a distributed storage system provided in an embodiment of the present invention may specifically include: a metadata service and a metadata separation management service module (i.e., a module for executing the steps of the file processing method provided by the present invention). The metadata service is used for distributed metadata storage layout and efficient management. In a distributed storage system cluster, each node starts an optimized metadata service process, and uses a dual-consistent hash mapping multi-path method to distribute and store the directory tree to multiple metadata services. The first layer of consistent hash mapping constructs a directory subset, and the second layer of consistent hash mapping is used to map the directory subset to the metadata service process for efficient retrieval access and load balancing of metadata. In order to ensure the consistency and reliability of metadata, the present invention uses the RocksDB database (RocksDB is a high-performance embedded key-value storage database system) to record metadata updates and complete local persistent storage. The metadata separation management service is deployed and implemented on the first node of the distributed storage cluster, and centrally manages basic metadata information, metadata extension attributes, and permission attributes. Considering the capacity and performance bottlenecks and scalability issues of a single metadata separation management service, the metadata separation management service process only stores the basic metadata information, metadata extension attributes, and permission attributes of the directory, and directly retrieves the basic metadata information, metadata extension attributes, and permission attributes of the file from the metadata service cluster. The embodiment of the present invention can provide a client dynamic link library module for the distributed file system, and through the dynamic loading method, the application program can load and unload the functional module as needed, thereby improving flexibility and efficiency, and provides a different interface for accessing the distributed file system metadata method, which is divided into a fast retrieval interface and a normal interface, wherein the fast retrieval interface client can use the interface to quickly retrieve the extension attributes or permission attributes, and the normal interface is retrieved from the metadata cluster service according to the normal process. These two interfaces can be performed simultaneously to provide efficient metadata access.

[0099] The method embodiment of the present invention has substantial improvements on the performance of distributed storage file system metadata, especially for massive small file scenarios, in metadata-intensive IO requests, the method of the present invention reduces the average search delay of metadata extension attributes and permission attributes, especially operations such as directory quotas and directory inheritance permission retrieval can reduce the metadata load pressure by 47%, while improving efficient access to metadata. Therefore, the method of the present invention is highly efficient in reducing the access delay of distributed file system metadata and improving metadata-intensive storage IP operations in large-scale massive data processing environments.

[0100] Specifically, the present invention has the following advantages:

[0101] 1. Performance. Based on the same hardware resources, for the scenario of massive small files, in metadata-intensive IO requests, the method of the present invention reduces the average search delay of metadata extension attributes and permission attributes, especially operations such as directory quotas and directory inheritance permission retrieval, which can reduce the metadata load pressure by 47% and improve the efficient access to metadata. Therefore, the method of the present invention is highly efficient in reducing the access delay of distributed file system metadata and improving metadata-intensive storage IP operations in large-scale massive data processing environments.

[0102] 2. Stability: The metadata organization and management method of the present invention is decoupled from the distributed storage module, is transparent to the business client, and has stability.

[0103] 3. Security: The components and method modules of the present invention are packaged and safe.

[0104] 4. Low cost: The metadata multi-path processing logic and metadata information and structured attribute information storage form proposed in the present invention can improve the competitiveness and maintenance cost of distributed file storage.

[0105] 5. Compatibility: The metadata multi-path management method of the embodiment of the present invention is portable and universal in different hardware devices.

[0106] In order to make the present invention easier to understand, please refer to Figure 4 , Figure 4 A flowchart of a file processing method provided by an embodiment of the present invention may specifically include:

[0107] S201, determining basic metadata information, metadata extension attributes, and permission attributes corresponding to each target file.

[0108] S202, constructing a prefix target tree based on basic metadata information, metadata extension attributes and permission attributes corresponding to each target file, and a permission inheritance method.

[0109] S203, processing the directory tree based on the first layer consistent hash mapping of the dual consistent hash mapping method to obtain a directory subset.

[0110] The dual consistent hash mapping method in this embodiment corresponds to the dual consistent hash mapping multi-path method described above.

[0111] S204, mapping the target subset to the metadata service based on the second layer consistent hash mapping of the dual consistent hash mapping method.

[0112] The embodiment of the present invention adopts a method of separating metadata attributes to provide a fast path for client application requests, which includes centralized management of metadata extension attributes and permission attributes to efficiently perform queries such as directory quotas or permissions. Metadata is managed in a multi-path manner based on dual consistent hash mapping, ensuring metadata load balancing and reducing the migration cost of metadata subtrees.

[0113] The following is an introduction to a file processing device provided by an embodiment of the present invention. The file processing device described below and the file processing method described above can be referenced to each other.

[0114] Figure 5 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention may include:

[0115] The key metadata attribute information determination module 100 is used to determine the key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes;

[0116] The target tree construction module 200 is used to construct a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree.

[0117] Further, based on any embodiment, the target tree construction module 200 may include:

[0118] A parent directory and subdirectory determination unit, used to determine a parent directory and a subdirectory corresponding to the parent directory based on the target file;

[0119] A key metadata attribute information determining unit, configured to determine that the key metadata attribute information of the subdirectory inherits the key metadata attribute information of the parent directory when it is determined that the key metadata attribute information of the subdirectory is consistent with the key metadata attribute information of the parent directory;

[0120] The prefix tree determination unit is used to delete the prefix of the key metadata attribute information in the subdirectory that is the same as that in the parent directory, and construct a target tree of the prefix tree structure.

[0121] Further, based on any of the above embodiments, the basic metadata information includes at least one of the file's location information, file name, user group to which the file belongs, file size, file creation time, and file access time; the metadata extended attributes include at least one of file directory quota, file audit, and file metadata description; and the permission attributes include at least one of access rights and operation rights.

[0122] Further, based on any of the above embodiments, the above file processing device may further include:

[0123] A first judgment module, used for judging whether the path name of the inserted new directory permission already exists in the target tree when the inserted information is a target permission;

[0124] The error reporting module is used to determine if an error is reported if the path name exists;

[0125] The second judgment module is used to determine the current directory permission information of the parent directory corresponding to the inserted new directory permission when the path name does not exist, and to judge whether the new directory permission is the same as the current directory permission information;

[0126] A non-update module is used to determine not to update the target tree when they are the same;

[0127] A new node creation unit, used to create a new node under the child node of the current target tree to represent the inserted new directory authority when they are not the same;

[0128] A subdirectory deletion module, used for, when determining to delete a parent directory in the target tree, determining to delete a subdirectory having the same attribute as the parent directory;

[0129] A target tree traversal module, used for, when it is determined to update the permission attribute of the target tree, recursively traversing the entire target tree starting from the root directory or the specified target directory, and applying the same permission attribute setting to each node;

[0130] The permission update module is used to adopt an error handling mechanism and / or a rollback mechanism when performing a recursive permission update operation; wherein the error handling mechanism is a mechanism for recording an error state, and the rollback mechanism is a mechanism for rolling back to a previous permission attribute state.

[0131] Further, based on any of the above embodiments, the above file processing device may further include:

[0132] A directory subset determination module, used for slicing the target tree to obtain sub-directory slices, and constructing a directory subset based on the sub-directory slices;

[0133] A mapping module is used to map the directory subset to multiple metadata services according to the total number of metadata services and the number of each subset in the directory subset.

[0134] Further, based on any of the above embodiments, the directory subset determination module may include:

[0135] A maximum file quantity determination unit, used to determine the directory scale based on the sub-directory fragments, and determine the maximum number of files carried by a single directory subset;

[0136] A directory subset determination unit is used to determine the number of constructed directory subsets based on the directory size and the maximum number of files; wherein the target subset includes a unique identifier, a main subset and a subset number; the unique identifier is an identifier for distinguishing subsets of different directories, the main subset is a set of the total number of files under the storage directory, and the subset number is the number of the file.

[0137] Further, based on any of the above embodiments, the above file processing device may further include:

[0138] The persistent storage module is used to first update the corresponding key metadata attribute information in the metadata storage service when it is determined to update the key metadata attribute information, and then update the target tree based on the key metadata attribute information; wherein the metadata storage service is a service for persisting all key metadata attribute information.

[0139] It should be noted that the order of the modules and units in the above-mentioned file processing device can be changed without affecting the logic.

[0140] Figure 5 The description of the features in the corresponding embodiments can be found in Figure 5 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0141] A file processing device provided by an embodiment of the present invention may include: a key metadata attribute information determination module 100, used to determine the key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes; a target tree construction module 200, used to construct a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree. Compared with the current storage of metadata in fragments in each storage node, the present invention determines the key metadata attribute information corresponding to the target file and constructs a target tree corresponding to the key metadata attribute information, thereby improving the efficiency of subsequent key metadata attribute information search and the performance of accessing metadata.

[0142] A file processing device provided by an embodiment of the present invention is introduced below. The file processing device described below and the file processing method described above can refer to each other.

[0143] Figure 6 A schematic diagram of the structure of a file processing device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the file processing device includes: a memory 60 for storing a computer program;

[0144] The processor 61 is used to implement the steps of the file processing method in the above embodiment when executing a computer program.

[0145] The file processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0146] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0147] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the file processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data required for file processing, etc.

[0148] In some embodiments, the file processing device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0149] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the file processing device, and may include more or fewer components than those shown in the figure.

[0150] It is understandable that if the file processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program codes.

[0151] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned file processing method are implemented.

[0152] The above is a detailed introduction to a file processing method provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0153] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0154] The above is a detailed introduction to a file processing method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A file processing method, characterized in that: include: Determine key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes; A target tree is constructed based on the key metadata attribute information of each target file, so as to query metadata corresponding to the target file based on the target tree.

2. The file processing method according to claim 1, characterized in that: The constructing a target tree based on the key metadata attribute information of each target file includes: Determine a parent directory based on the target file, and a subdirectory corresponding to the parent directory; When it is determined that the key metadata attribute information of the subdirectory is consistent with that of the parent directory, it is determined that the key metadata attribute information of the subdirectory inherits the key metadata attribute information of the parent directory; The prefix of the key metadata attribute information in the subdirectory that is the same as that in the parent directory is deleted to construct a target tree of prefix tree structure.

3. The file processing method according to claim 1, characterized in that: The basic metadata information includes at least one of the file's location information, file name, user group to which the file belongs, file size, file creation time, and file access time; the metadata extended attributes include at least one of file directory quota, file audit, and file metadata description; and the permission attributes include at least one of access rights and operation rights.

4. The file processing method according to claim 1, characterized in that: After constructing the target tree based on the key metadata attribute information of each target file, the method further includes: When the inserted information is a target permission, determining whether the path name of the inserted new directory permission already exists in the target tree; If the path name exists, make sure to report an error; When the path name does not exist, determine the current directory permission information of the parent directory corresponding to the inserted new directory permission, and determine whether the new directory permission is the same as the current directory permission information; If they are the same, it is determined that the target tree is not updated; If they are not the same, a new node is created under the child node of the current target tree to represent the inserted new directory permission; When it is determined to delete the parent directory in the target tree, it is determined to delete the subdirectory having the same attribute as the parent directory; When it is determined to update the permission attribute of the target tree, starting from the root directory or the specified target directory, recursively traverse the entire target tree and apply the same permission attribute setting to each node; When performing a recursive permission update operation, an error handling mechanism and / or a rollback mechanism is used; wherein the error handling mechanism is a mechanism for recording an error state, and the rollback mechanism is a mechanism for rolling back to a previous permission attribute state.

5. The file processing method according to any one of claims 1 to 4, characterized in that: After building a target tree based on the key metadata attribute information of each target file to query metadata corresponding to the target file based on the target tree, the method further includes: Slicing the target tree to obtain sub-directory slices, and constructing a directory subset based on the sub-directory slices; The directory subset is mapped to a plurality of metadata services according to the total number of metadata services and the number of each subset in the directory subset.

6. The file processing method according to claim 5, characterized in that: Slicing the target tree to obtain sub-directory slices, and constructing a directory subset based on the sub-directory slices, including: Determine the directory size based on the sub-directory fragments, and determine the maximum number of files carried by a single directory subset; The number of constructed directory subsets is determined based on the directory size and the maximum number of files; wherein the target subset includes a unique identifier, a main subset and a subset number; the unique identifier is an identifier for distinguishing subsets of different directories, the main subset is a set of the total number of files under the storage directory, and the subset number is the number of the file.

7. The file processing method according to claim 1, characterized in that: After constructing the target tree based on the key metadata attribute information of each target file, the method further includes: When it is determined to update key metadata attribute information, the corresponding key metadata attribute information in the metadata storage service is first updated, and then the target tree is updated based on the key metadata attribute information; wherein the metadata storage service is a service for persisting all key metadata attribute information.

8. A file processing device, characterized in that: include: A key metadata attribute information determination module, used to determine the key metadata attribute information corresponding to each target file; wherein the key metadata attribute information includes basic metadata information, metadata extension attributes and permission attributes; The target tree construction module is used to construct a target tree based on the key metadata attribute information of each target file, so as to query the metadata corresponding to the target file based on the target tree.

9. A file processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the file processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the file processing method according to any one of claims 1 to 7 are implemented.

Images