Directory path analysis method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a directory path analysis method, which comprises the following steps: acquiring directory information of a file system, and creating a metadata cache in a metadata client for storing the directory information; receiving a directory path analysis request, and judging whether a directory accessed by the directory path analysis request exists in the metadata cache or not based on the directory information; if the directory accessed by the directory path analysis request exists in the metadata cache, acquiring an entity identifier corresponding to the directory from the metadata cache; sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key value database through the metadata server; and returning result data corresponding to the directory path analysis request. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and directory information and result data can be stored in a block chain. According to the method, the directory path analysis time delay can be reduced, and the problem of near root hotspots is solved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a directory path parsing method, device, equipment and medium. Background Art

[0002] In computer storage systems, directory path resolution is an indispensable part of accessing files or directories. However, as storage systems continue to expand in size and complexity, path resolution faces many challenges.

[0003] First, the problem of increasing directory path resolution latency is becoming increasingly prominent. Traditional directory path resolution methods require step-by-step access from the root directory to the permissions and other attributes of the target directory item. For an N-level directory structure, N-1 path resolution requests need to be initiated. As the depth of the directory increases, this step-by-step resolution method causes the latency to increase linearly, seriously affecting the access efficiency of the storage system. Secondly, the near-root hotspot problem cannot be ignored. Since directory path resolution requires step-by-step requests, the access frequency of high-level directories (such as the root directory) is extremely high, making them hotspots for back-end access. This not only increases the system load, but may also lead to uneven resource allocation and performance bottlenecks. In summary, there is an urgent need for a method that can solve the problems of increasing directory path resolution latency and near-root hotspots. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a directory path resolution method, apparatus, device and medium to solve the existing directory path resolution delay growth and near-root hotspot problems.

[0005] In order to solve the above technical problems, the present application embodiment provides a directory path parsing method, which adopts the following technical solutions:

[0006] Obtain directory information of the file system, create a metadata cache inside the metadata client, and use the metadata cache to store directory information; receive a directory path resolution request, and determine whether the directory accessed by the directory path resolution request exists in the metadata cache based on the directory information; if the directory accessed by the directory path resolution request exists in the metadata cache, obtain the entity identifier corresponding to the directory from the metadata cache; send the entity identifier to the metadata server to query the result data corresponding to the directory from the key-value database through the metadata server; and return the result data corresponding to the directory path resolution request.

[0007] Furthermore, the steps of creating a metadata cache inside the metadata client specifically include:

[0008] Obtain metadata client configuration information from the configuration center and create a metadata cache based on the metadata client configuration information.

[0009] Further, the directory information is stored in the metadata cache using a hash table structure; based on the directory information, the step of determining whether the directory accessed by the directory path resolution request exists in the metadata cache specifically includes:

[0010] Get the names of directories at all levels of the directory path in the directory path resolution request; based on the directory information, use the directory name as the search key and perform a hash search to determine whether the directories at all levels of the directory path exist in the metadata cache.

[0011] Further, based on the directory information, the directory name is used as a search key, and a hash search is performed to determine whether the directories at all levels of the directory path exist in the metadata cache, specifically including:

[0012] Use the directory name as the search key and match the search key with the directory name key of the directory information; if the search key matches the directory name key, it is determined that the directories at all levels of the directory path exist in the metadata cache; if the search key does not match the directory name key, it is determined that the directories at all levels of the directory path do not exist in the metadata cache.

[0013] Furthermore, the step of obtaining the entity identifier corresponding to the directory from the metadata cache specifically includes:

[0014] Based on the directory path information of the directory path, a hash algorithm is used to perform a hash operation on the directory path to obtain a hash value of the directory path; according to the hash value of the directory path, a corresponding cache entry is searched in the metadata cache; if a cache entry corresponding to the directory path is found in the metadata cache, an entity identifier corresponding to the directory is obtained from the cache entry.

[0015] Further, after the step of determining whether the directory accessed by the directory path resolution request exists in the metadata cache based on the directory information, the method further includes:

[0016] If the directory accessed by the directory path resolution request does not exist in the metadata cache, the target result data corresponding to the directory path is queried from the key-value database; the target result data corresponding to the directory path resolution request is returned, and the metadata cache is updated based on the target result data.

[0017] Furthermore, after the step of sending the entity identifier to the metadata server to query the result data corresponding to the directory from the key-value database through the metadata server, it also includes:

[0018] When a directory update operation of the metadata server is monitored, the directory update operation message broadcast by the master node of the metadata server is obtained; the directory update operation message is parsed to obtain the modified directory path and the modified target version number, and the modified directory path is added to the cache invalidation list; the cache check request initiated by the metadata client is received, and the cache version number and the access file path carried in the cache check request are obtained, and the cache check request is triggered by the directory update operation message; the cache version number is compared with the target version number, and if the cache version number is less than the target version number, it is determined whether the access file path exists in the cache invalidation list; if so, a response message containing the cache invalidation list is returned to the metadata client, so that the metadata client invalidates the corresponding cache entry in the metadata cache according to the cache invalidation list.

[0019] In order to solve the above technical problems, the embodiment of the present application also provides a directory path parsing device, which adopts the following technical solutions:

[0020] The information acquisition module is used to obtain the directory information of the file system and create a metadata cache inside the metadata client. The metadata cache is used to store the directory information.

[0021] A request receiving module, used to receive a directory path resolution request, and based on the directory information, determine whether the directory accessed by the directory path resolution request exists in the metadata cache;

[0022] An identifier acquisition module, used for acquiring an entity identifier corresponding to the directory from the metadata cache if the directory accessed by the directory path resolution request exists in the metadata cache;

[0023] A sending module, used for sending the entity identifier to the metadata server, so as to query the result data corresponding to the directory from the key-value database through the metadata server;

[0024] Return module, used to return the result data corresponding to the directory path resolution request.

[0025] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above directory path resolution method are implemented.

[0026] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above directory path resolution method.

[0027] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: by introducing a metadata cache mechanism (metadata cache), the directory information of the file system is cached inside the metadata client, thereby significantly optimizing the process of directory path resolution. When a directory path resolution request is received, the directory can be quickly found in the metadata cache, avoiding the cumbersome process of accessing directory items step by step in the traditional method. If the directory exists in the metadata cache, its corresponding entity identifier can be directly obtained, and a query request can be quickly initiated to the metadata server, and then the required result data can be retrieved from the key-value database. This innovative design not only greatly reduces the delay of directory path resolution, but also effectively solves the problem of near-root hotspots. Since the directory information is stored in the metadata cache, for frequently accessed near-root directories, a quick response can be given without the need for repeated path resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0030] Figure 2 It is a flowchart of a directory path parsing method provided by this application;

[0031] Figure 3 It is a structural schematic diagram of a directory path parsing device provided by the present application;

[0032] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0037] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0038] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0039] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0040] It should be noted that the directory path resolution method provided in the embodiment of the present application is generally executed by a server, and accordingly, the directory path resolution device is generally arranged in the server.

[0041] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] Continue to refer Figure 2 , shows a flow chart of an embodiment of a directory path parsing method according to the present application. The directory path parsing method comprises the following steps:

[0043] Step S201, obtain directory information of the file system, create a metadata cache inside the metadata client, and the metadata cache is used to store the directory information.

[0044] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can obtain the directory information of the file system through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0045] The file system refers to the mechanism used in the operating system to organize and access data files. It is responsible for organizing the data on the storage device into a directory structure and provides management functions such as creating, deleting, reading, and writing the directory structure.

[0046] Directory information refers to detailed data about the directory structure in the file system, including the directory's name, path, hierarchical relationship, and subdirectories, which is the basis for file system navigation and access.

[0047] The metadata client is a software component that is specifically used to handle metadata-related operations. It is responsible for obtaining, caching, and updating metadata from the metadata server, as well as providing metadata query and parsing services.

[0048] Among them, the metadata cache is a temporary storage mechanism established inside the metadata client to store directory information and other metadata to improve the efficiency and speed of metadata access.

[0049] Step S202: receiving a directory path resolution request, and determining whether the directory accessed by the directory path resolution request exists in the metadata cache based on the directory information.

[0050] The directory path resolution request refers to a request initiated by a user or an application program to resolve and determine the actual location or existence of a specified directory path in the file system.

[0051] Among them, the directory is a logical structure in the file system, which is used to organize and store files and other sub-directories. It constitutes the hierarchical structure of the file system, making it easier for users and administrators to manage and access files.

[0052] Step S203: If the directory accessed by the directory path resolution request exists in the metadata cache, an entity identifier corresponding to the directory is obtained from the metadata cache.

[0053] The entity identifier is a unique identifier assigned to each directory in the file system, which is used to uniquely identify and locate these entities in the key-value database.

[0054] Step S204: Send the entity identifier to the metadata server, so as to query the result data corresponding to the directory from the key-value database through the metadata server.

[0055] Among them, the metadata server is a system component responsible for storing, managing and providing metadata services. It receives requests from metadata clients and returns corresponding metadata based on the requests.

[0056] Among them, the key-value database is a non-relational database that stores data in the form of key-value pairs, where each key is uniquely mapped to a value, supporting efficient data access and query operations.

[0057] The result data refers to the directory or file-related data that is queried from the key-value database and returned to the requester based on the directory path resolution request, including the file's attributes, content, directory structure, etc.

[0058] Step S205, returning result data corresponding to the directory path resolution request.

[0059] The process of step S205 refers to that after the metadata client successfully resolves the directory path and obtains the corresponding entity identifier, it sends the identifier to the metadata server for query, and returns the query result data to the original directory path resolution requester.

[0060] The embodiment of the present application can cache the directory information of the file system inside the metadata client by introducing a metadata cache mechanism (metadata cache), thereby significantly optimizing the process of directory path resolution. When a directory path resolution request is received, the directory can be quickly found in the metadata cache, avoiding the cumbersome process of accessing directory items step by step in the traditional method. If the directory exists in the metadata cache, its corresponding entity identifier can be directly obtained, and a query request can be quickly initiated to the metadata server, and then the required result data can be retrieved from the key-value database. This innovative design not only greatly reduces the delay of directory path resolution, but also effectively solves the problem of near-root hotspots. Since the directory information is stored in the metadata cache, for frequently accessed near-root directories, a quick response can be given without the need for repeated path resolution.

[0061] In some optional implementations of this embodiment, step 201, creating a metadata cache inside the metadata client, specifically includes the following steps:

[0062] Obtain metadata client configuration information from the configuration center and create a metadata cache based on the metadata client configuration information.

[0063] Among them, the configuration center is a system that centrally manages and distributes application configuration information. It allows dynamic modification and updating of configurations and ensures that these configurations can be synchronized to all related applications and services in a timely manner.

[0064] The metadata client configuration information refers to the detailed information stored in the configuration center about how the metadata client connects to and operates the metadata server. The metadata client configuration information may include the service address, port number, authentication information, etc.

[0065] In one example, first, the configuration information of the metadata client can be obtained from the configuration center. The configuration center is a component that centrally manages configuration information, which stores various parameters and settings required for the operation of the metadata client. The configuration information may include but is not limited to the size of the metadata cache, the cache strategy, the cache validity period, etc. Next, a metadata cache is created based on the metadata client configuration information obtained. The metadata cache is a temporary storage area for storing file system directory information, which can improve the efficiency of directory path resolution. According to the cache size and cache strategy in the configuration information, the corresponding memory space is allocated, and the update and expiration rules of the cached data are set. For example, assuming that the metadata cache size set in the configuration center is 10MB and the cache strategy is the LRU (Least Recently Used) algorithm, the metadata client will create a cache area of ​​10MB in size and use the LRU algorithm to manage the data in the cache.

[0066] The embodiment of the present application can achieve high flexibility and configurability in metadata management by dynamically acquiring the configuration information of the metadata client from the configuration center. The solution first ensures that the creation process of the metadata cache is based on the latest and accurate configuration information, which effectively avoids the problem of inconsistent metadata caused by configuration lags or errors. By automatically pulling configurations from the configuration center, manual intervention is reduced and the maintainability and reliability of the system are improved.

[0067] In some optional implementations of this embodiment, the directory information is stored in the metadata cache using a hash table structure; step S202, based on the directory information, determines whether the directory accessed by the directory path resolution request exists in the metadata cache, specifically including the following steps:

[0068] Get the names of directories at all levels of the directory path in the directory path resolution request; based on the directory information, use the directory name as the search key and perform a hash search to determine whether the directories at all levels of the directory path exist in the metadata cache.

[0069] Among them, "directory information is stored in the metadata cache using a hash table structure." This is a storage optimization strategy. By storing directory information in the metadata cache in the form of a hash table, fast search and access to directory information can be achieved.

[0070] The retrieval key refers to a keyword used when performing a hash search, which is usually associated with the data item to be searched and is used to locate the data item in the hash table.

[0071] Among them, hash search is an efficient search algorithm that uses a hash function to map a search key to a certain position in a hash table, thereby achieving fast location and access to data.

[0072] In one example, the specific application of the technical solution of this embodiment is described in detail with the online trading system in the financial field as the background. In the financial trading system, it is often necessary to process a large amount of transaction data, customer information, market analysis reports and other files, which are classified and stored in the multi-level directories of the server. In order to improve the efficiency of file access, the system uses metadata caching technology to accelerate the parsing process of the directory path. When the trading system receives a directory path parsing request, it first parses the directory names of each level in the path. For example, if the request path is " / financial market / stock / 2023 annual report", the parsed directory names are "financial market", "stock" and "2023 annual report". Next, the system uses the constructed metadata cache for hash search. The cache stores metadata of directories at all levels, with the directory name as the search key. The system searches with "financial market", "stock" and "2023 annual report" as keys in turn to determine whether these directories exist in the metadata cache.

[0073] The embodiment of the present application can achieve detailed division and identification of the directory structure by accurately obtaining the directory names at all levels in the directory path resolution request, providing an accurate data basis for subsequent operations. Subsequently, the metadata cache is used as a carrier for information retrieval, combined with the efficiency of hash search, and the directory name is used as the search key for rapid matching. This strategy significantly improves the efficiency of directory existence judgment and reduces system response time. Through the introduction of hash search, not only the complexity of traditional linear search is reduced, but also the uniformity and rapid positioning ability of the hash function in data distribution are fully utilized, making the retrieval of large-scale metadata cache faster and more accurate. In addition, the scheme also implies the effective use of the metadata cache, which reduces the overhead of repeated queries through the caching mechanism, greatly reduces the delay of directory path resolution, effectively solves the problem of near-root hotspots, and further improves the performance and stability of the overall system.

[0074] In some optional implementations of this embodiment, the step of "based on the directory information, using the directory name as the search key, through hash search, determining whether the directories at all levels of the directory path exist in the metadata cache" specifically includes the following steps:

[0075] Use the directory name as the search key and match the search key with the directory name key of the directory information; if the search key matches the directory name key, it is determined that the directories at all levels of the directory path exist in the metadata cache; if the search key does not match the directory name key, it is determined that the directories at all levels of the directory path do not exist in the metadata cache.

[0076] The directory name key refers to a key used to uniquely identify a directory name in a hash table. It usually corresponds to the directory name itself and is used to find and locate metadata of the directory in the hash table.

[0077] In one example, the specific application of the technical solution of this embodiment is described in detail with the financial data backup system in the financial field as the background. In the financial data backup system, a large number of directory paths need to be monitored and managed. The system first obtains the directory path resolution request to be checked and extracts the directory names at all levels. For example, a directory path is " / financial data / 2023 / transaction records". The system uses the directory names at all levels of the path, such as "financial data", "2023" and "transaction records", as retrieval keys. Subsequently, in the metadata cache, the system uses these directory names as keys for matching and searching. If the directory names at all levels match the keys in the cache, the system determines that the directories at all levels of the directory path exist in the metadata cache, indicating that these directories have been backed up by the system. Conversely, if there are unmatched keys, it is determined that they do not exist.

[0078] The embodiment of the present application can use the directory name as the search key to accurately match the directory name key in the metadata cache, thereby achieving efficient judgment on the existence of directories at all levels of the directory path. Specifically, when the search key and the directory name key are completely consistent, it can quickly confirm that all levels of directories in the directory path have been cached, thereby avoiding unnecessary disk access or network requests, and significantly improving the speed and efficiency of data retrieval. Conversely, if the search key does not match any level of directory name key, it can be immediately determined that the corresponding level of directory of the directory path does not exist in the cache. By reducing redundant search operations, the overall load of the system is reduced, and the stability and response speed of the system are improved.

[0079] In some optional implementations of this embodiment, step S203, obtaining an entity identifier corresponding to the directory from the metadata cache, specifically includes the following steps:

[0080] Based on the directory path information of the directory path, a hash algorithm is used to perform a hash operation on the directory path to obtain a hash value of the directory path; according to the hash value of the directory path, a corresponding cache entry is searched in the metadata cache; if a cache entry corresponding to the directory path is found in the metadata cache, an entity identifier corresponding to the directory is obtained from the cache entry.

[0081] The hash value of the directory path is a unique value calculated by applying a hash function to the directory path as input, and is used to quickly locate metadata entries related to the directory path in the metadata cache.

[0082] Among them, a cache entry refers to a data unit stored in the metadata cache, which contains metadata related to a directory or file, such as an entity identifier, a hash value of a directory path, etc., and is the basic component unit of the metadata cache.

[0083] In one example, the specific application of the technical solution of this embodiment is described in detail with the financial data archiving system in the financial field as the background. In the financial data archiving system, it is necessary to efficiently manage a large number of data files. Each data file is stored in a specific directory path. In order to quickly locate and access these files, the system uses the method of this embodiment to perform hash retrieval of the directory path. First, the system receives a directory path to be retrieved, for example: " / financial data / 2023 / transaction record / monthly / 01 month". Then, the system performs a hash operation on the directory path and uses an efficient hash algorithm (such as SHA-256) to generate a unique hash value. Next, the system uses the hash value to search in the metadata cache. The metadata cache is a mapping table that stores the hash value of the directory path and the corresponding entity identifier (such as file ID, storage location, etc.). The system quickly locates the corresponding entry in the cache through the hash value. If the system successfully finds a cache entry that matches the hash value of the directory path in the metadata cache, it directly reads and returns the entity identifier corresponding to the directory from the entry. For example, the system can return a URL or file ID pointing to the storage location of the actual data file, thereby achieving fast access to the data file.

[0084] The embodiment of the present application can convert a lengthy directory path into a fixed-length hash value through a hash algorithm operation based on the directory path information. This conversion process not only simplifies the subsequent retrieval steps, but also ensures the uniqueness of the hash value corresponding to each directory path, effectively avoiding the retrieval confusion that may be caused by similar directory paths. Compared with the traditional string matching method, the retrieval speed is significantly improved by using the hash value to search in the metadata cache, which reduces the system processing time and improves the overall operating efficiency. At the same time, the introduction of the metadata cache enables the mapping relationship between the directory path and the entity identifier to be quickly stored and read, further speeding up the data access speed.

[0085] In some optional implementations of this embodiment, in step S202, after determining whether the directory accessed by the directory path resolution request exists in the metadata cache based on the directory information, the following steps are specifically included:

[0086] If the directory accessed by the directory path resolution request does not exist in the metadata cache, the target result data corresponding to the directory path is queried from the key-value database; the target result data corresponding to the directory path resolution request is returned, and the metadata cache is updated based on the target result data.

[0087] The target result data refers to the detailed data obtained by accessing the key-value database query for the directory path that is not in the metadata cache, including the directory entity identifier, structure, file list and attributes and other directly related information.

[0088] In one example, the specific application of the technical solution of this embodiment is described in detail with the financial transaction system in the financial field as the background. In the financial transaction system, users may need to access key information such as transaction records and market data stored in different directory paths. In order to quickly respond to these access requests, when a user initiates a directory path resolution request, the system will first check whether the directory accessed by the directory path resolution request exists in the metadata cache. If not, the system will automatically turn to the key-value database for query. The key-value database stores all possible directory paths and their corresponding target result data. After the system retrieves the target result data in the database, it will return it to the user. Importantly, while returning the target result data, the system will also update the metadata cache based on this data.

[0089] When faced with a directory path resolution request, the embodiment of the present application can automatically retrieve target result data that matches the directory path from the key-value database if the requested directory does not exist in the metadata cache in advance. This mechanism ensures that even in the face of a cache miss, the data required by the user can still be quickly located and returned, thereby greatly improving the flexibility and reliability of data retrieval. At the same time, after returning the target result data, the metadata cache will be automatically updated based on this data, and the newly accessed directory path will be included in the cache management. This update strategy not only enriches the cache content, but also improves the response speed of subsequent identical or similar requests, effectively reducing system load and response delays.

[0090] In some optional implementations of this embodiment, in step S204, after sending the entity identifier to the metadata server so as to query the result data corresponding to the directory from the key-value database through the metadata server, the following steps are specifically included:

[0091] When a directory update operation of the metadata server is monitored, the directory update operation message broadcast by the master node of the metadata server is obtained; the directory update operation message is parsed to obtain the modified directory path and the modified target version number, and the modified directory path is added to the cache invalidation list; the cache check request initiated by the metadata client is received, and the cache version number and the access file path carried in the cache check request are obtained, and the cache check request is triggered by the directory update operation message; the cache version number is compared with the target version number, and if the cache version number is less than the target version number, it is determined whether the access file path exists in the cache invalidation list; if so, a response message containing the cache invalidation list is returned to the metadata client, so that the metadata client invalidates the corresponding cache entry in the metadata cache according to the cache invalidation list.

[0092] Among them, in the distributed metadata service architecture, the master node (for example, the leader node) is the core node responsible for coordinating and managing metadata update operations. It receives update requests from metadata clients and broadcasts update operation messages to other nodes to ensure the consistency and synchronization of metadata.

[0093] The modified directory path refers to the path of the directory that is modified or affected when performing a directory update operation. It identifies the specific location of the update operation and is key information for subsequent cache invalidation processing and update synchronization.

[0094] The modified target version number refers to the new version number assigned by the metadata server to the updated directory after the directory update operation. It is used to identify the latest changes in the directory status and serves as the basis for cache checking and invalidation processing.

[0095] The cache invalidation list is a list of cache entries that need to be invalidated due to metadata updates. It contains directory paths or file information that are no longer valid due to update operations and is an important mechanism to ensure metadata cache consistency.

[0096] The cache check request is initiated by the metadata client to check whether the metadata entry in its local cache is still valid. It usually contains information such as the cache version number and the access file path as the basis for cache consistency verification.

[0097] The cache version number is used to identify the status of the cache entry in the metadata cache and is compared with the target version number of the metadata server to determine whether the cache entry needs to be invalidated.

[0098] The access file path refers to the path of the file or directory whose cache validity needs to be verified, which is specified by the metadata client when initiating a cache check request. It is an important part of the cache check request.

[0099] Among them, the response message is the result message returned to the metadata client by the metadata server after receiving the cache check request, judging whether the cache entry is valid based on the cache version number and access file path in the cache check request. It can contain key information such as the cache invalidation list.

[0100] Among them, invalidation processing refers to the process in which the metadata client, after receiving a response message containing a cache invalidation list, marks the corresponding cache entry in its local cache as invalid or deleted according to the information in the list. It is an important step to ensure the consistency and validity of the metadata cache.

[0101] In an example, the specific application of the technical solution of this embodiment is described in detail with the financial transaction system in the financial field as the background. In the financial transaction system, the metadata server is responsible for managing the directory structure and file metadata of the system. In order to improve data access efficiency, the directory information of the file system can be stored in the metadata cache using a hash table structure. However, when a directory update operation occurs on the metadata server, how to ensure that the data in the metadata cache remains consistent and valid becomes a key issue. This embodiment proposes a solution: when a directory update operation is detected, a directory update operation message broadcast by the metadata server master node is obtained. The directory update operation message contains the modified directory path and the modified target version number. After receiving this message, the system parses the message and adds the modified directory path to the cache invalidation list. Subsequently, when the metadata client initiates a cache check request, the system obtains the cache version number and the access file path carried in the cache check request. This cache check request is usually triggered by a directory update operation message to ensure that the metadata client can obtain the latest cache status in a timely manner. The system can compare the cache version number provided by the metadata client with the target version number. If the cache version number is less than the target version number, it means that the data in the metadata cache is outdated and needs further inspection. At this time, the system will determine whether the accessed file path exists in the cache invalidation list. If it does, it means that the corresponding directory has been invalidated due to metadata update, and the system will return a response message containing the cache invalidation list to the client.

[0102] The embodiment of the present application can ensure the timeliness and accuracy of the metadata cache update mechanism by real-time monitoring of the directory update operation of the metadata server. When the directory is updated, the message broadcast by the master node is effectively captured and parsed, so as to accurately obtain the modified directory path and target version number. This mechanism avoids the problem of data inconsistency caused by cache data lag. By adding the modified directory path to the cache invalidation list, the present embodiment realizes the accurate marking of invalid cache entries. When the metadata client initiates a cache check request, the system can quickly compare the cache version number with the target version number, and judge the validity of the access file path based on the cache invalidation list. If the cache data is outdated and the access path exists in the invalidation list, the system can immediately return a response message containing the invalidation list to the client. This feedback mechanism prompts the metadata client to clean up and update the metadata cache in a timely manner, thereby ensuring the accuracy and efficiency of data access. Overall, the present embodiment can significantly improve the reliability and system performance of metadata management in scenarios such as finance and big data processing.

[0103] It should be emphasized that in order to further ensure the privacy and security of the directory information and result data of the above-mentioned file system, the directory information and result data of the above-mentioned file system can also be stored in a node of a blockchain.

[0104] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0106] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0107] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a directory path parsing device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0108] like Figure 3 As shown, the directory path parsing device 400 of this embodiment includes: an information acquisition module 401, a request receiving module 402, an identifier acquisition module 403, a sending module 404 and a returning module 405. Among them:

[0109] The information acquisition module 401 is used to acquire the directory information of the file system and create a metadata cache inside the metadata client, and the metadata cache is used to store the directory information;

[0110] The request receiving module 402 is used to receive a directory path resolution request and determine whether the directory accessed by the directory path resolution request exists in the metadata cache based on the directory information;

[0111] The identifier acquisition module 403 is used to acquire the entity identifier corresponding to the directory from the metadata cache if the directory accessed by the directory path resolution request exists in the metadata cache;

[0112] The sending module 404 is used to send the entity identifier to the metadata server, so as to query the result data corresponding to the directory from the key-value database through the metadata server;

[0113] The return module 405 is used to return the result data corresponding to the directory path resolution request.

[0114] The embodiment of the present application can cache the directory information of the file system inside the metadata client by introducing a metadata cache mechanism (metadata cache), thereby significantly optimizing the process of directory path resolution. When a directory path resolution request is received, the directory can be quickly found in the metadata cache, avoiding the cumbersome process of accessing directory items step by step in the traditional method. If the directory exists in the metadata cache, its corresponding entity identifier can be directly obtained, and a query request can be quickly initiated to the metadata server, and then the required result data can be retrieved from the key-value database. This innovative design not only greatly reduces the delay of directory path resolution, but also effectively solves the problem of near-root hotspots. Since the directory information is stored in the metadata cache, for frequently accessed near-root directories, a quick response can be given without the need for repeated path resolution.

[0115] In one embodiment, the information acquisition module 401 includes:

[0116] The first acquisition submodule is used to acquire metadata client configuration information from the configuration center and create a metadata cache according to the metadata client configuration information.

[0117] The embodiment of the present application can achieve high flexibility and configurability in metadata management by dynamically acquiring the configuration information of the metadata client from the configuration center. The solution first ensures that the creation process of the metadata cache is based on the latest and accurate configuration information, which effectively avoids the problem of inconsistent metadata caused by configuration lags or errors. By automatically pulling configurations from the configuration center, manual intervention is reduced and the maintainability and reliability of the system are improved.

[0118] In one embodiment, the request receiving module 402 includes:

[0119] The second acquisition submodule is used to obtain the directory names of all levels of the directory path in the directory path resolution request;

[0120] The judgment submodule is used to judge whether the directories at all levels of the directory path exist in the metadata cache based on the directory information and the directory name as the search key through hash search.

[0121] The embodiment of the present application can achieve detailed division and identification of the directory structure by accurately obtaining the directory names at all levels in the directory path resolution request, providing an accurate data basis for subsequent operations. Subsequently, the metadata cache is used as a carrier for information retrieval, combined with the efficiency of hash search, and the directory name is used as the search key for rapid matching. This strategy significantly improves the efficiency of directory existence judgment and reduces system response time. Through the introduction of hash search, not only the complexity of traditional linear search is reduced, but also the uniformity and rapid positioning ability of the hash function in data distribution are fully utilized, making the retrieval of large-scale metadata cache faster and more accurate. In addition, the scheme also implies the effective use of the metadata cache, which reduces the overhead of repeated queries through the caching mechanism, greatly reduces the delay of directory path resolution, effectively solves the problem of near-root hotspots, and further improves the performance and stability of the overall system.

[0122] In one embodiment, the judgment submodule is also used to use the directory name as a search key to match the search key with the directory name key of the directory information; if the search key matches the directory name key, it is determined that the directories at all levels of the directory path exist in the metadata cache; if the search key does not match the directory name key, it is determined that the directories at all levels of the directory path do not exist in the metadata cache.

[0123] The embodiment of the present application can use the directory name as the search key to accurately match the directory name key in the metadata cache, thereby achieving efficient judgment on the existence of directories at all levels of the directory path. Specifically, when the search key and the directory name key are completely consistent, it can quickly confirm that all levels of directories in the directory path have been cached, thereby avoiding unnecessary disk access or network requests, and significantly improving the speed and efficiency of data retrieval. Conversely, if the search key does not match any level of directory name key, it can be immediately determined that the corresponding level of directory of the directory path does not exist in the cache. By reducing redundant search operations, the overall load of the system is reduced, and the stability and response speed of the system are improved.

[0124] In one embodiment, the identifier acquisition module 403 includes:

[0125] An operation submodule, for performing a hash operation on the directory path based on the directory path information of the directory path by using a hash algorithm to obtain a hash value of the directory path;

[0126] The search submodule is used to search for the corresponding cache entry in the metadata cache according to the hash value of the directory path;

[0127] The third acquisition submodule is used to obtain the entity identifier corresponding to the directory from the cache entry if a cache entry corresponding to the directory path is found in the metadata cache.

[0128] The embodiment of the present application can convert a lengthy directory path into a fixed-length hash value through a hash algorithm operation based on the directory path information. This conversion process not only simplifies the subsequent retrieval steps, but also ensures the uniqueness of the hash value corresponding to each directory path, effectively avoiding the retrieval confusion that may be caused by similar directory paths. Compared with the traditional string matching method, the retrieval speed is significantly improved by using the hash value to search in the metadata cache, which reduces the system processing time and improves the overall operating efficiency. At the same time, the introduction of the metadata cache enables the mapping relationship between the directory path and the entity identifier to be quickly stored and read, further speeding up the data access speed.

[0129] In one embodiment, the directory path parsing device further includes:

[0130] A query module, used for querying target result data corresponding to the directory path from the key-value database if the directory accessed by the directory path resolution request does not exist in the metadata cache;

[0131] The update module is used to return the target result data corresponding to the directory path resolution request and update the metadata cache based on the target result data.

[0132] When faced with a directory path resolution request, the embodiment of the present application can automatically retrieve target result data that matches the directory path from the key-value database if the requested directory does not exist in the metadata cache in advance. This mechanism ensures that even in the face of a cache miss, the data required by the user can still be quickly located and returned, thereby greatly improving the flexibility and reliability of data retrieval. At the same time, after returning the target result data, the metadata cache will be automatically updated based on this data, and the newly accessed directory path will be included in the cache management. This update strategy not only enriches the cache content, but also improves the response speed of subsequent identical or similar requests, effectively reducing system load and response delays.

[0133] In one embodiment, the directory path parsing device further includes:

[0134] A monitoring module, used for obtaining a directory update operation message broadcasted by a master node of the metadata server when a directory update operation of the metadata server is detected;

[0135] A parsing module is used to parse the directory update operation message, obtain the modified directory path and the modified target version number, and add the modified directory path to the cache invalidation list;

[0136] A receiving module is used to receive a cache check request initiated by a metadata client, and obtain a cache version number and an access file path carried in the cache check request, wherein the cache check request is triggered by a directory update operation message;

[0137] A comparison module is used to compare the cache version number with the target version number. If the cache version number is smaller than the target version number, it is determined whether the access file path exists in the cache invalidation list.

[0138] The message return module is used to return a response message containing a cache invalidation list to the metadata client if it exists, so that the metadata client invalidates the corresponding cache entry in the metadata cache according to the cache invalidation list.

[0139] The embodiment of the present application can ensure the timeliness and accuracy of the metadata cache update mechanism by real-time monitoring of the directory update operation of the metadata server. When the directory is updated, the message broadcast by the master node is effectively captured and parsed, so as to accurately obtain the modified directory path and target version number. This mechanism avoids the problem of data inconsistency caused by cache data lag. By adding the modified directory path to the cache invalidation list, the present embodiment realizes the accurate marking of invalid cache entries. When the metadata client initiates a cache check request, the system can quickly compare the cache version number with the target version number, and judge the validity of the access file path based on the cache invalidation list. If the cache data is outdated and the access path exists in the invalidation list, the system can immediately return a response message containing the invalidation list to the client. This feedback mechanism prompts the metadata client to clean up and update the metadata cache in a timely manner, thereby ensuring the accuracy and efficiency of data access. Overall, the present embodiment can significantly improve the reliability and system performance of metadata management in scenarios such as finance and big data processing.

[0140] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0141] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0142] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0143] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the directory path resolution method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0144] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a directory path resolution method.

[0145] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0146] The embodiment of the present application can cache the directory information of the file system inside the metadata client by introducing a metadata cache mechanism (metadata cache), thereby significantly optimizing the process of directory path resolution. When a directory path resolution request is received, the directory can be quickly found in the metadata cache, avoiding the cumbersome process of accessing directory items step by step in the traditional method. If the directory exists in the metadata cache, its corresponding entity identifier can be directly obtained, and a query request can be quickly initiated to the metadata server, and then the required result data can be retrieved from the key-value database. This innovative design not only greatly reduces the delay of directory path resolution, but also effectively solves the problem of near-root hotspots. Since the directory information is stored in the metadata cache, for frequently accessed near-root directories, a quick response can be given without the need for repeated path resolution.

[0147] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the directory path resolution method as described above.

[0148] The embodiment of the present application can cache the directory information of the file system inside the metadata client by introducing a metadata cache mechanism (metadata cache), thereby significantly optimizing the process of directory path resolution. When a directory path resolution request is received, the directory can be quickly found in the metadata cache, avoiding the cumbersome process of accessing directory items step by step in the traditional method. If the directory exists in the metadata cache, its corresponding entity identifier can be directly obtained, and a query request can be quickly initiated to the metadata server, and then the required result data can be retrieved from the key-value database. This innovative design not only greatly reduces the delay of directory path resolution, but also effectively solves the problem of near-root hotspots. Since the directory information is stored in the metadata cache, for frequently accessed near-root directories, a quick response can be given without the need for repeated path resolution.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0150] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0151] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A directory path parsing method, characterized in that: The steps include: Acquire directory information of the file system, and create a metadata cache inside the metadata client, wherein the metadata cache is used to store the directory information; receiving a directory path resolution request, and determining, based on the directory information, whether a directory accessed by the directory path resolution request exists in the metadata cache; If the directory accessed by the directory path resolution request exists in the metadata cache, obtaining an entity identifier corresponding to the directory from the metadata cache; Sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key-value database through the metadata server; The result data corresponding to the directory path resolution request is returned.

2. The method according to claim 1, characterized in that The directory information is stored in the metadata cache using a hash table structure; The step of determining, based on the directory information, whether the directory accessed by the directory path resolution request exists in the metadata cache specifically includes: Obtain the directory names of each level of the directory path in the directory path resolution request; Based on the directory information, the directory name is used as a search key, and a hash search is performed to determine whether directories at all levels of the directory path exist in the metadata cache.

3. The method according to claim 2, characterized in that The step of determining whether directories at all levels of the directory path exist in the metadata cache based on the directory information and using the directory name as a search key through a hash search specifically includes: Using the directory name as a search key, matching the search key with the directory name key of the directory information; If the search key matches the directory name key, determining that directories at all levels of the directory path exist in the metadata cache; If the search key does not match the directory name key, it is determined that the directories at all levels of the directory path do not exist in the metadata cache.

4. The method according to claim 2, characterized in that: The step of obtaining the entity identifier corresponding to the directory from the metadata cache specifically includes: Based on the directory path information of the directory path, a hash operation is performed on the directory path using a hash algorithm to obtain a hash value of the directory path; According to the hash value of the directory path, searching for a corresponding cache entry in the metadata cache; If a cache entry corresponding to the directory path is found in the metadata cache, an entity identifier corresponding to the directory is obtained from the cache entry.

5. The method according to claim 2, characterized in that: After the step of determining, based on the directory information, whether the directory accessed by the directory path resolution request exists in the metadata cache, the method further includes: If the directory accessed by the directory path resolution request does not exist in the metadata cache, querying the target result data corresponding to the directory path from the key-value database; The target result data corresponding to the directory path resolution request is returned, and the metadata cache is updated based on the target result data.

6. The method according to claim 1, characterized in that The step of creating a metadata cache inside the metadata client specifically includes: The metadata client configuration information is obtained from the configuration center, and a metadata cache is created according to the metadata client configuration information.

7. The method according to claim 1, characterized in that After the step of sending the entity identifier to the metadata server to query the result data corresponding to the directory from the key-value database through the metadata server, the method further includes: When a directory update operation of the metadata server is monitored, obtaining a directory update operation message broadcasted by a master node of the metadata server; Parsing the directory update operation message to obtain a modified directory path and a modified target version number, and adding the modified directory path to a cache invalidation list; Receive a cache check request initiated by the metadata client, and obtain a cache version number and an access file path carried in the cache check request, wherein the cache check request is triggered by the directory update operation message; Compare the cache version number with the target version number, and if the cache version number is smaller than the target version number, determine whether the access file path exists in the cache invalidation list; If so, a response message including the cache invalidation list is returned to the metadata client, so that the metadata client invalidates the corresponding cache entry in the metadata cache according to the cache invalidation list.

8. A directory path parsing device, characterized in that: include: An information acquisition module, used to acquire directory information of the file system, and to create a metadata cache inside the metadata client, wherein the metadata cache is used to store the directory information; A request receiving module, configured to receive a directory path resolution request, and determine, based on the directory information, whether a directory accessed by the directory path resolution request exists in the metadata cache; an identifier acquisition module, configured to acquire an entity identifier corresponding to the directory from the metadata cache if the directory accessed by the directory path resolution request exists in the metadata cache; A sending module, used for sending the entity identifier to a metadata server, so as to query result data corresponding to the directory from a key-value database through the metadata server; The return module is used to return the result data corresponding to the directory path resolution request.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the directory path resolution method as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the directory path resolution method according to any one of claims 1 to 7 are implemented.

Images