Directory traversal method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a directory traversal method and device, a storage medium and electronic equipment, and relates to the field of computers.The directory traversal method comprises the steps that a directory traversal request sent by a client side is received, the directory traversal request carries an offset parameter, and the offset parameter is used for indicating the initial position of traversal of a directory; and under the condition that the offset parameter is greater than 0 and the file index information matched with the offset parameter exists in the cache, reading N directory entries from the cache based on the file index information, and returning the N directory entries to the client, N being a positive integer greater than 1.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a directory traversal method and device, a storage medium, and an electronic device. Background Art

[0002] In the related technology, after the client executes the Is command (used to list the files and subdirectories under the current directory), it sends a readdir (the NFS server's folder reading interface) request to the nfs (network file system) server; after nfs receives the request, it first determines whether the directory cache is invalid. If invalid, it first clears the directory cache and then rebuilds the nfs directory cache; then the nfs server determines the starting position from the cache based on the offset (indicating the offset of the starting position of the directory read) sent by the client, and then reads several files sequentially and sends them to the client.

[0003] In the above process, since the interface needs to be called once for each file when rebuilding the cache, and a cache needs to be created for each file, multiple interfaces need to be called according to the file names of each file during the cache creation process. As a result, when there are too many directory files, the cache building process takes too long, thereby affecting the directory access speed, and further resulting in low directory traversal efficiency.

[0004] Therefore, there is a technical problem of low efficiency of directory traversal in the related technology. Summary of the invention

[0005] The embodiments of the present application provide a directory traversal method and device, a storage medium, and an electronic device to at least solve the technical problem of poor directory traversal effect in the related art.

[0006] According to one embodiment of the present application, a directory traversal method is provided, including: receiving a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate the starting position of the traversal directory; when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, N directory entries are read from the cache based on the file index information, and the N directory entries are returned to the client, wherein N is a positive integer greater than 1.

[0007] As an optional solution, after receiving a directory traversal request sent by the client, the method also includes: when the offset parameter is greater than 0 and there is no file index information matching the offset parameter in the cache, using the metadata service interface to obtain file index information matching the offset parameter; using the file index information to update the cache, and based on the file index information, reading N directory entries from the cache, and returning the N directory entries to the client.

[0008] As an optional solution, after receiving a directory traversal request sent by the client, the method also includes: when the offset parameter is equal to 0, using the metadata service interface to obtain file index information that matches the offset parameter; based on the file index information, reading N directory entries from the cache, and returning the N directory entries to the client.

[0009] As an optional solution, after using the metadata service interface to obtain file index information matching the offset parameter, the method also includes: when a new cache identifier is generated in the cache, updating the offset parameter using the cache identifier, wherein the first three fields of the updated offset parameter are the cache identifier.

[0010] As an optional solution, before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, the method also includes: establishing a key-value pair relationship between the offset parameter and the file index information, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information; and storing the key-value pair relationship in the cache.

[0011] As an optional solution, after reading N directory items from the cache based on the file index information and returning the N directory items to the client, the method also includes: recording the historical time information of the most recent reading of the N directory items; performing a timestamp check on the N directory items at every preset time period, wherein the timestamp check is used to check whether the time difference between the historical time information and the current time exceeds a preset time value; and deleting from the cache directory items whose time difference exceeds the preset time value.

[0012] As an optional solution, N directory entries are read from the cache based on the file index information and the N directory entries are returned to the client, including: batch reading the N directory entries indicated by the file index information from the cache and filling the specified data structure with the N directory entries; sending the specified data structure to the client.

[0013] According to another embodiment of the present application, a directory traversal device is provided, including: a receiving unit, used to receive a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate the starting position of the traversal directory; a traversal unit, used to read N directory entries from a cache based on the file index information when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, and return the N directory entries to the client, wherein N is a positive integer greater than 1.

[0014] As an optional solution, the device also includes: a first acquisition module, which is used to use the metadata service interface to obtain file index information matching the offset parameter after receiving a directory traversal request sent by the client, when the offset parameter is greater than 0 and there is no file index information matching the offset parameter in the cache; a first traversal module, which is used to update the cache using the file index information after receiving a directory traversal request sent by the client, and read N directory items from the cache based on the file index information, and return the N directory items to the client.

[0015] As an optional solution, the device also includes: a second acquisition module, which is used to use the metadata service interface to obtain file index information matching the offset parameter after receiving the directory traversal request sent by the client and when the offset parameter is equal to 0; a second traversal module, which is used to read N directory items from the cache based on the file index information after receiving the directory traversal request sent by the client, and return the N directory items to the client.

[0016] As an optional solution, the device also includes: an update module, which is used to update the offset parameter using the cache identifier after using the metadata service interface to obtain file index information matching the offset parameter and generating a new cache identifier in the cache, wherein the first three fields of the updated offset parameter are the cache identifier.

[0017] As an optional solution, the device also includes: an establishment module, which is used to establish a key-value pair relationship between the offset parameter and the file index information before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information; a storage module, which is used to store the key-value pair relationship in the cache before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client.

[0018] As an optional solution, the device also includes: a recording module, which is used to record the historical time information of the most recent reading of the N directory items after reading N directory items from the cache based on the file index information and returning the N directory items to the client; a checking module, which is used to perform a timestamp check on the N directory items every preset time period after reading N directory items from the cache based on the file index information and returning the N directory items to the client, wherein the timestamp check is used to check whether the time difference between the historical time information and the current time exceeds a preset time value; a deleting module, which is used to delete the directory items whose time difference exceeds the preset time value from the cache after reading N directory items from the cache based on the file index information and returning the N directory items to the client.

[0019] As an optional solution, the traversal unit includes: a filling module for batch reading N directory entries indicated by file index information from the cache and filling the N directory entries with a specified data structure; a sending module for sending the specified data structure to the client.

[0020] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0021] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0022] Through the embodiment provided by the present application, a directory traversal request sent by a client is received, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate the starting position of the traversal directory; when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, N directory items are read from the cache based on the file index information, and the N directory items are returned to the client, wherein N is a positive integer greater than 1. In this process, the server cache construction process is omitted, and the directory items indicated by the N file index information are directly read in batches from the file cache, thereby avoiding the defect of low directory traversal speed due to time-consuming cache components, achieving the purpose of accelerating directory access speed, thereby achieving the technical effect of improving the efficiency of directory traversal, and solving the technical problem of low efficiency of directory traversal. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a hardware structure block diagram of a directory traversal method according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of a directory traversal method according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a directory traversal method according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a directory traversal method according to an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a directory traversal method according to an embodiment of the present application;

[0028] Figure 6 It is a structural block diagram of a directory traversal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a directory traversal method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the mapping relationship in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] As an optional solution, the directory traversal method includes the following steps:

[0035] S202, receiving a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate a starting position of the traversal directory;

[0036] S204: When the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, N directory entries are read from the cache based on the file index information and the N directory entries are returned to the client, where N is a positive integer greater than 1.

[0037] Optionally, in this embodiment, the directory traversal method can be applied to, but not limited to, a scenario of traversing a large directory based on a read folder interface. In this scenario, after the client executes the ls command (used to list the files and subdirectories under the current directory), a readdir (read folder interface) request can be sent to the nfs server. After the nfs (network file system) receives the request, it first determines whether the cache of the directory is invalid. If invalid, the directory cache is first cleared, and then the nfs-side directory cache is rebuilt.

[0038] Optionally, the directory cache construction process is: first call opendir to open the directory, then call the icfs_readdirplus_r interface in a loop, read a file from 1ib each time, and obtain the file's inode pointer by further calling the icfs_lIookup interface according to the file name, and build the corresponding cache. When all files are read, call releasedir to close the directory; then the nfs server determines the starting position from the cache according to the offset sent by the client, and then reads several files in sequence, calls getattr to refresh the file attributes, and then sends the read file information to the client; after the client receives the readdir return, it continues to send readdir requests to read the remaining files in the directory.

[0039] In this process, since the icfs_readdirplus_r interface needs to be called once for each file when rebuilding the cache, and a cache needs to be created for each file, the getattr and Iookup interfaces need to be called according to the file name of each file during the cache creation process. This will cause the cache building process to take too long when there are too many directory files, thus affecting the directory access speed.

[0040] For the above defects, the directory traversal method provided in this embodiment is used to omit the process of building a cache for each file on the nfs side, and directly read files in batches from the lib library cache and return them to the client, thereby avoiding the defect of low directory traversal speed due to time-consuming cache components, achieving the purpose of speeding up directory access speed, and thus realizing the technical effect of improving the efficiency of directory traversal.

[0041] Optionally, in this embodiment, the directory traversal request sent by the client may be, but is not limited to, a Readdir request triggered when the client executes an ls command, for requesting the nfs server to read and traverse N directory entries, where N is a positive integer greater than 1.

[0042] Optionally, in this embodiment, the directory traversal request may, but is not limited to, carry an offset parameter (offset) for indicating the starting position of the traversal directory.

[0043] It is understood that the offset parameter refers to an offset relative to the starting position of the data, usually in bytes. When reading a file or directory, the offset parameter can be used to specify the position in the file from which to start reading data.

[0044] Optionally, in this embodiment, when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, N directory entries are read from the lib library cache based on the file index information and the N directory entries are returned to the client.

[0045] It is understandable that if Figure 3 As shown in the figure, the process of readdir in the lib library follows the following flow: when readingdir, first determine whether the offset parameter offset is equal to 0. If offset>0, first determine whether it hits the cache, that is, determine whether there is file index information matching the offset parameter from the lib library cache. If it exists in the cache, it hits the cache, and returns the read dir information (directory entry information) from the cache.

[0046] It should be noted that a key-value pair relationship between the offset parameter and the file index information is pre-established, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information; and the key-value pair relationship is stored in the cache.

[0047] To further illustrate, Figure 3 As shown in the figure, when offset>0, if there is no matching file index information in the cache, that is, the cache is not hit, the metadata service interface mds is used to read the matching file index information, and then the lib library cache is updated, and the dir information is returned from the cache. If offset is equal to 0, the dir information is obtained first, and then the lib library generates a new readdir cache identifier and puts the identifier code in the highest 3 bits of offset.

[0048] Through the embodiments provided in the present application, the server-side cache building process is omitted, and directory items indicated by N file index information are directly read in batches from the file cache, thereby avoiding the defect of low directory traversal speed due to time-consuming cache components, achieving the purpose of speeding up directory access speed, and realizing the technical effect of improving the efficiency of directory traversal.

[0049] As an optional solution, after receiving the directory traversal request sent by the client, the method further includes:

[0050] When the offset parameter is greater than 0 and there is no file index information matching the offset parameter in the cache, the metadata service interface is used to obtain the file index information matching the offset parameter;

[0051] The cache is updated using the file index information, and N directory entries are read from the cache based on the file index information, and the N directory entries are returned to the client.

[0052] Optionally, in this embodiment, the metadata service interface MDS is responsible for managing and providing metadata information of the file system in the distributed file system. Metadata refers to data that describes data, such as the attributes of files and directories, including information such as permissions, owners, creation time, modification time, size and location. In the context of NFS (Network File System), MDS can be, but is not limited to, specifically referring to a server component used to process these metadata requests, maintain an index of the file system directory structure, and provide this information when a client requests the attributes of a file or directory.

[0053] It can be understood that in this embodiment, when processing the readdir request of the client, the metadata service interface MDS is used to ensure that the directory information is up-to-date, and can read the information from the underlying file system and update the cache when it is not in the cache.

[0054] Through the embodiments provided by the present application, when the offset parameter is greater than 0 and there is no file index information matching the offset parameter in the cache, the metadata service interface is used to obtain the file index information matching the offset parameter, ensuring that the directory information is up to date, and when there is no file index information in the cache, the information can be read from the underlying file system and the cache can be updated. This process does not require the construction of additional components, thereby avoiding the defect of low directory traversal speed due to time-consuming cache components, achieving the purpose of speeding up directory access speed, and thus realizing the technical effect of improving the efficiency of directory traversal.

[0055] As an optional solution, after receiving the directory traversal request sent by the client, the method further includes:

[0056] When the offset parameter is equal to 0, the metadata service interface is used to obtain the file index information matching the offset parameter;

[0057] Based on the file index information, N directory entries are read from the cache and returned to the client.

[0058] Optionally, in this embodiment, when the offset parameter is equal to 0, it indicates that a new client requests a directory traversal from the lib library cache, and uses the metadata service interface to obtain file index information matching the offset parameter.

[0059] As an optional solution, after using the metadata service interface to obtain file index information matching the offset parameter, the method further includes:

[0060] When a new cache identifier is generated in the cache, the cache identifier is used to update the offset parameter, wherein the first three fields of the updated offset parameter are the cache identifier.

[0061] Optionally, in this embodiment, when a new cache identifier is generated in the cache, the cache identifier is determined as the highest 3 bits of the offset parameter, that is, the first three bits of the updated offset parameter are the cache identifier.

[0062] It is understandable that if Figure 4 As shown in the figure, the client starts reading from offset = 0, and the lib library will consider it as a new client. First, the readdir cache identifier in the lib library is obtained; if size = 0, the 0×001 identifier is directly generated; if there is a client identifier, that is, size is not 0, it is further determined whether the readdir identifier cache is full. If it is full, the offset is no longer updated, and the highest 3 bits of the offset are 0×000. If the readdir identifier cache is not full, a minimum identifier code that does not exist in the cache will be generated.

[0063] Through the embodiment provided by the present application, when a new cache identifier is generated in the cache, the offset parameter is updated using the cache identifier, wherein the first three fields of the updated offset parameter are the cache identifier. This process does not require the construction of additional components, thereby avoiding the defect of low directory traversal speed due to time-consuming cache components, achieving the purpose of speeding up directory access speed, and thus achieving the technical effect of improving the efficiency of directory traversal.

[0064] As an optional solution, before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, the method further includes:

[0065] Establishing a key-value pair relationship between the offset parameter and the file index information, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information;

[0066] Store key-value pairs in cache.

[0067] Optionally, in this embodiment, a key-value pair relationship between the offset parameter and the file index information can be established in the form of, but not limited to, a key-value pair, and the key-value pair relationship is stored in the cache as a strategy for optimizing data storage and retrieval efficiency. A key-value pair is a form of data structure, in which a "key" is an index used to uniquely identify a data item, and a "value" is specific data associated with the key.

[0068] Through the embodiments provided by the present application, since the key is unique, the corresponding value can be directly searched through the key, avoiding the need to traverse the entire data set, thereby improving the retrieval speed. The key-value pair storage structure can effectively reduce storage redundancy, especially when processing a large amount of metadata, storing in this way can save space. This optimization is particularly important for improving the efficiency of traversing large directories with the ls command (listing directory contents) because it avoids repeated data processing and transmission, reducing network latency and server load. By quickly retrieving and verifying metadata information, client requests can be responded to more quickly, thereby improving user experience and system performance.

[0069] As an optional solution, after reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, the method further includes:

[0070] Record the historical time information of the last read of N directory entries;

[0071] Perform a timestamp check on N directory entries at every preset time period, wherein the timestamp check is used to check whether the time difference between the historical time information and the current time exceeds a preset time value;

[0072] Delete directory entries from the cache whose time difference exceeds the preset time value.

[0073] Optionally, in this embodiment, whenever the client sends a readdir request and processes the directory information, the lib library will record the last access time or update time of the directory. A timer (tick mechanism) will run inside the lib library. Normally, this timer will run every 1 second, but it can be configured to meet the needs of different systems. Every time the tick runs, the lib library will check the time difference between the last access time and the current time of all directory caches. If this time difference exceeds the set expiration time (the default is 10 seconds, but it can also be configured according to demand), the directory cache is considered expired. Once the directory cache is detected to be expired, the lib library will clear the cache information of the directory and release the corresponding memory space to make room for new or more frequently accessed directory caches.

[0074] Through the embodiments provided by the present application, the lib library can dynamically manage the cache to ensure that the data in the cache is the latest, while also avoiding excessive occupation of limited memory resources by cached data, thereby improving the overall efficiency and performance of the system. By setting the expiration policy, it can be adjusted according to the actual needs of the system. For example, in scenarios with high concurrency and frequent access to large directories, the expiration time can be appropriately reduced to update the cached data faster and improve the response speed. In an environment where cache space is tight, the frequent updates of the cache can be reduced by extending the expiration time to save memory resources.

[0075] As an optional solution, N directory entries are read from the cache based on the file index information and returned to the client, including:

[0076] Batch read N directory entries indicated by file index information from the cache, and fill the specified data structure with the N directory entries;

[0077] Sends the specified data structure to the client.

[0078] Optionally, in this embodiment, the N directory items (such as file name, file size, file type, file permission, etc.) indicated by the file index information can be organized into a specified data structure, but not limited to, for subsequent processing or display. The specified data structure can be an array, a linked list, a hash table, a tree structure (such as a directory tree), etc. The specific choice depends on the specific application scenario and performance requirements.

[0079] In the context of NFS (Network File System) or similar file systems, the specified data structure may include, but is not limited to, at least one of the following:

[0080] Directory Entry List: Store the read file information in the form of directory entries in an array or linked list. Each directory entry contains the file name and some basic file attributes.

[0081] Hash Map: A hash table can be used to store file information, with the file name as the key and other file information (such as inode, file size, etc.) as the value, in order to quickly find files.

[0082] Tree Structure: Use a tree structure to reflect the relationship between directories and subdirectories, as well as the files under each directory, to facilitate efficient processing of directory hierarchies. It is very efficient for displaying file system trees.

[0083] Custom Structured Data: Based on specific application requirements, you can design a custom structure to store the read file information. For example, if you need to store additional custom metadata, you can add corresponding fields to the structure.

[0084] It should be noted that the selection of a specified data structure can take into account, but is not limited to, the following factors: Performance requirements: For example, whether it is necessary to quickly find files, whether it is necessary to quickly traverse the file list, etc. Memory usage: Different data structures have different performances in memory usage, and memory efficiency should be considered when selecting. Subsequent processing: Select an appropriate data structure based on the subsequent processing method of the file information, for example, whether sorting, filtering, etc. are required. System compatibility: The selection of the data structure should take into account the compatibility and optimization of the target operating system.

[0085] Through the embodiments provided in the present application, batch-read file information is filled into a specified structure, which can more efficiently manage and transmit file information, reduce network transmission and processing delays, and improve the efficiency of directory traversal.

[0086] As an optional solution, the above directory traversal method is applied to the scenario of optimizing the read folder interface to improve the efficiency of ls traversing large directories. In this scenario, a method of optimizing Readdir to improve the efficiency of ls traversing large directories is designed. When the client executes ls, a readdir request is sent to the nfs server. After receiving the readdir request, the nfs server first calls opendir to open the directory, locates the read position according to the offset sent by the client through the seekdir interface, and then calls the icfs_readdirplus_r interface to read a certain number of files from the lib library. After a certain number of files are read in batches, releasedir is called to close the directory; then the nfs server sends the read files to the client; after the client receives the response, it continues to send readdir to read the remaining files.

[0087] It should be noted that in the above process, the nfs module removes the cache building process. Seekdir is optimized. When the offset is inconsistent with the last returned offset, the cache data is directly used in the cache and the cache is no longer cleared. An offset and dname index cache mechanism is established to reduce the probability of rereading the directory due to the offset not matching the corresponding dname.

[0088] Alternatively, if Figure 5As shown in the figure, when using ls, the client sends a readdir request to nfs, and the offset starts from 0; then the nfs module calls the opendir interface, and then locates the reading position through seekdir. If it is in the cache, it reads from the cache. If it is not in the cache, it sends a request to mds to obtain it and updates the lib library cache. After Seekdir is completed, the lib library readdir interface is called to read. After the batch reading is completed, the releasedir interface is called to close the directory. After receiving the readdir return, the nfs client continues to send readdir requests. The offset may not be the last offset returned last time.

[0089] Specifically, an optional directory traversal method includes the following steps:

[0090] Step S1, first determine whether the offset starts from 0; if so, read the dir information from mds, and after the reading is completed, continue to step S4; if it does not start from 0, continue to step S2.

[0091] Step S2, determine whether it is in the cache. If it is in the cache, continue to step S6. If it is not in the cache, read mds from 0 and continue to step S3.

[0092] Step S3, determine whether a new identification code needs to be generated, if so, proceed to step S4, if not, update the cache, and proceed to step S6.

[0093] Step S4, determine whether the lib library readdir cache identifier is full, if full (the highest 3 digits of the offfset cache identifier are all occupied), continue to step S6, if not full, continue to step S5.

[0094] Step S5, generate a readdir cache identification code, update the cache information, and continue to execute step S6.

[0095] Step S6, update and return nfsoffset, and put back the read information.

[0096] It should be noted that the readdir process of the lib library will be as follows:

[0097] 1eaddir first checks whether offset is equal to 0. If offset is equal to 0, the lib library generates a new readdir cache identifier and puts the identifier in the highest 3 bits of offset. If offset is greater than 0, it is first obtained from the lib library cache. If it does not exist in the cache, it is read from mds and then updates the lib library cache, returning the read information from the cache.

[0098] It is understandable that this embodiment can optimize the readdir operation in the NFS module by removing the cache building process, directly processing the client's readdir request from the library file, and using the seekdir function to locate the read position according to the offset sent by the NFS client, and reading the file information from the library cache. This method will improve the efficiency of traversing large directories.

[0099] Through the embodiments provided by the present application, by omitting the NFS server cache construction process, a method of directly reading files in batches from the library file cache is used to speed up directory access and improve the efficiency of traversing large directories. At the same time, when the lib cache is updated, the directory with updated readdir cache information will be saved. In the lib library tick logic, every 1 second, it will be obtained whether these directories are expired for 10 seconds (this time can be configured). If it exceeds 10 seconds, the readdir cache related information will be deleted. This will also increase the available cache space to a certain extent, and improve the efficiency and availability of the product to a certain extent.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0101] In this embodiment, a directory traversal device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0102] Figure 6 is a structural block diagram of a directory traversal device according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0103] The receiving unit 602 is used to receive a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate a starting position of the traversal directory;

[0104] The traversal unit 604 is used to read N directory entries from the cache based on the file index information and return the N directory entries to the client when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, where N is a positive integer greater than 1.

[0105] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0106] As an optional solution, the device also includes:

[0107] A first acquisition module is used to, after receiving a directory traversal request sent by a client, use a metadata service interface to acquire file index information matching the offset parameter when the offset parameter is greater than 0 and the file index information matching the offset parameter does not exist in the cache;

[0108] The first traversal module is used to update the cache using the file index information after receiving the directory traversal request sent by the client, and read N directory entries from the cache based on the file index information, and return the N directory entries to the client.

[0109] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0110] As an optional solution, the device also includes:

[0111] The second acquisition module is used to, after receiving the directory traversal request sent by the client, use the metadata service interface to acquire file index information matching the offset parameter when the offset parameter is equal to 0;

[0112] The second traversal module is used to read N directory entries from the cache based on the file index information after receiving the directory traversal request sent by the client, and return the N directory entries to the client.

[0113] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0114] As an optional solution, the device also includes:

[0115] The update module is used to update the offset parameter using the cache identifier after obtaining the file index information matching the offset parameter using the metadata service interface and generating a new cache identifier in the cache, wherein the first three fields of the updated offset parameter are the cache identifier.

[0116] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0117] As an optional solution, the device also includes:

[0118] An establishment module is used to establish a key-value pair relationship between the offset parameter and the file index information before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information;

[0119] The storage module is used to store the key-value pair relationship in the cache before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client.

[0120] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0121] As an optional solution, the device also includes:

[0122] A recording module, for recording the most recent historical time information of the N directory entries after reading the N directory entries from the cache based on the file index information and returning the N directory entries to the client;

[0123] A checking module, configured to read N directory entries from the cache based on the file index information and return the N directory entries to the client, and then perform a timestamp check on the N directory entries at every preset time period, wherein the timestamp check is used to check whether the time difference between the historical time information and the current time exceeds a preset time value;

[0124] The deletion module is used to delete the directory entries whose time difference exceeds a preset time value from the cache after reading N directory entries from the cache based on the file index information and returning the N directory entries to the client.

[0125] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0126] As an optional solution, the traversal unit 604 includes:

[0127] A filling module is used to batch read N directory entries indicated by the file index information from the cache, and fill the N directory entries into a specified data structure;

[0128] The sending module is used to send the specified data structure to the client.

[0129] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0131] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0132] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0133] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0134] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0135] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0136] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores the computer program product, and when the computer program is executed by a processor, the steps of the method in each embodiment of the present application are implemented.

[0137] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0138] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0139] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A directory traversal method, characterized in that: include: Receive a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate a starting position of the traversal directory; When the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, N directory entries are read from the cache based on the file index information and returned to the client, where N is a positive integer greater than 1.

2. The method according to claim 1, characterized in that After receiving the directory traversal request sent by the client, the method further includes: When the offset parameter is greater than 0 and the file index information matching the offset parameter does not exist in the cache, using a metadata service interface to obtain the file index information matching the offset parameter; The cache is updated using the file index information, and the N directory entries are read from the cache based on the file index information, and the N directory entries are returned to the client.

3. The method according to claim 2, characterized in that After receiving the directory traversal request sent by the client, the method further includes: When the offset parameter is equal to 0, using the metadata service interface to obtain the file index information matching the offset parameter; The N directory entries are read from the cache based on the file index information, and the N directory entries are returned to the client.

4. The method according to claim 3, characterized in that After using the metadata service interface to obtain the file index information matching the offset parameter, the method further includes: In the case where a new cache identifier is generated in the cache, the offset parameter is updated using the cache identifier, wherein the first three fields of the updated offset parameter are the cache identifier.

5. The method according to any one of claims 1 to 4, characterized in that: Before reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, the method further includes: Establishing a key-value pair relationship between the offset parameter and the file index information, wherein the key-value pair relationship is used to indicate a one-to-one correspondence between the offset parameter and the file index information; The key-value pair relationship is stored in the cache.

6. The method according to any one of claims 1 to 4, characterized in that: After reading N directory entries from the cache based on the file index information and returning the N directory entries to the client, the method further includes: Record the historical time information of the last time the N directory entries were read; Performing a timestamp check on the N directory entries at every preset time period, wherein the timestamp check is used to check whether the time difference between the historical time information and the current time exceeds a preset time value; The directory entry whose time difference exceeds the preset time value is deleted from the cache.

7. The method according to any one of claims 1 to 4, characterized in that: The reading N directory entries from the cache based on the file index information and returning the N directory entries to the client includes: Batch read the N directory entries indicated by the file index information from the cache, and fill the specified data structure with the N directory entries; The specified data structure is sent to the client.

8. A directory traversal device, characterized in that: include: A receiving unit, configured to receive a directory traversal request sent by a client, wherein the directory traversal request carries an offset parameter, and the offset parameter is used to indicate a starting position of the traversal directory; A traversal unit is used to read N directory entries from the cache based on the file index information when the offset parameter is greater than 0 and there is file index information matching the offset parameter in the cache, and return the N directory entries to the client, wherein N is a positive integer greater than 1.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images