Adaptive memory conversion method and apparatus for distributed file system

Abstract

Embodiments of the present disclosure disclose an adaptive memory conversion method and device for a distributed file system. A specific implementation of the method comprises: dividing a metadata set in a target directory to obtain a metadata subset group; setting a corresponding memory data structure form for each metadata subset; determining a target metadata subset; executing a corresponding access processing flow on the target metadata subset to obtain and output a final access result; obtaining access feature sets and system resource states of each metadata subset in the metadata subset group in real time to obtain a directory index set; determining memory data structure form conversion instructions corresponding to metadata subsets to be converted in the metadata subset group to obtain a conversion instruction set and a to-be-converted metadata subset group; and performing data reconstruction on the to-be-converted metadata subset group to obtain a converted metadata subset group. The implementation can improve memory utilization, reduce redundant memory occupation, and reduce unnecessary data access overhead.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to an adaptive memory translation method and apparatus for a distributed file system. Background Technology

[0002] In large-scale distributed file systems, directory-associated metadata (such as directory entries and their associated inode identifiers, lightweight attribute caches, etc.) is typically resident in the computer's main memory. Traditional methods usually store directory metadata uniformly in a memory structure that supports random operations (such as hash tables), or use methods such as global compression for storage and transformation.

[0003] However, when using the above method to convert memory, the following technical problems often occur: Low memory utilization. The metadata associated with directories can reach billions or even trillions of records. Using online data structures that support random addition, deletion, modification, and querying (such as hash tables and balanced trees) will result in low memory utilization due to inherent pointer overhead, reserved space, and memory fragmentation issues. Furthermore, in typical file system workloads, many directories are inactive most of the time, yet they occupy memory resources for extended periods due to their costly online structure.

[0004] Redundant memory consumption. Existing tiered caching schemes suffer from dual-copy consistency issues. The current system employs a tiered architecture of "compressed storage layer + online hash cache layer" to balance memory efficiency and access performance. The lower layer stores metadata in compressed form, while the upper layer maintains an online hash cache of active data. This means that when data in the cache is modified, the compressed copy at the lower layer must be updated or invalidated synchronously, introducing consistency maintenance overhead and redundant memory consumption.

[0005] This generates unnecessary data access overhead. Existing systems often use probabilistic data structures such as Bloom filters to speed up searches and avoid unnecessary overhead. However, such filtering structures usually exist only as independent optimization components on the search path. Their query statistics (such as the frequency of negative decisions) are not incorporated into the data storage strategy decision-making process, which makes it impossible for the system to determine whether it is necessary to maintain a certain data partition in a compact form with low memory overhead based on the negative search ratio.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide adaptive memory translation methods, apparatuses, electronic devices, and computer-readable media for distributed file systems to address one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide an adaptive memory conversion method for a distributed file system. The method includes: in response to the number of files in a target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, partitioning the metadata dataset in the target directory to obtain a metadata subset group; setting a corresponding memory data structure form for each metadata subset in the metadata subset group according to preset state conditions, wherein the memory data structure form includes online state, flat state, and archive state; determining a target metadata subset based on the metadata operation request; performing a corresponding access processing flow on the target metadata subset according to the memory data structure form corresponding to the target metadata subset, obtaining a final access result, and outputting the final access result; acquiring the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain a directory indicator set; determining a memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group based on the directory indicator set and preset conversion conditions, obtaining a conversion instruction set and a metadata subset group to be converted; and reconstructing the metadata subset group to be converted based on the conversion instruction set to obtain a converted metadata subset group.

[0010] Secondly, some embodiments of this disclosure provide an adaptive memory conversion apparatus for a distributed file system. The apparatus includes: a partitioning unit configured to partition the metadata dataset in the target directory to obtain a metadata subset group in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory; a setting unit configured to set a corresponding memory data structure form for each metadata subset in the metadata subset group according to preset state conditions, wherein the memory data structure form includes an online state, a flat state, and an archive state; a first determining unit configured to determine a target metadata subset based on the metadata operation request; and an execution unit configured to determine the target metadata subset based on the target metadata subset. Based on the memory data structure form corresponding to the subset, the corresponding access processing flow is executed on the target metadata subset to obtain the final access result and output the final access result; the real-time acquisition unit is configured to acquire the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain the directory index set; the second determination unit is configured to determine the memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group based on the directory index set and preset conversion conditions to obtain the conversion instruction set and the metadata subset group to be converted; the data reconstruction unit is configured to reconstruct the metadata subset group to be converted based on the conversion instruction set to obtain the converted metadata subset group.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] The above embodiments of this disclosure have the following beneficial effects: the adaptive memory transformation method for distributed file systems according to some embodiments of this disclosure can improve memory utilization, reduce redundant memory occupation, and reduce unnecessary data access overhead. Specifically, the reasons for low memory utilization, redundant memory occupation, and unnecessary data access overhead are as follows: Low memory utilization. The metadata associated with a directory can reach billions or even trillions of records. Using online data structures that support random addition, deletion, modification, and query (such as hash tables and balanced trees) will result in low memory utilization due to inherent pointer overhead, reserved space, and memory fragmentation problems. In typical file system workloads, a large number of directories are inactive most of the time, yet they occupy memory resources for a long time with high-cost online structures. Redundant memory occupation. Existing layered caching schemes have a dual-copy consistency problem. Existing systems use a layered architecture of "compressed storage layer + online hash cache layer" to balance memory efficiency and access performance. The lower layer stores metadata in compressed form, and the upper layer maintains an online hash cache of active data. When the data in the cache is modified, the compressed copy at the lower layer needs to be updated or invalidated synchronously, introducing consistency maintenance overhead and redundant memory occupation. Unnecessary data access overhead is generated. Existing systems often use probabilistic data structures such as Bloom filters to accelerate searches and avoid unnecessary overhead. However, such filtering structures usually exist only as independent optimization components on the search path, and their query statistics (such as the frequency of negative decisions) are not incorporated into the data storage strategy decision-making process. This makes it impossible for the system to determine whether it is necessary to maintain a data partition in a compact form with low memory overhead based on the negative search ratio. Based on this, some embodiments of the adaptive memory conversion method for distributed file systems disclosed in this disclosure firstly, in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, the metadata dataset in the target directory is partitioned to obtain metadata subset groups. Thus, massive metadata can be divided into smaller, logically independent metadata subsets, avoiding the constraints of unified management of the entire directory and laying the foundation for adaptive management. Secondly, according to preset state conditions, a corresponding memory data structure form is set for each metadata subset in the metadata subset group, wherein the memory data structure form includes online state, flat state, and archive state. Thus, the most suitable storage representation can be selected according to the specific situation of the metadata subset. Secondly, based on the aforementioned metadata operation requests, a target metadata subset is determined. This avoids traversing the entire metadata dataset, narrowing the scope of data processing. Thirdly, according to the memory data structure corresponding to the target metadata subset, the corresponding access processing flow is executed on the target metadata subset to obtain the final access result, which is then output. This avoids unnecessary lookup or decompression overhead, significantly reducing the system's average latency in negative lookup scenarios.Next, the access characteristic set and system resource status of each metadata subset in the aforementioned metadata subset group are acquired in real time to obtain a directory indicator set. This allows for the quantification of the access status and system resource status of each subset. Then, based on the aforementioned directory indicator set and preset conversion conditions, the memory data structure transformation instruction corresponding to at least one metadata subset to be converted in the aforementioned metadata subset group is determined, resulting in a conversion instruction set and the metadata subset group to be converted. This allows for the reduction of long-term idle subsets to a flat state to save memory, or the conversion of flattened subsets to an archive state when memory is scarce. Finally, based on the aforementioned conversion instruction set, the metadata subset group to be converted is reconstructed to obtain the converted metadata subset group. Ultimately, this improves memory utilization, reduces redundant memory usage, and minimizes unnecessary data access overhead. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 This is a flowchart of some embodiments of the adaptive memory translation method for a distributed file system according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the adaptive memory translation apparatus for a distributed file system according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0021] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Figure 1 A flow 100 of some embodiments of the adaptive memory translation method for a distributed file system according to this disclosure is shown. The adaptive memory translation method for a distributed file system includes the following steps: Step 101: In response to the number of files in the target directory exceeding the preset partitioning threshold or receiving a metadata operation request for the target directory, the metadata dataset in the target directory is partitioned to obtain a metadata subset group.

[0023] In some embodiments, the execution entity (e.g., a server) of the adaptive memory transformation method for a distributed file system may, in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, partition the metadata dataset in the target directory to obtain a metadata subset group. The target directory may be a specific directory in the distributed file system containing a large number of files and requiring memory management. For example, the " / datasets / images" directory storing a large number of image files. The metadata dataset may be a collection of metadata corresponding to all files or subdirectories under the target directory. For example, the metadata in the metadata set may include, but is not limited to, directory entries (dentries) and associated inodes (inodes). The preset partitioning threshold may be a pre-set threshold for the number of files used to determine whether the metadata dataset needs to be subsetted. For example, the preset partitioning threshold may be 1000 directory entries. The metadata subset group may be multiple independent metadata subsets obtained after partitioning the metadata. The metadata subsets in the metadata subset group are the basic units for memory management and state transitions. The metadata operation request may be an internal instruction received by the distributed file system to operate on a file or directory. For example, it could be operation commands such as create, query, or delete.

[0024] In practice, the aforementioned execution entity can respond to a situation where the number of files in the target directory exceeds a preset partitioning threshold or a metadata operation request for the target directory is received by dividing the metadata dataset in the target directory into metadata subset groups through the following steps: Step one: In response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, a target partitioning instruction is determined. In practice, the execution entity may generate a target partitioning instruction in response to the number of files in the target directory exceeding the preset partitioning threshold or receiving a metadata operation request for the target directory. This target partitioning instruction may be an instruction to partition the target directory into metadata sets. The target partitioning instruction may include the identifier of the target directory to be partitioned.

[0025] Step two: Based on the aforementioned target partitioning instructions, the metadata dataset in the target directory is partitioned to obtain metadata subset groups. In practice, the executing entity can locate the target directory corresponding to the aforementioned target partitioning instructions. Then, it iterates through each file in the target directory, determines the subset identifier for each file, and groups the files in the target directory according to the subset identifier to obtain metadata subset groups. As an example, the executing entity can take the hash value of the filename modulo N to obtain the subset identifier, and group the metadata of files with the same subset identifier into the same metadata subset. Here, N can represent the number of metadata subsets to be divided into in the target directory.

[0026] Step 102: Based on the preset state conditions, set the corresponding memory data structure form for each metadata subset in the metadata subset group.

[0027] In some embodiments, the aforementioned execution entity can set corresponding memory data structure forms for each metadata subset in the aforementioned metadata subset group according to preset state conditions. These memory data structure forms include online state, flat state, and archived state. The preset state conditions can be pre-set to determine the initial memory data structure form of a newly created metadata subset. For example, the initial memory data structure form of a newly created metadata subset can be online state. Online state can represent a memory data structure form that supports random, efficient CRUD operations, such as using hash tables, balanced trees, or a combination of hash tables and linked lists. Flat state can represent a memory data structure form that stores metadata in contiguous memory blocks arranged in an ordered manner according to keys (such as filenames). Flat state supports precise queries, sequential traversal, and marked deletion, but does not support in-situ addition and in-situ modification. Marked deletion can be done without physically moving metadata; instead, a deletion flag is added to the deleted metadata, and metadata with existing deletion flags is skipped during search or traversal operations. In-situ addition can be done by directly allocating a position for the new metadata and inserting it into the memory data structure instance. The aforementioned in-situ modification can be performed by directly updating the attribute fields of metadata within the in-memory data structure instance. The aforementioned archived state can be a compressed binary block obtained by losslessly compressing a contiguous memory region of flattened data. The aforementioned archived state has the smallest memory footprint and requires decompression before access.

[0028] In some optional implementations of certain embodiments, the aforementioned execution entity may, according to preset state conditions, set the corresponding memory data structure form for each metadata subset in the aforementioned metadata subset group through the following steps: Step 1: Extract the metadata attribute information of each metadata subset in the above metadata subset group to obtain a metadata attribute information set. The metadata attribute information in the above metadata attribute information set may include: the number of file items, creation timestamp, and subset identifier.

[0029] Step two: Based on the preset state conditions and the aforementioned metadata attribute information set, determine the initial memory data structure form of each metadata subset in the aforementioned metadata subset group, and generate an initial memory data structure form mapping table. In practice, the aforementioned execution entity can create a key-value pair between the subset identifier of each metadata subset in the aforementioned metadata subset group and the aforementioned initial memory data structure form to obtain the initial memory data structure form mapping table. The aforementioned preset state conditions can be pre-set to determine the initial memory data structure form of the newly created metadata subset. For example, it could be: setting the initial memory data structure form of the newly created metadata subset to an online state. The aforementioned initial memory data structure form mapping table can be a hash table used to represent the initial memory data structure form of the metadata subset.

[0030] Step 3: Based on the initial memory data structure morphology mapping table described above, create memory data structure instances for each metadata subset in the metadata subset group, resulting in a set of memory data structure instances. In practice, the execution entity can allocate memory space in memory for each key-value pair in the initial memory data structure morphology mapping table and initialize the corresponding memory data structure instance, thus obtaining a set of memory data structure instances. For example, a new hash table can be initialized as a memory data structure instance. The memory data structure instances include a first memory data structure instance, a second memory data structure instance, and a third memory data structure instance. The first memory data structure instance can be created for the metadata subset representing the online state. The second memory data structure instance can be created for the metadata subset representing the flat state. The third memory data structure instance can be created for the metadata subset representing the archived state.

[0031] As an example, the aforementioned execution entity can create a first in-memory data structure instance through the following steps: First, encode the metadata records of the metadata subset into an operable list of records. Second, initialize a new data structure instance (such as a hash table or a balanced binary tree) and insert the aforementioned list of records into the data structure instance.

[0032] The aforementioned execution entity can create a second in-memory data structure instance through the following steps: First, sort the metadata records of the metadata subset by filename to obtain sorted metadata records. Second, allocate a contiguous block of physical memory and encode the aforementioned metadata records sequentially into the aforementioned physical memory. Third, generate or update the associated probabilistic digest structure for the aforementioned physical memory.

[0033] The aforementioned execution entity can create a third memory data structure instance through the following steps: First, obtain the flat, contiguous memory region corresponding to the metadata subset. Second, compress the aforementioned flat, contiguous memory region using a lossless compression algorithm (such as LZ4 or Zstandard) to obtain an archive-state compressed binary block, which serves as the third memory data structure instance.

[0034] Step four involves associating the aforementioned set of in-memory data structure instances with the aforementioned metadata subset group to complete the setting of the in-memory data structure format. In practice, the executing entity can perform an association operation between the aforementioned set of in-memory data structure instances and the aforementioned metadata subset group to obtain an association hash table. This association operation can establish a reference relationship between the metadata subset and its corresponding in-memory data structure instance. For example, an access handle (such as a memory pointer) can be set in the metadata subset. This access handle can directly point to the in-memory data structure instance corresponding to the metadata subset. The aforementioned association hash table can be a hash table used to store the reference relationship between the metadata subset and its corresponding in-memory data structure instance.

[0035] Step 103: Determine the target metadata subset based on the metadata operation request.

[0036] In some embodiments, the execution entity may determine the target metadata subset based on the metadata operation request.

[0037] In practice, the aforementioned executing entity can determine the target metadata subset based on the aforementioned metadata operation request through the following steps: Step one: Parse the aforementioned metadata operation request to obtain the target filename. In practice, the executing entity can respond to the metadata operation request, which represents a precise query, existence check, or modification request for specific target metadata, by extracting the filename to be operated on from the metadata operation request to obtain the target filename.

[0038] Step two: Determine the target subset identifier corresponding to the target filename. In practice, the execution entity can determine the MD5 hash value of the target filename, and then take the remainder of the hash value with respect to N to obtain the target subset identifier.

[0039] Step 3: Query the associated hash table for the target metadata subset that matches the target subset identifier described above. In practice, the executing entity can find the metadata subset that matches the target subset identifier in the associated hash table and use it as the target metadata subset.

[0040] Step 104: Based on the memory data structure of the target metadata subset, perform the corresponding access processing flow on the target metadata subset to obtain the final access result and output the final access result.

[0041] In some embodiments, the execution entity may perform a corresponding access processing flow on the target metadata subset according to the memory data structure form corresponding to the target metadata subset, obtain the final access result, and output the final access result.

[0042] In addressing the technical problems mentioned above by adopting technical solutions, considering the application scenario: in file system directory metadata access scenarios, a large number of path lookup requests may exhibit negative lookups (e.g., during incremental builds of large-scale projects, the compiler may search for header files along multiple include paths, leading to multiple queries for non-existent filenames). This often involves the following technical issues: traditional file system access to directory metadata may trigger unnecessary scanning, decompression, or upgrades of directory data, resulting in ineffective access overhead. Considering the following requirements for this application scenario: extremely high lookup efficiency and low access latency, we decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may perform the corresponding access processing flow on the target metadata subset according to the memory data structure form corresponding to the target metadata subset through the following steps, obtain the final access result, and output the final access result: Step 1: In response to the above metadata operation request representing a non-traversal type operation, perform the following steps: In sub-step one, in response to the online state of the in-memory data structure representation corresponding to the aforementioned target metadata subset, the aforementioned metadata operation request is executed in the in-memory data structure instance corresponding to the aforementioned target metadata subset, obtaining a first operation result, and determining the aforementioned first operation result as the final access result. In practice, the aforementioned execution entity can, in response to the online state of the in-memory data structure representation corresponding to the aforementioned target metadata subset, directly apply the aforementioned metadata operation request to the in-memory data structure instance corresponding to the aforementioned target metadata subset, obtaining a first operation result, and determining the aforementioned first operation result as the final access result. The aforementioned metadata operation request can represent a traversal-type operation or a non-traversal-type operation. The aforementioned non-traversal-type operation can be an operation that performs a precise query or existence check request for a specific target file. For example, the aforementioned non-traversal-type operation can be lookup, stat, or a request to determine whether a target filename exists. The aforementioned traversal-type operation can be an operation that obtains a list of all or a batch of file entries in a directory.

[0043] Sub-step two, in response to the offline state represented by the memory data structure corresponding to the above-mentioned subset of target metadata, performs the following steps: The first step is to input the target filename into the probabilistic digest structure corresponding to the aforementioned target metadata subset to obtain the digest query results. The offline state can be one of the following: flat state or archived state. The probabilistic digest structure can be used to determine whether an element definitely does not exist in a certain set. The probabilistic digest structure can be a Bloom filter. The digest query results can be the determination results obtained after querying the probabilistic digest structure. The digest query results can include "deterministic negative determination" (the target definitely does not exist) and "non-deterministic negative determination" (the target may exist).

[0044] The second step involves terminating access to the target metadata subset in response to the deterministic negation determination of the summary query result. In practice, this is done by terminating access to the target metadata subset in response to the deterministic negation determination of the summary query result and returning an access result indicating "file does not exist".

[0045] The third step, in response to the non-deterministic negative determination represented by the above summary query results, is to perform the following sub-steps: The first sub-step involves inputting the target filename into the flattened data structure corresponding to the target metadata subset to obtain the second operation result. In practice, the executing entity, responding to the flattened nature of the memory data structure corresponding to the target metadata subset, obtains a pointer to the memory block of the target metadata subset through the access handle of the associated hash table, and searches for the operation result corresponding to the target filename in the memory block using a search algorithm to obtain the second operation result. The search algorithm can be a binary search or a sequential traversal.

[0046] The second sub-step involves decompressing the archived state data structure of the target metadata subset in response to its corresponding memory data structure morphology. This yields a decompressed archived state data structure, into which the target filename is input to obtain the third operation result. In practice, the executing entity decompresses the memory data structure instance of the target metadata subset in response to its corresponding memory data structure morphology. This yields a decompressed memory data structure instance. Then, using the aforementioned search algorithm, the operation result corresponding to the target filename is searched within the decompressed memory data structure instance to obtain the third operation result.

[0047] The third sub-step is to determine the result of the second or third operation as the final access result.

[0048] Sub-step three: Output the final access result.

[0049] The above steps one through three and their related content, as an inventive point of this disclosure, solve the technical problem that "traditional file systems may trigger unnecessary scanning, decompression, or upgrade conversion of directory data when accessing directory metadata, resulting in invalid access overhead." The reason for this invalid access overhead is that traditional file systems may trigger unnecessary scanning, decompression, or upgrade conversion of directory data when accessing directory metadata, leading to invalid access overhead. Solving this problem solves the issue of invalid access overhead. To achieve this, firstly, in response to the online state of the memory data structure corresponding to the target metadata subset, the metadata operation request is executed in the memory data structure instance corresponding to the target metadata subset to obtain a first operation result, which is then determined as the final access result. This allows direct operation on the data structure to obtain the corresponding result. Secondly, in response to the offline state of the memory data structure corresponding to the target metadata subset, the following steps are executed: the target filename is input into the probabilistic digest structure corresponding to the target metadata subset to obtain a digest query result. This reduces invalid operations in subsequent operations. In response to the deterministic negation judgment of the above summary query result, access to the above target metadata subset is terminated. This avoids noisy access to the directory. In response to the non-deterministic negation judgment of the above summary query result, the following steps are performed: In response to the flat state of the memory data structure corresponding to the above target metadata subset, the target filename is input into the flat state data structure corresponding to the above target metadata subset to obtain the second operation result. This utilizes the continuity of memory to achieve efficient retrieval. In response to the archive state of the memory data structure corresponding to the above target metadata subset, the archive state data structure of the above metadata subset is decompressed to obtain the decompressed archive state data structure, and the target filename is input into the decompressed archive state data structure to obtain the third operation result. This keeps the decompression overhead within a limited range. The second or third operation result is determined as the final access result; in the third step, the final access result is output. Ultimately, invalid access overhead is reduced.

[0050] Optionally, the above method further includes: Step two, in response to the above metadata operation request representing the traversal class operation, perform the following steps: First, in response to the online state of the memory data structure representation corresponding to the above-mentioned target metadata subset, the memory data structure instances of the above-mentioned target metadata subset are traversed to obtain the final access result.

[0051] Second, in response to the flattened state of the memory data structure representation corresponding to the above target metadata subset, the metadata in the memory data structure instance of the above target metadata subset, excluding the deletion identifier, is traversed to obtain the final access result.

[0052] Third, in response to the memory data structure morphology representation of the above target metadata subset, the archive state data structure of the above target metadata subset is decompressed to obtain the decompressed archive state data structure, and the above decompressed archive state data structure is traversed to obtain the final access result.

[0053] Fourth, output the final access result.

[0054] Step 105: Obtain the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain the directory indicator set.

[0055] In some embodiments, the execution entity can obtain the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain the directory indicator set.

[0056] In practice, the aforementioned execution entity can obtain the access characteristic set and system resource status of each metadata subset in the aforementioned metadata subset group in real time, thus obtaining a directory indicator set. The access characteristics in the access characteristic set include read request frequency, write request frequency, idle time, access type ratio, negation judgment ratio, and subset size. The system resource status includes memory utilization and CPU load. The read request frequency represents the number of read requests to the metadata subset within a preset time window (e.g., 5 seconds). The write request frequency represents the number of write requests to the metadata subset within a preset time window. The idle time represents the timestamp difference between the last access to the metadata subset and the current access to the metadata subset. The access type ratio represents the proportion of query operations, traversal operations, and modification operations to the total number of operations. The negation judgment ratio represents the proportion of queries that return a "negative judgment" from the probabilistic digest structure to the total number of queries. The subset size represents the number of entries in the metadata subset. The memory utilization represents the proportion of physical memory occupied by the current node. The CPU load described above can characterize the processor utilization of the current node.

[0057] Step 106: Based on the directory index set and preset conversion conditions, determine the memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group, and obtain the conversion instruction set and the metadata subset group to be converted.

[0058] In some embodiments, the execution entity can determine the memory data structure conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group based on the directory index set and preset conversion conditions, thereby obtaining a conversion instruction set and a metadata subset group to be converted. The preset conversion conditions can be predefined and used to determine whether the metadata subset needs to change its current memory data structure and which memory data structure it needs to be converted to. The preset conversion conditions may include: demotion conditions, promotion conditions, and resource pressure conditions.

[0059] The aforementioned downgrade conditions can include: Condition 1: If the write request frequency of the metadata subset representing the online state of the memory data structure is 0 and the idle time exceeds a time threshold, the metadata subset is marked as the metadata subset to be converted, and the target conversion state is flat. Condition 2: If the read request frequency of the metadata subset representing the flat state of the memory data structure is 0 and the idle time exceeds a time threshold, the metadata subset is marked as the metadata subset to be converted, and the target conversion state is archived. For example, the time threshold for triggering the downgrade from online state to flat state could be 200 milliseconds. The time threshold for triggering the downgrade from flat state to archived state could be 10 minutes.

[0060] The aforementioned upgrade conditions may include: Condition 1: When a subset of metadata representing a flat or archived state in the memory data structure receives a metadata operation request (such as adding a file) that is not supported by the current state, the metadata subset is marked as a subset to be converted, and the target conversion state is online. Condition 2: When the frequency of read requests for a subset of metadata representing an archived state in the memory data structure exceeds a preset frequency (which could be once per second), the metadata subset is marked as a subset to be converted, and the target conversion state is flat.

[0061] The aforementioned resource pressure conditions can be in response to memory utilization exceeding a preset value (e.g., 85%), marking a subset of metadata that is flat in terms of memory data structure representation, or whose idle time exceeds a preset idle value (e.g., 10 minutes), or whose access frequency (e.g., read request frequency, write request frequency) is lower than a preset access value (e.g., once per second) as a subset of metadata to be converted, with the target conversion form being archived.

[0062] In practice, the aforementioned execution entity can determine the memory data structure format conversion instruction corresponding to at least one metadata subset to be converted in the aforementioned metadata subset group based on the aforementioned directory index set and preset conversion conditions through the following steps, thereby obtaining the conversion instruction set and the metadata subset group to be converted: Step 1: Based on the above-mentioned preset conversion conditions and the above-mentioned directory index set, determine at least one metadata subset to be converted in the above-mentioned metadata subset group, obtain the metadata subset group to be converted, and determine the target memory data structure form of each metadata subset to be converted in the above-mentioned metadata subset group to be converted, obtain the target structure form information group.

[0063] Step two: Generate a conversion instruction set based on the target memory data structure format information group mentioned above. The conversion instructions in this conversion instruction set may include the target subset identifier, the source memory data structure format, and the target memory data structure format.

[0064] Step 107: Based on the conversion instruction set, reconstruct the metadata subset group to be converted to obtain the converted metadata subset group.

[0065] In some embodiments, the execution entity may reconstruct the metadata subset group to be converted based on the conversion instruction set to obtain the converted metadata subset group.

[0066] In addressing the technical challenges mentioned above, the application scenario of distributed file directory association metadata often presents the following issues: the access activity of directory association metadata subsets varies over time; traditional solutions use homogeneous in-memory representations for each subset, making it difficult to balance memory usage, access latency, and operational capabilities; and the inability to dynamically adjust based on actual load leads to significant memory overhead. Considering the following requirements for this application scenario—high conversion efficiency and real-time response—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may reconstruct the metadata subset group to be converted based on the conversion instruction set through the following steps to obtain the converted metadata subset group: Step 1: Based on the above conversion instruction set, perform the following data reconstruction steps for each subset of metadata to be converted in the above metadata subset group: The first step is to determine the target memory data structure instance for the aforementioned subset of metadata to be converted. In practice, the execution entity can traverse the aforementioned conversion instruction set to determine the target memory data structure instance for each subset of metadata to be converted within the aforementioned group of metadata subsets. The target memory data structure instance can include any one of the following: a first target memory data structure instance, a second target memory data structure instance, and a third target memory data structure instance.

[0067] The second step is to update the memory data structure instance of the aforementioned subset of metadata to be converted to the aforementioned target memory data structure instance. In practice, the execution entity can update the memory data structure instance of the aforementioned subset of metadata to be converted to the aforementioned target memory data structure instance through the following sub-steps: In the first sub-step, in response to the online state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted, a first target memory data structure instance is constructed as the target memory data structure instance, and the memory data structure instance of the aforementioned subset of metadata to be converted is updated to the aforementioned target memory data structure instance. In practice, firstly, the executing entity can initialize a new data structure instance (such as a hash table or a balanced binary tree) as the first target memory data structure instance. Secondly, in response to the online state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted and the flat state of the source memory data structure corresponding to the aforementioned subset of metadata to be converted, each metadata element that has not been deleted in the aforementioned subset of metadata to be converted is encoded into an operable record list and inserted into the aforementioned first target memory data structure instance. In response to the online state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted and the archived state of the source memory data structure corresponding to the aforementioned subset of metadata to be converted, the metadata corresponding to the aforementioned subset of metadata to be converted is decompressed to obtain a decompressed metadata dataset, and each decompressed metadata element in the aforementioned decompressed metadata dataset is encoded into an operable record list and inserted into the aforementioned first target memory data structure instance. Third, update the access handle of the aforementioned subset of metadata to be converted to point to the aforementioned target first memory data structure instance.

[0068] Sub-step two: In response to the flattened state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted, a target second memory data structure instance is constructed as the target memory data structure instance, and the memory data structure instance of the aforementioned subset of metadata to be converted is updated to the aforementioned target memory data structure instance. In practice, the aforementioned execution entity can respond to the flattened state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted and the online state of the source memory data structure corresponding to the subset of metadata to be converted. First, a contiguous physical storage area is allocated as the second memory data structure instance. Second, the metadata records of the subset of metadata to be converted are sorted by filename to obtain sorted metadata records. Third, the aforementioned metadata records are sequentially encoded into the aforementioned target second memory data structure instance. Fourth, a probabilistic digest structure associated with the aforementioned physical memory is generated or updated. Fifth, the access handle of the aforementioned subset of metadata to be converted is updated to point to the aforementioned target second memory data structure instance. In response to the fact that the target memory data structure corresponding to the aforementioned subset of metadata to be converted is in a flat state and the source memory data structure corresponding to the subset of metadata to be converted is in an archived state, firstly, a decompression operation is performed on the archived state compressed binary block corresponding to the aforementioned subset of metadata to be converted to obtain a target second memory data structure instance, and the probabilistic digest structure already associated with the aforementioned subset of metadata to be converted is retained. Secondly, the access handle of the aforementioned subset of metadata to be converted is updated to point to the aforementioned target second memory data structure instance.

[0069] Sub-step three: In response to the archived state of the target memory data structure representation corresponding to the aforementioned subset of metadata to be converted, a third memory data structure instance is constructed as the target memory data structure instance, and the memory data structure instance of the aforementioned subset of metadata to be converted is updated to the aforementioned target memory data structure instance. In practice, firstly, in response to the online state of the source memory data structure representation corresponding to the aforementioned subset of metadata to be converted, the valid metadata records in the aforementioned subset of metadata to be converted are traversed, and flattened contiguous memory regions are generated by encoding according to filename or key-value order. In response to the flattened state of the source memory data structure representation corresponding to the aforementioned subset of metadata to be converted, the flattened contiguous memory region is used as the input to be compressed. Secondly, a lossless compression algorithm is applied to the entire flattened contiguous memory region to obtain archived compressed binary blocks. Thirdly, the archived compressed binary blocks are stored as the target third memory data structure instance, and the probabilistic digest structure corresponding to the metadata subset is retained or associated. Fourthly, the access handle of the aforementioned subset of metadata to be converted is updated to point to the aforementioned target third memory data structure instance. The aforementioned valid metadata records can be metadata records that have not been deleted.

[0070] Step two: Release the memory resources occupied by each subset of metadata to be converted in the aforementioned subset group to generate the converted metadata subset group. In practice, the execution entity can release the memory resources occupied by the source data structure instances of each subset of metadata to be converted in the aforementioned subset group in response to the aforementioned access handle update and the absence of concurrent access to the source data structure instances of the aforementioned subset of metadata to be converted.

[0071] Steps one and two above, and their related content, constitute an inventive point of this disclosure, solving the technical problem that "the access activity of directory-associated metadata subsets changes over time; traditional solutions use homogeneous memory representation for each subset, making it difficult to balance memory usage, access latency, and operational capability." The reason for this imbalance is that the access activity of directory-associated metadata subsets changes over time; and traditional solutions use homogeneous memory representation for each subset, making it difficult to balance memory usage, access latency, and operational capability. Solving these factors would allow for a better balance of memory usage, access latency, and operational capability. To achieve this, the first step involves performing the following data reconstruction steps for each metadata subset in the aforementioned metadata subset group, based on the aforementioned conversion instruction set: determining the target memory data structure instance for the aforementioned metadata subset; and updating the memory data structure instance of the aforementioned metadata subset to the aforementioned target memory data structure instance. This enables on-demand storage. The second step involves releasing the memory resources occupied by each metadata subset in the aforementioned metadata subset group to generate the converted metadata subset group. Ultimately, this approach can balance issues of memory usage, access latency, and operational capabilities.

[0072] The aforementioned access handle update can be achieved through the following atomic operation steps: First, obtain the current access handle and reference count of the current memory data structure instance for the aforementioned subset of metadata to be converted. The reference count can be an integer value associated with the current memory data structure instance, used to record the number of times the current accessor (such as a thread or process) is using the current memory data structure instance.

[0073] Second, in response to the completion of the creation of the target memory data structure instance of the aforementioned subset of metadata to be converted, the reference count of the aforementioned target memory data structure instance is initialized to zero. In practice, the aforementioned execution entity can create and associate a reference count variable with the aforementioned target memory data structure instance, and set the initial value of the reference count variable to zero.

[0074] Third, the target of the current access handle is switched to the target memory data structure instance of the metadata subset to be converted, resulting in an updated access handle. In practice, the execution entity can use atomic operations to switch the target of the current access handle to the target memory data structure instance of the metadata subset to be converted, thus obtaining an updated access handle.

[0075] Fourth, associate the updated access handle with the aforementioned subset of metadata to be converted.

[0076] Fifth, in response to detecting that the reference count of the current memory data structure instance is zero, the memory resources occupied by the current memory data structure instance are released. In practice, the executing entity can continuously acquire the reference count of the current memory data structure instance. When the current visitor completes access to the current memory data structure instance and releases the reference, the reference count decreases. In response to detecting that the reference count of the current memory data structure instance is zero, the memory resources occupied by the current memory data structure instance are released.

[0077] The above embodiments of this disclosure have the following beneficial effects: the adaptive memory transformation method for distributed file systems according to some embodiments of this disclosure can improve memory utilization, reduce redundant memory occupation, and reduce unnecessary data access overhead. Specifically, the reasons for low memory utilization, redundant memory occupation, and unnecessary data access overhead are as follows: Low memory utilization. The metadata associated with a directory can reach billions or even trillions of records. Using online data structures that support random addition, deletion, modification, and query (such as hash tables and balanced trees) will result in low memory utilization due to inherent pointer overhead, reserved space, and memory fragmentation problems. In typical file system workloads, a large number of directories are inactive most of the time, yet they occupy memory resources for a long time with high-cost online structures. Redundant memory occupation. Existing layered caching schemes have a dual-copy consistency problem. Existing systems use a layered architecture of "compressed storage layer + online hash cache layer" to balance memory efficiency and access performance. The lower layer stores metadata in compressed form, and the upper layer maintains an online hash cache of active data. When the data in the cache is modified, the compressed copy at the lower layer needs to be updated or invalidated synchronously, introducing consistency maintenance overhead and redundant memory occupation. Unnecessary data access overhead is generated. Existing systems often use probabilistic data structures such as Bloom filters to accelerate searches and avoid unnecessary overhead. However, such filtering structures usually exist only as independent optimization components on the search path, and their query statistics (such as the frequency of negative decisions) are not incorporated into the data storage strategy decision-making process. This makes it impossible for the system to determine whether it is necessary to maintain a data partition in a compact form with low memory overhead based on the negative search ratio. Based on this, some embodiments of the adaptive memory conversion method for distributed file systems disclosed in this disclosure firstly, in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, the metadata dataset in the target directory is partitioned to obtain metadata subset groups. Thus, massive metadata can be divided into smaller, logically independent metadata subsets, avoiding the constraints of unified management of the entire directory and laying the foundation for adaptive management. Secondly, according to preset state conditions, a corresponding memory data structure form is set for each metadata subset in the metadata subset group, wherein the memory data structure form includes online state, flat state, and archive state. Thus, the most suitable storage representation can be selected according to the specific situation of the metadata subset. Secondly, based on the aforementioned metadata operation requests, a target metadata subset is determined. This avoids traversing the entire metadata dataset, narrowing the scope of data processing. Thirdly, according to the memory data structure corresponding to the target metadata subset, the corresponding access processing flow is executed on the target metadata subset to obtain the final access result, which is then output. This avoids unnecessary lookup or decompression overhead, significantly reducing the system's average latency in negative lookup scenarios.Next, the access characteristic set and system resource status of each metadata subset in the aforementioned metadata subset group are acquired in real time to obtain a directory indicator set. This allows for the quantification of the access status and system resource status of each subset. Then, based on the aforementioned directory indicator set and preset conversion conditions, the memory data structure transformation instruction corresponding to at least one metadata subset to be converted in the aforementioned metadata subset group is determined, resulting in a conversion instruction set and the metadata subset group to be converted. This allows for the reduction of long-term idle subsets to a flat state to save memory, or the conversion of flattened subsets to an archive state when memory is scarce. Finally, based on the aforementioned conversion instruction set, the metadata subset group to be converted is reconstructed to obtain the converted metadata subset group. Ultimately, this improves memory utilization, reduces redundant memory usage, and minimizes unnecessary data access overhead.

[0078] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an adaptive memory translation device for a distributed file system, these device embodiments being similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0079] like Figure 2As shown, an adaptive memory conversion device 200 for a distributed file system in some embodiments includes: a partitioning unit 201, a setting unit 202, a first determining unit 203, an execution unit 204, a real-time acquisition unit 205, a second determining unit 206, and a data reconstruction unit 207. The partitioning unit 201 is configured to partition the metadata dataset in the target directory to obtain a metadata subset group in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory. The setting unit 202 is configured to set a corresponding memory data structure form for each metadata subset in the metadata subset group according to preset state conditions, wherein the memory data structure form includes online state, flat state, and archive state. The first determining unit 203 is configured to determine the target metadata subset based on the metadata operation request. The execution unit 204 is configured to perform real-time acquisition on the target metadata subset according to the memory data structure form corresponding to the target metadata subset. The metadata subset executes the corresponding access processing flow to obtain the final access result and outputs the final access result; the real-time acquisition unit 205 is configured to acquire the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain the directory indicator set; the second determination unit 206 is configured to determine the memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group based on the directory indicator set and preset conversion conditions to obtain the conversion instruction set and the metadata subset group to be converted; the data reconstruction unit 207 is configured to reconstruct the metadata subset group to be converted based on the conversion instruction set to obtain the converted metadata subset group.

[0080] It is understandable that the units described in the device 200 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0081] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0082] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0083] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0084] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0085] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0086] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0087] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: respond to a target directory where the number of files exceeds a preset partitioning threshold or receive a metadata operation request for the target directory, partition the metadata dataset in the target directory to obtain metadata subset groups; according to preset state conditions, set a corresponding memory data structure form for each metadata subset in the metadata subset group, wherein the memory data structure form includes online state, flat state, and archive state; determine the target metadata subset based on the aforementioned metadata operation request; and determine the target metadata subset based on the aforementioned target metadata... Based on the memory data structure of the subset, the corresponding access processing flow is executed on the target metadata subset to obtain the final access result, and the final access result is output; the access feature set and system resource status of each metadata subset in the metadata subset group are obtained in real time to obtain the directory indicator set; based on the directory indicator set and preset conversion conditions, the memory data structure conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group is determined to obtain the conversion instruction set and the metadata subset group to be converted; based on the conversion instruction set, the metadata subset group to be converted is reconstructed to obtain the converted metadata subset group.

[0088] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0090] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a partitioning unit, a setting unit, a first determining unit, an execution unit, a real-time acquisition unit, a second determining unit, and a data reconstruction unit. The names of these units do not necessarily limit the specific unit itself; for example, the partitioning unit may also be described as "in response to the number of files in a target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, partitioning the metadata dataset in the target directory to obtain a metadata subset group."

[0091] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0092] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An adaptive memory translation method for a distributed file system, comprising: In response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, the metadata dataset in the target directory is partitioned to obtain a metadata subset group; According to preset state conditions, a corresponding memory data structure form is set for each metadata subset in the metadata subset group. The memory data structure form includes online state, flat state and archive state. The online state is a memory data structure form that uses hash table, balanced tree or a combination of hash table and linked list to support CRUD operations on the metadata subset. The flat state is a memory data structure form that stores metadata by storing contiguous memory blocks that are arranged in an orderly manner according to keys. The archive state is a compressed binary block form obtained by losslessly compressing the entire contiguous memory area of ​​the flat state data. Based on the metadata operation request, determine the target metadata subset; Based on the memory data structure of the target metadata subset, the corresponding access processing flow is executed on the target metadata subset to obtain the final access result, and the final access result is output. The access feature set and system resource status of each metadata subset in the metadata subset group are obtained in real time to obtain the directory indicator set; Based on the directory index set and preset conversion conditions, determine the memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group, and obtain the conversion instruction set and the metadata subset group to be converted; Based on the conversion instruction set, the metadata subset group to be converted is reconstructed to obtain the converted metadata subset group.

2. The method according to claim 1, wherein, The access features in the access feature set include read request frequency, write request frequency, idle time, access type ratio, negative judgment ratio, and subset size.

3. The method according to claim 1, wherein, The system resource status includes memory usage and CPU load.

4. The method according to claim 1, wherein, Based on the directory index set and preset conversion conditions, the step of determining the memory data structure format conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group, thereby obtaining a conversion instruction set and a metadata subset group to be converted, includes: Based on the preset conversion conditions and the directory index set, at least one metadata subset to be converted is determined in the metadata subset group to obtain the metadata subset group to be converted, and the target memory data structure form of each metadata subset to be converted in the metadata subset group to be converted is determined to obtain the target structure form information group. A set of conversion instructions is generated based on the target memory data structure morphology information group.

5. The method according to claim 1, wherein, In response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, the metadata dataset in the target directory is partitioned to obtain metadata subset groups, including: In response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory, a target partitioning instruction is determined; Based on the target partitioning instruction, the metadata dataset in the target directory is partitioned to obtain a metadata subset group.

6. The method according to claim 1, wherein, The step of determining the target metadata subset based on the metadata operation request includes: The metadata operation request is parsed to obtain the target file name; Determine the target subset identifier corresponding to the target file name; Search the associated hash table for the subset of target metadata that matches the target subset identifier representation.

7. The method according to claim 1, wherein, The step of setting a corresponding memory data structure for each metadata subset in the metadata subset group according to preset state conditions includes: Extract the metadata attribute information of each metadata subset in the metadata subset group to obtain a metadata attribute information set; Based on the preset state conditions and the metadata attribute information set, determine the initial memory data structure form of each metadata subset in the metadata subset group, and generate an initial memory data structure form mapping table. Based on the initial memory data structure morphology mapping table, memory data structure instances are created for each metadata subset in the metadata subset group to obtain a set of memory data structure instances. The memory data structure instances include a first memory data structure instance, a second memory data structure instance, and a third memory data structure instance. The first memory data structure instance represents a metadata subset in the online state, the second memory data structure instance represents a metadata subset in the flat state, and the third memory data structure instance represents a metadata subset in the archive state. Associating the set of in-memory data structure instances with the set of metadata subsets completes the setting of the in-memory data structure format.

8. An adaptive memory translation device for a distributed file system, comprising: The partitioning unit is configured to partition the metadata dataset in the target directory to obtain a metadata subset group in response to the number of files in the target directory exceeding a preset partitioning threshold or receiving a metadata operation request for the target directory. The setting unit is configured to set a corresponding memory data structure form for each metadata subset in the metadata subset group according to preset state conditions. The memory data structure form includes online state, flat state and archive state. The online state is a memory data structure form that uses a hash table, a balanced tree or a combination of a hash table and a linked list to support CRUD operations on the metadata subset. The flat state is a memory data structure form that stores metadata by storing contiguous memory blocks that are arranged in an orderly manner according to keys. The archive state is a compressed binary block form obtained by losslessly compressing the entire contiguous memory area of ​​the flat state data. The first determining unit is configured to determine a subset of target metadata based on the metadata operation request; The execution unit is configured to perform a corresponding access processing flow on the target metadata subset according to the memory data structure form corresponding to the target metadata subset, obtain the final access result, and output the final access result; The real-time acquisition unit is configured to acquire the access feature set and system resource status of each metadata subset in the metadata subset group in real time to obtain the directory indicator set; The second determining unit is configured to determine, based on the directory index set and preset conversion conditions, the memory data structure form conversion instruction corresponding to at least one metadata subset to be converted in the metadata subset group, thereby obtaining a conversion instruction set and a metadata subset group to be converted. The data reconstruction unit is configured to reconstruct the metadata subset group to be converted based on the conversion instruction set, so as to obtain the converted metadata subset group.

9. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images