Hybrid storage method and system for unified metadata layer

Abstract

The application provides a unified metadata layer hybrid storage method and system, the method comprises receiving a storage access request of each node, parsing the protocol type and the target data path corresponding to the request; based on the preset directory table, the index node table and the object table, the metadata operation corresponding to the protocol type is executed; the metadata operation is encapsulated as an atomic transaction submission, and the corresponding records of the directory table, the index node table and the object table are synchronously updated, and the read and write operations of the target data are completed. The application not only solves the problems of semantic fragmentation, poor consistency and performance bottleneck existing in the traditional hybrid storage system for a long time, but also solves the problems of resource contention, business insecurity and poor cluster stability in the multi-training task parallel scene according to the special needs of the AI large model training scene.

Description

Technical Field

[0001] This application belongs to the field of operating systems, and in particular relates to a hybrid storage method and system for a unified metadata layer. Background Technology

[0002] With the rapid growth in the training scale of large AI models, training systems need to simultaneously handle massive small-file sample datasets and large-size model weights and checkpoint files. This places demands on the underlying storage to achieve both POSIX-compliant high-frequency random access and S3-compliant high-throughput sequential read / write. Currently, the mainstream industry architecture employs a layered deployment of file storage and object storage, with their metadata layers operating independently, maintaining separate inode / directory entry and object path index systems. Existing convergence solutions often achieve cross-system access through protocol bridging and middleware synchronization, but the underlying layer still fails to break down the barriers between the two metadata management logics. This results in semantic fragmentation, complex synchronization, performance degradation, and difficulty in guaranteeing consistency, making it unsuitable for the stringent requirements of frequent updates to TB-level sample sets and concurrent training across multiple nodes in large-scale model training scenarios. Furthermore, existing storage architectures lack resource management capabilities for parallel scenarios with multiple training tasks on large models, easily leading to issues such as resource preemption for production tasks and drastic fluctuations in training performance. Summary of the Invention

[0003] This application provides a hybrid storage method and system for a unified metadata layer to address the underlying issues in training large AI models.

[0004] This application provides a hybrid storage method for a unified metadata layer, including the following steps:

[0005] Receive storage access requests from each node and parse the protocol type and target data path corresponding to the request;

[0006] Based on the preset directory table, index node table, and object table, perform metadata operations corresponding to the protocol type;

[0007] Metadata operations are encapsulated into atomic transaction commits, synchronously updating the corresponding records in the directory table, inode table, and object table to complete the read and write operations of the target data.

[0008] Optionally, the preset directory table, index node table, and object table are constructed in the following ways:

[0009] Construct a directory table using the file system identifier, parent directory inode number, and file name as the row key, and store the inode numbers corresponding to the sub-files;

[0010] An inode table is constructed using the file system identifier and inode number as row keys to store the distribution information of file data blocks;

[0011] An object table is constructed using the bucket name and object path as row keys, storing the index node number and object extension information corresponding to the data.

[0012] Optionally, the step of encapsulating metadata operations into atomic transaction commits and synchronously updating the corresponding records in the directory table, inode table, and object table includes:

[0013] The write, update, and delete operations of metadata for the directory table, index node table, and object table are encapsulated into the same distributed transaction;

[0014] Execute all metadata operations within the transaction and verify the operation results for each table;

[0015] If all operations are executed successfully, the transaction is committed to complete the synchronization update of the directory table, inode table, and object table;

[0016] If any operation fails, all operations within the transaction will be rolled back.

[0017] Optionally, the node is an AI training node, and the method further includes:

[0018] Each AI training node is assigned a corresponding resource token. The initial number of resource tokens for each AI training node is determined based on its storage credit limit. All AI training nodes and their corresponding resource tokens are managed in a temporary intermediary. The higher the storage credit limit of an AI training node, the more resource tokens it will have in the temporary intermediary at the beginning.

[0019] Based on a preset random competition function, a competitive selection is made among all AI training nodes that meet the competition conditions in the temporary intermediary. The random competition function satisfies the following condition: the more resource tokens an AI training node has, the higher its probability of being selected.

[0020] The selected AI training nodes are added to the scheduling body and the tasks are scheduled and executed in a preset order;

[0021] When the selected AI training node is added to the scheduling body, the number of resource tokens of the AI ​​training node in the temporary intermediate body is reduced by a preset ratio.

[0022] Collect the training task convergence of each AI training node, and update the number of resource tokens of the corresponding AI training node in the temporary intermediate based on the change in training convergence: when the training result convergence increases, increase the resource tokens of the corresponding AI training node; when the training result convergence decreases, decrease the resource tokens of the corresponding AI training node, until all AI training tasks are scheduled and executed.

[0023] Optionally, the random competition function is implemented using a weighted random roulette wheel algorithm. The probability of a single AI training node being selected in a single competition is the ratio of the number of valid resource tokens currently held by the AI ​​training node in the temporary intermediary to the total number of valid resource tokens held by all AI training nodes participating in this competition in the temporary intermediary. Based on the calculated selection probability of each AI training node, a continuous random number interval of corresponding length is assigned to each node. The AI ​​training node selected in this competition is determined by the interval in which the generated globally unique random number falls.

[0024] Optionally, the step of deducting the current number of resource tokens of the AI ​​training node in the temporary intermediate according to a preset ratio specifically includes:

[0025] The configurable range for the preset token deduction ratio is 10%-30%;

[0026] When an AI training node is selected and completes the enqueue operation of the scheduling body, the number of tokens corresponding to the product of the current balance and the preset deduction ratio is deducted from the resource token balance of the node in the temporary intermediate body. The resource token balance after deduction is not less than 0. When the balance after deduction is 0, the AI ​​training node is marked as suspended from competition until it replenishes its resource tokens to a value greater than 0 through training convergence, and then regains its eligibility to compete.

[0027] Optionally, updating the number of resource tokens for the corresponding AI training node in the temporary intermediary based on changes in training convergence specifically includes:

[0028] The training convergence is at least one quantifiable indicator among the loss function decline rate of the training task and the improvement rate of model validation accuracy.

[0029] Using a preset training iteration cycle as the unit, collect the training convergence value of each AI training node in the current iteration cycle and compare it with the convergence benchmark value of the previous iteration cycle.

[0030] When the convergence value of the current iteration period is higher than the baseline value of the previous iteration period, the corresponding number of resource tokens are added to the temporary intermediate for the AI ​​training node.

[0031] When the convergence value of the current iteration period is lower than the baseline value of the previous iteration period, the corresponding number of resource tokens are deducted from the resource token balance of the AI ​​training node in the temporary intermediate; wherein, the change in convergence is linearly positively correlated with the increase or decrease in the number of resource tokens.

[0032] Optionally, the step of adding the selected AI training nodes to the scheduling body and scheduling and executing tasks according to a preset order specifically includes:

[0033] The scheduling body adopts a first-in-first-out ordered queue structure. Selected AI training nodes are added to the tail of the ordered queue in the order of their selection time, and the order of nodes in the queue cannot be tampered with.

[0034] The scheduler retrieves AI training nodes sequentially from the head of the ordered queue, allocates matching training storage and computing resources to them, and executes the corresponding training tasks.

[0035] After the training task of the AI ​​training node is scheduled and executed, if the node still has valid resource tokens remaining in the temporary intermediary and the task has not reached the preset convergence termination threshold, it will be reinstated into the competitive selection pool to participate in the next round of random competition.

[0036] If a task reaches the preset convergence termination threshold, the remaining resource tokens of that node will be cleared and it will be removed from the competition selection pool.

[0037] Optionally, the temporary intermediary is a token management container with access control, used to maintain the unique identifier of all online AI training nodes, the corresponding resource token balance, the storage credit limit, real-time training convergence data and task execution status; the scheduling body is an ordered scheduling queue with a sequence lock, used to store AI training nodes that have been selected through competition and are waiting to be scheduled for resource execution.

[0038] This application also provides a hybrid storage system with a unified metadata layer, including:

[0039] The receiving module receives storage access requests from each node and parses the corresponding protocol type and target data path of the request.

[0040] The execution module performs metadata operations corresponding to the protocol type based on the preset directory table, inode table, and object table.

[0041] The synchronization module encapsulates metadata operations into atomic transaction commits, synchronously updating the corresponding records in the directory table, index table, and object table to complete the read and write operations of the target data.

[0042] As can be seen from the above technical solutions, the hybrid storage method and system for the unified metadata layer provided in this application, through the unified metadata architecture of INODE_TABLE, DIR_TABLE, and OBJ_TABLE, natively supports POSIX and S3 dual-protocol semantics within the same metadata structure, achieving the unification of file and object storage namespaces. This eliminates the redundant metadata storage and cross-system synchronization overhead of traditional layered architectures. Simultaneously, it provides a temporary intermediary and a scheduling body. The higher the storage trust of each AI training node, the more resource tokens it holds in the temporary intermediary. A random competition function is required in the temporary intermediary to ensure that the more resource tokens a node has, the higher its probability of being selected. Subsequently, the selected AI training node... The AI ​​training node is added to the scheduling body and scheduled sequentially. Simultaneously, the current resource token count of that AI training node is reduced by a certain percentage (e.g., one-fifth). Meanwhile, the convergence rate of the training results is used to increase resource tokens in the temporary intermediate unit; a decrease in convergence rate reduces resource tokens. In this way, the scheduling of training tasks for AI training nodes is based on resource token control, ensuring that high-quality training tasks are selected from other training tasks more quickly. Furthermore, the resource tokens in the temporary intermediate unit are continuously updated based on the training convergence rate, ensuring that AI training tasks with faster convergence have higher priority. This method eliminates the need to set a priority for each AI training node, enabling priority updates for each AI training node and preventing low-priority tasks from preempting resources. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This paper illustrates a flowchart of a hybrid storage method for a unified metadata layer in an embodiment of this application.

[0045] Figure 2 This application illustrates a unified metadata architecture diagram for object storage and file systems in an embodiment of the present application.

[0046] Figure 3 This illustration shows a design diagram of the unified metadata table structure for object storage and file system in an embodiment of this application;

[0047] Figure 4 This illustration shows a storage scheduling diagram of the nth round of AI training nodes in an embodiment of this application;

[0048] Figure 5This illustration shows a storage scheduling diagram of the AI ​​training node in the (n+1)th round in an embodiment of this application.

[0049] Figure 6 A schematic diagram of a hybrid storage system architecture with a unified metadata layer is shown in an embodiment of this application. Detailed Implementation

[0050] It should be noted that the embodiments of this application relate to the field of distributed storage technology, and in particular to a hybrid storage method and system with a unified metadata layer. It is applicable to hybrid storage scenarios that require simultaneous support for file storage semantics and object storage semantics, such as training large artificial intelligence models, high-performance computing, and management of massive unstructured data. It is especially suitable for the hybrid access requirements of high-frequency random access to massive small file sample sets and high-throughput sequential read and write of large-size model checkpoint files during the training of large AI models.

[0051] It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0052] In this application, technical terms are first defined in a unified and clear manner to ensure that those skilled in the art can clearly and unambiguously understand the technical solutions of this application.

[0053] The directory table, or DIR_TABLE, is one of the key-value tables in the unified metadata architecture of this application. It is used to implement hierarchical directory structure management in file system semantics, support high-frequency metadata operations such as lookup and readdir, and is the carrier for implementing POSIX protocol path resolution.

[0054] The index node table, or INODE_TABLE, is used to store the metadata attributes and data block distribution information of files / objects.

[0055] The object table, or OBJ_TABLE, is used to implement flat index management for object storage semantics, supporting S3 protocol operations such as object PUT, GET, and LIST.

[0056] Distributed key-value storage, also known as distributed KV storage, is preferred in this application embodiment as TiKV as the underlying distributed KV storage. Other key-value storage systems with distributed transaction capabilities, such as RocksDB, LevelDB, and RADOS, can also be used. This application does not impose any special restrictions on this.

[0057] Storage credit refers to the amount of metadata caching resources and scheduling service priority granted to a single AI large model training task based on its full lifecycle business attributes, operational performance, resource utilization compliance, and fault impact scope by a unified metadata hybrid storage system.

[0058] Metadata resource tokens, or resource tokens for short, are the smallest rights carrier for storage trust. One unit of storage trust corresponds to one resource token. Each token carries two rights: first, the right to occupy a fixed-size metadata cache page; second, the right to allow training requests to enter the global scheduling queue, which is the unique interaction credential between the executor and the cache shard.

[0059] The execution-cache shard architecture refers to a unique pair of task-specific metadata execution bodies and task-specific sample cache shards corresponding to each independent AI training task.

[0060] The POSIX protocol, or Portable Operating System Interface, is the file storage semantic standard protocol supported by this application. It is used to provide a standard file system access interface for AI training frameworks and high-performance computing applications, including standard file operations such as create, delete, lookup, readdir, open, read, and write.

[0061] The S3 protocol, or Simple Storage Service Protocol, is the object storage semantic standard protocol supported by this application. It provides standard object storage access interfaces for scenarios such as model archiving, sample set management, and log storage, including standard object operations such as PUT, GET, DELETE, and LIST.

[0062] In this application embodiment, the hybrid storage method for the unified metadata layer is described in conjunction with... Figures 1 to 3 By configuring the directory table, inode table, and object table in a coordinated manner, this method achieves native compatibility with both POSIX and S3 protocols within the same metadata structure, eliminating the problems of fragmented file and object semantics, metadata redundancy, complex synchronization, and poor consistency in traditional hierarchical storage architectures. The execution flow of this method includes three steps, and the specific implementation of each step is described in detail below.

[0063] Step 1: Receive storage access requests from each node and parse the corresponding protocol type and target data path.

[0064] In the embodiments of this application, such as Figure 2As shown, the hybrid storage system with a unified metadata layer provides both POSIX protocol gateways and S3 protocol gateways to the outside world. The two gateways share the same unified metadata processing layer. Nodes can initiate storage access requests through either protocol gateway. The system can handle concurrent requests from different protocol gateways at the same time and ensure the consistency of metadata when accessing the same dataset through different protocols.

[0065] It should be noted that in this embodiment, the POSIX protocol gateway is implemented based on the FUSE (User Space File System) framework, or it can be implemented based on the kernel-mode VFS interface. This application does not make any special restrictions on this. The S3 protocol gateway is implemented based on the HTTP / HTTPS protocol, is compatible with the Amazon S3 protocol specification, supports the standard S3 API interface, and can seamlessly connect to various nodes, AI training frameworks and data processing tools that are compatible with the S3 protocol.

[0066] Next, after receiving the storage access request initiated by the node through the protocol gateway, the system first performs protocol type parsing and identification. Specifically, the system identifies the protocol type of the request as either a POSIX protocol request or an S3 protocol request based on the access port, request header characteristics, and request format.

[0067] Those skilled in the art will understand that POSIX protocol requests are typically initiated based on file system mounting, and the request is transmitted to the gateway in the form of file operation instructions, including information such as path name, file operation type, and operation parameters; S3 protocol requests are typically initiated based on HTTP / HTTPS requests, and the request includes information such as request method, URI, request header, and request body, wherein the URI includes the bucket name and object path, which will not be elaborated in this application.

[0068] After completing protocol type identification and legality verification, the system performs target data path parsing. For requests of different protocol types, the corresponding path parsing rules are used to extract a unified metadata operation target path.

[0069] For POSIX protocol requests, the system parses the absolute path string in the request, and according to the hierarchical directory rules of the POSIX file system, splits the path into the root directory, each level of parent directory, and the target file name / directory name. At the same time, it determines the file system instance identifier corresponding to the path, i.e., FSID.

[0070] It should be noted that in the embodiments of this application, each independent file system instance corresponds to a unique FSID. The FSID is a unique identifier that distinguishes different file system instances and is also a component of the row key of the directory table and the inode table, thereby realizing the isolation management of multi-tenant and multi-file system instances.

[0071] For an S3 protocol request, the system first parses the bucket name (Bucket) and the complete path of the object (ObjectKey) from the request URI. Then, the system maps the bucket name to the corresponding file system instance FSID and splits the object path into parent directories and target object names according to the POSIX hierarchical directory rules, thereby achieving a unified mapping between S3 object paths and POSIX file system paths.

[0072] Those skilled in the art will understand that, through this path mapping rule, this application achieves the unification of the flat namespace of S3 object storage and the hierarchical namespace of the POSIX file system. The same dataset can be accessed via the S3 protocol using an object path or via the POSIX protocol using a file path, without any data migration or metadata synchronization, thus eliminating the semantic fragmentation problem in traditional layered architectures.

[0073] It should be noted that, in this embodiment of the application, for the common prefix enumeration operation in the S3 protocol, namely the LIST operation with delimiter, the system can quickly complete the enumeration of objects under a specified prefix by means of the row key prefix matching feature of the directory table. Unlike traditional object storage systems, it does not need to perform full key-value traversal and string parsing, which greatly improves the execution efficiency of LIST operation. It is especially suitable for the directory traversal requirements of millions of small file sample sets in AI training scenarios.

[0074] Step 2: Based on the preset directory table, index node table, and object table, perform metadata operations corresponding to the protocol type.

[0075] In this embodiment, the preset directory table, index node table, and object table are strongly associated through index node numbers, i.e., Ino, as follows: Figure 2 As shown. Those skilled in the art will understand that in traditional hybrid storage solutions, file storage and object storage maintain independent metadata indexing systems. The file system relies on the inode / dentry mechanism, while object storage relies on a flat index with path strings as keys. There is a lack of direct mapping between the two, resulting in semantic fragmentation and complex synchronization. However, this application, through the linked configuration of the directory table, inode table, and object table, natively supports both POSIX and S3 dual-protocol semantics within the same metadata structure, eliminating the various problems caused by the two metadata systems.

[0076] In this embodiment, the directory table, index node table, and object table are constructed in the following ways:

[0077] (1) Construction of the directory table (DIR_TABLE)

[0078] In this embodiment, a directory table is constructed using the file system identifier FSID, the parent directory inode number PIno, and the file name FileName as the row key. The value of the directory table stores the inode number Ino corresponding to the sub-file / subdirectory, as well as the extended attribute information of the directory entry.

[0079] It should be noted that the row key configuration of the directory table adopts a three-segment combination structure. FSID is the first prefix, ensuring that directory entry data of different file system instances are isolated in the underlying KV storage; PIno is the second prefix, i.e., the inode number of the parent directory, ensuring that all subdirectory entries under the same parent directory are stored sequentially and continuously in the KV storage; FileName is the third segment, i.e., the name of the subfile / subdirectory, ensuring that directory entries under the same parent directory are unique. Those skilled in the art will understand that this three-segment row key configuration can adapt to the lookup and readdir operation requirements of the POSIX file system: for lookup operations, the corresponding row key can be accurately located based on the parent directory PIno and the target file name FileName, and the inode of the subfile can be quickly obtained, with a time complexity of O(1); for readdir operations, a range scan using FSID+PIno as a prefix can be performed to sequentially obtain all subdirectory entries under the parent directory without performing a full table traversal, greatly improving the execution efficiency of directory traversal, especially adapting to the high-frequency readdir operation requirements of DataLoader for massive small file directories in AI training scenarios.

[0080] In addition to the sub-file Ino number, the value fields of the directory table can also store extended information such as the type identifier (file / directory / symbolic link), creation time, and permission bits of the directory item. This further improves the efficiency of lookup operations, avoids frequent access to the index node table, and reduces the access pressure on the underlying key-value storage.

[0081] (2) Construction of the index node table (INODE_TABLE)

[0082] In this embodiment, an inode table is constructed using the file system identifier FSID and the inode number Ino as the row key. The values ​​in the inode table store the data block distribution information of the file / object, as well as the full metadata attributes of the file / object.

[0083] It should be noted that the inode table is the carrier of the unified semantics of the two protocols in this application. Whether it is a file created by the POSIX protocol or an object uploaded by the S3 protocol, it will correspond to a unique record in the inode table and share the same inode metadata system. The row key of the inode table adopts a two-part combination structure. FSID is the first prefix to ensure the isolation of inode data in different file system instances; Ino is the second prefix, which is the globally unique number of the inode. Each file / object corresponds to a unique Ino number, which is the anchor point connecting the directory table and the object table.

[0084] In this embodiment, the value fields of the inode table store information including, but not limited to: file / object size (Size), file type (Type), creation time (Ctime), modification time (Mtime), access time (Atime), permission bits (Mode), user ID (UID), user group ID (GID), hard link count (Nlink), data block index information (Blocks), and extended attribute (xattr). The data block index information (Blocks) records the actual distribution location of the file / object data in the underlying distributed block storage, serving as the basis for data read / write operations. The extended attribute (xattr) can be used to store S3 protocol object extended information, including Content-Type, ETag, and custom metadata (UserMeta), thereby achieving unified storage of S3 object metadata and POSIX file metadata, eliminating the need to maintain two separate metadata systems.

[0085] Those skilled in the art will understand that this configuration of the index node table enables the fusion of file and object metadata. Regardless of whether the access request is made using the POSIX protocol or the S3 protocol, it will ultimately point to the same record in the index node table, ensuring the consistency of metadata for dual-protocol access and eliminating the overhead and consistency risks caused by synchronizing two sets of metadata in traditional solutions.

[0086] (3) Construction of the object table (OBJ_TABLE)

[0087] In this embodiment of the application, an object table is constructed using the bucket name (Bucket) and the object path (Object) as the row keys. The values ​​in the object table store the index node number (Ino) corresponding to the data, as well as the S3 protocol extension information of the object.

[0088] It should be noted that the object table is the carrier for implementing the S3 protocol semantics in this application. Its row key adopts a two-part composite structure: Bucket is the first prefix, ensuring data isolation between objects in different storage buckets; Object is the second prefix, which is the complete path of the object, consistent with the ObjectKey in the S3 protocol, thus enabling accurate object lookup and prefix enumeration operations according to the S3 protocol standard. The value field of the object table stores the Ino number corresponding to the object. Through this Ino number, it is possible to link to the corresponding record in the index node table to obtain the object's metadata attributes and data block distribution information, thereby realizing the mapping from the S3 protocol to a unified metadata system.

[0089] Furthermore, the value fields of the object table can also store S3 protocol-specific extended information, including the object's storage type, version number, lifecycle configuration, encryption information, etc., thus ensuring full compatibility with various advanced features of the S3 protocol. Those skilled in the art will understand that the object table and directory table are strongly associated through Ino numbers. An object uploaded via the S3 protocol will simultaneously create an object index in the object table, a corresponding directory entry in the directory table, and an inode record in the inode table, thereby enabling dual-protocol access to the same object. Similarly, a file created using the POSIX protocol will also synchronously create a corresponding object index in the object table, enabling S3 protocol access to the same file, thus achieving the effect of write-once, dual-semantic access.

[0090] Next, after constructing the directory table, inode table, and object table, the system executes the corresponding metadata operations based on the parsed protocol type and target data path. In this embodiment, metadata operations include, but are not limited to, file / object creation, deletion, modification, querying, directory traversal, and object prefix enumeration. The system executes differentiated metadata operation processes for different protocol types and operation types.

[0091] The following provides a detailed description of the file / object creation, deletion, modification, query, directory traversal, and object prefix enumeration types in the embodiments of this application.

[0092] Metadata operation process for file creation request (create)

[0093] When a file creation request is received according to the POSIX protocol, the system performs the following metadata operation process:

[0094] The first step is path resolution and parent directory verification. The system resolves the absolute path of the target file, queries the directory table level by level to obtain the PIno number corresponding to the parent directory of the target file, and verifies the existence of the parent directory and the write permissions of the node to the parent directory. If the parent directory does not exist or the permissions are insufficient, the corresponding error code is returned directly and the operation is terminated.

[0095] The second step is to assign a globally unique Ino number. The system uses its built-in inode allocator to assign a globally unique Ino number to the file to be created. This number is unique under the corresponding FSID and will not duplicate existing inode numbers. It should be noted that in this embodiment, the inode allocator adopts a pre-allocation segmented allocation strategy. Each metadata node pre-allocates a continuous range of Ino numbers to avoid number conflicts in a distributed environment and improve allocation efficiency.

[0096] The third step is directory table operations. The system inserts a new record into the directory table. The key of this row is FSID+PIno+FileName, and the value is the newly assigned Ino number and directory entry extension information, thereby adding a directory entry for the target file in the parent directory.

[0097] The fourth step is inode table operations. The system inserts a new record into the inode table. The key of this row is FSID+Ino, and the value is the initialized file metadata attributes, including file type, size, creation time, modification time, permission bits, data block index, and other information, thus completing the inode initialization.

[0098] Step 5: Object table operations. The system inserts a corresponding record into the object table. The key of this row is the full path of the Bucket and the value is the newly assigned Ino number and S3 extension information. This creates an access index for the file using the S3 protocol, ensuring that the file can be accessed directly via the S3 protocol after its creation.

[0099] Metadata operation process for file lookup requests

[0100] When a POSIX protocol file lookup request is received, the system performs the following metadata operation:

[0101] The first step is path resolution and prefix matching. The system parses the absolute path of the target file, breaks it down into the parent directory path and the target file name, and queries the directory table level by level to obtain the PIno number corresponding to the parent directory.

[0102] The second step is accurate directory table lookup. The system uses FSID+PIno+FileName as the row key to accurately search the directory table and obtain the Ino number corresponding to the target file. If the record does not exist, an error code indicating that the file does not exist is returned.

[0103] The third step is to query the inode table. The system uses FSID+Ino as the row key to query the inode table, obtain the full metadata attributes of the file, including file size, permissions, data block distribution, etc., and complete the lookup operation.

[0104] Those skilled in the art will understand that this lookup operation process only requires two accurate queries of the KV storage, with a time complexity of O(1). Compared with the multi-level directory traversal of the traditional file system and the remote call of MDS, it greatly reduces the query latency. In particular, it is suitable for the high-frequency lookup operation requirements of DataLoader on sample files in AI training scenarios, which can significantly reduce the sample loading time and reduce the idle time of GPU due to data waiting.

[0105] Metadata operation flow of directory traversal request (readdir)

[0106] When a POSIX-compliant directory traversal request (readdir) is received, the system performs the following metadata operations:

[0107] The first step is to obtain the Ino number of the target directory. The system parses the path of the target directory and obtains the corresponding Ino number, i.e., PIno, through a lookup operation.

[0108] The second step is a range scan of the directory table prefix. The system uses FSID+PIno as the row key prefix to perform a range scan of the directory table, obtaining all subdirectory entries under the parent directory, including the names of all subfiles / subdirectories and their corresponding Ino numbers.

[0109] The third step is metadata population and result return. Based on the Ino numbers obtained from the scan, the system performs batch queries on the inode table to obtain the metadata attributes of each subdirectory item, formats them according to the POSIX protocol specification, and returns them to the node, completing the readdir operation.

[0110] It should be noted that, because the row key configuration of the directory table ensures that all subdirectory entries under the same parent directory are stored contiguously in the KV storage, the efficiency of this range scan operation is extremely high. There is no need to perform a full table traversal. Even if there are millions of small files in the directory, the traversal operation can be completed in milliseconds, which solves the performance bottleneck of traditional object storage and file systems in the scenario of traversing a large number of small file directories.

[0111] Metadata manipulation process for file deletion requests (unlink)

[0112] When a POSIX protocol file deletion request (unlink) is received, the system performs the following metadata operation:

[0113] The first step is path resolution and parent directory verification. The system resolves the absolute path of the target file, queries the directory table level by level to obtain the PIno number of the parent directory and the Ino number of the target file, verifies the existence of the file and the deletion permissions of the node, and returns an error code if the verification fails.

[0114] The second step is the directory table deletion operation. The system deletes the row key record corresponding to FSID+PIno+FileName in the directory table and removes the file directory entry from the parent directory.

[0115] The third step is the object table deletion operation. The system deletes the row key record corresponding to the full path of Bucket+Object in the object table and removes the S3 access index of that file.

[0116] The fourth step is the inode table update operation. The system decrements the hard link count of the inode record corresponding to the file in the inode table by 1. If the hard link count reaches zero, the inode is deleted, and the underlying data block reclamation process is triggered.

[0117] In this embodiment of the application, the various operations of the S3 protocol are described in detail, taking the four most typical operations—object upload, object download, object enumeration, and object deletion—as examples:

[0118] Metadata operation process for object upload request (PUT)

[0119] When an S3 protocol object upload request (PUT) is received, the system performs the following metadata operation:

[0120] The first step is request parsing and path mapping. The system parses the PUT request to obtain the bucket name, object full path ObjectKey, object metadata, and data content. It maps the bucket to the corresponding FSID and the objectKey to the hierarchical path of the POSIX file system, separating the parent directory path and the target object name.

[0121] The second step is parent directory checking and automatic creation. The system queries the directory table level by level to check if the parent directories corresponding to the object path exist at each level. If an intermediate directory does not exist, the system automatically creates the corresponding directory inode and directory table record to ensure the integrity of the hierarchical path. It should be noted that this automatic directory creation operation is also encapsulated in an atomic transaction to ensure consistency between directory creation and object creation.

[0122] The third step is to assign a globally unique Ino number. The system uses the inode allocator to assign a globally unique Ino number to the object to be uploaded.

[0123] The fourth step is object table operations. The system inserts a new record into the object table with the row key Bucket+ObjectKey and the value being the newly assigned Ino number and S3 extended information, including Content-Type, ETag, and custom metadata.

[0124] Step 5: Directory table operations. The system inserts the corresponding directory entry record into the directory table, with the row key being FSID + parent directory PIno + object name, and the value being the newly assigned Ino number. This creates a POSIX protocol directory entry index for the object.

[0125] Step 6: Inode table operations. The system inserts the corresponding inode record into the inode table, with the row key being FSID+Ino and the value being the initialized metadata attributes, including object size, creation time, modification time, permission bits, and S3 extension information synchronized from the object table. At the same time, the distribution information of the object data blocks is written to the Blocks field.

[0126] After the operation is completed, the object can be accessed via both the S3 protocol GET request and the POSIX protocol open / read request, achieving seamless interoperability between the two semantics.

[0127] Metadata manipulation flow for object download request (GET)

[0128] When a GET request for object download using the S3 protocol is received, the system performs the following metadata operation:

[0129] The first step is request parsing and object table lookup. The system parses the GET request to obtain the Bucket name and ObjectKey. Using Bucket + ObjectKey as the row key, it accurately queries the object table to obtain the corresponding Ino number. If the record does not exist, it returns an error code indicating that the object does not exist.

[0130] The second step is to query the inode table. The system uses FSID+Ino as the row key to query the inode table, obtains the full metadata attributes of the object, including object size, data block distribution information, ETag, Content-Type, etc., and verifies the node's download permissions.

[0131] The third step is data reading and result return. Based on the data block distribution information in the index node table, the system reads the complete data of the object from the underlying distributed block storage, encapsulates the HTTP response according to the S3 protocol specification, and returns it to the node, completing the GET operation.

[0132] Those skilled in the art will understand that this GET operation only requires two accurate KV queries to complete the metadata acquisition. The process is simple and efficient with extremely low latency. At the same time, since the metadata is shared with the POSIX protocol, objects uploaded via the S3 protocol can be directly accessed via the POSIX protocol without any metadata synchronization, eliminating the data migration and synchronization overhead of cross-protocol access in traditional solutions.

[0133] Metadata operation flow of object enumeration request (LIST)

[0134] When an S3 protocol object enumeration request (LIST) is received, the system performs the following metadata operation:

[0135] The first step is request parsing and prefix matching. The system parses the LIST request, obtains parameters such as Bucket name, prefix, delimiter, and maximum number of entries (MaxKeys), maps the Bucket to the corresponding FSID, and maps the prefix to the directory path of the POSIX file system.

[0136] The second step is a range scan of the directory table. The system uses FSID + parent directory PIno + Prefix as the row key prefix to perform a range scan of the directory table, obtain all directory entry records that match the prefix, and aggregate the directory hierarchy according to the delimiter parameter to distinguish between ordinary objects and subdirectory prefixes.

[0137] The third step is result formatting and return. Based on the Ino numbers obtained from the scan, the system performs batch queries on the index node table to obtain the metadata attributes of each object, including object size, last modified time, ETag, etc. After formatting according to the S3 protocol specification, it is encapsulated into an XML / JSON response and returned to the node, completing the LIST operation.

[0138] It should be noted that traditional object storage LIST operations require traversing, matching, and aggregating all object keys in the entire Bucket, resulting in extremely poor performance in scenarios with massive amounts of small files. In contrast, this application transforms LIST operations into prefix range scanning of the directory table by configuring the hierarchical row keys of the directory table, improving execution efficiency by 1-2 orders of magnitude. This can meet the prefix enumeration requirements of massive sample sets in AI training scenarios.

[0139] Metadata operation flow of object deletion request (DELETE)

[0140] When a DELETE request for an object is received via the S3 protocol, the system performs the following metadata operation:

[0141] The first step is request parsing and object table lookup. The system parses the DELETE request, obtains the Bucket name and ObjectKey, and uses Bucket + ObjectKey as the row key to query the object table, obtains the corresponding Ino number, and verifies the existence of the object and the deletion permissions of the node.

[0142] The second step is the object table deletion operation. The system deletes the corresponding row key record in the object table and removes the object's S3 access index.

[0143] The third step is the directory table deletion operation. The system parses the parent directory PIno corresponding to the object path and the object name, deletes the row key record corresponding to FSID+PIno+object name in the directory table, and removes the POSIX directory entry index.

[0144] The fourth step is the inode table update operation. The system updates the inode records in the inode table, decrements the hard link count by 1, and deletes the inode record if the count reaches zero, triggering the underlying data block reclamation process.

[0145] After the operation is completed, the object is deleted synchronously in both the POSIX and S3 protocols, and there will be no consistency problem where the object is deleted on one side but still accessible on the other side.

[0146] Step 3: Encapsulate the metadata operations into atomic transaction commits, synchronously update the corresponding records in the directory table, index table, and object table, and complete the read and write operations of the target data.

[0147] It should be noted that in this embodiment, atomic transactions are implemented based on the distributed transaction capabilities of the underlying distributed KV storage, preferentially adopting TiKV's Percolator distributed transaction model. This model supports both optimistic and pessimistic transaction modes, which can be flexibly selected according to the business scenario. For high-frequency operations such as creating and deleting small files with high concurrency in AI training scenarios, the optimistic transaction mode is preferred to reduce transaction overhead and improve concurrency performance. For complex transactions involving multiple tables and rows, such as writing large files and renaming directories, the pessimistic transaction mode is preferred to avoid retry overhead caused by transaction conflicts and ensure the success rate of operations. This application does not impose any restrictions on this.

[0148] As can be seen, the hybrid storage method of the unified metadata layer provided in this application, through the unified metadata architecture of INODE_TABLE, DIR_TABLE, and OBJ_TABLE, is natively compatible with POSIX and S3 dual-protocol semantics in the same metadata structure, realizes the unification of file and object storage namespaces, eliminates the redundant metadata storage and cross-system synchronization overhead of traditional layered architectures, and ensures strong consistency of cross-protocol access by atomically committing all metadata operations through distributed transactions. At the same time, it greatly simplifies the path resolution and query process, significantly reduces the latency of high-frequency metadata operations such as lookup and list, and achieves an order-of-magnitude improvement in training data loading efficiency in AI large model training scenarios with millions of small files. The key-value configuration based on distributed KV storage also supports flexible sharding and horizontal scaling, breaking through the performance bottleneck of traditional centralized metadata nodes.

[0149] Furthermore, for existing large-model training hybrid storage clusters, i.e., in embodiments where nodes are AI training nodes, in order to prevent low-priority tasks from preempting resources, the method further includes:

[0150] Each AI training node is assigned a corresponding resource token. The initial number of resource tokens for each AI training node is determined based on its storage credit limit. All AI training nodes and their corresponding resource tokens are managed in a temporary intermediary. The higher the storage credit limit of an AI training node, the more resource tokens it will have in the temporary intermediary at the beginning.

[0151] Based on a preset random competition function, a competitive selection is made among all AI training nodes that meet the competition conditions in the temporary intermediary. The random competition function satisfies the following condition: the more resource tokens an AI training node has, the higher its probability of being selected.

[0152] The selected AI training nodes are added to the scheduling body and the tasks are scheduled and executed in a preset order;

[0153] When the selected AI training node is added to the scheduling body, the number of resource tokens of the AI ​​training node in the temporary intermediate body is reduced by a preset ratio.

[0154] Collect the training task convergence of each AI training node, and update the number of resource tokens of the corresponding AI training node in the temporary intermediate based on the change in training convergence: when the training result convergence increases, increase the resource tokens of the corresponding AI training node; when the training result convergence decreases, decrease the resource tokens of the corresponding AI training node, until all AI training tasks are scheduled and executed.

[0155] In this embodiment, the AI ​​training node involved is a physical server or virtual computing instance in a distributed AI training cluster that has model training computing power and supporting storage resources. Each AI training node can correspond to an independent training task instance, but this application is not limited to this.

[0156] In this embodiment of the application, a corresponding resource token is allocated to each AI training node. It should be noted that the resource token is a quantitative identifier used to represent the scheduling competition authority of the AI ​​training node. Each token corresponds to a fixed weight percentage in a single scheduling competition. The number of tokens is directly related to the scheduling competition capability of the node. Moreover, the resource token is only effective within the control scope of the temporary intermediary and does not have the direct exchange attribute of cluster resources. It is only used for probability weight calculation in the scheduling competition process.

[0157] The initial number of resource tokens for an AI training node is determined based on its storage credit limit. It should be noted that the storage credit limit is a quantitative evaluation value of the AI ​​training node's storage resource capabilities and historical task performance, used to define the initial scheduling authority benchmark for the node. In this embodiment, the evaluation dimensions of the storage credit limit may include one or more of the node's storage hardware capacity, storage IO bandwidth, data persistence availability level, historical task scheduling performance rate, and data read / write failure rate. This application is not limited to these. The higher the storage credit limit of the AI ​​training node, the more resource tokens it initially has in the temporary intermediary. The two are positively correlated, thereby achieving the matching of initial scheduling authority and node storage resource capabilities.

[0158] All AI training nodes and their corresponding resource tokens are managed within a temporary intermediary. It should be noted that this temporary intermediary is a token management container with access control, an independent token management unit separate from the scheduling execution chain. It maintains the unique identifier of all online AI training nodes, their corresponding resource token balance, storage credit limit, real-time training convergence data, and task execution status. In this embodiment, the temporary intermediary can be implemented using a distributed key-value storage architecture to ensure strong consistency and real-time update capability of the token data. This application is not limited to this. Furthermore, the temporary intermediary can perform full-chain access control on token operations for each AI training node, allowing only authorized scheduling core modules to perform token addition, deletion, query, and locking operations, thus preventing unauthorized operations from tampering with token data.

[0159] In this embodiment, the random competition function is implemented using a weighted random roulette wheel algorithm. The probability of a single AI training node being selected in a single competition is the ratio of the number of valid resource tokens currently held by the AI ​​training node in the temporary intermediary to the total number of valid resource tokens held by all AI training nodes participating in this competition in the temporary intermediary. Based on the calculated selection probability of each AI training node, a continuous random number interval of corresponding length is assigned to each node. The AI ​​training node selected in this competition is determined by the interval in which the generated globally unique random number falls.

[0160] Specifically, based on a preset random competition function, a competitive selection is made among all AI training nodes in the temporary intermediary that meet the competition conditions. It should be noted that the random competition function is an algorithmic rule used to determine the selected object in a single scheduling among multiple AI training nodes to be scheduled. The random competition function satisfies the following: the more resource tokens an AI training node has, the higher its probability of being selected. In this embodiment, the random competition function is implemented using a weighted random roulette algorithm. Specifically, in a single scheduling competition, the probability of a single AI training node being selected in a single competition is the ratio of the number of valid resource tokens currently held by the AI ​​training node in the temporary intermediary to the total number of valid resource tokens held by all AI training nodes participating in this competition in the temporary intermediary. The sum of the selection probabilities of all participating AI training nodes is 1. The probability proportion of each node is completely determined by the proportion of its valid tokens in the total token pool. The more tokens a node has, the higher its corresponding probability proportion, thereby achieving a positive correlation between the number of tokens and the selection probability.

[0161] It should be noted that the weighted random roulette wheel algorithm is a well-known technology in the field. The core concept of this embodiment is not the weighted random roulette wheel algorithm itself, but the configuration of a random competition function to realize the storage scheduling of AI training nodes. In other embodiments, other feasible methods can be selected to realize random competition, and this application does not limit or elaborate on this.

[0162] Based on the calculated selection probability of each AI training node, a continuous random number interval of corresponding length is assigned to each node. The AI ​​training node selected in this competition is determined by the interval in which the generated globally unique random number falls. In this embodiment, the random number interval can be allocated sequentially according to the lexicographical order of the node's unique identifier, or sequentially according to the order of token quantity from high to low. This application is not limited to this. It should be noted that the globally unique random number can be generated using a true random number generator that conforms to cryptographic security standards to ensure the randomness and fairness of a single competition process and avoid unexpected scheduling bias caused by fixed rules.

[0163] In this embodiment, the selected AI training nodes are added to the scheduling body and executed according to a preset order. It should be noted that the scheduling body is an ordered scheduling queue with a sequence lock, serving as an execution buffer unit for storing AI training nodes that have been selected through contention and are awaiting resource scheduling. In this embodiment, the scheduling body adopts a first-in-first-out ordered queue structure. The selected AI training nodes are added to the tail of the ordered queue in the order of their selection. The order of nodes within the queue cannot be tampered with. It should be noted that the sequence lock is used to ensure the serialized execution of write and read operations in the queue, avoiding data corruption in multi-threaded concurrent scheduling scenarios and ensuring the uniqueness and immutability of the scheduling order. The scheduler sequentially retrieves AI training nodes from the head of the ordered queue, allocates matching training storage resources and computing resources to them, and executes the corresponding training tasks. In this embodiment, the allocated training storage resources may include distributed storage mounting permissions and IO bandwidth quotas matching the node's storage trust. The computing resources may include one or more of the following: CPU core quota, GPU computing power quota, and memory quota. This application is not limited to these.

[0164] Furthermore, after the training task of an AI training node is scheduled and executed, if the node still has remaining valid resource tokens in the temporary intermediary and the task has not reached the preset convergence termination threshold, it will be reinstated into the competitive selection pool to participate in the next round of random competition; if the task reaches the preset convergence termination threshold, the remaining resource tokens of the node will be cleared and it will be removed from the competitive selection pool. It should be noted that the preset convergence termination threshold is a preset completion condition for the training task, which may include one or more of the following: the loss function converges to the preset threshold, the model verification accuracy reaches the preset target, and the number of training iterations reaches the preset upper limit. This application is not limited to these.

[0165] It should be noted that training convergence is a standardized numerical value that quantifies the degree to which an AI model approaches a preset target during training, and thus the quality of the training effect. Training convergence can be measured in units of complete training epochs, calculating the model's business metrics (classification accuracy, detection mAP, F1 score, etc.) on the validation set and comparing them with the metrics of the previous epoch to obtain the metric improvement rate. A higher metric improvement rate indicates higher convergence; a decrease in metrics directly leads to a decrease in convergence. Alternatively, it can be measured in units of fixed training iteration periods, calculating the average loss value of the current period and comparing it with the average loss value of the previous period to obtain the loss reduction rate. A higher loss reduction rate indicates higher convergence; an increase in loss directly leads to a decrease in convergence. This application does not elaborate on or limit this aspect.

[0166] In this embodiment, the step of deducting the current resource token quantity of the AI ​​training node in the temporary intermediary according to a preset ratio specifically includes: the preset token deduction ratio is configurable within a range of 10%-30%; when the AI ​​training node is selected and completes the queuing operation of the scheduling body, the number of tokens corresponding to the product of the current balance and the preset deduction ratio is simultaneously deducted from the resource token balance of the node in the temporary intermediary; wherein, the resource token balance after deduction is not less than 0, and when the balance after deduction is 0, the AI ​​training node is marked as in a suspended competition state until it replenishes its resource tokens to a value greater than 0 through training convergence, and then regains its competition qualification.

[0167] Specifically, when a selected AI training node is added to the scheduling body, the number of resource tokens held by that AI training node in the temporary intermediary is reduced by a preset ratio. This preset token reduction ratio is configurable from 10% to 30%, and can be flexibly configured based on the overall scheduling frequency of the cluster, the scale of the training tasks, and the number of nodes; this application is not limited to this. It should be noted that when an AI training node is selected and completes its enqueue operation in the scheduling body, the number of tokens corresponding to the product of the current balance and the preset reduction ratio is simultaneously deducted from the node's resource token balance in the temporary intermediary. This token consumption after a single selection during scheduling avoids a single node with a high number of tokens continuously occupying scheduling resources for an extended period, achieving a fair distribution of scheduling resources among multiple nodes. In addition, the resource token balance after deduction is not less than 0. When the balance after deduction is 0, the AI ​​training node is marked as suspended from competition until it replenishes its resource tokens to a value greater than 0 through training convergence. It should be noted that AI training nodes in the suspended competition state cannot participate in the subsequent scheduling competition process. They can only restore their competition rights through token replenishment operations. This is to restrict the scheduling rights of nodes that have no training effect improvement.

[0168] In this embodiment of the application, updating the resource token quantity of the corresponding AI training node in the temporary intermediate based on the change in training convergence specifically includes:

[0169] The training convergence is at least one quantifiable indicator among the loss function decline rate of the training task and the improvement rate of model validation accuracy.

[0170] Using a preset training iteration cycle as the unit, collect the training convergence value of each AI training node in the current iteration cycle and compare it with the convergence benchmark value of the previous iteration cycle.

[0171] When the convergence value of the current iteration period is higher than the baseline value of the previous iteration period, the corresponding number of resource tokens are added to the temporary intermediate for the AI ​​training node.

[0172] When the convergence value of the current iteration period is lower than the baseline value of the previous iteration period, the corresponding number of resource tokens are deducted from the resource token balance of the AI ​​training node in the temporary intermediate; wherein, the change in convergence is linearly positively correlated with the increase or decrease in the number of resource tokens.

[0173] Specifically, the convergence of training tasks for each AI training node is collected, and the number of resource tokens for the corresponding AI training node in the temporary intermediary is updated based on the changes in training convergence: when the training convergence increases, the resource tokens for the corresponding AI training node are increased; when the training convergence decreases, the resource tokens for the corresponding AI training node are decreased, until all AI training tasks are scheduled and executed. It should be noted that training convergence is a quantifiable indicator used to characterize the training progress and training effect of AI training tasks. The training convergence is at least one quantifiable indicator among the loss function decline rate and the model validation accuracy improvement rate of the training task, and this application is not limited to this.

[0174] In this embodiment, a preset training iteration cycle is used as the unit to collect the training convergence value of each AI training node in the current iteration cycle, and compare it with the convergence benchmark value of the previous iteration cycle. It should be noted that the preset training iteration cycle can be flexibly configured according to the type of training task, model size, and iteration duration. It can be set as a single training iteration, multiple training iterations, or a statistical cycle of fixed duration, and this application is not limited to this. When the convergence value of the current iteration cycle is higher than the benchmark value of the previous iteration cycle, the corresponding number of resource tokens is added to the temporary intermediate for the AI ​​training node; when the convergence value of the current iteration cycle is lower than the benchmark value of the previous iteration cycle, the corresponding number of resource tokens is deducted from the resource token balance of the AI ​​training node in the temporary intermediate. The change in convergence is linearly positively correlated with the increase or decrease in the number of resource tokens. Through the dynamic binding of training convergence and token quantity, the real-time linkage between training effect and scheduling authority is realized. The better the training effect, the more tokens can be replenished, thereby increasing the probability of being selected for scheduling and obtaining more scheduling execution opportunities. Conversely, the scheduling authority is gradually reduced.

[0175] like Figure 4 As shown in the storage scheduling of the AI ​​training nodes in the nth round, node 1 has 1000 resource tokens, node 2 has 1500, node 3 has 1835, node 4 has 2300, node 5 has 1680, and node 6 has 1730. Then, based on random competition, the scheduling sequence is determined to be node 4, node 5, and node 3. At this point, AI training data is first stored on node 4, followed by nodes 5 and 3 in sequence, as follows. Figure 5As shown in the storage scheduling of the AI ​​training nodes in the (n+1)th round, the convergence of node 4 in the nth round of training decreases. At this time, based on the above embodiment, it is known that the resource token decreases by 10-30%, for example, by 30%. Then, in the storage scheduling of the AI ​​training nodes in the (n+1)th round, the resource token of node 4 is updated to 1610, thus reducing the selection probability of the random competition in the (n+1)th round. Therefore, based on the configuration of random competition combined with temporary intermediate and scheduling body in this embodiment, the storage scheduling of AI training data can be realized based on resource token control. Thus, high-quality AI training data can be automatically updated to higher priority and have a higher probability of being selected in subsequent training storage. Conversely, AI training data with reduced training convergence is automatically updated to lower priority and has a lower probability of being selected by random competition. Thus, through the method of this embodiment, each AI training node can spontaneously form a cyclic update.

[0176] It should be understood that the above is merely one example for ease of understanding, and the resource tokens and other data in this application are not limited to the above data, and will not be exhaustively listed here.

[0177] This application also discloses a hybrid storage system with a unified metadata layer, such as... Figure 6 As shown, it includes:

[0178] Receive module 1 receives storage access requests from each node and parses the protocol type and target data path corresponding to the request;

[0179] Execution module 2 performs metadata operations corresponding to the protocol type based on the preset directory table, index node table, and object table.

[0180] Synchronization module 3 encapsulates metadata operations into atomic transaction commits, synchronously updating the corresponding records in the directory table, index node table, and object table to complete the read and write operations of the target data.

[0181] It should be noted that although TiKV is used as the underlying distributed KV storage and AI large model training is described in detail in the embodiments of this application, the technical solution of this application is not limited to this. The underlying layer can adopt any KV storage system with distributed transaction capabilities, and the application scenario can be extended to any hybrid storage scenario that needs to support dual protocol access of files and objects, such as high-performance computing, massive unstructured data management, and converged media content storage. Any scenario adaptation, parameter adjustment, or structure replacement based on the technical principles of this application shall be within the protection scope of this application.

Claims

1. A hybrid storage method for a unified metadata layer, characterized in that, Includes the following steps: Receive storage access requests from each node and parse the protocol type and target data path corresponding to the request; Based on the preset directory table, index node table, and object table, perform metadata operations corresponding to the protocol type; Metadata operations are encapsulated into atomic transaction commits, which synchronously update the corresponding records in the directory table, index table, and object table to complete the read and write operations of the target data. The node is an AI training node, and the method further includes: Each AI training node is assigned a corresponding resource token. The initial number of resource tokens for each AI training node is determined based on its storage credit limit. All AI training nodes and their corresponding resource tokens are managed in a temporary intermediary. The higher the storage credit limit of an AI training node, the more resource tokens it will have in the temporary intermediary at the beginning. Based on a preset random competition function, a competitive selection is made among all AI training nodes that meet the competition conditions in the temporary intermediary. The random competition function satisfies the following condition: the more resource tokens an AI training node has, the higher its probability of being selected. The selected AI training nodes are added to the scheduling body and the tasks are scheduled and executed in a preset order; When the selected AI training node is added to the scheduling body, the number of resource tokens of the AI ​​training node in the temporary intermediate body is reduced by a preset ratio. Collect the training task convergence of each AI training node, and update the number of resource tokens of the corresponding AI training node in the temporary intermediate based on the change in training convergence: when the training result convergence increases, increase the resource tokens of the corresponding AI training node; when the training result convergence decreases, decrease the resource tokens of the corresponding AI training node, until all AI training tasks are scheduled and executed. The random competition function is implemented using a weighted random roulette wheel algorithm. The probability of a single AI training node being selected in a single competition is the ratio of the number of valid resource tokens currently held by the AI ​​training node in the temporary intermediary to the total number of valid resource tokens held by all AI training nodes participating in this competition in the temporary intermediary. Based on the calculated selection probability of each AI training node, a continuous random number interval of corresponding length is assigned to each node. The AI ​​training node selected in this competition is determined by the interval in which the generated globally unique random number falls. The update of the resource token quantity of the corresponding AI training node in the temporary intermediate based on the change in training convergence specifically includes: The training convergence is at least one quantifiable indicator among the loss function decline rate of the training task and the improvement rate of model validation accuracy. Using a preset training iteration cycle as the unit, collect the training convergence value of each AI training node in the current iteration cycle and compare it with the convergence benchmark value of the previous iteration cycle. When the convergence value of the current iteration period is higher than the baseline value of the previous iteration period, the corresponding number of resource tokens are added to the temporary intermediate for the AI ​​training node. When the convergence value of the current iteration period is lower than the baseline value of the previous iteration period, the corresponding number of resource tokens are deducted from the resource token balance of the AI ​​training node in the temporary intermediate; wherein, the change in convergence is linearly positively correlated with the increase or decrease in the number of resource tokens. The process of adding the selected AI training nodes to the scheduler and executing tasks according to a preset order specifically includes: The scheduling body adopts a first-in-first-out ordered queue structure. Selected AI training nodes are added to the tail of the ordered queue in the order of their selection time, and the order of nodes in the queue cannot be tampered with. The scheduler retrieves AI training nodes sequentially from the head of the ordered queue, allocates matching training storage and computing resources to them, and executes the corresponding training tasks. After the training task of the AI ​​training node is scheduled and executed, if the node still has valid resource tokens remaining in the temporary intermediary and the task has not reached the preset convergence termination threshold, it will be reinstated into the competitive selection pool to participate in the next round of random competition. If a task reaches the preset convergence termination threshold, the remaining resource tokens of that node will be cleared and it will be removed from the competition selection pool.

2. The method according to claim 1, characterized in that, The preset directory table, index node table, and object table are constructed in the following ways: Construct a directory table using the file system identifier, parent directory inode number, and file name as the row key, and store the inode numbers corresponding to the sub-files; An inode table is constructed using the file system identifier and inode number as row keys to store the distribution information of file data blocks; An object table is constructed using the bucket name and object path as row keys, storing the index node number and object extension information corresponding to the data.

3. The method according to claim 1, characterized in that, The process of encapsulating metadata operations into atomic transaction commits, synchronously updating the corresponding records in the directory table, inode table, and object table, includes: The write, update, and delete operations of metadata for the directory table, index node table, and object table are encapsulated into the same distributed transaction; Execute all metadata operations within the transaction and verify the operation results for each table; If all operations are executed successfully, the transaction is committed to complete the synchronization update of the directory table, inode table, and object table; If any operation fails, all operations within the transaction will be rolled back.

4. The method according to claim 1, characterized in that, The step of deducting the current resource token quantity of the AI ​​training node in the temporary intermediary according to a preset ratio specifically includes: the preset token deduction ratio can be configured within a range of 10%-30%; when the AI ​​training node is selected and completes the enqueue operation of the scheduling body, the number of tokens corresponding to the product of the current balance and the preset deduction ratio is simultaneously deducted from the resource token balance of the node in the temporary intermediary; wherein, the resource token balance after deduction is not less than 0, and when the balance after deduction is 0, the AI ​​training node is marked as in a suspended competition state until it replenishes its resource tokens to a value greater than 0 through training convergence, and then regains its competition qualification.

5. The method according to claim 1, characterized in that, The temporary intermediary is a token management container with access control, used to maintain the unique identifier of all online AI training nodes, the corresponding resource token balance, the storage credit limit, real-time training convergence data and task execution status; the scheduling body is an ordered scheduling queue with a sequence lock, used to store AI training nodes that have been selected through competition and are waiting to be scheduled for resource execution.

6. A hybrid storage system with a unified metadata layer, characterized in that, include: The receiving module receives storage access requests from each node and parses the corresponding protocol type and target data path of the request. The execution module performs metadata operations corresponding to the protocol type based on the preset directory table, inode table, and object table. The synchronization module encapsulates metadata operations into atomic transaction commits, synchronously updating the corresponding records in the directory table, index table, and object table to complete the read and write operations of the target data. The hybrid storage system of the unified metadata layer is used to perform the method as described in any one of claims 1-5.

Images