Data block processing method, system and device, storage medium and product

Abstract

The invention discloses a data block processing method, system and device, a storage medium and a product, and relates to the technical field of data storage, and the method comprises the steps: obtaining a tree structure storage table, a leaf node storage table and a data block storage table, and when a node identifier matched with a first node identifier is found in each of the leaf node storage table and the data block storage table, sending a first node identifier to the leaf node storage table; establishing a first hierarchical relationship between the first node and the attribute information and the data block information of the first node; determining first small file list information and adding the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship; and generating a metadata snapshot according to the second hierarchical relationship and at least one third hierarchical relationship of other nodes, and sending the metadata snapshot to the client. According to the method, the metadata snapshot is issued to the client, so that the client directly reads the metadata snapshot from the local cache, the access pressure on a metadata management center is reduced, the metadata access efficiency is improved, and the problem of low supply efficiency of a small file data set in deep learning training operation is solved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data block processing method, system, device, storage medium and product. Background Art

[0002] Currently, more and more AI training jobs are running in cloud data centers. The elastic GPU computing resource capabilities provided by cloud data centers can meet the diverse computing resource requirements of AI training jobs. The supply efficiency of training data, especially large-scale small file data sets, is directly related to the computing performance and efficiency of the distributed deep learning job system in the cloud data center.

[0003] In the related art, for large-scale small file data sets, high-performance storage is usually equipped on the computing nodes of deep learning jobs, and the small file data sets in the remote storage system are copied to the high-performance storage to provide data for deep learning jobs for training. However, due to the limited capacity of high-performance storage, small file data sets that exceed the capacity cannot use this method to accelerate data supply; or, the small file data sets are stored in a parallel file system, and the data in the small file data sets are pulled from the parallel file system and fed to the computing nodes of deep learning jobs. However, due to the large performance bottleneck of the parallel file system in data query and random reading, there will be a mismatch between the data supply efficiency and the computing speed of the computing nodes. Therefore, neither of the above two methods can effectively improve the supply efficiency of small file data sets in deep learning training jobs. Summary of the invention

[0004] The present application provides a data block processing method, system, device, storage medium and product to at least solve the problem of low supply efficiency of small file data sets in deep learning training operations in related technologies.

[0005] The present application provides a data block processing method, which is applied to a metadata management center. The method includes: obtaining a tree structure storage table, a leaf node storage table and a data block storage table, wherein the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and each node identifier of each leaf node in a tree structure corresponding to a plurality of data blocks, the data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of the plurality of leaf nodes stored in each leaf node of the plurality of leaf nodes, the first node is a root node or any leaf node in the tree structure, and the plurality of data blocks are obtained by aggregating a plurality of small files required for a training job.

[0006] Search the leaf node storage table and the data block storage table respectively to see whether there is a node identifier that matches the first node identifier; if matching node identifiers are found, establish a first hierarchical relationship between the first node and the attribute information of the first node and the data block information of the first node; determine the first small file list information included in the data block corresponding to the first node according to the first data block identifier of the data block corresponding to the first node, and add the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship; generate a metadata snapshot according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, and send the metadata snapshot to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job starts, and the other nodes are at least one leaf node other than the first node in the tree structure.

[0007] The present application also provides another data block processing method, which is applied to a client, and the method includes: when a training job is started, loading metadata snapshots corresponding to multiple data blocks from a local cache, the metadata snapshot is generated based on a tree structure storage table, a leaf node storage table and a data block storage table, the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and each node identifier of each leaf node in the tree structure, the data block storage table includes data block information, a data block identifier of each data block stored in each leaf node of the multiple leaf nodes, and a node identifier of each leaf node of the multiple leaf nodes, the first node is a root node or any leaf node in the tree structure, and the multiple data blocks are obtained by aggregating multiple small files required for the training job; detecting whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center; if so, after the metadata snapshot is loaded, based on the metadata snapshot, reading multiple data blocks from the backend aggregate file system; performing a deaggregation operation on the multiple data blocks to obtain multiple small files.

[0008] The present application also provides a data block processing device, which includes: a first transceiver module, used to obtain a tree structure storage table, a leaf node storage table and a data block storage table, the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and each node identifier of each leaf node in the tree structure corresponding to multiple data blocks, the data block storage table includes data block information, data block identifiers, and node identifiers of each leaf node of the multiple leaf nodes stored in each leaf node of the multiple leaf nodes, the first node is the root node or any leaf node in the tree structure, and the multiple data blocks are obtained by aggregating multiple small files required for the training job.

[0009] The first processing module is used to search in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier; if matching node identifiers are found, a first hierarchical relationship is established between the first node and the attribute information of the first node and the data block information of the first node; according to the first data block identifier of the data block corresponding to the first node, the first small file list information included in the data block corresponding to the first node is determined, and the first small file list information is added to the first hierarchical relationship to obtain a second hierarchical relationship; according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, a metadata snapshot is generated, and the metadata snapshot is sent to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job is started, and the other nodes are at least one leaf node other than the first node in the tree structure.

[0010] The present application also provides another data block processing device, which includes: a second transceiver module, which is used to load metadata snapshots corresponding to multiple data blocks from a local cache after a training job is started, and the metadata snapshots are generated based on a tree structure storage table, a leaf node storage table, and a data block storage table. The tree structure storage table stores a first node identifier of a first node, and the leaf node storage table includes attribute information and each node identifier of each leaf node in the tree structure. The data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of the multiple leaf nodes for each data block stored in each leaf node of the multiple leaf nodes. The first node is a root node or any leaf node in the tree structure, and the multiple data blocks are obtained by aggregating multiple small files required for the training job.

[0011] The second processing module is used to detect whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center; if so, after the metadata snapshot is loaded, based on the metadata snapshot, multiple data blocks are read from the backend aggregate file system; the multiple data blocks are deaggregated to obtain multiple small files.

[0012] The present application also provides a data block processing system, which includes a metadata management center and a client. The metadata management center is used to obtain a tree structure storage table, a leaf node storage table, and a data block storage table, wherein the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and node identifiers of each leaf node in a tree structure corresponding to a plurality of data blocks, the data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of the plurality of leaf nodes, the first node is a root node or any leaf node in the tree structure, and the plurality of data blocks are obtained by aggregating a plurality of small files required for a training job.

[0013] The metadata management center is also used to search in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier; if matching node identifiers are found, a first hierarchical relationship is established between the first node and the attribute information of the first node and the data block information of the first node; according to the first data block identifier of the data block corresponding to the first node, the first small file list information included in the data block corresponding to the first node is determined, and the first small file list information is added to the first hierarchical relationship to obtain a second hierarchical relationship; according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, a metadata snapshot is generated, and the metadata snapshot is sent to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job is started, and the other nodes are at least one leaf node other than the first node in the tree structure.

[0014] The client is used to load metadata snapshots corresponding to multiple data blocks from a local cache after a training job is started. The metadata snapshot is generated based on a tree structure storage table, a leaf node storage table, and a data block storage table. The tree structure storage table stores a first node identifier of a first node. The leaf node storage table includes attribute information and node identifiers of each leaf node in the tree structure. The data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of each leaf node. The first node is a root node or any leaf node in the tree structure. The multiple data blocks are obtained by aggregating multiple small files required for the training job. It is detected whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of a metadata management center. If so, after the metadata snapshot is loaded, multiple data blocks are read from a backend aggregate file system based on the metadata snapshot. The multiple data blocks are deaggregated to obtain multiple small files.

[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data block processing methods when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data block processing methods are implemented.

[0017] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data block processing methods when executed by a processor.

[0018] Through this application, since the metadata management center can send the metadata snapshot of the training data set (multiple data blocks) to the computing node where the client is located (which is also the computing node where the application is located), the application can directly read the metadata snapshot from the local cache, reducing the access pressure on the metadata management center and accelerating the metadata access efficiency. The client can quickly obtain the metadata snapshot from the local cache and read the required metadata information from it without frequently accessing the metadata management center, thereby reducing network overhead and improving the efficiency of metadata access, thereby improving the supply efficiency of small file data sets.

[0019] In addition, during the execution of a training job, the dataset is usually read-only, and metadata snapshot updates due to training dataset updates will not occur during this period. Therefore, localized metadata snapshot cache is suitable for deep learning training jobs. Through the metadata snapshot mechanism, efficient metadata access performance can be maintained during large-scale small file dataset training, while reducing the consumption of storage and computing resources, and speeding up the parsing and supply efficiency of data block-based training datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A topological structure diagram of a data block processing system provided in an embodiment of the present application; Figure 2 A schematic diagram of a high-performance data reading pipeline provided in an embodiment of the present application; Figure 3 A flowchart of a data block processing method provided in an embodiment of the present application; Figure 4 A schematic diagram of metadata construction for data blocks provided in an embodiment of the present application; Figure 5 A flowchart of another data block processing method provided in an embodiment of the present application; Figure 6 A schematic diagram of the number of small batches corresponding to reading data blocks according to an embodiment of the present application; Figure 7 A structural view of a data block provided in an embodiment of the present application; Figure 8 A structural block diagram of a data block processing device provided in an embodiment of the present application; Fig. 9A structural block diagram of another data block processing device provided in an embodiment of the present application; Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0024] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] In conjunction with a specific application environment architecture or a specific hardware architecture on which the execution of a data block processing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0026] The embodiments of the present application are applied to a scenario where a storage system needs to provide a small file data set as a training set to a distributed deep learning operation system for model training.

[0027] For large-scale small file data sets, the mainstream processing method is to re-aggregate a large number of small files into a new data format to improve data reading performance. However, this data format is only supported by a few programming frameworks and is not universal. In addition, the original data needs to be formatted before the training job starts. After the training starts, the data format needs to be converted back to the original data for calculation. This method limits its application scenarios.

[0028] For deep learning training jobs, private data is usually downloaded or collected from public libraries and converted into a dataset format that can be used for deep learning training. The prepared small file datasets are then fed in batches to the computing process of the computing node for calculation. For small file datasets, the file access characteristics of the training job include: each training task only operates on a small file dataset, and only read-only access to the small file dataset is performed during the training process. The small file dataset remains unchanged during this process; the small file dataset is read one by one by epoch, and all small files are read once in each epoch; between different epochs, multiple small files are randomly shuffled (shuffle) to avoid model overfitting.

[0029] Specifically, at the beginning of a typical deep learning training job, all file names of the small file dataset will be loaded into memory. Before each epoch calculation, the training framework (such as PyTorch) will randomly shuffle the file name index to generate a random file data reading order. After that, the mini-batch datasets are read in batches in an iterative form. Each mini-batch dataset is fed to a computing node (for example, the computing node can be a graphics processing unit (GPU)) to perform parameter update calculations for the deep neural network. After the calculation is completed, the next batch of batch datasets is read until all small files are read once to complete the current epoch calculation. Before the model parameters converge, the training job often requires multiple rounds of epoch calculations. The overall training time depends on the training data supply efficiency and GPU computing efficiency. In a large computing cluster, many training jobs are run concurrently, and each training job may use a different dataset. These training jobs will issue a large number of small file read requests to the storage system where the dataset is located. At this time, if the storage system is inefficient in obtaining small files and the training data supply efficiency is slow, it will cause the GPU to wait frequently, resulting in a slow training process.

[0030] In order to solve the above technical problems, an embodiment of the present application proposes a data block processing method, which includes: obtaining a tree structure storage table, a leaf node storage table and a data block storage table, and when a node identifier matching the first node identifier is found in both the leaf node storage table and the data block storage table, establishing a first hierarchical relationship between the attribute information and data block information of the first node and the first node; determining the first small file list information included in the data block corresponding to the first node according to the first data block identifier of the data block corresponding to the first node, and adding the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship; generating a metadata snapshot according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, and sending the metadata snapshot to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job starts, and based on the metadata snapshot, quickly obtains multiple data blocks required for the training job from the file system, thereby obtaining the required small file data set, thereby improving the supply efficiency of small file data sets in the training job.

[0031] Below Figure 1 Taking the data block processing system shown as an example, the method provided in the embodiment of the present application is described.

[0032] like Figure 1 As shown, Figure 1 A topological diagram of a data block processing system provided in an embodiment of the present application. Figure 1 In the embodiment, the data block processing system 100 includes a metadata management center 101, a client 102, and a backend aggregate file system 103. Optionally, the data block processing system 100 also includes a global shared file system 104.

[0033] The metadata management center 101 (also called CacheFS-Server) in the embodiment of the present application is used to manage and store metadata information of the entire small file data set. In the process of aggregating and writing the small file data set to the back-end aggregate file system 103, the metadata management center 101 receives metadata information extracted from the client 102 and stores the metadata information in a distributed database. In order to improve the access speed, the metadata management center 101 also caches a copy of the metadata information in the memory. The metadata management center 101 supports the metadata snapshot function, constructs a metadata snapshot in a preset format, and sends it to the client 102.

[0034] The metadata management center 101 has backup and fault tolerance functions. In a distributed storage system, the availability and integrity of metadata directly affect the operation of the entire system. Therefore, the metadata management center ensures the stability and reliability of the metadata management center, minimizes the impact of system failures on business, ensures the security of metadata, and maintains the continuous service capability of the file system through measures such as regular backup, transaction log recording, data replication, and multi-node deployment.

[0035] The metadata management center 101 is configured with full backup and incremental backup strategies to meet different recovery requirements. Full backup completely copies all metadata in the database and is usually performed during periods of low system load and with a low execution frequency to reduce the impact on normal business. Incremental backup only records data that has changed since the last backup and is performed more frequently, which saves storage space and improves backup efficiency.

[0036] In terms of fault tolerance, the metadata management center relies on the database replication function and its master-slave switching mechanism to achieve metadata synchronization and fault tolerance. The database replication function allows metadata to be synchronized between multiple database instances, providing data redundancy and load balancing. When the primary database instance fails, the system can automatically switch to the backup database instance to ensure service continuity. In the master-slave replication mode, the primary database synchronizes the updated metadata to the slave database in real time. When the primary database is unavailable, the slave database can take over immediately to avoid service interruption.

[0037] Understandably, when the metadata management center encounters data corruption or other unforeseen system failures, the backup and fault tolerance mechanism can quickly start the recovery process. By using the backup data, the metadata can be restored in a short time to ensure that the business is not interrupted.

[0038] The client 102 (also called CacheFS-Client) in the embodiment of the present application is used to perform read and write operations on small file data sets. The client 102 consists of three parts: a write operation module, a read operation module and a general operation module, each of which has different functions.

[0039] Among them, the write operation module is used to scan the small files stored in the global shared file system, and aggregate the small files into larger data blocks according to a specific aggregation algorithm, and store the aggregated data blocks in the back-end aggregation file system, and report the metadata information of each data block to the metadata management center 101 in real time.

[0040] The read operation module is used to read the aggregated data blocks directly from the backend aggregate file system and de-aggregate them using the built high-performance data reading pipeline when the training task starts. In addition, the read operation module also integrates the data randomization function, so that the data can meet the data randomization requirements of deep learning training during the de-aggregation process. Data reading, de-aggregation and randomization can be integrated into a continuous and efficient operation process, which improves the efficiency of accessing small file data sets and ensures efficient data supply during model training.

[0041] Optionally, the high-performance data reading pipeline integrates the data block parser (Data Reader), metadata snapshot interpreter (Meta Interpreter), data block address parser (Chunk Path Parser) and data block-based shuffle mechanism (Chunk-Based Shuffle) to achieve efficient reading of training data. Figure 2 As shown, Figure 2 A schematic diagram of a high-performance data reading pipeline provided in an embodiment of the present application. Figure 2 In the data reading pipeline, the data reading pipeline obtains metadata snapshots from the CacheFS metadata in the CacheFS metadata management center, and parses the original multiple small file data from the stored data blocks in real time based on the metadata snapshot to form a mini-batch of data, and feeds it to the training job through the POSIX interface. The data reading pipeline also integrates a data block-based shuffle mechanism, which randomly shuffles the order of all small files at the beginning of each epoch of training.

[0042] The general operation module is used to perform general file system operations, such as data update, cache management, and directory structure reconstruction. The general operation module can periodically evict expired small file data sets and provide a conversion from a data block storage view to a user-friendly tree structure view. After the training job is started, the general operation module reads data from the backend aggregate file system 103 with data blocks as the minimum granularity, that is, pre-reads small file data sets, and performs cache management on the pre-read small file data sets.

[0043] The back-end aggregate file system 103 in the embodiment of the present application may be a high-performance parallel file system formed by aggregating high-speed storage resources dispersed on various computing nodes. For example, the parallel file system may be a BeeOND system.

[0044] The global shared file system 104 in the embodiment of the present application can be any storage system. The global shared file system 104 is used to store a large number of small file data sets.

[0045] In the embodiment of the present application, the metadata management center 101 can be deployed as a container service on the Master node in the cluster, which can make full use of the Master node's full life cycle management capabilities for the service. The client 102 can be directly deployed on the computing node of the cluster and interact with the metadata management center through the Restful API. The back-end aggregate file system 103 can also be deployed on the computing node of the cluster to provide data block storage functions through the client.

[0046] When the training job is scheduled to the target computing node, before the container corresponding to the training job is started, the job management system of the container cloud platform aggregates the small file data set required for the training job from the remote global shared file system 104 into a large data block and writes it to the backend aggregate file system 103 by calling the client 102. When the container corresponding to the training job is started, the data directory corresponding to the large data block is mounted to the container, and the training program in the container can access the data blocks stored in the backend aggregate file system 103 as if it were a local directory. The high-performance data reading pipeline constructed by the client's read operation can efficiently and quickly supply data to the training job.

[0047] In the embodiment of the present application, the metadata management center 101, the client 102, and the backend aggregate file system 103 together constitute a distributed cache system (CacheFS).

[0048] Figure 1 The data block processing system shown is only used as an example and is not used to limit the technical solution of the present application. Those skilled in the art should understand that in the specific implementation process, the data block processing system can also include more devices without limitation.

[0049] The method is described in detail below in conjunction with the execution flow of the data block processing method.

[0050] The embodiment of the present application provides a data block processing method, which is applied to Figure 1 The metadata management center shown, such as Figure 3 As shown, Figure 3 A flowchart of a data block processing method provided in an embodiment of the present application, the data block processing method comprises the following steps: S301: Obtain a tree structure storage table, a leaf node storage table, and a data block storage table.

[0051] Among them, the tree structure storage table is also called the Leaf table. The Leaf table is used to store the leaf node information in the tree structure corresponding to the data set directory of multiple data blocks in the backend aggregate file system. Leaf nodes include data block files and directories. The Leaf table stores the first node identifier of the first node, the node identifiers (also called inodes) of other nodes, the leaf name of each leaf node, the node identifier and node type of the parent directory node. The inode in the Leaf table is a unique identifier for each data block file or directory, and the inode of the parent directory is used to represent the hierarchical relationship in the tree structure. The node type is used to indicate whether each leaf node is a data block file or a directory. Each data block file and directory node is associated with the parent directory node through a node identifier.

[0052] The leaf node storage table is also called the LeafAttributes table. The LeafAttributes table is an extension of the Leaf table, which stores the attribute information and node identifiers of each leaf node in the tree structure corresponding to multiple data blocks. The attribute information includes the inode of the data block file or directory, the inode of the parent directory, the node type, modification time, creation time, size, and read and write permissions.

[0053] As can be understood, the LeafAttributes table is associated with the Leaf table through the inode to ensure that all relevant metadata information of the file or directory can be queried.

[0054] The data block storage table is also called the Chunk table. The Chunk table is used for metadata information of multiple data blocks. The Chunk table stores the data block information of each data block stored in each leaf node among multiple leaf nodes and the node identifier of each leaf node of multiple leaf nodes. The data block information includes the data block identifier (chunkid), the data block name, and the data block size. One data block corresponds to the node identifier of a leaf node. The data block identifier is used to quickly locate a specific data block.

[0055] It can be understood that the Chunk table is associated with the Leaf table and the LeafAttributes table through inode.

[0056] The first node is the root node or any leaf node in the tree structure.

[0057] Multiple data blocks are obtained by aggregating multiple small files required for the training job.

[0058] It can be understood that the metadata snapshot is generated based on the update timestamp of the metadata management center. Whenever the metadata of the data set changes, the metadata management center will automatically refresh the update timestamp. When it is detected that the update timestamp has not changed within a time window (such as 1 hour), it means that the metadata of multiple data blocks have been constructed, that is, the tree structure storage table, leaf node storage table and data block storage table have been constructed, which can trigger the metadata management center to obtain the tree structure storage table, leaf node storage table and data block storage table.

[0059] S302: Searching in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier.

[0060] S303: If matching node identifiers are found, a first-level relationship is established between the first node, the attribute information of the first node, and the data block information of the first node.

[0061] In one example, based on an initial hierarchical structure in a preset format, a metadata management center writes attribute information of a first node and data block information of the first node into a position corresponding to a preset first key field in the initial hierarchical structure to establish a first hierarchical relationship.

[0062] The preset format may be a JSON format.

[0063] The preset first key field (also called Key field) may be the attr field.

[0064] Exemplarily, the metadata management center writes the first node identifier, leaf name, parent directory's inode, node type, modification time, creation time, file size, and read / write permissions into corresponding positions in the attr field of the initial hierarchical structure in JSON format based on the initial hierarchical structure in JSON format to establish a first-level relationship.

[0065] It can be understood that since the Chunk table is associated with the Leaf table and the LeafAttributes table through inode, the first node can be associated with the attribute information of the first node and the data block information of the first node based on the first node identifier to establish a first-level relationship.

[0066] S304: Determine first small file list information included in the data block corresponding to the first node according to the first data block identifier of the data block corresponding to the first node, and add the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship.

[0067] In one example, a metadata management center obtains a data block attribute storage table; searches the data block attribute storage table for a data block identifier that matches a first data block identifier; if a matching data block identifier is found, obtains the first small file list information included in the data block corresponding to the first data block identifier in the data block attribute storage table.

[0068] The data block attribute storage table is an extension of the Chunk table content. The data block attribute storage table includes the data block identifier of each data block and the small file list information included in each data block. The small file list information includes the name, small file size, and small file modification time of each small file in each data block. The small file list information is a series of strings. If a data block contains a large number of small files, the metadata management center compresses the string of the small file list information into large object (Blob) binary format data through the zlib compression algorithm.

[0069] It can be understood that the compressed small file list information can reduce storage space usage and is also beneficial to improving metadata query efficiency.

[0070] In one example, the metadata management center writes the first small file list information into a position corresponding to a preset second key field in the first hierarchical relationship to obtain a second hierarchical relationship.

[0071] Among them, the preset second key field can be the chunks field.

[0072] Exemplarily, the metadata management center writes the inode, data block identifier, data block name and data block size, and small file list information of the data block into corresponding positions in the chunks field to establish a second-level relationship.

[0073] It can be understood that the ChunkAttributes table is associated with the Chunk table through chunkid, so that the metadata information of each small file can be accurately tracked while managing data blocks.

[0074] In one example, Figure 4 As shown, Figure 4 A schematic diagram of metadata construction for data blocks provided in an embodiment of the present application. Figure 4 In the data block metadata information is stored in the form of a tree structure storage table (Table 1), a leaf node storage table (Table 2), a data block storage table (Table 3) and a data block attribute storage table (Table 4). The tree structure storage table, the leaf node storage table, the data block storage table and the data block attribute storage table correspond to the tree directory structure (also called the tree structure) in the file system under Linux.

[0075] All metadata stored in the tree structure storage table, leaf node storage table, data block storage table and data block attribute storage table are stored in a database relational database, which can be deployed on the high-speed storage medium of a single server node or distributed on the high-speed storage medium of multiple server nodes.

[0076] It is understandable that the database is deployed on high-speed storage media of multiple server nodes to avoid metadata loss caused by single point failure and ensure high concurrency and scalability of metadata query. The metadata management center uses a database with sharded table storage, which improves metadata query efficiency and system scalability by distributing different types of metadata into multiple storage tables. Each storage table is dedicated to storing specific metadata information, reducing unnecessary data access and speeding up query speed. At the same time, the structure of sharded table storage allows flexible expansion when the amount of data increases, avoiding the complexity of reconstructing the entire database. In addition, sharded table storage also enhances maintainability and ease of use, helping to reduce the risk of errors.

[0077] S305: Generate a metadata snapshot according to the second-level relationship and at least one third-level relationship of other nodes, and send the metadata snapshot to the client, so that the client stores it in a local cache and obtains the metadata snapshot when the training job starts.

[0078] Among them, the other nodes are at least one leaf node other than the first node in the tree structure.

[0079] In one example, the tree directory structure of the metadata snapshot in JSON format is as follows: cachefs├── attr (inode: 2, type: "directory", mode: 493, ...)└── entries├── imagenet│ ├── attr (inode: 39051, type: "directory", mode: 493, ...)│ └── entries│ ├── imagenet-train_100│ │ ├── attr(inode: 39152, type: "regular", mode: 420, ...)│ │ └── chunks│ │ ├──chunkid: 49283│ │ └── slices│ │ ├── size: 4054528│ │ └── files│ │ ├── "imagenet / train / n10148035 / n10148035_11304.JPEG"│ │ ├── "imagenet / train / n10148035 / n10148035_11195.JPEG" │ │ ├── "imagenet / train / n10148035 / n10148035_11268.JPEG" The tree-like directory structure is used to represent the hierarchical relationship in the metadata snapshot. In this tree-like directory structure, starting from the root node (cachefs), it expands to the directory imagenet, then to the aggregated data chunk imagenet-train_100, and finally displays the small file list information of the data chunk, for example, "imagenet / train / n10148035 / n10148035_11304.JPEG".

[0080] In some optional implementations, the metadata management center obtains the number of times the metadata snapshot is accessed within a first time period and the latest update time of the metadata snapshot; when a second time period between the latest update time and the current moment is greater than or equal to a first threshold and the number of accesses is less than or equal to a second threshold, the metadata snapshot is cleared.

[0081] Based on the above Figure 3According to the method shown, the metadata management center can obtain a tree structure storage table, a leaf node storage table, and a data block storage table, and when a node identifier matching the first node identifier is found in both the leaf node storage table and the data block storage table, a first hierarchical relationship between the first node and the attribute information of the first node and the data block information of the first node is established; and based on the first data block identifier of the data block corresponding to the first node, the first small file list information included in the data block corresponding to the first node is determined, and the first small file list information is added to the first hierarchical relationship to obtain a second hierarchical relationship; a metadata snapshot is generated based on the second hierarchical relationship and at least one third hierarchical relationship with other nodes, and the metadata snapshot is sent to the client.

[0082] Since the metadata management center can send the metadata snapshot of the training data set (multiple data blocks) to the computing node where the client is located (which is also the computing node where the application is located), the application can directly read the metadata snapshot from the local cache, reducing the access pressure on the metadata management center and accelerating metadata access efficiency.

[0083] When the training job is started, the client first downloads the metadata snapshot with the latest timestamp from the metadata management center to the local storage. All metadata information query operations after the model training job is started are queried from the metadata snapshot cached locally. Through the metadata snapshot mechanism, efficient metadata access performance can be maintained during large-scale small file data set training, while reducing the consumption of storage and computing resources, and speeding up the parsing and supply efficiency of data block-based training data sets.

[0084] The embodiment of the present application provides a data block processing method, which is applied to Figure 1 The client shown, such as Figure 5 As shown, Figure 5 A flowchart of another data block processing method provided in an embodiment of the present application, the data block processing method comprises the following steps: S501: When the training job is started, metadata snapshots corresponding to multiple data blocks are loaded from the local cache.

[0085] In one example, when a training job is started, the client loads metadata snapshots corresponding to multiple data blocks from a local cache, and performs preprocessing operations during the metadata snapshot loading process.

[0086] The preprocessing operation includes file integrity check and format verification. It can be understood that the preprocessing operation can ensure that the metadata snapshot is not damaged and conforms to the predefined JSON format.

[0087] S502: Detect whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center.

[0088] In one example, the metadata snapshot interpreter of the client detects whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center.

[0089] It is understandable that, in order to ensure the real-time nature of the metadata snapshot, the metadata snapshot interpreter has a timestamp checking mechanism built in. The metadata snapshot interpreter can compare the snapshot timestamp of the metadata snapshot with the latest snapshot timestamp of the metadata management center.

[0090] If the snapshot time of the metadata snapshot is inconsistent with the latest snapshot timestamp, the client downloads the latest metadata snapshot from the metadata management center and updates the local cache to ensure that the training job obtains multiple small files based on the latest metadata snapshot.

[0091] If the snapshot time of the metadata snapshot is consistent with the latest snapshot timestamp, the client executes S503.

[0092] S503: If yes, after the metadata snapshot is loaded, multiple data blocks are read from the backend aggregate file system based on the metadata snapshot.

[0093] S504: Perform a de-aggregation operation on the multiple data blocks to obtain multiple small files.

[0094] In some optional implementations, the client parses the metadata snapshot to obtain small file list information and data block identifiers constituting each data block; and reads multiple data blocks from the backend aggregate file system based on the small file list information and data block identifiers.

[0095] In one example, the client presets a position corresponding to a first key field in a metadata snapshot, parses to obtain a data block identifier and a data block name for each of a plurality of data blocks, and stores the plurality of data block identifiers corresponding to the plurality of data blocks in a global data block list; presets a position corresponding to a second key field in the metadata snapshot, parses to obtain small file list information included in each of the plurality of data blocks, and the small file list information includes a data block identifier for each data block and a small file list included in each data block.

[0096] Understandably, during the process of client parsing metadata snapshots, the metadata snapshot interpreter also has a complete error handling mechanism. When encountering abnormal situations such as parsing failure or data inconsistency, the interpreter will trigger the error handling logic (for example, logging or warning prompts), and if necessary, re-obtain the latest metadata snapshot from the metadata management center for re-parsing.

[0097] In one example, the client may establish a first hash table between a data block identifier and multiple small file names; and establish a second hash table between a small file name and at least one data block, and a data block name of each data block in the at least one data block.

[0098] The first hash table is also called a hash_files_to_chunk table, and the second hash table is also called a hash_files_to_chunkid table.

[0099] In some optional implementations, the client obtains multiple file indexes; based on the multiple file indexes, generates a list of sub-files belonging to the first data block; obtains the total file quantity of multiple first small files; if the total file quantity is less than a first threshold, creates a data block object for the first data block, and obtains the small file header information and the small file header information length of each first small file; based on the small file size and the small file header information length of each first small file, obtains the address offset of each first small file in the first data block; based on the address offset, obtains multiple first small files, and stores the data block identifier of the first data block, the small file name and the address offset of each first small file in a global hash table corresponding to the data block object.

[0100] The multiple file indexes are the indexes of each first small file in the small file list included in the small batch data set required by the training program for executing the training job. The small batch data set is a mini-batch of training data.

[0101] The sub-file list corresponds to a plurality of first small files. The small file header information includes the small file size and the small file name of each first small file.

[0102] For example, Figure 6 As shown, Figure 6 This is a schematic diagram of the number of small batches corresponding to the reading of data blocks in the embodiment of the present application. Figure 6 In the training data parser of the client, the data chunks are parsed into original small file data after shuffling, forming mini-batch data to be fed to the model training program.

[0103] In the initialization phase of small file reading, the metadata snapshot interpreter generates a second hash table that maps small files to the data blocks they are in, with the key value being the small file name and the value being the data block identifier and the data block name. When the training program needs a mini-batch of training data, the client takes multiple file indexes of a new mini-batch from the randomized small file list, maps the multiple file indexes to the corresponding data blocks, and generates a list of sub-files belonging to the same data block, then creates a data block object (chunk object) for each hit data block, and traverses all small file header information in the data block when initializing the data block object.

[0104] As you can understand, the client records the addressing information (address offset) in a global hash table, where the key is the file name and the value is the object containing the addressing information. In this way, when you access the file later, you can quickly locate the location of the file data without parsing the header information again. The global addressing operation for each data block is performed only once, and the result is stored in the global hash table. When other processes of model training access the data block, they can directly extract the address offset of the small file from the cached chunk object, thereby avoiding repeated addressing operations. This can improve addressing efficiency and reduce unnecessary I / O operations.

[0105] Optionally, the client's training data parser supports batch addressing. When creating a chunk object, the client can pass a sub-file list parameter to determine whether the current file name is in the sub-list when traversing the data block header information. If it is a hit, its addressing information is obtained and an addressing object is generated. When all files in the sub-file list are addressed, the chunk object is initialized without triggering the actual data read operation.

[0106] In some optional embodiments, when the client needs to read the first target small file, it obtains the first target name of the first target small file, reads the first address offset of the first target small file corresponding to the first target name from the global hash table; based on the first address offset, loads the target data block including the first target small file into the local cache.

[0107] In some optional embodiments, when the client needs to read the second target small file in the target data block, it obtains the second target name of the second target small file, reads the second address offset of the second target small file corresponding to the second target name from the global hash table; based on the first address offset, reads the second target small file from the target data block stored in the local cache.

[0108] It is understandable that when any small file included in a data block is triggered to be read, the client will load the data of the entire data block into the local cache. If other small files in the data block are hit again in the current mini-batch or subsequent mini-batch, the corresponding file data will be read directly from the local cache without having to read it again from the backend aggregate file system. This method is equivalent to introducing a data pre-fetching mechanism, which improves the reading efficiency of small files and reduces the I / O burden of the system.

[0109] During the entire training cycle, once a data block is loaded into the local cache, subsequent access to the files in the data block no longer needs to interact with the backend aggregate file system unless the cached data block is cleared or invalidated. In subsequent epochs, if the local cache hits the target file, the system will load the data directly from the local cache, improving the efficiency of training data supply.

[0110] Based on the above Figure 5 According to the method shown, when the training job is started, the client loads metadata snapshots corresponding to multiple data blocks from the local cache, and detects whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center. If so, after the metadata snapshot is loaded, based on the metadata snapshot, multiple data blocks are read from the backend aggregate file system, and the multiple data blocks are deaggregated to obtain multiple small files.

[0111] Since the dataset is usually read-only during the execution of the training job, there will be no metadata snapshot update due to the update of the training dataset, so the localized metadata snapshot cache is suitable for deep learning training jobs. The client can quickly obtain the metadata snapshot from the local cache and read the required metadata information from it without frequently accessing the metadata management center, thereby reducing network overhead and improving the efficiency of metadata access, thereby improving the supply efficiency of small file datasets.

[0112] Optionally, the client can also aggregate multiple small files stored in the global shared file system into data blocks and store them in the backend aggregate file system. This includes the following stages: Phase 1: The client scans the dataset directory stored in the remote global shared file system and collects metadata information for each data block in the dataset directory.

[0113] It is understandable that the scanning process is performed in units of subdirectories of the dataset directory to avoid aggregating files in different directories into the same data block, thereby reducing the complexity of system implementation. In order to improve scanning efficiency, a multi-worker concurrent processing strategy can be adopted and multiple scanning workers can be started in parallel based on factors such as the size of the dataset, the number of files, the directory structure, and the preset data block size. Each scanning worker is responsible for scanning files in a part of the subdirectories and sending the collected file information to the scanning channel.

[0114] Phase 2: A dynamic channel allocation algorithm is used to enable each scanning worker to send file information to multiple channels in an interleaved manner to avoid channel congestion and data skew.

[0115] Specifically, the client maintains a channel list for each scanning worker and records the usage status of each channel. When the scanning worker needs to send file information, it selects a channel that is not currently in use from the channel list and sends the file information to the channel. After the sending is completed, the scanning worker updates the usage status of the channel and moves it to the end of the channel list. In addition, each channel has a buffer queue to cope with instantaneous data traffic peaks.

[0116] It can be understood that this dynamic channel allocation algorithm can improve scanning efficiency and scanning channel utilization.

[0117] Phase 3: The aggregation worker receives file information from the scanning channel and aggregates small files into a data block list based on the directory and the preset data block size.

[0118] It is understandable that the aggregation process only involves the processing of metadata information, and does not perform actual data reading and writing operations. Each aggregation worker is responsible for processing the file information under a part of the directory and sending the aggregated data block list to the packaging channel. Similar to the scanning channel, the packaging channel also uses a dynamic channel allocation algorithm to ensure load balancing and efficient utilization of the channel.

[0119] Phase 4: The packaging worker receives the data block list from the packaging channel, reads the corresponding small file data from the remote global shared file system according to the file information in the list, and generates the corresponding header information according to the data of each small file. Then, the header information and the small file data are packaged into a data block.

[0120] like Figure 7 As shown, Figure 7 A structural view of a data block provided in an embodiment of the present application; Figure 7In the data block size, the data block size is a pre-configured value (such as 4MB), and the header information of each small file includes: file name (name), file size (size), file modification time (mtime), and file read permission (mode). The length of the header file is fixed to 512 bytes, and the insufficient part is automatically filled.

[0121] Phase 5: The packaging worker writes the generated data blocks to the new data set directory of the backend aggregate file system deployed on the computing node through the client. The client uses the FUSE file system interface to synchronize the metadata information of the generated data blocks by intercepting the write operation, and reports it to the synchronization channel of the metadata management center in batch processing.

[0122] Phase 6: The synchronization worker receives metadata information of the data block from the synchronization channel and writes it in batches to the database of the metadata management center. The aggregated file list of each data block is compressed before being written to the database to improve efficiency and reduce storage space usage.

[0123] Understandably, the client uses multiple workers to concurrently process tasks at each stage in each stage to improve processing efficiency. Secondly, a dynamic channel allocation algorithm is used to ensure that each worker can use multiple channels evenly and staggeredly to avoid channel congestion and data skew, and improve channel utilization. Thirdly, a batch processing mechanism is used to reduce the number of interactions with the metadata center database and reduce database load pressure. Finally, metadata compression technology is used to reduce metadata storage space occupancy and data transmission volume, and improve metadata reading efficiency.

[0124] Optionally, the client can also convert the small file dataset originally stored in the tree-like directory result into a flat directory structure based on data blocks and store it in the backend aggregate file system through the small file aggregation operation. Accordingly, the client can also rebuild the directory structure. The directory structure reconstruction function is to re-parse the data blocks stored in the backend aggregate file system into the same small file dataset directory structure as the original one stored in the global shared file system, so as to provide a consistent dataset directory structure view to the application.

[0125] Specifically, the directory structure reconstruction is dynamically generated based on the data block metadata. The client obtains metadata information from the Leaf table, LeafAttributes table, Chunk table, and ChunkAttributes table of the metadata management center, and dynamically restores and constructs the tree-like directory structure of small files. For example, the tree structure hierarchy of the directory is first restored through the Leaf table, and the attribute information of each leaf node is obtained from the LeafAttributes table. For directories containing data blocks, the system uses inode to query the data block identifiers of all aggregated data blocks under the directory in the Leaf table and Chunk table through association query, and finds the aggregated small file list from the ChunkAttributes table through the data block identifier. The relative path information retained in the small file list in the data block is used to restore the directory hierarchy layer by layer, and the directories at all levels are added to the corresponding leaf nodes of the tree linked list to construct a tree directory structure. In accordance with the standard format of the Linux file system, the tree linked list structure is returned to complete the directory structure reconstruction.

[0126] In order to effectively handle the data inconsistency problem between the original small file dataset in the global shared file system and the data blocks stored in the backend aggregate file system, the client can perform a complete small file aggregation operation on the dataset used for training before starting a new training job. This full update method is simple and reliable, and can ensure that each training job uses the latest dataset.

[0127] However, if the new dataset is only a partial change to the old dataset, a full update may result in a large number of repeated small file aggregation operations, wasting unnecessary time. Therefore, in this case, the client can first detect whether the dataset used for training has been cached in the backend aggregate file system before starting a new training job. If the dataset with the same name has been cached, the modification time will be compared with the directory as the minimum granularity. If the modification time of a directory has changed, that is, the data in the directory has been modified, the small file aggregation operation will be re-executed on the directory. On the contrary, if the modification time of the directory has not changed, there is no need to re-aggregate the directory. In this way, by updating only the changed parts, the full update of the entire dataset can be avoided, thereby significantly improving efficiency.

[0128] Furthermore, the client can also perform cache data management. Cache data management includes cleaning operations for obsolete data, junk data, and invalid data to ensure that the data in the metadata management center and the backend data block storage always remains valid. First, mark the target small file or small file data set as "deleted" in the metadata. If all files in a data block are marked as "deleted", the data block is also marked as "deleted". When the entire data set is marked as "deleted", all data blocks under the data set are uniformly marked as "deleted". This allows effective tracking of data blocks that are no longer referenced, providing a basis for subsequent garbage collection operations.

[0129] The garbage data cleaning mechanism relies on regularly running garbage collection tasks. The client can periodically scan metadata to identify data blocks that are not referenced by any files and have been marked as "deleted". After confirming that these data blocks are no longer needed, they are marked as recyclable and the corresponding records are deleted from the metadata management center. To ensure the consistency of data deletion operations, the relevant metadata and data blocks are locked during the deletion process to prevent other operations from interfering with the deletion process. This locking mechanism ensures that deletion operations in concurrent environments are safe and reliable, avoiding the risk of inconsistent data or accidental loss.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0131] An embodiment of the present application also provides a data block processing system, which includes a metadata management center and a client.

[0132] A metadata management center is used to obtain a tree structure storage table, a leaf node storage table and a data block storage table. The tree structure storage table stores a first node identifier of a first node. The leaf node storage table includes attribute information and node identifiers of each leaf node in a tree structure corresponding to multiple data blocks. The data block storage table includes data block information, data block identifiers, and node identifiers of each leaf node in multiple leaf nodes. The first node is a root node or any leaf node in the tree structure. Multiple data blocks are obtained by aggregating multiple small files required for a training job.

[0133] The metadata management center is also used to search in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier; if matching node identifiers are found, a first hierarchical relationship is established between the first node and the attribute information of the first node and the data block information of the first node; according to the first data block identifier of the data block corresponding to the first node, the first small file list information included in the data block corresponding to the first node is determined, and the first small file list information is added to the first hierarchical relationship to obtain a second hierarchical relationship; according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, a metadata snapshot is generated, and the metadata snapshot is sent to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job is started, and the other nodes are at least one leaf node other than the first node in the tree structure.

[0134] The client is used to load metadata snapshots corresponding to multiple data blocks from a local cache after a training job is started. The metadata snapshot is generated based on a tree structure storage table, a leaf node storage table, and a data block storage table. The tree structure storage table stores a first node identifier of a first node. The leaf node storage table includes attribute information and node identifiers of each leaf node in the tree structure. The data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of each leaf node. The first node is a root node or any leaf node in the tree structure. The multiple data blocks are obtained by aggregating multiple small files required for the training job. It is detected whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of a metadata management center. If so, after the metadata snapshot is loaded, multiple data blocks are read from a backend aggregate file system based on the metadata snapshot. The multiple data blocks are deaggregated to obtain multiple small files.

[0135] The description of the features in the embodiment corresponding to the data block processing system can refer to the relevant description of the embodiment corresponding to the data block processing method, which will not be repeated here.

[0136] The embodiment of the present application also provides a data block processing device, such as Figure 8 As shown, Figure 8A structural block diagram of a data block processing device provided in an embodiment of the present application; the device includes: a first transceiver module 801, used to obtain a tree structure storage table, a leaf node storage table and a data block storage table, the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and each node identifier of each leaf node in a tree structure corresponding to multiple data blocks, the data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of the multiple leaf nodes for each data block stored in each leaf node of the multiple leaf nodes, the first node is a root node or any leaf node in the tree structure, and the multiple data blocks are obtained by aggregating multiple small files required for a training job.

[0137] The first processing module 802 is used to search in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier; if matching node identifiers are found, a first hierarchical relationship is established between the first node and the attribute information of the first node and the data block information of the first node; according to the first data block identifier of the data block corresponding to the first node, the first small file list information included in the data block corresponding to the first node is determined, and the first small file list information is added to the first hierarchical relationship to obtain a second hierarchical relationship; according to the second hierarchical relationship and at least one third hierarchical relationship with other nodes, a metadata snapshot is generated, and the metadata snapshot is sent to the client, so that the client stores it in the local cache and obtains the metadata snapshot when the training job is started, and the other nodes are at least one leaf node other than the first node in the tree structure.

[0138] In some optional implementations, the first processing module 802 is specifically used to obtain a data block attribute storage table, which includes a data block identifier for each data block and small file list information included in each data block; searching the data block attribute storage table for a data block identifier that matches the first data block identifier; if a matching data block identifier is found, obtaining the first small file list information included in the data block corresponding to the first data block identifier in the data block attribute storage table.

[0139] In some optional implementations, the first processing module 802 is further specifically used to write the attribute information of the first node and the data block information of the first node into a position corresponding to a preset first key field in the initial hierarchical structure based on the initial hierarchical structure in a preset format to establish a first hierarchical relationship.

[0140] In some optional implementations, the first processing module 802 is further specifically configured to write the first small file list information into a position corresponding to a preset second key field in the first hierarchical relationship to obtain a second hierarchical relationship.

[0141] In some optional implementations, the first transceiver module 801 is also used to obtain the number of times the metadata snapshot is accessed within a first time period and the latest update time of the metadata snapshot; the first processing module 802 is also used to clear the metadata snapshot when a second time period between the latest update time and the current moment is greater than or equal to a first threshold and the number of times the metadata snapshot is accessed is less than or equal to a second threshold.

[0142] The embodiment of the present application also provides another data block processing device, such as Fig. 9 As shown, Fig. 9 A structural block diagram of another data block processing device provided in an embodiment of the present application; the device includes: The second transceiver module 901 is used to load metadata snapshots corresponding to multiple data blocks from the local cache after the training job is started. The metadata snapshots are generated based on the tree structure storage table, the leaf node storage table and the data block storage table. The tree structure storage table stores the first node identifier of the first node. The leaf node storage table includes attribute information and each node identifier of each leaf node in the tree structure. The data block storage table includes data block information, data block identifiers, and node identifiers of each data block stored in each leaf node of the multiple leaf nodes. The first node is the root node or any leaf node in the tree structure. The multiple data blocks are obtained by aggregating multiple small files required for the training job.

[0143] The second processing module 902 is used to detect whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center; if so, after the metadata snapshot is loaded, based on the metadata snapshot, multiple data blocks are read from the back-end aggregate file system; the multiple data blocks are deaggregated to obtain multiple small files.

[0144] In some optional implementations, the second processing module 902 is specifically used to parse the metadata snapshot to obtain the small file list information and data block identifiers that constitute each data block; based on the small file list information and data block identifiers, read multiple data blocks from the backend aggregate file system.

[0145] In some optional embodiments, the second processing module 902 is further specifically used to preset a position corresponding to the first key field in the metadata snapshot, parse to obtain a data block identifier and a data block name for each of the multiple data blocks, and store multiple data block identifiers corresponding to the multiple data blocks in a global data block list; preset a position corresponding to the second key field in the metadata snapshot, parse to obtain small file list information included in each of the multiple data blocks, the small file list information including a data block identifier for each data block and a small file list included in each data block.

[0146] In some optional embodiments, the second transceiver module 901 is also used to obtain multiple file indexes, which are the indexes of each first small file included in the small batch data set required by the training program for executing the training job in the small file list; the second processing module 902 is also used to generate a sub-file list belonging to the first data block based on the multiple file indexes, and the sub-file list corresponds to multiple first small files; the second transceiver module 901 is also used to obtain the total file amount of multiple first small files; the second processing module 902 is also used to create a data block object of the first data block if the total file amount is less than the first threshold, and obtain the small file header information and the small file header information length of each first small file, the small file header information includes the small file size and the small file name of each first small file; according to the small file size and the small file header information length of each first small file, obtain the address offset of each first small file in the first data block; according to the address offset, obtain multiple first small files, and store the data block identifier of the first data block, the small file name and the address offset of each first small file in the global hash table corresponding to the data block object.

[0147] In some optional embodiments, the second transceiver module 901 is also used to obtain the first target name of the first target small file and read the first address offset of the first target small file corresponding to the first target name from the global hash table when the first target small file needs to be read; the second processing module 902 is also used to load the target data block including the first target small file into the local cache based on the first address offset.

[0148] In some optional embodiments, the second transceiver module 901 is also used to obtain the second target name of the second target small file and read the second address offset of the second target small file corresponding to the second target name from the global hash table when it is necessary to read the second target small file in the target data block; the second processing module 902 is also used to read the second target small file from the target data block stored in the local cache based on the first address offset.

[0149] For the description of the features in the embodiment corresponding to the data block processing device, reference can be made to the relevant description of the embodiment corresponding to the data block processing method, which will not be repeated here.

[0150] The embodiment of the present application also provides an electronic device, such as Fig.10 As shown, Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application; the electronic device includes a processor 10 and a memory 20, the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above-mentioned data block processing method embodiments.

[0151] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data block processing method embodiments when running.

[0152] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0153] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data block processing method embodiments are implemented.

[0154] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data block processing method embodiments are implemented.

[0155] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0156] The above is a detailed introduction to a data block processing method, system, device, storage medium and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data block processing method, characterized in that: Applied to a metadata management center, the method comprises: Obtain a tree structure storage table, a leaf node storage table, and a data block storage table, wherein the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and node identifiers of each leaf node in a tree structure corresponding to a plurality of data blocks, the data block storage table includes data block information, a data block identifier, and a node identifier of each leaf node of the plurality of leaf nodes, the first node is a root node or any leaf node in the tree structure, and the plurality of data blocks are obtained by aggregating a plurality of small files required for a training job; Searching in the leaf node storage table and the data block storage table respectively whether there is a node identifier matching the first node identifier; If matching node identifiers are found, a first hierarchical relationship between the first node and the attribute information of the first node and the data block information of the first node is established; Determine, according to the first data block identifier of the data block corresponding to the first node, first small file list information included in the data block corresponding to the first node, and add the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship; A metadata snapshot is generated based on the second hierarchical relationship and at least one third hierarchical relationship between other nodes, and the metadata snapshot is sent to the client so that the client stores the metadata snapshot in a local cache and obtains the metadata snapshot when the training job starts. The other nodes are at least one leaf node other than the first node in the tree structure.

2. The method according to claim 1, characterized in that The determining, according to the first data block identifier of the data block corresponding to the first node, the small file list information included in the data block corresponding to the first node comprises: Acquire a data block attribute storage table, wherein the data block attribute storage table includes a data block identifier of each data block and small file list information included in each data block; Searching in the data block attribute storage table whether there is a data block identifier that matches the first data block identifier; If a matching data block identifier is found, the first small file list information included in the data block corresponding to the first data block identifier is obtained in the data block attribute storage table.

3. The method according to claim 2, characterized in that The establishing of a first hierarchical relationship between the first node and the attribute information of the first node and the data block information of the first node includes: Based on an initial hierarchical structure in a preset format, the attribute information of the first node and the data block information of the first node are written into a position corresponding to a preset first key field in the initial hierarchical structure to establish the first hierarchical relationship.

4. The method according to claim 3, characterized in that The adding the first small file list information to the first hierarchical relationship to obtain a second hierarchical relationship includes: The first small file list information is written into a position corresponding to a preset second key field in the first hierarchical relationship to obtain the second hierarchical relationship.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining the number of times the metadata snapshot is accessed within a first time period and the latest update time of the metadata snapshot; When a second time period between the latest update time and the current time is greater than or equal to a first threshold, and the number of accesses is less than or equal to a second threshold, the metadata snapshot is cleared.

6. A data block processing method, characterized in that: Applied to a client, the method comprises: When the training job is started, metadata snapshots corresponding to multiple data blocks are loaded from the local cache, wherein the metadata snapshots are generated based on a tree structure storage table, a leaf node storage table, and a data block storage table, wherein the tree structure storage table stores a first node identifier of a first node, the leaf node storage table includes attribute information and node identifiers of each leaf node in the tree structure, and the data block storage table includes data block information, a data block identifier, and a node identifier of each of the multiple leaf nodes stored in each leaf node of the multiple leaf nodes, wherein the first node is a root node or any leaf node in the tree structure, and the multiple data blocks are obtained by aggregating multiple small files required for the training job; Detecting whether the snapshot timestamp of the metadata snapshot is consistent with the latest snapshot timestamp of the metadata management center; If so, after the metadata snapshot is loaded, the plurality of data blocks are read from the backend aggregate file system based on the metadata snapshot; A de-aggregation operation is performed on the multiple data blocks to obtain the multiple small files.

7. The method according to claim 6, characterized in that After the metadata snapshot is loaded, reading the multiple data blocks from the backend aggregate file system based on the metadata snapshot includes: Parsing the metadata snapshot to obtain small file list information constituting each data block and the data block identifier; Based on the small file list information and the data block identifiers, the multiple data blocks are read from a backend aggregate file system.

8. The method according to claim 7, characterized in that The step of parsing the metadata snapshot to obtain the small file list information and the data block identifier constituting each data block includes: parse the location corresponding to the first key field preset in the metadata snapshot to obtain a data block identifier and a data block name of each of the multiple data blocks, and store the multiple data block identifiers corresponding to the multiple data blocks in a global data block list; The small file list information included in each of the multiple data blocks is obtained by parsing the position corresponding to the preset second key field in the metadata snapshot, and the small file list information includes the data block identifier of each data block and the small file list included in each data block.

9. The method according to claim 8, characterized in that The method further comprises: Acquire multiple file indexes, where the multiple file indexes are indexes in the small file list of each first small file included in the small batch data set required for executing the training program of the training job; Based on the multiple file indexes, generating a sub-file list belonging to the first data block, the sub-file list corresponding to the multiple first small files; Obtain the total number of files of the plurality of first small files; If the total amount of the file is less than the first threshold, a data block object of the first data block is created, and small file header information and small file header information length of each of the first small files are obtained, wherein the small file header information includes a small file size and a small file name of each of the first small files; Obtaining an address offset of each of the first small files in the first data block according to the small file size and the small file header information length of each of the first small files; According to the address offset, the multiple first small files are obtained, and the data block identifier of the first data block, the small file name of each of the first small files and the address offset are stored in a global hash table corresponding to the data block object.

10. The method according to claim 9, characterized in that The method further comprises: When it is necessary to read the first target small file, obtain the first target name of the first target small file, and read the first address offset of the first target small file corresponding to the first target name from the global hash table; Based on the first address offset, a target data block including the first target small file is loaded into the local cache.

11. The method according to claim 10, characterized in that The method further comprises: When it is necessary to read the second target small file in the target data block, obtain the second target name of the second target small file, and read the second address offset of the second target small file corresponding to the second target name from the global hash table; Based on the first address offset, the second target small file is read from the target data block stored in the local cache.

12. A data block processing system, characterized in that: The data block processing system includes a metadata management center and a client; The metadata management center is used to execute the data block processing method described in any one of claims 1 to 5; The client is used to execute the data block processing method described in any one of claims 6-11.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data block processing method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data block processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data block processing method according to any one of claims 1 to 11 are implemented.

Images