Metadata processing method, device and equipment and nonvolatile storage medium

Abstract

The invention relates to the technical field of storage, and particularly discloses a metadata processing method, device and equipment and a nonvolatile storage medium. Small file data sets are aggregated and written into a storage system, aggregated file metadata are constructed, and the small file metadata and the aggregated file metadata are associated based on identifiers of aggregated files; calculating a storage position of the aggregation file in a storage system according to the identifier of the aggregation file; and sending the aggregation file metadata corresponding to the small file data set to data reading equipment, so that when the data reading equipment needs to read the small file data set, the position of a target aggregation file is directly calculated and read according to the aggregation file metadata. On the basis of reducing the reading and writing times of the small files through aggregated writing and aggregated reading of the small files, a data reading device does not need to access the metadata of the small files once every time the small files are read through the design of the metadata of the aggregated files, the metadata performance of the massive small files is improved, and the storage performance of the massive small files is improved.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a metadata processing method, device, equipment and non-volatile storage medium. Background Art

[0002] Artificial Intelligence (AI) model training for massive small file data sets requires localizing the data sets before training, which not only requires sufficient local storage space, but also takes a lot of time to wait for the data sets to be localized. Therefore, current data set processing mainly uses a framework that separates the computing environment and storage, but this has caused problems such as data access latency and high overhead for remote data set acquisition. In large-scale computing clusters, many training tasks are run concurrently, and the data sets used by each training task may be different. Therefore, based on the current general data set processing framework, when reading massive small file data sets, tens of millions of small files require more than tens of millions of metadata and data traffic interactions, which not only puts huge pressure on the storage system, but also greatly affects the efficiency of model training.

[0003] How to improve the metadata performance of storing massive small files, and then improve the reading performance of massive small files, is a technical problem that those skilled in the art need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a metadata processing method, device, equipment and non-volatile storage medium, which are used to improve the metadata performance of storing massive small files, and further improve the reading performance of massive small files.

[0005] In order to solve the above technical problems, the present invention provides a metadata processing method, comprising: Determine, according to the file directory of the small file data set to be written into the storage system, the small file metadata of each small file in the small file data set; Aggregate the small files into an aggregate file, construct aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on an identifier of the aggregate file; Calculate the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and write the aggregate file into the storage system according to the storage location; The aggregate file metadata corresponding to the small file data set is sent to a data reading device, so that when the data reading device needs to read a target small file in the small file data set, it queries the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculates the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

[0006] On the one hand, constructing the aggregate file metadata of the aggregate file, and associating the small file metadata and the aggregate file metadata based on the identifier of the aggregate file, including: Creating an aggregate file metadata table corresponding to the aggregate file, wherein the aggregate file metadata table at least includes an identifier of the aggregate file and an underlying storage index number of the aggregate file in the storage system; Creating an aggregate file fragmentation table corresponding to the aggregate file, the aggregate file fragmentation table at least including an identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; The aggregate file metadata table and the aggregate file fragment table are used as the aggregate file metadata.

[0007] On the other hand, creating an aggregate file shard table corresponding to the aggregate file includes: The small file metadata of the small files aggregated by the aggregate file are compressed and associated with the identifier of the aggregate file and stored in the aggregate file fragment table.

[0008] On the other hand, calculating the storage location of the aggregate file in the storage system according to the identifier of the aggregate file includes: Establishing a secondary directory structure for storing the aggregated files; The storage location of the first-level directory and the storage location of the second-level directory of the aggregate file in the secondary directory structure are calculated according to the size of the identifier of the aggregate file.

[0009] On the other hand, the identifier of the aggregated file is represented by a large integer number increasing from 1; Calculating the storage location of the aggregated file in the secondary directory structure according to the size of the identifier of the aggregated file includes: Calculate the identifier of the aggregated file and divide it by the total number of files that a first-level directory in the secondary directory structure is allowed to store and round it down to obtain the identifier of the aggregated file in the first-level directory; The identifier of the aggregated file is calculated and divided by the total number of files allowed to be stored in a second-level directory in the secondary directory structure and rounded down to obtain the identifier of the second-level directory of the aggregated file in the first-level directory.

[0010] On the other hand, it also includes: The file directory of the small file data set is digitized and then saved; intercepting a user space file system query command for the small file data set, and constructing a file view according to a file directory of the small file data set; The file view is displayed in response to the user space file system query command.

[0011] On the other hand, according to the file directory of the small file data set to be written into the storage system, determining the small file metadata of each small file in the small file data set includes: Reading attribute information of each of the small files according to the file directory of the small file data set, and constructing a small file metadata table and a small file metadata attribute table according to the attribute information of the small files; Metadata processing methods also include: intercepting a user space file system query command for the small file data set, and constructing a file view according to the small file metadata table and the small file metadata attribute table; displaying the file view in response to the user space file system query command; Aggregating the small files into an aggregate file, constructing aggregate file metadata of the aggregate file, and associating the small file metadata and the aggregate file metadata based on the identifier of the aggregate file, including: Aggregating the small files into the aggregated file; Creating an aggregate file metadata table corresponding to the aggregate file, wherein the aggregate file metadata table at least includes an identifier of the aggregate file and an underlying storage index number of the aggregate file in the storage system; Creating an aggregate file fragmentation table corresponding to the aggregate file, the aggregate file fragmentation table at least including an identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; The aggregate file metadata table and the aggregate file fragment table are used as the aggregate file metadata; Calculating the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and writing the aggregate file into the storage system according to the storage location, includes: Establishing a secondary directory structure for storing the aggregated files; Calculate the identifier of the aggregated file and divide it by the total number of files that a first-level directory in the secondary directory structure is allowed to store and round it down to obtain the identifier of the aggregated file in the first-level directory; Calculate the identifier of the aggregated file, divide it by the total number of files that a second-level directory in the secondary directory structure is allowed to store, and round it down to obtain the identifier of the second-level directory of the aggregated file in the first-level directory; The aggregate file is stored according to the identifier of the aggregate file in the first-level directory and the identifier of the second-level directory of the aggregate file in the first-level directory.

[0012] In order to solve the above technical problems, the present invention also provides a metadata processing method, comprising: Receiving aggregated file metadata corresponding to the small file data set, and storing the aggregated file metadata locally; When it is necessary to read a target small file in the small file data set, query the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculate the target storage location of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location; Among them, the aggregate file metadata is constructed based on the small file metadata of the small files of the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the aggregate file is calculated at the storage location of the storage system based on the identifier of the aggregate file.

[0013] In order to solve the above technical problems, the present invention further provides a metadata processing device, which is applied to a storage system, comprising: A determination unit, configured to determine small file metadata of each small file in the small file data set according to a file directory of the small file data set to be written into the storage system; A data set processing unit, configured to aggregate the small files into an aggregate file, construct aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on an identifier of the aggregate file; a data writing unit, configured to calculate a storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and write the aggregate file into the storage system according to the storage location; The first data reading unit is used to send the aggregate file metadata corresponding to the small file data set to the data reading device, so that when the data reading device needs to read the target small file in the small file data set, it can query the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculate the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

[0014] In order to solve the above technical problems, the present invention further provides a metadata processing device, which is applied to a data reading device, comprising: A receiving unit, configured to receive aggregated file metadata corresponding to the small file data set, and store the aggregated file metadata locally; A second data reading unit is used for, when it is necessary to read a target small file in the small file data set, querying an identifier of a target aggregate file where the target small file is located according to the aggregate file metadata, and calculating a target storage location of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location; Among them, the aggregate file metadata is constructed based on the small file metadata of the small files of the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the aggregate file is calculated at the storage location of the storage system based on the identifier of the aggregate file.

[0015] In order to solve the above technical problems, the present invention further provides a metadata processing device, comprising: Memory for storing computer programs; The processor is used to execute the computer program, and the computer program, when executed by the processor, implements the steps of any one of the metadata processing methods described above.

[0016] In order to solve the above technical problems, the present invention further provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata processing method as described in any one of the above items are implemented.

[0017] The metadata processing method provided by the present invention has the beneficial effect that when a small file data set is written into a storage system, the small files in the small file data set are aggregated and processed into an aggregate file, and the aggregate file metadata of the aggregate file is constructed, the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file, the storage position of the aggregate file in the storage system is calculated according to the identifier of the aggregate file, and the aggregate file is written into the storage system according to the storage position; the aggregate file metadata corresponding to the small file data set is sent to the data reading device, so that when the data reading device needs to read the small file data set, the position of the target aggregate file is directly calculated according to the aggregate file metadata and read. On the basis of reducing the number of read and write times of small files by aggregating and reading small files, the design of the aggregate file metadata makes it unnecessary for the data reading device to access the metadata of the small file once for each small file read, and only needs to read the aggregate file metadata corresponding to the small file data set once, and thereafter, when a read data demand is generated, it only needs to directly access the storage location where the small file is located, thereby greatly improving the metadata performance of massive small files, and further improving the storage performance of massive small files.

[0018] The metadata processing device, equipment and non-volatile storage medium provided by the present invention have the above-mentioned beneficial effects, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a metadata processing method provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a tree-shaped file system structure; Figure 3 A timing diagram of metadata query provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a metadata processing device provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a metadata processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The core of the present invention is to provide a metadata processing method, device, equipment and non-volatile storage medium, which are used to improve the metadata performance of massive small file storage, thereby improving the reading performance of massive small files.

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the early stages of artificial intelligence (AI) model training, the data set is localized before training. This not only requires sufficient local storage space, but also takes a long time to wait for the data set to be localized. With the development of big data and artificial intelligence technologies, artificial intelligence model training for complex scenarios and artificial intelligence model training that requires massive amounts of data have begun to emerge. A better processing method is needed between training and data sets to solve the scenario of separating data set storage from training and to improve the efficiency of training and reading data sets.

[0024] In order to solve the storage problem of massive data sets, related fields have proposed to use distributed storage systems to store data sets, so as to aggregate storage resources and store massive data sets. In order to solve the training problem of applying massive data sets, related fields are also developing solutions to the problem of reading massive data in training tasks. However, the currently widely used data set processing frameworks have all been developed into general data processing frameworks in order to take into account different storage, different data type files, and different application scenarios. As a result, the metadata (meta) processing performance of massive small file data sets at the scale of tens of millions and hundreds of millions is not particularly good, and there are many problems that need to be optimized. Among them, for small files at the kilobyte (kB) level used in training, the smaller the file, the higher the time consumption of metadata operations, and the latency and query rate per second (QPS) exceed the throughput, which becomes the main contradiction. During the model training process, the shuffle sampling and reading of small files, latency and query rate per second are the key to improving performance.

[0025] In addition, in large-scale computing clusters, many training tasks are run concurrently, and each training task may use a different data set. Training tasks send a large number of small file read requests to the underlying storage system. This access method is a challenge to the storage system. The storage system is inefficient in processing small files, resulting in slower training speed. Currently, there are a large number of metadata-intensive workloads in the storage system.

[0026] For massive small file datasets with tens of thousands of directories and tens of millions or even billions of small files, current processing methods mainly solve the problem from the following three aspects: (1) Use a shared distributed storage system to mount the dataset directory to each computing node, and the training task directly reads data from the shared storage. However, due to the high frequency of metadata and storage interactive access, as well as high concurrency and continuous data input / output (I / O) reading, tens of millions of small files require more than tens of millions of metadata and data traffic interactions, which brings great load pressure to the storage system and seriously affects the efficiency of model training.

[0027] (2) Encapsulate a large number of small files in an optimized data format. For example, by compressing a small file dataset and writing it to a storage system, the compressed dataset is read locally when the computing node needs it for decompression, and then the metadata of the small file dataset is used to operate the small file dataset. Although the customized format can reduce the metadata workload, it is not conducive to the management and maintenance of metadata, and the workload of reading the dataset has not changed, and it still cannot solve the performance bottleneck of limited bandwidth between the storage system and the computing nodes. At the same time, for the encapsulated small files, it is necessary to define their own encapsulation and interpretation rules, and it is necessary to maintain a management system between the storage system and the training. It takes a lot of time to compress and decompress the small file dataset, which increases the complexity.

[0028] (3) Use high-performance hardware to expand the scale and solve the performance bottleneck problem, but this brings cost issues.

[0029] Since the data set processing tools currently launched in related fields are mainly connected to object storage systems, or use distributed file systems (Hadoop Distributed File System, HDFS) at the bottom, they are basically unable to effectively solve metadata-related problems of massive small files.

[0030] Based on the commonly used dataset processing solutions, the datasets used in model training tasks are basically stored in remote storage or cloud storage. In actual model training tasks, computing nodes need to localize the datasets. However, the current common dataset processing solutions have many problems in processing massive small file datasets.

[0031] Operations on data sets mainly involve reading and writing. The writing process is mainly reflected in pulling remote data sets and writing them to local storage (memory, NVME, SSD, HDD, locally built shared storage system, distributed storage system or object storage system, etc.). The reading process is reflected in reading the pulled data sets according to iterative training batches when the model training task is running. However, if the remote storage data set is pre-pulled to the locally built storage system, the metadata information of the data set needs to be recorded, and the metadata management method affects the efficiency of obtaining the training file list during training, as well as the efficiency of out-of-order sampling of the data set. Even after localization, the metadata interaction method of massive small files affects the reading efficiency of the training task, and thus affects the training efficiency.

[0032] At present, most of the massive small file data sets are based on an architecture that separates metadata from storage data. While this architecture model has its advantages, it also leads to high-density metadata interactions for massive small file data sets, as well as high-frequency small file IO overhead, which results in the file system designed based on this model having basically no performance when facing massive small files. In the process of writing small file data sets, it is necessary to synchronize the metadata of small files, solve the problems of metadata consistency, cache mechanism and lifecycle management, and design the data structure and format of metadata. In the process of reading small file data sets, it is also necessary to maintain metadata consistency, which also involves issues such as intensive metadata interaction, cache management, cache hits, out-of-order sampling, parsing data set lists, and interaction with data storage systems.

[0033] Then, when using massive small file datasets, the problems you may face include: (1) Disk I / O bottlenecks caused by the reading and writing of massive small files. A large amount of disk I / O overhead slows down data loading and processing time, affecting overall training performance. (2) The metadata storage and access overhead of massive small file datasets is high. Massive small files usually come with additional metadata, such as file names, timestamps, etc., which not only consume storage space but also lead to intensive metadata interactions. The performance of metadata operations affects the reading and writing performance of massive small files. (3) File system limitations. Some file systems have limits on the maximum number of files stored in a directory or partition. When processing millions or billions of small files, these limitations may bring challenges in file organization and storage management.

[0034] In order to solve the metadata related problems of massive small file data sets in artificial intelligence model training tasks, the metadata processing method provided by the embodiment of the present invention provides innovative metadata construction solutions and metadata query solutions along with the reading and writing of small file data sets, so as to improve the performance of metadata operations of small file data sets, thereby improving the efficiency of artificial intelligence model training.

[0035] The metadata processing method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0036] Figure 1 The present invention provides a flowchart of a metadata processing method.

[0037] like Figure 1 As shown, the metadata processing method provided by the embodiment of the present invention includes: S101: Determine small file metadata of each small file in the small file data set according to a file directory of the small file data set to be written into the storage system; S102: Aggregate the small files into an aggregate file, construct aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on an identifier of the aggregate file; S103: Calculate the storage location of the aggregated file in the storage system according to the identifier of the aggregated file, and write the aggregated file into the storage system according to the storage location; S104: Send the aggregate file metadata corresponding to the small file data set to the data reading device, so that when the data reading device needs to read the target small file in the small file data set, it queries the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculates the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

[0038] In a specific implementation, the metadata processing method provided in the embodiment of the present invention can be applied to a storage system, and the specific steps can be implemented by deploying a metadata service in a storage node in the storage system. In an embodiment of the present invention, the metadata service can be called a metadata center. The metadata center can use a mysql or redis database.

[0039] The storage system in the embodiment of the present invention may be a shared distributed storage system mounted to each computing node. The data reading device in the embodiment of the present invention may refer to a computing node that performs an artificial intelligence model training task.

[0040] For S101, the remote small file data set is written into the storage system. According to the directory structure of the small file data set in the file system, the small file metadata of each small file in the small file data set can be determined.

[0041] The original directory structure of the small file data set often has the problem of low query efficiency. For example, in the Linux file system, a tree-like directory structure is used to organize and manage files. With the rapid increase in the amount of data, the tree-like structure has limitations, low search efficiency, and is difficult to adapt to large-scale data and dynamic changes. To solve this problem, the metadata processing method provided by the embodiment of the present invention can also include: reading the attribute information of each small file according to the file directory of the small file data set, and constructing a small file metadata table and a small file metadata attribute table according to the attribute information of the small file.

[0042] According to the POSIX interface characteristics of Linux's Virtual File System (VFS), when a large number of small files are written, the tree directory structure is parsed, the key information of the operating directory and the file is extracted, and all the leaf nodes in the tree directory structure are digitized and saved in the metadata center in the form of data.

[0043] Among them, the small file metadata table is used to store the digitized small file metadata, which may include the name of the leaf node, the underlying storage index number (inode) of the small file in the storage system, the underlying storage index number of the parent node, and the type (directory, file or other), thereby constructing a small file metadata table (leaf) that records the key information of the leaf node.

[0044] Figure 2 A schematic diagram of a tree-shaped file system structure is shown in FIG. Figure 2 As shown, the branches are: directory 1 (dir1) → directory 2 (dir2) → file 3 (file3), directory 1 (dir1) → file 1 (file1), directory 1 (dir1) → file 2 (file2), directory 1 (dir1) → directory 3 (dir3) → file 4 (file4). Then Figure 2 The tree-like directory structure shown is digitized and recorded in the small file metadata table, as shown in Table 1 (a small file metadata table).

[0045] Table 1

[0046] Type 1 indicates a file, and type 2 indicates a directory.

[0047] Each leaf node, that is, directory or file has its own attribute information (such as read and write permissions, modification time, size, etc.). It is necessary to construct a data format to record relevant attribute information. The attribute information of each directory and file is recorded in the small file metadata attribute table (leaf_attr) with the underlying storage index number (inode) as the association point. The key fields may include: underlying storage index number (inode), underlying storage index number (inode) of the parent directory, type, modification time, data size, read and write permissions, etc.

[0048] After completing the processing of small file metadata, it is also necessary to design the data storage structure and the association with the metadata center.

[0049] For S102, for massive small files, in order to reduce metadata overhead and improve storage efficiency, the small file data set is stored in an aggregate write manner. Multiple small files can be aggregated into an aggregate file, and the aggregate file metadata of the aggregate file can be constructed. In some optional implementations of the embodiments of the present invention, small files are aggregated into aggregate files, which can be aggregated into lossless compressed code blocks (chunks), each chunk can store a total of 4MB of small files, for example, for small files of 4KB in size, each chunk can aggregate 1024 small files. The aggregate file metadata corresponding to each chunk is used to record the small file metadata of the corresponding number of small files.

[0050] In the metadata center, a unique identifier is generated for each aggregated file, that is, the identifier of the aggregated file in the embodiment of the present invention. For example, a chunk ID with a unique identification code (UUID) number can be generated in units of chunks. The structure of the aggregated file metadata is designed to store the information of the aggregated file metadata, and realize the association of small file metadata and aggregated file metadata based on the identifier of the aggregated file.

[0051] For S103, a storage structure of an aggregated file data set obtained by aggregating small file data sets is designed. In order to quickly access the storage system to obtain the aggregated file, the embodiment of the present invention provides a solution for calculating the storage location of the aggregated file in the storage system according to the identifier of the aggregated file, so that the storage location of the aggregated file can be quickly determined according to the aggregated file metadata when reading small files.

[0052] For S104, in an embodiment of the present invention, the data reading device reads the small file data set in units of aggregate files, so that when the data reading device needs to read other small files in the aggregate files locally stored by the data reading device, there is no need to access the storage system and the data can be directly hit from the local storage.

[0053] In some optional implementations of the embodiments of the present invention, the aggregated file metadata may be generated into a snapshot and then sent to a data reading device to improve metadata transmission efficiency and reduce storage usage of the data reading device.

[0054] The aggregate file metadata corresponding to the small file data set is sent to the data reading device. That is, for the small file data set, the metadata only needs to be sent to the data reading device once. When the data reading device subsequently generates a read data demand for the small file data set, the identifier of the target aggregate file where the target small file is located can be determined according to the aggregate file metadata, and the target storage location of the target aggregate file in the storage system can be calculated according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location.

[0055] The metadata processing method provided by the embodiment of the present invention, when writing a small file data set into a storage system, aggregates the small files in the small file data set into an aggregate file, constructs the aggregate file metadata of the aggregate file, associates the small file metadata and the aggregate file metadata based on the identifier of the aggregate file, calculates the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and writes the aggregate file into the storage system according to the storage location; sends the aggregate file metadata corresponding to the small file data set to a data reading device, so that when the data reading device needs to read the small file data set, it directly calculates the location of the target aggregate file according to the aggregate file metadata and reads it. On the basis of reducing the number of reads and writes to small files by aggregating and reading small files, the design of the aggregate file metadata allows the data reading device to not need to access the metadata of the small file once for each small file read, but only needs to read the aggregate file metadata corresponding to the small file data set once, and thereafter, when a read data demand is generated, it only needs to directly access the storage location where the small file is located, thereby greatly improving the metadata performance of massive small files, and further improving the storage performance of massive small files.

[0056] Based on the above embodiment, the embodiment of the present invention continues to illustrate the method for constructing metadata.

[0057] In the metadata processing method provided in the embodiment of the present invention, constructing aggregate file metadata of the aggregate file in S102, associating small file metadata and aggregate file metadata based on the identifier of the aggregate file, can include: creating an aggregate file metadata table (meta_chunk table) corresponding to the aggregate file, the aggregate file metadata table at least including the identifier of the aggregate file and the underlying storage index number of the aggregate file in the storage system; creating an aggregate file slicing table (chunk_slice table) corresponding to the aggregate file, the aggregate file slicing table at least including the identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; using the aggregate file metadata table and the aggregate file slicing table as the aggregate file metadata.

[0058] In the above embodiment, it is introduced that for massive small files, in order to reduce the overhead of small file reading and writing and the overhead of small file metadata, the small files are aggregated into aggregate files, and the aggregate file metadata is constructed accordingly. In the embodiment of the present invention, an aggregate file metadata form using an aggregate file metadata table and an aggregate file shard table is provided.

[0059] The aggregate file metadata table may be used to maintain information such as the identifier of the aggregate file (chunk ID), the underlying storage index number of the aggregate file in the storage system (inode), the name of the aggregate file (name), and the size of the aggregate file (size).

[0060] When recording the aggregate file metadata, it is also necessary to record the information of the small files aggregated by each aggregate file. The aggregate file fragment table can be used to record the relevant information of the small files aggregated by the aggregate file. The main fields may include the aggregate file identifier (chunk ID), the small file metadata list and the small file metadata (files) and other information.

[0061] The aggregate file metadata table and the aggregate file fragmentation table are associated with each other through the identifier of the aggregate file, thereby associating the aggregate file metadata with the small file metadata.

[0062] In order to reduce the data volume of the aggregate file metadata, an aggregate file sharding table corresponding to the aggregate file is created, which may include: compressing the small file metadata of the small files aggregated by the aggregate file and associating it with the identifier of the aggregate file and storing it in the aggregate file sharding table. The above-mentioned files field can be used to record the list of small files aggregated by the aggregate file (file name) and key attribute information (data size, modification time, etc.) in the small file metadata. Files can be constructed in json format and compressed into BLOB format data through the zlib compression algorithm to reduce the data storage size and improve metadata query and retrieval efficiency.

[0063] Based on the above embodiment, the embodiment of the present invention continues to explain the storage method of massive small file data sets.

[0064] In the above embodiment, it is introduced that for massive small files, in order to reduce the overhead of small file reading and writing and the overhead of small file metadata, a method of aggregating small files into aggregate files is adopted. In order to realize reading and writing in units of aggregate files, the embodiment of the present invention adopts a storage solution of calculating the storage location of the aggregate file in the storage system according to the identifier of the aggregate file.

[0065] In the metadata processing method provided in an embodiment of the present invention, in order to limit the number of files under each directory, the storage position of the aggregate file in the storage system is calculated according to the identifier of the aggregate file in S103, which may include: establishing a secondary directory structure for storing aggregate files; and calculating the storage position of the first-level directory and the storage position of the second-level directory of the aggregate file in the secondary directory structure according to the size of the identifier of the aggregate file.

[0066] In the embodiment of the present invention, the metadata center names the aggregate file in the storage system according to the identifier of the aggregate file and the size of the aggregate file.

[0067] In some optional implementations of the embodiments of the present invention, the identifier of the aggregated file may be represented by a large integer (bigint) that increases from 1. Calculating the storage position of the aggregated file in the secondary directory structure according to the size of the identifier of the aggregated file may include: calculating the identifier of the aggregated file divided by the total number of files allowed to be stored in a first-level directory in the secondary directory structure and rounding down to obtain the identifier of the aggregated file in the first-level directory; calculating the identifier of the aggregated file divided by the total number of files allowed to be stored in a second-level directory in the secondary directory structure and rounding down to obtain the identifier of the second-level directory of the aggregated file in the first-level directory.

[0068] In the embodiment of the present invention, the identifier of the aggregated file can be represented by a large integer (bigint) starting from 1 and increasing in increments, and the maximum value can be , you can design a calculation rule for the secondary directory storing aggregate files, the generated first-level directory starts from / 0, the second-level directory also starts from / 0, and the second-level directory under the first-level directory / 0 is from / 0 to / 999.

[0069] For example, in the / 0 directory (calculation rule: chunk ID / 1000000), chunks with chunk ID numbers 1 to 999999 are all in this directory, and there are 0 to 999 secondary directories in the / 0 directory (calculation rule: chunk ID / 1000), and each secondary directory stores 999 chunk files. Then, in the / 0 / 0 directory, files with chunk ID numbers 1 to 999 are stored, and in the / 0 / 1 directory, files with chunk ID numbers 1000 to 1999 are stored, and so on. Chunk files with numbers 1 to 999999 are evenly distributed in the / 0 / 0 to / 0 / 999 directories under the / 0 directory, and each secondary directory stores 999 files.

[0070] The above directory calculation and storage rules can effectively reduce the number of aggregate files managed under each directory and improve the efficiency of aggregate file management, query, retrieval, update and maintenance.

[0071] The aggregate file storage solution based on the identification of aggregate files provided by the embodiment of the present invention enables the metadata center to establish an effective association with the storage system through the calculation rule on the basis of realizing effective data management of the metadata center and orderly generation and management of storage system data. At the same time, when accessing data under high concurrent load, the high-intensity IO access of metadata and storage system is greatly reduced.

[0072] Based on the above embodiment, the embodiment of the present invention continues to explain the metadata query method.

[0073] Queries for metadata of small file datasets mainly include scenarios where users query directory information of small file datasets and batch training data in sampling iterative training of computing nodes used for artificial intelligence model training.

[0074] Figure 3 A timing diagram of metadata query provided by an embodiment of the present invention.

[0075] like Figure 3 As shown, by applying the metadata processing method provided by an embodiment of the present invention, a data reading device sends a metadata query request to a virtual file system (Virtual File System, VFS) of a storage system, the virtual file system sends the metadata query request to a user space file system (Filesystem in Userspace, FUSE), the user space file system sends the metadata query request to a metadata center of a cache file system (CacheFS), and then the metadata center returns the queried metadata query result to the user space file system, then to the virtual file system, and finally to the data reading device.

[0076] To achieve user-unawareness, the metadata processing method provided by an embodiment of the present invention may also include: digitizing the file directory of the small file data set and saving it; intercepting the user space file system query command for the small file data set, and constructing a file view based on the file directory of the small file data set; displaying the file view in response to the user space file system query command.

[0077] In a specific implementation, in order to realize a user view of a small file data set without user perception, an embodiment of the present invention intercepts the relevant query interface of the file system in userspace (FUSE), and uses the relevant parsing structure through the pre-saved directory information of the small file data set to construct a file view of the small file data set in the storage system from the metadata center, so that the file view is consistent with the original file system view of the small file data set (such as the original Linux file system view). The pre-saved directory information of the small file data set can refer to the small file metadata table (leaf), small file metadata attribute table (leaf_attr), aggregate file metadata table (meta_chunk table) and aggregate file slice table (chunk_slice table) introduced in the above embodiment.

[0078] The file system is constructed through the metadata center. As long as the user does not add / delete / modify files, there is no interaction with the storage system. By applying the metadata processing method provided by the embodiment of the present invention, the file system view constructed in the metadata center of the storage system can be cached in the node memory. Only when the relevant files change, the view will be updated, that is, load once and use multiple times.

[0079] In the AI ​​model training scenario, the data reading device (computing node) needs to obtain a list of all files in the small file data set. It can obtain the underlying storage index number (inode) of the data set based on the data set name through the metadata center, query the underlying storage index number (inode) of all aggregated files of the data set in the small file metadata table (leaf), and query the identifiers (chunk ID) of all aggregated files through the aggregate file metadata table (meta_chunk table); parse the small file data aggregated by all aggregated files from the aggregate file slicing table (chunk_slice table) through the identifier (chunk ID) of the aggregate file, generate a complete small file list of the data set, and generate an index list of the data set based on the total length of the list for use in training tasks.

[0080] In each iterative training (epoch) cycle of the training task, when the sampler of the data reader (computing node) samples data, in order to increase the probability of batch sampling (batch_size) hitting the same aggregate file and ensure that the same aggregate file is cached to the same computing node as much as possible, it is necessary to design a sampler. In an embodiment of the present invention, the data reader reads the target small file in the small file data set, which may include: when shuffle sampling is required for iterative training, determining the sampling range in the shuffled (shuffle) data set index list according to the level number of the local computing node, so as to perform batch sampling within the sampling range.

[0081] Take the computing node as a graphics processing unit (GPU) as an example, GPU rank number 0 (gpu ranks are numbered from 0 in ascending order according to the number of nodes and the number of GPUs in each node). After sampling the data of a specified size, let the GPU with rank number 1 sample, and so on. The rule can be: the dataset index list sampled by each rank = the dataset index list [rank number * number of samples for each rank: (rank number * number of samples for each rank + number of samples for each rank)]. For example, if there are 256 indexes in total and the sampling size is 64, then rank0 samples the list of indexes 0 to 63, rank1 samples the list of indexes 64 to 127, and rank3 samples the list of indexes 128 to 255. Each index corresponds to a related dataset file.

[0082] The model training sampling scheme provided by the embodiment of the present invention can not only improve the probability of hitting the same aggregate file in the same batch batch_size, but also improve the probability of caching the same aggregate file to only one computing node in each iterative training cycle. After testing, the model training sampling scheme provided by the embodiment of the present invention does not affect the accuracy of the Top-1 accuracy index (the accuracy of the most likely category (with the highest probability) predicted by the model) and the Top-5 Accuracy index (the probability of the correct category being included in the top five most likely categories predicted by the model) of the deep residual network ResNet50.

[0083] Based on the above embodiments, in the metadata processing method provided in the embodiments of the present invention, S101 determines the small file metadata of each small file in the small file data set according to the file directory of the small file data set to be written to the storage system, which may include: reading the attribute information of each small file according to the file directory of the small file data set, and constructing a small file metadata table and a small file metadata attribute table according to the attribute information of the small file.

[0084] The metadata processing method provided by the embodiment of the present invention may also include: intercepting a user space file system query command for a small file data set, and constructing a file view based on a small file metadata table and a small file metadata attribute table; and displaying the file view in response to the user space file system query command.

[0085] In S102, small files are aggregated into an aggregate file, and aggregate file metadata of the aggregate file is constructed. The small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file, which may include: aggregating small files into an aggregate file; creating an aggregate file metadata table corresponding to the aggregate file, the aggregate file metadata table at least including the identifier of the aggregate file and the underlying storage index number of the aggregate file in the storage system; creating an aggregate file shard table corresponding to the aggregate file, the aggregate file shard table at least including the identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; and using the aggregate file metadata table and the aggregate file shard table as the aggregate file metadata.

[0086] In S103, the storage position of the aggregate file in the storage system is calculated according to the identifier of the aggregate file, and the aggregate file is written into the storage system according to the storage position, which may include: establishing a secondary directory structure for storing the aggregate file; calculating the identifier of the aggregate file and dividing it by the total number of files allowed to be stored in a first-level directory in the secondary directory structure and rounding down to obtain the identifier of the aggregate file in the first-level directory; calculating the identifier of the aggregate file and dividing it by the total number of files allowed to be stored in a second-level directory in the secondary directory structure and rounding down to obtain the identifier of the second-level directory of the aggregate file in the first-level directory; storing the aggregate file according to the identifier of the aggregate file in the first-level directory and the identifier of the second-level directory of the aggregate file in the first-level directory.

[0087] The specific implementation of the embodiments of the present invention can refer to the description of the above embodiments.

[0088] The metadata processing method provided by the embodiment of the present invention can significantly improve the efficiency and reliability of metadata processing of massive small file data sets in metadata operations of massive small files. Through the data format design of metadata, the association design of metadata and underlying data, metadata query, metadata storage, metadata chunk-based training sampling design, metadata cache and life cycle management, metadata update consistency maintenance, metadata visualization and fault-tolerant recovery and other aspects, the storage and access efficiency of metadata can be significantly improved, I / O overhead and response time can be reduced, the consistency and integrity of metadata can be better guaranteed, and the risk of data errors can be reduced. The design of a visual interface for the storage system facilitates the user's understanding and operation of metadata and improves the user experience. At the same time, the design of a fault-tolerant recovery mechanism enhances the stability and reliability of the system and ensures the security of data. Improve the overall pull and read efficiency of massive small file data sets, thereby improving training efficiency.

[0089] The above embodiment introduces a metadata processing method applied to a storage system. In addition, an embodiment of the present invention provides a metadata processing method applied to a data reading device.

[0090] The metadata processing method provided by an embodiment of the present invention may include: receiving aggregate file metadata corresponding to a small file data set, and saving the aggregate file metadata locally; when it is necessary to read a target small file in the small file data set, querying the identifier of a target aggregate file where the target small file is located according to the aggregate file metadata, and calculating the target storage location of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location.

[0091] Among them, the aggregate file metadata is constructed based on the small file metadata of the small files in the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the storage location of the aggregate file in the storage system is calculated based on the identifier of the aggregate file.

[0092] The specific implementation of the embodiments of the present invention can refer to the description of the above embodiments.

[0093] It should be noted that in the embodiments of the metadata processing methods of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the metadata processing methods provided in the embodiments of the present invention.

[0094] The above describes in detail various embodiments corresponding to the metadata processing method. On this basis, the present invention also discloses a metadata processing device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0095] Figure 4 A schematic diagram of the structure of a metadata processing device provided by an embodiment of the present invention.

[0096] like Figure 4 As shown, applied to a storage system, the metadata processing device provided by the embodiment of the present invention may include: A determination unit 401 is used to determine small file metadata of each small file in the small file data set according to a file directory of the small file data set to be written into the storage system; The data set processing unit 402 is used to aggregate the small files into an aggregate file, construct the aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on the identifier of the aggregate file; The data writing unit 403 is used to calculate the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and write the aggregate file into the storage system according to the storage location; The first data reading unit 404 is used to send the aggregate file metadata corresponding to the small file data set to the data reading device, so that when the data reading device needs to read the target small file in the small file data set, it queries the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculates the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

[0097] Based on the above embodiment, the embodiment of the present invention also provides a metadata processing device applied to a data reading device, which may include: A receiving unit, configured to receive aggregated file metadata corresponding to the small file data set, and store the aggregated file metadata locally; The second data reading unit is used to query the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata when it is necessary to read the target small file in the small file data set, and calculate the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position; Among them, the aggregate file metadata is constructed based on the small file metadata of the small files in the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the storage location of the aggregate file in the storage system is calculated based on the identifier of the aggregate file.

[0098] It should be noted that in each implementation of the metadata processing device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. Separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided by the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0099] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0100] Figure 5 A schematic diagram of the structure of a metadata processing device provided by an embodiment of the present invention.

[0101] like Figure 5As shown, the metadata processing device provided by the embodiment of the present invention includes: a memory 510, used to store a computer program 511; a processor 520, used to execute the computer program 511, and when the computer program 511 is executed by the processor 520, the steps of the metadata processing method provided by any of the above embodiments are implemented.

[0102] Among them, the processor 520 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 520 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 520 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 520 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 520 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0103] The memory 510 may include one or more non-volatile storage media, which may be non-transitory. The memory 510 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 510 is at least used to store the following computer program 511, wherein the computer program 511, after being loaded and executed by the processor 520, can implement the relevant steps in the metadata processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 510 may also include an operating system 512 and data 513, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 512 may be Windows or other types of operating systems. Data 513 may include, but is not limited to, the data involved in the above method.

[0104] In some embodiments, the metadata processing device may further include a display screen 530 , a power supply 540 , a communication interface 550 , an input / output interface 560 , a sensor 570 , and a communication bus 580 .

[0105] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the metadata processing device, and may include more or fewer components than those shown in the figure.

[0106] The metadata processing device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the metadata processing method provided in the above embodiment, and the effect is the same as above.

[0107] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the metadata processing method provided in any one of the above embodiments can be implemented.

[0108] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0109] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the metadata processing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0110] An embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the metadata processing method provided in any one of the above embodiments when executed by a processor.

[0111] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the metadata processing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0112] The above is a detailed introduction to a metadata processing method, device, equipment and non-volatile storage medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0113] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A metadata processing method, characterized in that: include: Determine, according to the file directory of the small file data set to be written into the storage system, the small file metadata of each small file in the small file data set; Aggregate the small files into an aggregate file, construct aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on an identifier of the aggregate file; Calculate the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and write the aggregate file into the storage system according to the storage location; The aggregate file metadata corresponding to the small file data set is sent to a data reading device, so that when the data reading device needs to read a target small file in the small file data set, it queries the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculates the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

2. The metadata processing method according to claim 1, characterized in that: Constructing aggregate file metadata of the aggregate file, and associating the small file metadata and the aggregate file metadata based on the identifier of the aggregate file, including: Creating an aggregate file metadata table corresponding to the aggregate file, wherein the aggregate file metadata table at least includes an identifier of the aggregate file and an underlying storage index number of the aggregate file in the storage system; Creating an aggregate file fragmentation table corresponding to the aggregate file, the aggregate file fragmentation table at least including an identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; The aggregate file metadata table and the aggregate file fragment table are used as the aggregate file metadata.

3. The metadata processing method according to claim 2, characterized in that: Creating an aggregate file shard table corresponding to the aggregate file includes: The small file metadata of the small files aggregated by the aggregate file are compressed and associated with the identifier of the aggregate file and stored in the aggregate file fragment table.

4. The metadata processing method according to claim 1, characterized in that: Calculating the storage location of the aggregate file in the storage system according to the identifier of the aggregate file includes: Establishing a secondary directory structure for storing the aggregated files; The storage location of the first-level directory and the storage location of the second-level directory of the aggregate file in the secondary directory structure are calculated according to the size of the identifier of the aggregate file.

5. The metadata processing method according to claim 4, characterized in that: The identifier of the aggregated file is represented by a large integer number increasing from 1; Calculating the storage location of the aggregated file in the secondary directory structure according to the size of the identifier of the aggregated file includes: Calculate the identifier of the aggregated file and divide it by the total number of files that a first-level directory in the secondary directory structure is allowed to store and round it down to obtain the identifier of the aggregated file in the first-level directory; The identifier of the aggregated file is calculated and divided by the total number of files allowed to be stored in a second-level directory in the secondary directory structure and rounded down to obtain the identifier of the second-level directory of the aggregated file in the first-level directory.

6. The metadata processing method according to claim 4, characterized in that: Also includes: The file directory of the small file data set is digitized and then saved; intercepting a user space file system query command for the small file data set, and constructing a file view according to a file directory of the small file data set; The file view is displayed in response to the user space file system query command.

7. The metadata processing method according to claim 1, characterized in that: Determining small file metadata of each small file in the small file data set according to the file directory of the small file data set to be written into the storage system includes: Reading attribute information of each of the small files according to the file directory of the small file data set, and constructing a small file metadata table and a small file metadata attribute table according to the attribute information of the small files; Metadata processing methods also include: intercepting a user space file system query command for the small file data set, and constructing a file view according to the small file metadata table and the small file metadata attribute table; displaying the file view in response to the user space file system query command; Aggregating the small files into an aggregate file, constructing aggregate file metadata of the aggregate file, and associating the small file metadata and the aggregate file metadata based on the identifier of the aggregate file, including: Aggregating the small files into the aggregated file; Creating an aggregate file metadata table corresponding to the aggregate file, wherein the aggregate file metadata table at least includes an identifier of the aggregate file and an underlying storage index number of the aggregate file in the storage system; Creating an aggregate file fragmentation table corresponding to the aggregate file, the aggregate file fragmentation table at least including an identifier of the aggregate file and the small file metadata of the small files aggregated by the aggregate file; The aggregate file metadata table and the aggregate file fragment table are used as the aggregate file metadata; Calculating the storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and writing the aggregate file into the storage system according to the storage location, includes: Establishing a secondary directory structure for storing the aggregated files; Calculate the identifier of the aggregated file and divide it by the total number of files that a first-level directory in the secondary directory structure is allowed to store and round it down to obtain the identifier of the aggregated file in the first-level directory; Calculate the identifier of the aggregated file, divide it by the total number of files that a second-level directory in the secondary directory structure is allowed to store, and round it down to obtain the identifier of the second-level directory of the aggregated file in the first-level directory; The aggregate file is stored according to the identifier of the aggregate file in the first-level directory and the identifier of the second-level directory of the aggregate file in the first-level directory.

8. A metadata processing method, characterized in that: include: Receiving aggregated file metadata corresponding to the small file data set, and storing the aggregated file metadata locally; When it is necessary to read a target small file in the small file data set, query the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculate the target storage location of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location; Among them, the aggregate file metadata is constructed based on the small file metadata of the small files of the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the aggregate file is calculated at the storage location of the storage system based on the identifier of the aggregate file.

9. A metadata processing device, characterized in that: Applied to storage systems, including: A determination unit, configured to determine small file metadata of each small file in the small file data set according to a file directory of the small file data set to be written into the storage system; A data set processing unit, configured to aggregate the small files into an aggregate file, construct aggregate file metadata of the aggregate file, and associate the small file metadata with the aggregate file metadata based on an identifier of the aggregate file; a data writing unit, configured to calculate a storage location of the aggregate file in the storage system according to the identifier of the aggregate file, and write the aggregate file into the storage system according to the storage location; The first data reading unit is used to send the aggregate file metadata corresponding to the small file data set to the data reading device, so that when the data reading device needs to read the target small file in the small file data set, it can query the identifier of the target aggregate file where the target small file is located according to the aggregate file metadata, and calculate the target storage position of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage position.

10. A metadata processing device, characterized in that: Applicable to data reading devices, including: A receiving unit, configured to receive aggregated file metadata corresponding to the small file data set, and store the aggregated file metadata locally; A second data reading unit is used for, when it is necessary to read a target small file in the small file data set, querying an identifier of a target aggregate file where the target small file is located according to the aggregate file metadata, and calculating a target storage location of the target aggregate file in the storage system according to the identifier of the target aggregate file, so as to read the target aggregate file from the target storage location; Among them, the aggregate file metadata is constructed based on the small file metadata of the small files of the small file data set, and the small file metadata and the aggregate file metadata are associated based on the identifier of the aggregate file; the aggregate file is obtained by aggregating the small files; the aggregate file is calculated at the storage location of the storage system based on the identifier of the aggregate file.

11. A metadata processing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the metadata processing method according to any one of claims 1 to 8 are implemented.

12. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the metadata processing method according to any one of claims 1 to 8 are implemented.

Images