An optimization method and system for metadata extension

Abstract

This invention relates to an optimized method and system for metadata expansion. The method includes adding multiple cluster nodes when the number of files stored on a central node reaches a set limit. Data from the central node is distributed to the cluster nodes, and subsequent new files are synchronized from the central node to the newly added cluster nodes. Metadata information of files in the cluster nodes is uniformly aggregated and stored in the central node. When a client retrieves a batch of files, it sends relevant file information. Upon receiving this information, the central cluster node queries all metadata information based on key information and compiles the addresses of the cluster nodes containing these files, obtaining a list of routing addresses for the cluster nodes containing the files. When a client requests the central node, it connects to the corresponding cluster node based on the routing address list and directly retrieves the corresponding file data. This invention enables almost unlimited expansion of cluster nodes, improving the overall system's storage capacity and processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer communication technology, and in particular to an optimization method and system for metadata extension. Background Technology

[0002] Artificial intelligence (AI) is a popular technology, and its core lies in large-scale models. Training a suitable large model requires massive amounts of training data. These training files vary in size, and each file contains corresponding metadata. Typically, the training system accesses a central cluster node, which returns the relevant data. When the number of files is below a hundred million, the service can be provided relatively normally. However, when the number of files exceeds a hundred million, the central node becomes a bottleneck, and processing throughput drops sharply, impacting overall training efficiency. Summary of the Invention

[0003] To overcome the above problems, the purpose of this invention is to provide an optimized method and system for metadata expansion, which can expand cluster nodes almost infinitely, thereby increasing the overall system storage capacity and processing efficiency.

[0004] This invention is implemented using the following scheme: an optimization method for metadata extension, the method comprising the following steps: Step S1: When the number of files stored in the central node reaches the set number, multiple new cluster nodes are added. The data of the central node is distributed to the cluster nodes, and subsequent new files will be synchronized from the central node to the newly added cluster nodes. Step S2: The metadata information of the files in the newly added cluster nodes is uniformly summarized and stored in the central node; Step S3: When the client obtains a batch of file data, the client will send relevant information about this batch of files. After receiving the information, the central cluster node will query all metadata information based on the key information with file attributes, organize the addresses of the cluster nodes where these files are located based on the queried metadata information, and obtain the routing address list of the cluster nodes where the files in this batch of training are located.

[0005] Step S4: After the client requests the central node, it obtains the corresponding list of routing addresses. The client will then connect to the corresponding cluster node based on this list of routing addresses and directly obtain the corresponding file data.

[0006] Furthermore, the addition of multiple cluster nodes in step S1 further involves adding multiple servers as new cluster nodes, which are directly used to store files.

[0007] Furthermore, the relevant information in step S3 includes the primary key ID, the file storage MD5 value, and the file tag.

[0008] Furthermore, the acquisition in step S4 further involves: calling the interface provided by the central cluster to access the data through the interface.

[0009] An optimization system for metadata extension, the system comprising a cluster node addition module, a data aggregation module, an address query module, and a data acquisition module; The cluster node addition module: When the number of files stored by the central node reaches a set number, multiple cluster nodes are added, and the data of the central node is distributed to the cluster nodes. Subsequent new files will be synchronized from the central node to the newly added cluster nodes. The data aggregation module: The metadata information of files in the newly added cluster nodes is uniformly aggregated and stored in the central node; The address query module works as follows: When a client obtains a batch of file data, the client sends relevant information about this batch of files. After receiving the information, the central cluster node queries all metadata information based on key information with file attributes, organizes the addresses of the cluster nodes where these files are located based on the queried metadata information, and obtains a list of routing addresses of the cluster nodes where this batch of files are located.

[0010] The data acquisition module: When the client requests the central node, it obtains the corresponding list of routing addresses. The client will then connect to the corresponding cluster node based on this list of routing addresses and directly obtain the corresponding file data.

[0011] Furthermore, the addition of multiple cluster nodes in the cluster node addition module further involves adding multiple servers as new cluster nodes, which are directly used to store files.

[0012] Furthermore, the relevant information in the address query module includes the primary key ID, the file storage MD5 value, and the file tag.

[0013] Furthermore, the acquisition process in the data acquisition module further involves calling the interface provided by the central cluster to access the data through the interface.

[0014] The beneficial effects of this invention are as follows: File data is stored using multiple cluster nodes, with a central node storing all metadata. When data is retrieved, the central node analyzes the data to obtain a list of file data routing addresses and returns it to the client. The client directly connects to each cluster node based on this routing address list and retrieves the file data directly from each node. This method, by unifying metadata storage, distributing file data across multiple cluster nodes, and allowing clients to directly connect to cluster nodes for file data retrieval via routing, enables virtually unlimited expansion of the cluster nodes, thereby increasing the overall system's storage capacity and processing efficiency. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a system principle block diagram of the present invention. Detailed Implementation

[0016] The invention will now be further described with reference to the accompanying drawings.

[0017] Please see Figure 1 As shown, the present invention provides an optimization method for metadata expansion, the method comprising the following steps: Step S1: When the number of files stored in the central node reaches the set number, multiple new cluster nodes are added. The data of the central node is distributed to the cluster nodes, and subsequent new files will be synchronized from the central node to the newly added cluster nodes. Step S2: The metadata information of the files in the newly added cluster nodes is uniformly summarized and stored in the central node; Step S3: When the client retrieves a batch of file data, it sends relevant information about these files. Upon receiving this information, the central cluster node queries all metadata based on key file attribute information. Using the retrieved metadata, it compiles the addresses of the cluster nodes where these files reside, thus obtaining a list of routing addresses for the cluster nodes containing the files in this training batch. Key information includes file attributes such as file IDs, which are easily retrieved.

[0018] Step S4: After the client requests the central node, it obtains the corresponding list of routing addresses. The client will then connect to the corresponding cluster node based on this list of routing addresses and directly obtain the corresponding file data.

[0019] The present invention will be further described below with reference to a specific embodiment: An optimization method for metadata extension, 1. When the number of files stored on the central node reaches a set limit, multiple new cluster nodes are added. The data from the central node is distributed to the cluster nodes, and subsequent new files will be synchronized from the central node to the newly added cluster nodes. Previously, all files were stored on the central cluster node for faster read speeds. When the storage capacity reaches a certain level, expansion is required, and the data from the central node is distributed to the expanded nodes. The central node then only stores the metadata after the expansion.

[0020] 1.1 Adding multiple cluster nodes further involves adding multiple servers as new cluster nodes, directly used for file storage. Adding multiple servers is typically done by operations and maintenance personnel, or the node expansion is automated by a program.

[0021] For example: 1. Suppose the original central node contained 90 million files. At this point, the client accesses the central node via an interface, directly queries files based on corresponding conditions, and receives the file stream directly from the interface. When the central node's file storage exceeds the set limit of 100 million files, an expansion operation is initiated. Now, suppose 5 new nodes are added. Each node can store 100 million files.

[0022] 2. The metadata information of files in newly added cluster nodes is uniformly aggregated and stored in the central node; For example: 2. At this point, the files from the central node will be smoothly migrated to these 5 newly added nodes, with 20 million files per node. Afterward, the central node will only store the metadata information for each file.

[0023] 3. When a client retrieves a batch of file data, it sends relevant information about these files. Upon receiving this information, the central cluster node queries all metadata based on key file attribute information. Using this metadata, it compiles the addresses of the cluster nodes containing these files, obtaining a list of routing addresses for the cluster nodes containing the files in this training batch. Key information includes file attributes such as file IDs, which allow for rapid retrieval.

[0024] The relevant information in section 3.1 includes the primary key ID, the file storage MD5 value, and the file tag. The primary key ID, the file storage MD5 value, the file tag, or other business-related attributes can all serve as file-related information.

[0025] For example, in step 3, when the client accesses the central node, it will not directly query the file stream. Instead, it will use the query conditions provided by the client, based on the primary key ID, the file storage MD5 value, the file tag, or other business-related attributes, to first query which expansion nodes these files are on, and then only return the addresses of these expansion nodes.

[0026] 4. When the client requests the central node, it obtains the corresponding list of routing addresses. The client will then connect to the corresponding cluster node based on this list of routing addresses and directly obtain the corresponding file data.

[0027] 4.4 Further steps involve calling the interface provided by the central cluster to access the data.

[0028] For example, after the client obtains the expansion nodes in step 3, it directly accesses these expansion nodes and obtains the file stream directly from these nodes through the interface.

[0029] When the capacity reaches the 500 million limit again using the above method, it will continue to expand proportionally, for example, by doubling the number of cluster nodes. After expansion, similar to step 2, files need to be migrated evenly, and the routing address information of each file on the central node needs to be adjusted to ensure that the storage of each expanded node is approximately equal.

[0030] Please see Figure 2 As shown, the present invention also provides an optimization system for metadata expansion, the system including a cluster node addition module, a data aggregation module, an address query module, and a data acquisition module; The cluster node addition module works as follows: When the number of files stored on the central node reaches a set limit, multiple new cluster nodes are added. Data from the central node is distributed to these cluster nodes, and subsequent new files will be synchronized from the central node to the newly added cluster nodes. Previously, all files were stored on the central cluster node for faster read speeds. When the storage capacity reaches a certain level, expansion is required, and data from the central node is distributed to the expanded nodes. The central node then only stores the metadata after the expansion.

[0031] The addition of multiple cluster nodes in the cluster node addition module described in 1.1 further involves adding multiple servers as new cluster nodes, which are directly used to store files. Adding multiple servers is generally performed by operations and maintenance personnel, or the node expansion is automatically performed by the program.

[0032] The data aggregation module: The metadata information of files in the newly added cluster nodes is uniformly aggregated and stored in the central node; The address query module works as follows: When a client obtains a batch of file data, the client sends relevant information about this batch of files. After receiving the information, the central cluster node queries all metadata information based on key information with file attributes, organizes the addresses of the cluster nodes where these files are located based on the queried metadata information, and obtains a list of routing addresses of the cluster nodes where this batch of files are located.

[0033] The relevant information in the address query module described in 3.1 includes the primary key ID, the file storage MD5 value, and the file tag. The primary key ID, the file storage MD5 value, the file tag, or other business-related attributes can all serve as file-related information.

[0034] The data acquisition module: When the client requests the central node, it obtains the corresponding list of routing addresses. The client will then connect to the corresponding cluster node based on this list of routing addresses and directly obtain the corresponding file data.

[0035] The acquisition process in the data acquisition module described in 4.1 further involves calling the interface provided by the central cluster to access the data through the interface.

[0036] For example, after the client obtains the expansion nodes in step 3, it directly accesses these expansion nodes and obtains the file stream directly from these nodes through the interface.

[0037] When the capacity reaches the 500 million limit again using the above method, it will continue to expand proportionally, for example, by doubling the number of cluster nodes. After expansion, similar to step 2, files need to be migrated evenly, and the routing address information of each file on the central node needs to be adjusted to ensure that the storage of each expanded node is approximately equal.

[0038] In summary, file data is stored across multiple cluster nodes, with a central node storing all metadata. When data is retrieved, the central node analyzes the data to obtain a list of file data routing addresses, which is then returned to the client. The client directly connects to each cluster node based on this routing address list and retrieves the file data directly from them. This approach, by centrally storing metadata, distributing file data across multiple cluster nodes, and allowing clients to directly connect to cluster nodes for file data retrieval via routing, enables virtually unlimited expansion of the cluster nodes, thereby increasing the overall system's storage capacity and processing efficiency.

[0039] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A method of optimizing metadata extension, characterized by: The method comprises the following steps: Step S1, when the number of files stored by the center node reaches a set number, a plurality of cluster nodes are added, the data of the center node is distributed to the cluster nodes, and subsequent added files will be synchronized to the added cluster nodes by the center node; Step S2, the metadata information of the files in the added cluster nodes is uniformly stored in the center node; Step S3, when a client obtains a batch of file data, the client sends relevant information of the batch of files, the center cluster node receives the information, and according to the key information with the file attribute, the center cluster node queries all the metadata information, arranges the addresses of the cluster nodes where the files are located according to the queried metadata information, and obtains a routing address list of the cluster nodes where the batch of training files are located; Step S4, when the client requests the center node, the corresponding routing address list is obtained, and the client is connected to the corresponding cluster nodes according to the routing address list and directly obtains the corresponding file data.

2. The method of claim 1, wherein, The plurality of added cluster nodes in the step S1 are further added to a plurality of servers as the added cluster nodes for directly storing files.

3. The method of claim 1, wherein, The relevant information in the step S3 comprises a primary key id, a file storage md5 value and a file label.

4. The method of claim 1, wherein, The obtaining in the step S4 is further calling an interface provided by the center cluster and accessing data through the interface.

5. A system for optimization of metadata extension, characterized by: The system comprises a cluster node adding module, a data summarizing module, an address querying module and a data obtaining module; The cluster node adding module adds a plurality of cluster nodes when the number of files stored by the center node reaches a set number, distributes the data of the center node to the cluster nodes, and synchronizes subsequent added files to the added cluster nodes by the center node; The data summarizing module uniformly stores the metadata information of the files in the added cluster nodes in the center node; The address querying module sends relevant information of a batch of files when a client obtains the batch of file data, receives the information by the center cluster node, queries all the metadata information according to the key information with the file attribute, arranges the addresses of the cluster nodes where the files are located according to the queried metadata information, and obtains a routing address list of the cluster nodes where the batch of training files are located; The data obtaining module obtains the corresponding routing address list when the client requests the center node, and is connected to the corresponding cluster nodes according to the routing address list and directly obtains the corresponding file data.

6. The method of claim 5, wherein: The plurality of added cluster nodes in the cluster node adding module are further added to a plurality of servers as the added cluster nodes for directly storing files.

7. The method of claim 5, wherein: The relevant information in the address querying module comprises a primary key id, a file storage md5 value and a file label.

8. The method of claim 5, wherein: The obtaining in the data obtaining module is further calling an interface provided by the center cluster and accessing data through the interface.

Images