Platform data file distributed management method and system
By storing files in chunks on multiple servers and backing them up, combined with distributed development kits to manage metadata, the problem of unbalanced load and low reliability in traditional distributed file storage systems is solved, achieving efficient file management and fault tolerance, making it suitable for large files and batch processing applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-27
AI Technical Summary
Traditional distributed file storage systems suffer from problems such as unbalanced load, low data reliability, and difficulty in file management, especially with large files and a large number of files, resulting in high single-machine load, high risk of data loss, and complex management.
The file is divided into multiple blocks and stored on different servers. Each block is backed up multiple times. Metadata is managed through a distributed development kit, a unified file access interface is provided, and a directory tree is used to record the correspondence between file blocks and servers.
It achieves load balancing, improves data reliability and management efficiency, is suitable for large files and batch processing scenarios, and ensures high availability and fast access to data in the event of hardware failure.
Smart Images
Figure CN121743301A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of distributed data management, in particular to a platform data file distributed management method and system. BACKGROUND
[0002] A file system is a disk space management service provided by an operating system, which only needs to specify the storage location of a file and the file reading path, and does not need to understand how the file is stored on the disk. However, when the required space of a file is greater than the local disk space, a disk is generally added or a machine is added to solve the problem, but adding a disk has limitations to a certain extent. Adding a machine means providing networked storage in the form of a remote shared directory, which can be understood as the rudiment of a distributed file system. It can put different files into different machines, and can continue to add machines when the space is insufficient, thereby breaking through the limitation of storage space.
[0003] Problems exist in traditional distributed file storage:
[0004] (1) The load of each storage node is unbalanced, and the load of a single machine can be extremely high
[0005] If a certain file is a popular file, many users will frequently read the file, which will cause extremely high access pressure on the machine where the file is located.
[0006] (2) Low data reliability
[0007] When a machine where a certain file is located fails, the file cannot be accessed, and sometimes data loss occurs.
[0008] (3) Difficult file management
[0009] Adjusting the storage location of a file requires checking whether the target machine has enough space, and an administrator needs to maintain the file location. In the case of a large number of machines, such an operation is extremely complex. SUMMARY
[0010] The technical solution of the present application introduces the concept of distributed management in platform data file management, solves the problems existing in traditional distributed file storage, and realizes corresponding functions through a distributed development kit, specifically including problems such as low data reliability and difficult file management. Platform data file distributed management includes a data basic framework, file reading, and file data storage.
[0011] The embodiment of the present application provides a platform data file distributed management method, which comprises the following steps:
[0012] Divide a file into multiple blocks;
[0013] Store the file blocks on different servers;
[0014] a plurality of backups are made for each file block, and the backups are stored on different servers;
[0015] metadata of the file is abstracted as a directory tree for management;
[0016] a uniform file management function is provided externally through a distributed development kit, users access files through the distributed development kit, and the distributed development kit is responsible for reading files from the underlying corresponding servers and returning the files to the users.
[0017] In some embodiments, the file is divided into a plurality of blocks, including adjusting the size of the file block according to the storage capacity of the server and the size of the file.
[0018] In some embodiments, a plurality of backups are made for each file block, including adjusting the number of backups according to the importance of the file and the reliability of the server.
[0019] In some embodiments, the metadata of the file is abstracted as a directory tree for management, including that the directory tree records the correspondence between the file block and the server.
[0020] In some embodiments, a uniform file management function is provided externally through a distributed development kit, including that the distributed development kit includes a data basic framework, a file reading function module, and a file data storage function module.
[0021] The embodiment of the present application provides a platform data file distributed management system, characterized in that the system comprises:
[0022] a file segmentation module, configured to divide a file into a plurality of blocks;
[0023] a file storage module, configured to store the file blocks on different servers;
[0024] a backup module, configured to make a plurality of backups for each file block, and store the backups on different servers;
[0025] a metadata management module, configured to abstract metadata of the file as a directory tree for management;
[0026] a distributed development kit, configured to provide a uniform file management function externally, users access files through the distributed development kit, and the distributed development kit is responsible for reading files from the underlying corresponding servers and returning the files to the users.
[0027] In some embodiments, the file segmentation module adjusts the size of the file block according to the storage capacity of the server and the size of the file.
[0028] In some embodiments, the backup module adjusts the number of backups according to the importance of the file and the reliability of the server.
[0029] In some embodiments, the metadata management module's directory tree records the correspondence between the file blocks and the servers.
[0030] In some embodiments, the distributed development kit includes a data basic framework, a file reading module, and a file data storage module.
[0031] The beneficial effects of the above embodiments include:
[0032] The design concept of the platform data file distributed management is to run on ordinary machines and store files in a streaming data manner, with one-time writing and multiple-time querying. Specifically, there are the following points:
[0033] (1) Can be built on inexpensive machines
[0034] One of the design concepts of the platform data file distributed management is to run on ordinary hardware. Even if the hardware fails, the high availability of data can be guaranteed through fault-tolerant strategies.
[0035] (2) High fault tolerance
[0036] The platform data file distributed management automatically saves multiple copies of data. After the loss of a copy, it is automatically recovered, thereby achieving high fault tolerance of data.
[0037] (3) Suitable for batch processing
[0038] The platform data file distributed management is suitable for one-time writing and multiple-time querying (reading). After the data set is generated, various analyses need to be performed on this data set for a long time. Each analysis will involve most of the data or even all of the data of the data set, so the time delay of reading the entire data set is more important than the time delay of reading the first record.
[0039] (4) Suitable for storing large files
[0040] Large files include two types: one is files with a value file size exceeding 100 MB and reaching GB or even TB, PB levels; the other is files with a number exceeding one million. BRIEF DESCRIPTION OF DRAWINGS
[0041] The accompanying drawings, which are included to provide a further understanding of the various embodiments discussed herein and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments.
[0042] Figure 1 The figure is a schematic diagram of the data basic architecture;
[0043] Figure 2 The figure is a schematic diagram of the file reading process;
[0044] Figure 3 A schematic diagram of file data storage. DETAILED DESCRIPTION
[0045] In order to enable a person skilled in the art to more fully understand the features and technical content of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present application.
[0046] In the description of the embodiments of the present application, it should be noted that unless otherwise stated and limited, the term "connection" should be understood broadly, for example, it can be an electrical connection, or a connection between two elements, it can be directly connected, or indirectly connected through an intermediate medium. For a person skilled in the art, the specific meaning of the above-mentioned term can be understood according to the specific circumstances.
[0047] It should be noted that the terms "first", "second", "third" involved in the embodiments of the present application are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first", "second", "third" can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by "first", "second", "third" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.
[0048] The data distributed management development kit depends on many independent servers, and provides a unified file management function. The basic architecture of writing data by the distributed management development kit is as shown in Figure 1 .
[0049] When a user accesses a file in the data distributed development kit, the distributed development kit is responsible for reading the file from the corresponding server at the bottom, and then returning it to the user, so that the user only needs to deal with the distributed development kit, and does not need to care about how the file is stored as shown in Figure 2 .
[0050] In order to solve the problem of uneven load of storage nodes, the distributed development kit first divides a file into multiple blocks, and then stores these file blocks on different servers. The advantage of this way is that it is not afraid of too large files, and the pressure of reading files will not be concentrated on one server, so as to avoid the problem of too high single machine load caused by a hot file. For example, when a user needs to save a file / a / b / xxx.avi, the distributed development kit will first divide the file, for example, into 4 blocks, and then store them on different servers.
[0051] If a server is broken, the file cannot be read completely. If the disk cannot be recovered, the data stored on it will be lost. To ensure the reliability of the file, the distributed development kit makes multiple backups for each file block, generally 3 backups. If a file / a / b / xxx.avi is to be stored on a distributed development kit composed of storage nodes of server data node 1, data node 2, data node 3 and data node 4, the distributed development kit divides the file into 4 blocks, namely block 1, block 2, block 3 and block 4. To ensure the reliability of the file, the distributed development kit stores the data blocks on the 4 servers as shown in FIG. 1. Figure 3
[0052] After adopting the block multi-copy storage mode, the reliability of the distributed development kit file is greatly enhanced, that is, even if a server fails, the file can still be read completely, and the concurrent access capability of the file is also increased.
[0053] When multiple users read this file, block 1 is read. The distributed development kit can select a server to read block 1 according to the busy degree of the server. To manage the file, the distributed development kit needs to record and maintain some metadata, that is, data about file data information, such as which files are stored in the distributed development kit, which blocks the file is divided into, which server each block is placed on and the like. The distributed development kit abstracts these metadata into a directory tree to record these complex correspondence relationships. These metadata are managed by a separate module.
[0054] The embodiment of the present application provides a platform data file distributed management method, comprising:
[0055] Dividing a file into multiple blocks.
[0056] Storing the file blocks on different servers.
[0057] Making multiple backups for each file block and storing the backups on different servers.
[0058] Abstracting metadata of the file into a directory tree for management.
[0059] Providing a unified file management function to the outside through the distributed development kit, a user accesses a file through the distributed development kit, and the distributed development kit is responsible for reading the file from a corresponding server at a bottom layer and returning the file to the user.
[0060] In some embodiments, dividing a file into multiple blocks comprises: the size of the file block is adjusted according to the storage capacity of the server and the size of the file.
[0061] In some embodiments, multiple backups are made for each file block, including: the number of backups is adjusted according to the importance of the file and the reliability of the server.
[0062] In some embodiments, the metadata of the file is abstracted as a directory tree for management, including: the directory tree records the correspondence between the file block and the server.
[0063] In some embodiments, a unified file management function is provided externally through a distributed development kit, including: the distributed development kit includes a data basic framework, a file reading function module and a file data storage function module.
[0064] The embodiment of the application provides a platform data file distributed management system, including:
[0065] A file splitting module is configured to split a file into multiple blocks.
[0066] A file storage module is configured to store the file blocks on different servers.
[0067] A backup module is configured to make multiple backups for each file block and store the backups on different servers.
[0068] A metadata management module is configured to abstract the metadata of the file as a directory tree for management.
[0069] A distributed development kit is configured to provide a unified file management function externally, and a user accesses the file through the distributed development kit, and the distributed development kit is responsible for reading the file from the corresponding server at the bottom layer and returning the file to the user.
[0070] In some embodiments, the file splitting module adjusts the size of the file block according to the storage capacity of the server and the size of the file.
[0071] In some embodiments, the backup module adjusts the number of backups according to the importance of the file and the reliability of the server.
[0072] In some embodiments, the directory tree of the metadata management module records the correspondence between the file block and the server.
[0073] In some embodiments, the distributed development kit includes a data basic framework, a file reading function module and a file data storage function module.
[0074] The design concept of the platform data file distributed management is to run on a general machine and store the file in a streaming data mode, that is, write once and query multiple times, and the following points are included:
[0075] (1) can be built on a cheap machine
[0076] One of the design principles of the platform's distributed data file management is to run on ordinary hardware, so that even if the hardware fails, the high availability of data can be guaranteed through fault tolerance strategies.
[0077] (2) High fault tolerance
[0078] The platform's distributed data file management automatically saves multiple copies of the data, and automatically recovers the data if a copy is lost, thus achieving high fault tolerance.
[0079] (3) Suitable for batch processing
[0080] The platform's distributed data file management is suitable for scenarios requiring write-once, query-multiple (read-once) operations. After a dataset is generated, various analyses are often performed on it over a long period. Each analysis will involve most or even all of the data in the dataset; therefore, the latency of reading the entire dataset is more critical than the latency of reading the first record.
[0081] (4) Suitable for storing large files
[0082] Large files include two types: one is files with a value exceeding 100MB and reaching GB or even TB or PB levels; the other is files with a quantity of over one million.
[0083] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.
[0084] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A distributed management method for platform data files, characterized in that, include: Split the file into multiple blocks; The file blocks are stored on different servers; Make multiple backups of each file block and store the backups on different servers; The file's metadata is abstracted into a directory tree for management; A unified file management function is provided to the outside world through a distributed development toolkit. Users can access files through the distributed development toolkit, which is responsible for reading files from the corresponding underlying servers and returning them to the users.
2. The platform data file distributed management method according to claim 1, characterized in that, The process of dividing a file into multiple blocks includes adjusting the size of the file blocks based on the server's storage capacity and the file size.
3. The platform data file distributed management method according to claim 1, characterized in that, The process of making multiple backups for each file block includes adjusting the number of backups based on the importance of the file and the reliability of the server.
4. The platform data file distributed management method according to claim 1, characterized in that, The method of abstracting file metadata into a directory tree for management includes: the directory tree recording the correspondence between file blocks and the server.
5. The platform data file distributed management method according to claim 1, characterized in that, The provision of unified file management functionality through a distributed development toolkit includes: the distributed development toolkit includes a basic data framework, file reading, and file data storage modules.
6. A platform data file distributed management system, characterized in that, include: The file splitting module is used to split a file into multiple blocks; The file storage module is used to store the file blocks on different servers; The backup module is used to make multiple backups of each file block and store the backups on different servers; The metadata management module is used to abstract the metadata of files into a directory tree for management. A distributed development toolkit is used to provide unified file management functionality. Users access files through the distributed development toolkit, which is responsible for reading files from the underlying corresponding servers and returning them to the user.
7. The platform data file distributed management system according to claim 6, characterized in that, The file segmentation module adjusts the size of file blocks based on the server's storage capacity and the file size.
8. The platform data file distributed management system according to claim 6, characterized in that, The backup module adjusts the number of backups based on the importance of the files and the reliability of the server.
9. The platform data file distributed management system according to claim 6, characterized in that, The directory tree of the metadata management module records the correspondence between file blocks and servers.
10. The platform data file distributed management system according to claim 6, characterized in that, The distributed development toolkit includes a basic data framework, file reading, and file data storage functional modules.