Data adaptive storage method under multi-disk

Abstract

The application discloses a data adaptive storage method under multiple disks, which comprises the following steps: arranging the disks in descending order according to the residual capacity, traversing files, preferentially using the disks with large capacity, sequentially generating storage information of each file and saving the storage information in a storage tree, and finally migrating multiple files to multiple disks according to the storage tree. The application integrates resources of multiple disks, matches files with the disks through cooperation of the multiple disks, guarantees the storage work of the data, and simulates data migration in order to guarantee the feasibility of storage. Only the reduction work of the disks is performed, and after all the files are completed, corresponding storage tree information is generated, which can effectively guarantee the feasibility of subsequent copying and reduce the rollback work of the system.

Description

Technical Field

[0001] This invention relates to the field of automated data storage technology, and more specifically to a method for adaptive data storage under multiple disks. Background Technology

[0002] With the rapid development of the internet, the total amount of global data has grown exponentially, becoming the foundation for the development of the data center industry, making data storage and backup a key focus. Both data storage and backup require calculations and statistics to determine the appropriate number of disks. Ideally, a single disk can store all the data; however, when multiple disks need to work together, the storage locations must be manually allocated. Summary of the Invention

[0003] To achieve automated data splitting and storage, this invention provides a multi-disk adaptive data storage method, comprising the following steps: Step S1: Obtain all the files that need to be stored, and get the total capacity required to store the files; Step S2: Obtain the disks that meet the conditions and obtain the total disk capacity; if the total disk capacity is greater than the total capacity required to store the files, perform the subsequent storage work; Step S3: Sort the disks in descending order of remaining capacity; Step S4: Traverse the files. For the currently traversed files, prioritize the disk with the largest remaining capacity as the directory structure disk. If no disk is found that can store the files, the process ends. If a disk is found, proceed with the next steps. Step S5: Determine whether the disk device where the file will be stored is consistent with the directory structure disk selected in Step S4. If they are consistent, write the storage information related to the current file into the storage tree. If they are inconsistent, create a soft link in the directory structure disk to link to the current file, and write the storage information related to the current file into the storage tree. The information stored in the storage tree includes: file information, the correspondence between each file and its storage location, and soft link information when the file cannot be stored on the directory structure disk. Step S6: Update the remaining disk capacity and re-sort in reverse order; Step S7: Repeat steps S4-S6 until all files have been traversed; during the traversal, if any file cannot be stored on a disk, the entire process ends; wherein, steps S4 to S6 are a simulated file processing process, in which no actual file copying operation is performed, only the storage tree is generated. Step S8: Based on the final storage tree, complete the migration of all files to multiple disks.

[0004] This invention integrates resources across multiple disks. By cooperating with multiple disks, files are matched with disks, ensuring data storage. To guarantee storage feasibility and simulate data migration, only disk reduction is performed. After all files are completed, corresponding storage tree information is generated, which effectively ensures the feasibility of subsequent copies and reduces system rollback. Attached Figure Description

[0005] Figure 1 This is a flowchart illustrating the implementation of the adaptive data storage method under multiple disks according to the present invention. Detailed Implementation

[0006] To gain a better understanding of the technical solution and beneficial effects of the present invention, the technical solution of the present invention and its beneficial effects are described in detail below with reference to the accompanying drawings.

[0007] This invention provides a multi-disk adaptive data storage method. To automate data splitting and storage, it calculates and statistically analyzes the stored data, identifies all storage devices, splits the data, simulates storage across various disk devices, and generates feasible storage tree information. Subsequent storage operations are performed according to this information. A complete implementation flowchart is shown below. Figure 1 As shown, the specific steps include the following: 1. Configure the required data storage path. The program will search for all files and directories based on the path, calculate the size of each file, and calculate the total required data capacity.

[0008] 2. Filter out unsuitable disk devices. You can specify disks and directories based on disk type, permissions, and filter, and calculate the remaining capacity of all disks, then sort them in descending order of remaining disk capacity.

[0009] 3. Check if the total data capacity is greater than the total remaining disk space. If it is, storage cannot be completed and the program will exit. If it is less, storage can proceed.

[0010] 4. Traverse all files and search for disks that can store the corresponding file based on its size, prioritizing disks with the most remaining space. If no disk can be found to store a file, the storage operation cannot be completed, and the entire process ends. If a disk can be found to store a file (hereinafter referred to as the "directory structure disk" corresponding to the file), then continue to determine the disk device where the file will be stored (i.e., the disk where the file will be stored). Figure 1 The "file storage device" in the directory structure is consistent with the disk in the directory structure. If they are consistent, the file can be stored on the disk in the directory structure. If they are inconsistent, it is assumed that the file needs to be stored on a disk in a non-directory structure. In this case, a symbolic link needs to be created on the disk in the directory structure to link to the file.

[0011] The data contains directories and files. The directory structure is stored on a designated maximum disk, and this directory structure completely preserves the relationships between directories and files. Files, however, may be stored on other disk devices, hence the aforementioned check. This method is only used when a single disk cannot completely store all the data. When a single disk can store the data, it will be saved directly to that single disk.

[0012] 5. For the traversed files, generate relevant storage information and store it in the storage tree. The storage information involves the backup directory structure and specifically includes: information about the files to be saved later, the correspondence between the backed-up files and their storage locations, and soft link information when the files cannot be stored on the disk in the directory structure.

[0013] 6. Accordingly reduce the remaining disk capacity information of the directory structure, reorder the size of each disk, and traverse the next file until all files have been traversed, resulting in a storage tree where each file is stored on multiple disks. During the file traversal, if the size of any file exceeds the remaining capacity of any disk, it means that the storage work for this round of files cannot be completed, and the storage tree will be cleaned up.

[0014] Steps 4, 5, and 6 simulate file processing; they do not actually copy the files. They are merely for generating the storage tree. The process will fail if any file cannot be stored on any disk. The storage tree only becomes effective and serves as the basis for subsequent file migrations when all files are stored on the disk.

[0015] 7. After traversing all files, a storage tree of all files is generated. Subsequent operations will use this storage tree to migrate data.

[0016] In summary, the key technical points of this invention are as follows: 1. Simulated data copying: When assessing storage feasibility, copy information is generated in the storage tree information to reduce the remaining disk space, but no actual data copying work is performed.

[0017] 2. Multi-disk collaborative storage: For large amounts of data, files are distributed and stored across multiple disks, and file and disk information is stored in the storage tree information.

[0018] 3. Use symbolic links to ensure directory integrity: Select a disk to simulate the creation of a data directory structure, create symbolic links for files on other disks, and write the corresponding information into the storage tree information.

[0019] 4. Storage tree information: This records information about the file and storage location, as well as symbolic links, during the evaluation process, and subsequent copying work depends on this information.

[0020] This invention integrates resources across multiple disks. By cooperating with multiple disks, files are matched with disks, ensuring data storage. To guarantee storage feasibility and simulate data migration, only disk reduction is performed. After all files are completed, corresponding storage tree information is generated, which effectively ensures the feasibility of subsequent copies and reduces system rollback.

[0021] Although the present invention has been described using the above preferred embodiments, it is not intended to limit the scope of protection of the present invention. Any changes and modifications made by those skilled in the art to the above embodiments without departing from the spirit and scope of the present invention shall still fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A method for adaptive data storage under multiple disks, characterized in that, Includes the following steps: Step S1: Obtain all the files that need to be stored, and get the total capacity required to store the files; Step S2: Obtain the disks that meet the conditions and obtain the total disk capacity; if the total disk capacity is greater than the total capacity required to store the files, perform the subsequent storage work; Step S3: Sort the disks in descending order of remaining capacity; Step S4: Traverse the files. For the currently traversed files, prioritize the disk with the largest remaining capacity as the directory structure disk. If no disk is found that can store the files, the process ends. If a disk is found, proceed with the next steps. Step S5: Determine whether the disk device where the file will be stored is consistent with the directory structure disk selected in step S4: If consistent, write the storage information involved in the current file into the storage tree; If there is a discrepancy, a symbolic link is created in the directory structure disk to link to the current file, and the storage information related to the current file is written into the storage tree; The information stored in the storage tree includes: file information, the correspondence between each file and its storage location, and soft link information when a file cannot be stored on a directory structure disk; Step S6: Update the remaining disk capacity and re-sort in reverse order; Step S7: Repeat steps S4-S6 until all files have been traversed; during the traversal, if any file cannot be stored on a disk, the entire process ends; wherein, steps S4 to S6 are a simulated file processing process, in which no actual file copying operation is performed, only the storage tree is generated. Step S8: Based on the final storage tree, complete the migration of all files to multiple disks.

Images