An aggregated storage calling method and device for an algorithm data set

Abstract

The application discloses an algorithm data set aggregation storage calling method and device. The method comprises the following steps: performing secondary tree directory logical fragmentation on the algorithm data set according to business logic; counting the space occupation and the total number of files by traversing the fragments one by one; creating an adaptive independent virtual disk file and formatting it into a log-type file system according to the above; creating a dedicated mounting point for each virtual disk and completing mounting, and migrating the corresponding fragment complete data to the virtual disk; uninstalling after checking the data integrity, completing single fragment aggregation storage, and repeatedly executing the whole process in batches. Through the virtual disk formatting and millisecond-level mounting / uninstalling, secondary tree directory logical fragmentation, fine permission control and multi-user parallel access mechanism, the application solves the problem of algorithm data set fragment management in the prior art, meets the requirement of flexible data modification and instant calling, and realizes efficient management in the whole life cycle without additional space loss.

Description

Technical Field

[0001] This invention relates to a method for aggregating, storing, and retrieving algorithmic datasets, as well as a corresponding aggregating, storing, and retrieving device, belonging to the field of data processing technology. Background Technology

[0002] In the entire process of algorithm development, model training, and engineering application, the storage, dissemination, and retrieval of algorithm datasets are fundamental and crucial. Currently, algorithm datasets in various fields such as computer vision, speech recognition, and natural language processing generally exhibit the characteristic of a massive number of small files. Core file types include image formats such as JPG, PNG, and BMP; video formats such as MP4, AVI, and FLV; and feature extraction files such as .npy and .mat. The size of these individual files typically ranges from tens of kilobytes to tens of megabytes. Due to the inherent characteristics of the data, it's impossible to reduce the number of files simply by merging content, unlike with text datasets. The distribution of massive numbers of small files presents numerous practical challenges to the full lifecycle management of datasets. In scenarios such as dataset storage backup, cross-device copying, and network transmission and sharing, the processing mode of massive numbers of small files leads to cumbersome procedures and extremely low time efficiency, becoming a significant bottleneck restricting algorithm development progress and hindering data sharing and collaboration.

[0003] Currently, the industry mainly uses two conventional methods for storing and distributing algorithm datasets, both of which have unavoidable technical flaws and cannot meet the needs of efficient and low-cost dataset management. The first method is to directly store and distribute the original files. In this method, each small file needs to independently complete the entire operation process of opening the file handle, reading the file data, and closing the file handle. The time spent opening and closing the file handle is determined by the operating system's file system and is a relatively fixed time cost. The data reading time of small files accounts for a very small percentage, but the cumulative operation of massive numbers of files will generate huge amounts of ineffective time. Moreover, during disk copying and network transmission, the bidirectional read and write operations will further double this ineffective time. The larger the data volume and the more files there are, the lower the time efficiency becomes. The second approach is to compress the dataset into a single file for storage and distribution. While this method can integrate a large number of small files into a single file, significantly reducing the number of file handle operations and minimizing wasted time, multimedia files such as images and videos inherently possess certain compression characteristics. When processed again using general compression algorithms, the compression ratio is extremely low, failing to achieve effective storage space saving and only achieving a superficial file integration. Furthermore, this method has significant drawbacks. The dataset requires complete decompression before use, with decompression taking roughly the same time as compression. The decompression process also generates twice the storage space required by the dataset itself. If modifications, additions, or deletions are needed, repeated decompression and recompression operations are necessary. Even minor changes cannot avoid additional time and space consumption, significantly increasing the cost and complexity of using the dataset.

[0004] Chinese invention patent CN116185308B discloses a dataset processing method. This solution reduces the number of write operations by writing small file datasets to the storage system as aggregated file datasets. During model training, when retrieving required small files from the storage system based on the iterative dataset information list generated from the metadata of the aggregated file dataset, the required small file data is only retrieved from the storage system if it is not already present on the local compute node. This reduces the number of read operations during model training, improving small file processing efficiency and thus enhancing both storage performance and the efficiency of AI project implementation. However, this solution still faces challenges in managing algorithm dataset shards, failing to meet the requirements for flexible data modification and immediate retrieval, and cannot achieve efficient full lifecycle management without additional space consumption. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide a method for aggregating, storing, and retrieving algorithm datasets.

[0006] Another technical problem to be solved by the present invention is to provide an aggregate storage and retrieval device for algorithm datasets.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution: According to a first aspect of the present invention, a method for aggregating, storing, and retrieving an algorithm dataset is provided, comprising the following steps: Divide the algorithm dataset into one or more dataset fragments; For each dataset shard, calculate its space usage and total number of files; Based on the space usage and the total number of files, a corresponding virtual disk file is created, wherein the size of the virtual disk file at least meets the storage requirements of the space usage, and the number of its inodes at least meets the indexing requirements of the total number of files; Format the virtual disk file into the target file system; Create a mount point and mount the formatted virtual disk file to that mount point; The corresponding dataset shards are migrated to the virtual disk under the mount point; After the data migration is complete, the virtual disk file is unloaded to obtain an aggregated virtual disk file containing the sharded data of the dataset.

[0008] Preferably, when dividing the algorithm dataset into one or more dataset shards, the logical sharding process is performed according to at least one of the following: business logic, data type, data collection time, algorithm application scenario, or data annotation type. Each dataset shard is an independent directory unit, and the data capacity within the shard is not required to be evenly distributed.

[0009] Preferably, when calculating the space usage and total number of files for each dataset segment, all subdirectories and files of each dataset segment are traversed separately, the space usage is accumulated, the total number of files is counted one by one, and standardized statistical information including segment name, storage path, space usage, and total number of files is generated.

[0010] Preferably, the number of index nodes is configured according to the total number of files, so that the index node resources are not exhausted when the virtual disk space is not full.

[0011] Preferably, the target file system is a journaling file system, including but not limited to ext4, XFS or Btrfs.

[0012] Preferably, the mounting operation is set to read-only permission or read-write permission according to usage requirements.

[0013] Preferably, the aggregated storage access method further includes the following steps: Create a parent directory for unified access; Under the parent directory, create a corresponding subdirectory mount point for each aggregated virtual disk file; Mount each aggregated virtual disk file to its corresponding subdirectory mount point; By accessing the parent directory and its subdirectories' mount points, data in the algorithm dataset can be read or modified. After use, uninstall all mounted aggregated virtual disk files.

[0014] Preferably, when mounting each aggregated virtual disk file to the corresponding subdirectory mount point, the mounting method includes temporary mounting and permanent mounting; wherein temporary mounting is automatically invalidated after system restart, and permanent mounting is achieved by modifying the operating system configuration file to automatically mount after system restart; if the mounting of an aggregated virtual disk file fails, the mounting of that segment is paused and an early warning is issued, and the segment is remounted separately after the problem is resolved.

[0015] Preferably, when mounting each aggregated virtual disk file to the corresponding subdirectory mount point, for dataset fragments that do not need to be modified, they are mounted with read-only permissions; for dataset fragments that need to be modified, they are mounted with read-write permissions.

[0016] According to a second aspect of the present invention, an aggregation storage and retrieval device for an algorithm dataset is provided, comprising a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, causes the processor to implement the above-described method.

[0017] Compared with existing technologies, this invention divides the algorithm dataset into logical fragments and creates virtual disk files for each fragment, each with its own space requirements and the total number of files. During the storage phase, only a one-time data migration is required. Subsequent dataset propagation and retrieval leverage the millisecond-level mounting / unmounting capabilities of the virtual disk, completely eliminating the significant time-consuming overhead caused by repeatedly opening and closing file handles for numerous small files in traditional raw file storage methods. Furthermore, compared to compressed package storage, this invention allows direct mounting and use of the dataset without decompression, avoiding the time overhead of compression and decompression and the doubled storage space consumption. It also supports flexible, real-time modification of data in the virtual disk without repackaging. In addition, for ultra-large-scale algorithm datasets, this invention effectively solves the fragment management problem through logical fragmentation of a two-level tree directory, batch aggregation storage, and a multi-fragment mounting and access mechanism under a unified parent directory. It supports read-only / read-write permission control, temporary / permanent mounting, and multi-user parallel access, thereby achieving efficient storage, rapid propagation, and real-time retrieval of the algorithm dataset throughout its entire lifecycle, significantly reducing time and space costs. Attached Figure Description

[0018] Figure 1A This is a schematic diagram of the aggregation and storage process of the algorithm dataset in the first embodiment of the present invention; Figure 1B This is a schematic diagram of the algorithm dataset calling process in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the algorithm dataset aggregation storage and retrieval device in the second embodiment of the present invention. Detailed Implementation

[0019] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0020] First Embodiment like Figure 1A and Figure 1B As shown, the first embodiment of the present invention provides a method for aggregating, storing, and retrieving algorithm datasets, which includes at least the following steps: S1: Perform logical sharding of the algorithm dataset into a two-level tree-like directory based on business logic.

[0021] S2: Iterate through each dataset piece by piece to count the space usage and the total number of files.

[0022] S3: Creates an adapted independent virtual disk file based on the space and number of files in the partition.

[0023] S4: Format each individual virtual disk file as a journaling file system.

[0024] S5: Create a dedicated mount point for each virtual disk file and complete the mount operation.

[0025] S6: Migrate the complete data of the dataset shards to the virtual disk of the corresponding mount point.

[0026] S7: After verifying the integrity of the virtual disk data, perform the unloading operation to complete the single shard aggregation storage.

[0027] S8: Iterate through all datasets in batches, repeatedly execute the aggregation storage operation to complete the overall processing.

[0028] S9: Create a unified parent access directory for the algorithm dataset.

[0029] S10: Create a mount point for the subdirectory corresponding to the fragment under the parent directory.

[0030] S11: Batch mount each virtual disk segment to the corresponding subdirectory mount point.

[0031] S12: Access the algorithm dataset as a whole from the parent directory according to the organizational logic.

[0032] S13: After use, unload all fragmented virtual disks in sequence.

[0033] It should be noted that steps S1 to S8 above are used to aggregate and store the algorithm dataset; steps S9 to S13 are used to call the aggregated virtual disk file generated by steps S1 to S8. The specific steps of the above method are explained in detail below: In step S1, a root directory is first created for the entire algorithm dataset. Then, according to the determined sharding rules, an independent first-level subdirectory is created under the root directory for each dataset to be divided. Each first-level subdirectory is an independent dataset shard. All first-level subdirectories and the root directory together constitute a two-level tree-like directory structure for the algorithm dataset. The directory name of each dataset shard can be named according to its logical attributes to facilitate subsequent management, identification, and retrieval. The number of shards can be reasonably controlled according to the actual application scenario, and its upper limit is mainly limited by two factors: First, the operating system's file system imposes a hard limit on the number of subdirectories under the root directory. Different file systems have different thresholds for the number of directories, and the partitioning must be carried out within the threshold range. Secondly, the complexity of manual operation and subsequent management limits the process. Too many shards will lead to an excessively long operation process and increased management costs for subsequent batch aggregation and storage. Too few shards will fail to solve the overall processing problem of the algorithm dataset. Therefore, it is necessary to find the optimal balance between the number of shards and processing efficiency while meeting the requirements of storage, propagation and subsequent calling.

[0034] Meanwhile, the data capacity in each slice does not need to be forced to be evenly distributed. It can be flexibly adjusted according to the logical attributes and data characteristics of each slice. For example, for algorithm datasets sliced ​​by data type, the capacity of image file slices may be much larger than that of feature extraction file slices. It is only necessary to ensure the independent integrity of each slice, so as to adapt to the slicing requirements of different types and structures of algorithm datasets.

[0035] After the sharding operation is completed, each dataset shard is an independent directory unit, which contains all files, subdirectories and complete directory structure under the corresponding logic. This ensures the integrity and independence of the data in each shard, and there is no data overlap or directory disorder. It provides a clear and standard processing unit for subsequent aggregation storage operations to be performed on each shard separately, ensuring that subsequent batch aggregation storage operations can be carried out in an orderly and efficient manner.

[0036] This step addresses the issue of large numbers of small files in the algorithm dataset. Based on actual business needs, logical sharding is performed. By constructing a two-level tree-like directory structure, the entire algorithm dataset is divided into multiple independent dataset shards. This provides standardized and independent processing units for subsequent batch-based aggregation and storage operations, solving the problem that algorithm datasets cannot be directly aggregated and stored as a whole. In practice, the sharding criteria are not fixed and can be flexibly set according to various dimensions such as the business logic of algorithm development, the data type of the dataset, the time point of data collection, the application scenario corresponding to the data, and the data annotation type. The only requirement is that each sharded sub-dataset has independent logical attributes and can be stored and retrieved as a complete unit.

[0037] In step S2, the data statistics operation needs to be performed separately for each dataset shard, rather than performing overall statistics on the entire algorithm dataset. The statistical process for each shard is independent, and the statistical results only correspond to that shard, ensuring the accuracy and relevance of the statistical data. When calculating the space usage of a single dataset shard, an automated traversal script or a professional file statistics tool is used to perform a full traversal of all contents under the shard's subdirectories, including subdirectories at all levels, files of different formats, hidden files, etc. The actual storage space size of each file is identified and counted one by one, and the storage space sizes of all files are summed to obtain the total storage space size of the dataset shard. During the statistics process, the operating system's file size calculation standards must be strictly followed, including the actual data size of the files and the metadata space occupied by the file system, to ensure the completeness and accuracy of the space usage calculation and avoid the problem of insufficient virtual disk storage space due to missed statistics. When calculating the total number of files in a single dataset fragment, the same full traversal method as for space usage statistics is used. This involves traversing all levels of directories under the fragment's subdirectories, counting each file individually, including regular files, hidden files, and files of different formats, regardless of file size or type. Accurate file count is crucial for the rational planning of virtual disk inode resources. Since an inode is a structure in the file system used to store file metadata, each file corresponds to a unique inode. Inaccurate file counts can lead to an unreasonable allocation of virtual disk inode resources, resulting in situations where disk storage space is not full but inode resources are exhausted, thus wasting the remaining virtual disk storage space. During the statistics process, real-time progress monitoring and data verification mechanisms can be set up to display the progress in real time, preventing interruptions due to excessive fragment data volume. After the statistics are completed, a secondary verification is performed, using random sampling to check the file count and size of some directories to ensure the accuracy of the results. After the statistics are completed, the statistical data for each dataset shard is recorded and stored separately, forming a standardized statistical table containing shard name, storage path, space usage, and total number of files. This table serves as the core reference for creating a dedicated virtual disk file for each shard, providing data support for the subsequent refined creation of virtual disks.

[0038] This step involves performing refined data statistics operations on each independent dataset shard completed in step S1. By performing a full traversal, the actual space usage and total number of files for each shard are accurately calculated. This provides precise and reliable parameters for creating suitable virtual disk files for each shard, fundamentally avoiding storage space waste or inode resource exhaustion caused by unreasonable virtual disk size settings, and ensuring that the virtual disk can highly match the actual data characteristics of the dataset shards.

[0039] In step S3, the size of the virtual disk file is determined by the total space usage of the corresponding dataset fragment and the total number of files. It must simultaneously meet the dual requirements of storage space and inode resources, rather than simply being set based on space usage. On one hand, the basic storage space of the virtual disk file must be greater than or equal to the actual space usage of the dataset fragment, reserving a certain amount of redundant space to ensure complete storage of all files, subdirectory structures at all levels, and file system metadata for that fragment. The proportion of redundant space can be flexibly set according to the subsequent update needs of the dataset, generally set to 5% to 10% of the actual space usage. If the dataset fragment subsequently requires data addition, modification, or other updates, the proportion of redundant space can be appropriately increased to avoid data updates being impossible due to insufficient storage space. On the other hand, the inode resources of the virtual disk need to be rationally planned according to the total number of files in the dataset fragment. For dataset fragments with a large number of small files, the demand for inode resources is much higher than for large file datasets. If only storage space is considered while ignoring inode resources, a situation may arise where the virtual disk has sufficient storage space but the inode resources are exhausted, resulting in new files being unable to be written and the remaining storage space becoming unusable. Therefore, when creating a virtual disk file, the total number of inode resources required needs to be calculated based on the total number of files. The inode ratio should be adjusted in the virtual disk creation parameters. For datasets with a large number of files and small individual files, the inode resource ratio should be appropriately increased to increase the total number of inodes. For datasets with a small number of files and large individual files, while ensuring sufficient inode resources, the focus should be on optimizing storage space utilization and reducing the redundancy of inode resources. In practice, virtual disk file creation can be achieved through automated scripts. The fragment space occupancy and total number of files calculated in step S2 are used as input parameters for the script. The script automatically calculates and determines the optimal size and inode ratio parameters of the virtual disk file based on a preset algorithm, and then calls the operating system's virtual disk creation command to execute the creation operation. During the creation process, a parameter verification mechanism can be set to verify the rationality of the creation parameters and avoid virtual disk creation failure due to incorrect parameter settings. After the virtual disk file for each dataset shard is created, it is stored in the specified directory. The virtual disk file is given a name that is consistent with the corresponding dataset shard, which facilitates subsequent identification, management and association. At the same time, information such as the storage path, file size, number of inodes, and corresponding shard name of each virtual disk file is recorded to form a standardized virtual disk creation ledger, which provides a reference for subsequent formatting, mounting and other operations.

[0040] This step, based on the space usage and total number of files for each dataset shard obtained in step S2, creates a dedicated virtual disk file for each shard. By dynamically adjusting the size of the virtual disk file, it achieves a dual rational planning of storage space and inode resources. This ensures that the virtual disk file can fully adapt to the storage requirements of the dataset shard, storing the shard data completely without wasting storage space and inode resources. Virtual disks are a technology in the computer field that simulates the function of a physical disk through files, mapping files to logical disk partitions. The virtual disk file created in this step is a single, independent file; each dataset shard corresponds to a unique virtual disk file. Each virtual disk file is independent of the others, each adapted to the characteristics of its corresponding shard data, with no data association or resource sharing.

[0041] In step S4, this process needs to be performed individually for each virtual disk file. The operation strictly follows the formatting standard of the target file system (e.g., ext4). Using the operating system's formatting commands, complete ext4 file system parameters are configured for the virtual disk file, including file system block size, inode size, journaling mode, and reserved space ratio. The file system block size can be adjusted based on the file size characteristics of the dataset fragments. For fragments primarily composed of small files, a smaller block size is set to reduce internal disk fragmentation; for fragments containing some large files, the block size is appropriately increased to improve file read / write efficiency. After the formatting operation is complete, the virtual disk file will have a complete ext4 file system structure, including the file system interface, a set of software for object manipulation and management, and objects and attributes. It can perform all common operations such as file creation, storage, reading, modification, dumping, and deletion, meeting all the underlying requirements for fragmented storage of the algorithm dataset. If the algorithm development equipment uses a different type of operating system or has specific file system requirements in the actual application scenario, the virtual disk files can be formatted to other compatible file system types, such as XFS or Btrfs. The key is to ensure that the selected file system can support the storage and access of massive amounts of small files and has good stability, performance, and compatibility. After formatting, perform an integrity check on the file system of each virtual disk file. Use professional file system testing tools to check whether the file system structure is complete, whether the parameters are configured reasonably, and whether there are any corruption issues. This ensures that the formatted virtual disk files can be used normally. After passing the check, record the file system type, configuration parameters, and other information for each virtual disk file, and update the virtual disk creation log.

[0042] This step involves formatting the individual virtual disk files created for each dataset shard in step S3. It prioritizes formatting them as ext4 file systems, giving the virtual disk files a complete file system structure that the operating system can recognize and manipulate. This provides stable and efficient underlying file system support for subsequent migration, storage, and retrieval of dataset shards. If other file system requirements exist in the actual application scenario, other compliant file system types can also be flexibly adapted. A file system is the method and data structure used by the operating system to define files on a storage device. It is the foundation for file storage, management, and access. An unformatted virtual disk file is merely a regular binary file and cannot achieve hierarchical file storage, directory management, or access control. Therefore, formatting is necessary to endow it with file system attributes.

[0043] It should be noted that the ext4 file system is preferred in this step because, as the fourth-generation extended file system under Linux, ext4 has many advantages in adapting to the storage of algorithm datasets: First, it has strong stability. The journaling file system writes file operation records to the log. When abnormal situations such as system crashes or power outages occur, the file system can be quickly restored through the log, avoiding data loss or damage and ensuring the storage security of the algorithm dataset. Secondly, it has excellent file processing performance. ext4 supports very large files and massive directories, and can efficiently process a large number of small files in the algorithm dataset, greatly improving file read and write speed and access efficiency. Third, it has good compatibility. ext4 is highly compatible with mainstream Linux distributions and backward compatible with file systems such as ext3 and ext2. It can adapt to different operating system environments and meet the usage needs of different devices in the process of algorithm development. Fourth, it supports fine-grained access control, which can be achieved by setting read, write and execute permissions for files and directories, thereby ensuring data security.

[0044] In step S5, the creation of mount points and the mounting of virtual disks must be performed separately for each virtual disk file. Each virtual disk file corresponds to a unique mount point. Multiple virtual disk files are not allowed to be mounted to the same mount point, nor is a single virtual disk file allowed to be mounted to multiple mount points, to avoid problems such as file conflicts, directory confusion, and access errors. When creating a mount point, the operating system's file system rules must be followed, and an appropriate directory location must be selected for creation. Generally, it is preferable to create the mount point in the system's temporary directory or a fixed directory specifically created for the algorithm dataset. The directory name of the mount point must be consistent with the corresponding virtual disk file and dataset fragment for easy identification, management, and association later. The mount point creation operation can be executed through the operating system's directory creation command. After creation, the directory is an empty mount point, waiting to be associated with the virtual disk file. After creating the mount point, execute the virtual disk mount operation. Using the operating system's virtual disk mount command, associate the formatted virtual disk file with the corresponding dedicated mount point. The mount command must specify the storage path of the virtual disk file and the directory path of the mount point. Temporary mount parameters, such as read / write permissions and caching policies, can also be set according to actual needs. After the mount operation is executed, the operating system will recognize the mount point as an independent logical disk partition. Users can access the mount point directory to read and write to the virtual disk file, just like accessing a regular physical disk partition. After the mount operation is complete, the mount status needs to be verified. Use the operating system's mount query command to check whether the virtual disk file has been successfully mounted to the specified mount point and whether the mount status is normal. You can also verify the read / write functionality of the virtual disk by creating and deleting test files in the mount point directory to ensure the mount operation is successful and subsequent data migration operations can be performed smoothly. If mounting fails or the mounting status is abnormal during the mounting process, the cause should be investigated in a timely manner, including whether the virtual disk file is corrupted, whether the file system is complete, whether the mount point exists, and whether the permissions are sufficient. After the problem is resolved, the mounting operation should be re-executed to ensure that each virtual disk file can be successfully and normally mounted to its corresponding dedicated mount point.

[0045] This step involves creating a unique mount point for each formatted virtual disk file within the operating system's file system and mounting the virtual disk file to the corresponding mount point. This establishes a valid access connection between the operating system and the virtual disk, providing a usable file access entry point for subsequently migrating dataset fragments to the virtual disk. It ensures that the operating system can perform normal read and write operations on the virtual disk. A mount point is the entry directory of the disk file system in Linux, similar to drive letters like C:, D:, and E: in Windows. It is the only path for the operating system to access disk partitions. Even if a virtual disk file is formatted, it cannot be recognized and accessed by the operating system if it is not mounted. Therefore, the mounting operation is a crucial step in enabling the actual use of virtual disk files.

[0046] In step S6, data copying needs to be performed separately for each dataset shard. All contents of the original dataset shard's root directory, including subdirectories, all file formats, hidden files, file metadata, and directory permission settings, must be completely copied to the corresponding virtual disk mount point directory. During the copying process, the original directory structure must be strictly maintained; that is, the directory structure under the mount point directory must be completely consistent with the original dataset shard's directory structure. This ensures that users can find and use data according to their original access habits without changing their operating methods. The copying operation can be performed in various ways, including using the operating system's built-in file copy command or professional file transfer tools. Given the characteristics of a large number of small files, tools or commands that support multi-threaded copying and breakpoint resumption are preferred. Enabling multi-threaded copying improves copying efficiency and reduces copying time. The breakpoint resumption function prevents copying operations from failing due to system interruptions, network failures, or other abnormal situations. If an abnormality occurs, the copying operation can resume from the breakpoint after recovery, without having to recopy all the data. During the copying process, a robust data verification mechanism must be established to ensure the accuracy of data migration. Verification methods can be divided into real-time verification and post-verification. Real-time verification involves immediately verifying each copied file during the copying process. By comparing the file size, hash value, and other characteristic information of the source and target files, it verifies whether the files have been copied completely and whether any corruption has occurred. Post-verification involves performing a full verification of all data in the mount point directory after the copying operation of the entire dataset fragments is completed. By writing an automated verification script, iterates through all files and directories in the mount point directory, comparing them one by one with the original dataset fragments in terms of file quantity, file size, directory structure, hash value, and other information, ensuring that all data has been successfully copied without any damage or errors. If problems such as missing files, corrupted files, or disordered directory structures are found during the verification process, the cause of the problem must be located promptly, and retransmission or repair operations should be performed on the missing or corrupted files to ensure the integrity of the data migration. Simultaneously, real-time progress monitoring and logging functions can be set up during the copying process, displaying the copying progress in real time, including the number of files copied, the size of the copied data, and the remaining copying time, allowing operators to easily monitor the copying status. All operations, anomalies, and verification results during the copying process are also recorded in a log file, facilitating subsequent troubleshooting and tracing. After the data migration operation is completed and verified to be error-free, it is confirmed that all data in the dataset shards has been completely and accurately stored in the corresponding virtual disk files. At this point, the virtual disk files have become independent storage carriers containing the complete shard data.

[0047] This step involves copying all files and subdirectories of a single dataset shard from step S1 to the corresponding mount point directory in step S5, ensuring a complete data migration of that dataset shard to its dedicated virtual disk file. This is the only major time-consuming step in the algorithm's dataset shard aggregation and storage process, and it is also the core step in ensuring the complete storage of sharded data. Since each dataset shard contains a massive number of small files, the core requirements for data migration are ensuring data integrity, directory structure consistency, and data accuracy. This ensures that the migrated dataset shard is completely consistent with the original data, without any file loss, file corruption, or directory structure errors, providing a reliable data foundation for subsequent storage, dissemination, and retrieval.

[0048] In step S7, the uninstallation operation must be performed only after data migration is complete and all data has been verified to be error-free. It is strictly prohibited to perform the uninstallation operation during data copying or when data verification issues exist to prevent data loss or corruption. Before performing the uninstallation operation, a pre-check must be performed. Using relevant operating system commands, check whether there are any open file handles or ongoing file read / write operations at the mount point corresponding to the virtual disk. If open file handles or ongoing read / write operations exist, the relevant processes must be closed, file handles released, and read / write operations terminated to ensure the virtual disk is in an idle state without any operations. Performing the uninstallation operation while file operations are active will prevent data from being properly synchronized to the virtual disk file, resulting in data loss or file system corruption. After the pre-check is completed, a standardized uninstallation operation is performed. Using the operating system's virtual disk uninstallation command, specify the mount point directory path to be uninstalled, and execute the uninstallation operation. The uninstallation command will automatically disconnect the virtual disk from the mount point, write the unsynchronized data in memory to the virtual disk file, release related system resources and file handles, and complete the entire uninstallation process. After the uninstallation operation is executed, the uninstallation result needs to be verified. Using the operating system's mount query command, check whether the virtual disk file has been successfully unmounted from the mount point and whether the mount point has reverted to a normal empty directory, ensuring the uninstallation operation was successful. After uninstallation, all aggregation storage operations for this dataset fragment are complete. At this point, the dataset fragment no longer exists as a large number of small files, but is completely aggregated into a single independent virtual disk file. This virtual disk file contains all the data and complete directory structure of the fragment, possessing the characteristics of independent storage and propagation. Compared to the original large number of small files, this virtual disk file, as a single file, only requires operations on one file during storage and propagation, significantly reducing the unnecessary time spent opening and closing file handles and improving storage and propagation efficiency. Simultaneously, the virtual disk file retains the complete directory structure and file attributes of the original dataset fragment, allowing for quick access via mount operations during subsequent calls, without the need for additional operations such as decompression. After unloading, the aggregated storage results of the dataset shards are archived and the statistical ledger is updated. Information such as the virtual disk file storage path, file size, data migration completion time, verification results, and unloading completion time of the shards are recorded to form a complete operation record, which is convenient for subsequent management and traceability. At the same time, the virtual disk file can be backed up according to actual needs to ensure data security.

[0049] This step, following the completion of the dataset shard migration and data verification in step S6, involves performing a standardized unloading operation on the virtual disk file. This securely unmounts the virtual disk from its corresponding dedicated mount point, completing all aggregated storage operations for that single dataset shard. This allows the shard to exist as an independent virtual disk file, ready for subsequent operations such as storage, backup, and distribution. The core requirement of this unloading operation is ensuring its security and standardization, preventing issues such as data synchronization problems, unreleased file handles, and file system corruption caused by abnormal unloading. This ensures the integrity and usability of the shard data within the virtual disk file.

[0050] In step S8, a reasonable batch execution order needs to be determined first. This can be based on factors such as the size, logical attributes, and storage path of the dataset fragments. For example, fragments with smaller data volumes can be divided into one batch, while fragments with larger data volumes can be divided into a separate batch. Alternatively, the fragments can be executed sequentially according to their logical attributes. The batch order should balance execution efficiency and ease of operation to avoid chaotic execution due to unreasonable batch division. During batch execution, steps S2 to S7 are repeated sequentially for each dataset fragment. This involves calculating the space usage and number of files for each fragment, creating a suitable virtual disk file, formatting it to the target file system, creating a dedicated mount point and mounting it, completely migrating the fragment data to the virtual disk, verifying data integrity, and then performing an unmounting operation. The processing of each fragment is independent; the processing result of one fragment does not affect the processing of the next. The virtual disk files corresponding to each fragment are also independent, storing the data of their respective fragments separately, without data association or resource contention. To ensure the stability and traceability of the batch execution process, a robust progress monitoring, anomaly alert, and logging mechanism must be established. First, real-time progress monitoring: by writing automated batch execution scripts, information such as the name of the currently processed shard, the current operation step, the processing progress, the number of shards that have been processed, and the number of shards that remain to be processed can be displayed in real time, so that operators can keep track of the overall execution status. Secondly, there is an anomaly warning mechanism. During batch execution, each operation step is monitored in real time. If an anomaly occurs in a certain operation step of a certain shard, such as statistics failure, virtual disk creation failure, formatting failure, mounting failure, data migration error, verification failure, or unloading failure, the script will immediately pause the processing of that shard and issue an anomaly warning message through pop-up windows, SMS, email, etc. At the same time, the cause of the anomaly is located and recorded. Operators can promptly investigate and resolve the problem based on the warning message. After the problem is resolved, the script can continue to execute the processing operation of that shard from the step where the anomaly occurred, avoiding the spread of anomalies and affecting the processing of the entire algorithm dataset. Third, full-process logging records all operation information during batch execution, including the processing time of each segment, the execution results of each operation step, the occurrence time, cause and solution of abnormal situations, data verification results, etc., forming a complete batch execution log file, which facilitates subsequent problem investigation, operation traceability and result acceptance.

[0051] During batch execution, a parallel processing mechanism can be appropriately configured based on the operating system's performance. Provided operating system resources allow, different operation steps can be executed simultaneously on multiple dataset shards. For example, while performing data migration operations on shard A, virtual disk creation and formatting operations can be performed on shard B. Parallel processing improves overall execution efficiency and reduces the overall aggregation and storage time of the algorithm dataset. Once all dataset shards have completed all operations from steps S2 to S7, and each shard has formed an independent, data-complete virtual disk file, the shard aggregation and storage operation of the entire algorithm dataset is complete.

[0052] At this point, the original algorithm dataset, which consisted of a massive number of small files, was split into multiple independent virtual disk files. Each virtual disk file corresponds to a dataset shard. This process preserves the complete data and directory structure of the original dataset while solving the problem of low efficiency in storing and distributing massive numbers of small files. Furthermore, the independence of each shard's virtual disk file facilitates fine-grained management, backup, and distribution. Subsequent operations can be performed on individual shards as needed, improving the flexibility of dataset management. After the overall processing is complete, a full acceptance test is performed on the aggregated storage results of all shards. The consistency between the virtual disk files of each shard and the original shard data is verified, and the batch execution logs are checked for any unresolved anomalies. Upon successful acceptance, a report on the results of the algorithm dataset sharded aggregated storage is generated. This report includes sharding details, virtual disk information for each shard, overall processing time, and data integrity verification results, providing a complete reference for subsequent dataset retrieval and distribution.

[0053] This step involves repeatedly executing all operations from steps S2 to S7 on each of the dataset fragments defined in step S1, following a predetermined batch order. This process aggregates and stores all dataset fragments, splitting the entire algorithm dataset into multiple independent virtual disk files, each corresponding to a dataset fragment. Ultimately, this achieves fragmented aggregated storage of the algorithm dataset, solving the problems of efficient storage, propagation, and retrieval of the algorithm dataset. The core of this step is ensuring the orderliness, stability, and completeness of the batch execution process, guaranteeing that all dataset fragments successfully complete the aggregated storage operation without omissions or errors. The final virtual disk files should be able to completely reconstruct the original algorithm dataset.

[0054] As previously mentioned, steps S9 to S13 are used to call the aggregated virtual disk file generated in steps S1 to S8. In step S9, the creation of the parent access directory must follow the operating system's file system rules and the principle of ease of dataset management, selecting appropriate storage paths and directory names. Storage paths should prioritize partitions with ample disk space and fast access speeds to avoid impacting the efficiency of subsequent dataset calls due to insufficient disk space or slow access speeds. The directory name must be clearly identifiable to directly reflect its function as a unified access point for the algorithm dataset. When creating the parent directory, reasonable directory permissions must be set to ensure that authorized users can access the directory, create subdirectories, mount virtual disks, etc., while restricting access permissions for unauthorized users to ensure dataset security.

[0055] After the parent directory is created, it needs to be configured, including setting the directory's caching strategy and access control lists to optimize access performance and security. Simultaneously, a configuration file can be created in the parent directory to record information such as the number of shards in the algorithm dataset, the name of each shard, the storage path of the corresponding virtual disk file, and the organization logic of the shards. This provides a reference for subsequent subdirectory creation, virtual disk mounting, and data access, achieving standardized dataset management. The configuration file can be in a common format such as JSON or XML, facilitating automated script reading and parsing, and supporting subsequent batch mounting operations.

[0056] In addition, the creation of the parent directory needs to be verified. This involves accessing the directory, creating temporary subdirectories, and modifying directory permissions to ensure the parent directory is functional and free from permission issues or file system abnormalities. The creation of the parent directory is fundamental to the overall access to the algorithm dataset; its stability and appropriateness directly affect the mounting and access of all subsequent shards. Therefore, it is crucial to strictly adhere to standard procedures to ensure the created parent directory meets the requirements for subsequent hierarchical mounting and overall access.

[0057] This step involves creating a unified parent access directory within the operating system's file system. This directory serves as the hierarchical mounting basis for all subsequent sharded virtual disks, providing a unified access point for the algorithm dataset and resolving the issues of data fragmentation, difficulty in unified management, and invocation after sharded storage. This parent directory will become the sole entry point for users to access the entire algorithm dataset. All sharded virtual disks will be integrated under this directory through subdirectories, forming a logically complete dataset structure.

[0058] In step S10, the creation of subdirectory mount points must strictly follow the sharding organization logic of the dataset to ensure that the naming and hierarchical relationship of subdirectories accurately reflect the logical attributes of the corresponding shards, facilitating user understanding and access. It is important to note that the naming of subdirectory mount points must be consistent with the name of the corresponding dataset shard to avoid mounting errors or access inconvenience caused by naming confusion.

[0059] The number of subdirectory mount points created must exactly match the number of dataset shards. Each shard corresponds to a unique subdirectory mount point; multiple shards sharing a single subdirectory mount point or one shard corresponding to multiple subdirectory mount points is not allowed. During creation, independent directory permissions must be set for each subdirectory mount point. Permission settings must match the usage requirements of the corresponding shard's virtual disk. For example, for shards that only require reading, the subdirectory mount point can be set to read-only permissions; for shards that require modification, read-write permissions can be set. This fine-grained permission management ensures the security of the dataset.

[0060] After creation, all subdirectory mount points need to be checked for integrity and consistency. This includes verifying that the number of subdirectories matches the number of shards, that subdirectory names match shard names, and that directory permissions are set appropriately. The usability of subdirectories is also verified by accessing them and creating test files. An automated script can be used to traverse the parent directories, generating a list of subdirectory mount points, which is then compared with the dataset shard list to ensure no omissions or errors. The creation of subdirectory mount points provides the foundation for accurate mounting of subsequent sharded virtual disks. The rationality and accuracy of their organization directly affect the user's access experience to the algorithm dataset; therefore, it is essential to strictly adhere to the shard organization logic to ensure that each subdirectory mount point accurately corresponds to each dataset shard.

[0061] The core of this step is to create corresponding subdirectory mount points for each dataset fragment under the unified parent access directory created in step S9, based on the number of fragments and the organization logic of the algorithm dataset. Each subdirectory mount point corresponds one-to-one with a virtual disk file of a dataset fragment. The hierarchical structure of the subdirectories reflects the fragment organization logic of the dataset, providing a precise directory entry for mounting the fragment virtual disks in the future.

[0062] In step S11, the mounting operation can be performed using a batch automated mounting method. Based on the configuration file created in step S9, an automated script is written to read the storage path of each virtual disk segment, the path of the corresponding subdirectory mount point, and the mount permission requirements, and executes the mount commands in batches. During script execution, the virtual disk files of each segment are mounted to the corresponding subdirectory mount points in a preset order. The mount command must explicitly specify the virtual disk file path and the subdirectory mount point path, and specify the corresponding read and write permissions (read-only or read-write) for each mount operation according to the permission settings in the configuration file, ensuring that the mount permissions match the usage requirements of the dataset.

[0063] During batch mounting, a real-time monitoring and anomaly handling mechanism for the mounting status must be set up. The mounting progress and status of each virtual disk shard should be monitored in real time. A mounting log should be output via a script, displaying information such as the currently mounted shard name, mounting command, and execution result. If the mounting operation of a shard fails (e.g., corrupted virtual disk file, non-existent subdirectory mount point, insufficient permissions), the script should immediately pause the mounting of that shard, record the cause of the error, and issue a warning. After the operator has investigated and resolved the issue, the shard can be remounted independently without affecting the mounting process of other shards.

[0064] After mounting is complete, a full verification of the mount status of all shards is required. Using the operating system's mount query command, check whether each shard's virtual disk has been successfully mounted to the specified subdirectory mount point and whether the mount permissions are correct. Simultaneously, access each subdirectory mount point to view its file and directory structure, verifying consistency with the original dataset shards and ensuring that the mounted shard data can be accessed normally. For shards that failed to mount or have abnormal status, troubleshoot and resolve the issues one by one until all shard virtual disks are successfully and correctly mounted to their corresponding subdirectory mount points.

[0065] Furthermore, batch mounting operations support flexible mounting strategies, allowing users to choose between temporary or permanent mounting based on actual usage needs. Temporary mounting is suitable for short-term use cases, and the mounting relationship automatically expires after a system restart. Permanent mounting is suitable for long-term use cases; by modifying the operating system's mounting configuration file, the mounting operation is automatically executed after a system restart, requiring no manual intervention. Users can choose the appropriate mounting strategy according to their actual needs, improving the convenience of data set retrieval.

[0066] This step involves mounting all dataset shards to their corresponding subdirectory mount points in batches, according to the correspondence between subdirectory mount points and sharded virtual disks. This hierarchical mounting achieves logical integration of all sharded data, enabling users to access the entire algorithm dataset from a unified parent directory and resolving the problem of data fragmentation after sharded storage.

[0067] In step S12, users do not need to access each subdirectory mount point individually. Instead, they can achieve logical integrated access to all fragmented data through the parent directory. The subdirectory mount points under the parent directory are arranged according to the organization logic of the dataset, forming a logical structure consistent with the original algorithm dataset. Users can access the corresponding subdirectory mount points layer by layer from the parent directory in a conventional file access manner to read or modify the required data.

[0068] For data reading operations, users can directly read data from parent or subdirectory mount points using tools such as algorithm programs and file browsers. The virtual disk mounting operation is completed in milliseconds, with no additional waiting time, and the data reading speed is comparable to accessing a physical disk, meeting the high-efficiency data requirements of scenarios such as algorithm training and data analysis. For data modification operations, if the subdirectory mount point is set to read-write permissions, users can directly modify file content, add new files, or delete unnecessary files within the corresponding subdirectory mount point. The modified content will be synchronized to the corresponding virtual disk file in real time, without additional synchronization operations, ensuring the timeliness and accuracy of data modifications.

[0069] During access, the operating system automatically handles I / O operations for virtual disk files corresponding to different subdirectory mount points. Users do not need to concern themselves with the underlying virtual disk management details; all operations are consistent with accessing a regular file system. Furthermore, because each virtual disk shard is independent, access to a single shard will not affect the performance of other shards. Even if a shard has a large amount of data or a high access frequency, it will not cause a decrease in the access speed of the entire dataset, ensuring the stability and efficiency of the access.

[0070] Furthermore, it supports multi-user parallel access scenarios. In a local area network or distributed environment, multiple authorized users can simultaneously access the algorithm dataset through the parent directory. Each user's access operations are independent, and data consistency is ensured through the operating system's file locking mechanism, avoiding data conflicts caused by multiple users modifying the same file simultaneously. Users can also mount only some frequently used dataset shards according to actual needs. Unmounted shards do not consume system resources, further improving system operating efficiency.

[0071] The core of this step is that after all the sharded virtual disks are successfully mounted to the corresponding subdirectory mount points, users can access the unified parent directory created in step S9 and perform overall access, data reading, or modification operations on the entire algorithm dataset according to the sharding organization logic of the dataset, enjoying the efficient and convenient experience brought by aggregated storage without having to worry about the underlying sharding storage details.

[0072] In step S13, the unloading operation must be performed in the reverse order of the mounting operation; that is, the last mounted virtual disk shard is unloaded first, and the first mounted one is unloaded last. This ordered unloading avoids access errors caused by unloading dependent shards first. The unloading operation can be performed in batches by an automated script. The script reads the configuration file from step S9 to obtain the mounting information of all virtual disk shards, and executes the unloading commands sequentially according to the preset unloading order. Each shard's unloading command must specify the corresponding subdirectory mount point path to ensure accurate unloading of the corresponding virtual disk file.

[0073] During the uninstallation process, an uninstallation status monitoring and data synchronization verification mechanism must be set up. Before executing the uninstallation command, the script checks whether there are any unclosed file handles or ongoing file read / write operations at the corresponding subdirectory mount point. If so, the uninstallation operation is paused and executed only after the file handles are released or the read / write operations are completed, to avoid data loss or file system corruption caused by forced uninstallation. After the uninstallation command is executed, the virtual disk is verified to have been successfully uninstalled using the operating system's mount query command, ensuring that the subdirectory mount point is restored to a normal empty directory with no residual mount associations.

[0074] For sharded virtual disks with read-write permissions, a data synchronization operation must be automatically performed before unloading. This synchronizes data in memory that has not yet been written to the virtual disk file to the disk, ensuring that all modifications are fully saved and preventing data loss. After unloading, the script records the unloading time and status of each sharded virtual disk, generating an unloading log for subsequent troubleshooting and system maintenance.

[0075] After all sharded virtual disks have been unloaded, the parent and subdirectory mount points need to be cleaned up. You can choose to keep or delete the subdirectory mount points based on your needs. If the dataset will be used frequently in the future, you can keep the subdirectory mount points and directly mount them the next time you need them. If it will not be used for a long time, you can delete the subdirectory mount points and the parent directory to free up storage space. Finally, use the operating system's resource monitoring tools to check the system's CPU, memory, disk I / O, and other resource usage to ensure that the unloading operation has completely released the relevant resources and the system has returned to normal operation.

[0076] This step involves unmounting all the fragmented virtual disks mounted on the subdirectory mount points in sequence, following the safety specifications, after the algorithm dataset has been used. This releases the operating system's mount resources, file handles, and system cache, preventing problems such as excessive system resource consumption and abnormal data synchronization caused by long-term mounting, and ensuring system stability and the integrity of virtual disk data.

[0077] Second Embodiment Based on the above method, a second embodiment of the present invention provides an aggregated storage and retrieval device for algorithm datasets. For example... Figure 2 As shown, the aggregated store-and-call device includes one or more processors and a memory. The memory stores one or more programs, which, when executed by the processor, implement the aggregated store-and-call method in the above embodiments.

[0078] The processor controls the overall operation of the aggregated memory access device to complete all or part of the steps of the aggregated memory access method. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the aggregated memory access device. This data may include, for example, instructions for any application or method operating on the aggregated memory access device, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0079] In one exemplary embodiment, the aggregated store-and-call device may be implemented by a computer chip or physical entity, or by a product with certain functions, to perform the above-described method and achieve the same technical effect as the described method. A typical embodiment is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interface device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0080] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions that, when executed by a processor, implement the steps of the method in any of the above embodiments. For example, the computer-readable storage medium may be the memory including the program instructions described above, which can be executed by a processor to complete the above method and achieve the same technical effects as the above method.

[0081] It should be noted that the above embodiments are merely illustrative examples. The technical solutions of each embodiment can be combined, and all are within the protection scope of this invention.

[0082] The above provides a detailed description of the method and apparatus for aggregating, storing, and retrieving algorithm datasets provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. An aggregated storage invocation method of an algorithm data set, characterized by Includes the following steps: Divide the algorithm dataset into one or more dataset fragments; For each dataset shard, calculate its space usage and total number of files; Based on the space usage and the total number of files, a corresponding virtual disk file is created, wherein the size of the virtual disk file at least meets the storage requirements of the space usage, and the number of its inodes at least meets the indexing requirements of the total number of files; Format the virtual disk file into the target file system; Create a mount point and mount the formatted virtual disk file to that mount point; The corresponding dataset shards are migrated to the virtual disk under the mount point; After the data migration is complete, the virtual disk file is unloaded to obtain an aggregated virtual disk file containing the sharded data of the dataset.

2. The method of claim 1, wherein, When dividing the algorithm dataset into one or more dataset shards, logical sharding is performed according to at least one of the following: business logic, data type, data collection time, algorithm application scenario, or data annotation type. Each dataset shard is an independent directory unit, and the data capacity within the shard is not required to be evenly distributed.

3. The method of claim 1, wherein, When calculating the space usage and total number of files for each dataset segment, all subdirectories and files of each dataset segment are traversed separately, the space usage is accumulated, the total number of files is counted one by one, and standardized statistical information including segment name, storage path, space usage, and total number of files is generated.

4. The aggregated storage and retrieval method as described in claim 1, characterized in that, The number of index nodes is configured according to the total number of files, so that the index node resources are not exhausted when the virtual disk space is not full.

5. The aggregated storage and retrieval method as described in claim 1, characterized in that, The target file system is a journaling file system, including but not limited to ext4, XFS, or Btrfs.

6. The aggregated storage and retrieval method as described in claim 1, characterized in that, The mounting operation can be set to read-only or read-write permissions according to usage requirements.

7. The aggregated store-and-call method as described in claim 1, characterized in that... It also includes the following steps: Create a parent directory for unified access; Under the parent directory, create a corresponding subdirectory mount point for each aggregated virtual disk file; Mount each aggregated virtual disk file to its corresponding subdirectory mount point; By accessing the parent directory and its subdirectories' mount points, data in the algorithm dataset can be read or modified. After use, uninstall all mounted aggregated virtual disk files.

8. The aggregated storage and retrieval method as described in claim 7, characterized in that, When mounting each aggregated virtual disk file to the corresponding subdirectory mount point, the mounting methods include temporary mounting and permanent mounting; Temporary mounts automatically expire after a system restart, while permanent mounts are achieved by modifying the operating system configuration file to automatically mount after a system restart. If a certain aggregated virtual disk file fails to mount, the mounting of that segment will be paused and an alert will be issued. Once the problem is resolved, the segment will be remounted individually.

9. The aggregated storage and retrieval method as described in claim 7, characterized in that, When mounting each aggregated virtual disk file to the corresponding subdirectory mount point, for dataset fragments that do not need to be modified, they are mounted with read-only permissions; For dataset shards that need to be modified, mount them with read and write permissions.

10. A device for aggregating, storing, and retrieving algorithm datasets, characterized in that... It includes a processor and a memory, the memory being coupled to the processor for storing a computer program, which, when executed by the processor, implements the aggregated store-call method according to any one of claims 1 to 9.

Images