Big model training resource optimization method and system based on hadoop ecosystem

Abstract

The application relates to the field of model training, and in particular to a large model training resource optimization method and system based on a Hadoop ecology. The method comprises the following steps: dividing data heat levels based on access features of to-be-processed files; combining content association features to configure corresponding copy storage strategies or erasure code storage strategies for to-be-processed files of different heat levels and content combinations, and then performing hybrid storage on the to-be-processed files; dividing the to-be-processed files into multiple small file clusters according to the content association features, and merging the small file clusters to obtain large files and index files; solving a resource layout with optimal communication cost in resource allocation based on the access features and the data heat levels; converting the hybrid storage to-be-processed files processed by Spark into Arrow tables; and calling GPU resources to reuse the large files and the index files to complete data type adaptation, storage optimization and preprocessing adaptation, and to complete zero-copy conversion from the Arrow tables to PyTorch tensors.

Description

Technical Field

[0001] This application relates to the field of model training, and in particular to a method and system for optimizing large model training resources based on the Hadoop ecosystem. Background Technology

[0002] The Hadoop ecosystem is a technical system centered on distributed storage (Hadoop, HDFS), resource scheduling (Yet Another Resource Negotiator, YARN), and distributed computing (Spark), integrating functions such as data preprocessing, storage management, and resource allocation. It is the underlying foundation supporting large-scale data processing and large model training.

[0003] In the current field of large-scale model training, the Hadoop ecosystem, with its core capabilities of distributed storage and computing, has become a crucial technological foundation supporting large-scale training tasks. However, existing large-scale model training processes based on the Hadoop ecosystem suffer from high storage costs, low GPU utilization, and significant preprocessing latency, severely limiting the training efficiency and resource utilization of large models. Therefore, there is an urgent need to propose a novel technical solution to address at least one of the technical problems in existing technologies. Summary of the Invention

[0004] This application addresses the technical problems existing in the prior art by providing a method and system for optimizing large model training resources based on the Hadoop ecosystem, thereby solving at least one of the aforementioned technical problems.

[0005] In a first aspect, embodiments of this application provide a method for optimizing large model training resources based on the Hadoop ecosystem, the method comprising:

[0006] Collect access characteristics and content association characteristics of the files to be processed; the files to be processed include multiple small files.

[0007] Based on the access characteristics, data popularity levels are divided into hot data, warm data, and cold data according to access frequency; combined with the content association characteristics, corresponding copy storage strategies or erasure coding storage strategies are configured for files to be processed with different popularity levels and different content combinations, and the files to be processed are stored in a mixed manner.

[0008] Based on the content association features, the file to be processed is divided into multiple small file clusters and a merging operation is performed to obtain a merged large file and an index file. The index file is used to record the location information of each small file in the merged large file.

[0009] Solve the resource layout with optimal communication cost in resource allocation based on the access feature and data heat level, optimize GPU resource allocation in the distributed training process of the large model;

[0010] Convert the mixed storage file to be processed after Spark processing into an Arrow table; call GPU resources through the resource layout, reuse the merged large file and the index file to complete data type adaptation, storage optimization and preprocessing adaptation, and complete zero-copy conversion from the Arrow table to the PyTorch tensor based on the Arrow memory direct mapping mechanism.

[0011] In a second aspect, the embodiments of the present application provide a large model training resource optimization system based on a Hadoop ecosystem, which at least includes the following units:

[0012] The mixed storage unit is configured to collect access features and content association features of the file to be processed; the file to be processed includes a plurality of small files; data heat levels are divided based on the access features, and the data heat levels are divided into hot data, warm data and cold data according to access frequencies; the file to be processed is mixed stored by configuring corresponding replica storage strategies or erasure code storage strategies for the file to be processed with different heat levels and different content combinations in combination with the content association features; the file to be processed is divided into a plurality of small file clusters and a merging operation is performed according to the content association features, to obtain a merged large file and an index file, and the index file is used to record positioning information of each small file in the merged large file;

[0013] The scheduling unit is configured to solve the resource layout with optimal communication cost in resource allocation based on the access feature and data heat level, to optimize GPU resource allocation in the distributed training process of the large model.

[0014] The acceleration unit is configured to convert the mixed storage file to be processed after Spark processing into an Arrow table; call GPU resources through the resource layout, reuse the merged large file and the index file to complete data type adaptation, storage optimization and preprocessing adaptation, and complete zero-copy conversion from the Arrow table to the PyTorch tensor based on the Arrow memory direct mapping mechanism.

[0015] In a third aspect, the embodiments of the present application provide an electronic device, which includes at least one processor, a memory and an input-output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the large model training resource optimization method based on the Hadoop ecosystem of the first aspect.

[0016] In a fourth aspect, a computer-readable storage medium is provided, which includes instructions that, when executed on a computer, cause the computer to perform the Hadoop-ecological-based large model training resource optimization method of the first aspect.

[0017] The beneficial effects of the present application are: a Hadoop-ecological-based large model training resource optimization method and system are provided. In the technical solution, first, the access features and content association features of the to-be-processed files are collected; the to-be-processed files include a plurality of small files. Further, the data heat levels are divided based on the access features, and the data heat levels are divided into hot data, warm data and cold data according to the access frequency; in combination with the content association features, the corresponding replica storage strategy or erasure code storage strategy is configured for the to-be-processed files of different heat levels and different content combinations, and the to-be-processed files are mixedly stored. Then, the to-be-processed files are divided into a plurality of small file clusters and a merging operation is performed according to the content association features, to obtain merged large files and index files, the index files being used to record the positioning information of each small file in the merged large files. Next, based on the access features and the data heat levels, the resource layout with the optimal communication cost is solved in resource allocation, and the GPU resource allocation in the large model distributed training process is optimized. Finally, the to-be-processed files mixedly stored after Spark processing are converted into Arrow tables; the GPU resources are called through the resource layout, the merged large files and the index files are reused to complete data type adaptation, storage optimization and preprocessing adaptation, and the zero-copy conversion from the Arrow table to the PyTorch tensor is completed based on the Arrow memory direct mapping mechanism. The technical solution of the present application improves the running efficiency and stability of the whole process of large model training through the collaborative mode of mixed storage optimization, resource scheduling and preprocessing acceleration, solves the technical problems of high storage cost, low GPU utilization and large preprocessing delay of large models under the Hadoop ecology, and helps to improve the training efficiency and resource utilization level of large models. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a process schematic diagram of a Hadoop-ecological-based large model training resource optimization method according to an embodiment of the present application;

[0019] Figure 2 is a structure schematic diagram of a Hadoop-ecological-based large model training resource optimization system according to an embodiment of the present application;

[0020] Figure 3 is a structure schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0022] In the existing large model training process based on the Hadoop ecosystem, the following technical problems seriously restrict the training efficiency and resource utilization level: first, in terms of storage efficiency, HDFS is responsible for the reliable storage of massive data and is the core carrier of parameter files, data sets, logs and other data in large model training. The current storage strategy of HDFS will generate high additional storage overhead, and in the large-scale training process, many data shards will be generated, resulting in a significant increase in memory occupancy, and the storage management cost and metadata maintenance pressure also increase synchronously. Second, in terms of GPU resource utilization, YARN is responsible for the allocation and scheduling of CPU, GPU and other hardware resources in the cluster, and determines the resource supply efficiency of the training task. YARN Capacity Scheduler lacks native support for GPU resources, and the static queue configuration method easily causes a high resource fragmentation rate, so that the average waiting time of large model training tasks accounts for a high proportion of the total cycle, and a large amount of GPU resources are idle, resulting in low GPU utilization. Third, in the data preprocessing link, Spark, as the mainstream computing framework in the ecosystem, is responsible for data set cleaning, feature extraction and other preprocessing tasks before large model training, and the three work together to form the underlying architecture of large model training. There is a lot of serialization overhead between the data interfaces of Spark and PyTorch. For example, the time-consuming proportion of converting ImageNet data set from Spark DataFrame to PyTorch Tensor accounts for up to 38% of the total training time, which becomes a key bottleneck restricting the end-to-end training efficiency and greatly increases the preprocessing delay.

[0023] Therefore, it is urgent to propose a new technical solution to solve the technical problems of high storage cost, low GPU utilization and large preprocessing delay in the large model training process under the Hadoop ecosystem.

[0024] In the technical solution of the present application, in view of the technical problems of high storage cost, low GPU utilization and large preprocessing delay in the existing Hadoop ecological large model training, the technical solution of the present application forms a systematic solution through mutual cooperation of storage optimization, resource scheduling and preprocessing acceleration, as follows: First, in view of the problem of high HDFS storage overhead and large memory and metadata pressure caused by multiple small file fragments, the scheme first collects the access features and content association features of the to-be-processed files, divides the hot, warm and cold data heat levels based on the access features, and configures a replica or erasure code hybrid storage strategy for different combinations of data according to the content association features. Hot data is stored by replica to ensure high-frequency access efficiency, cold and warm data is stored by erasure code to reduce additional storage overhead, and at the same time, small files are clustered and merged into large files according to the content association features and index files are generated to record positioning information. This greatly reduces the number of data fragments, reduces memory usage and metadata maintenance pressure, and reduces storage management cost from the aspects of storage strategy and file form.

[0025] Second, in view of the problem of insufficient support for YARN scheduling GPU resources and high resource fragmentation rate leading to GPU idling, the scheme solves the resource layout with optimal communication cost in the resource allocation process based on the access features and data heat levels collected in the early stage. High-value training tasks (hot data associated tasks) are identified by data heat levels, and GPU resources are allocated to them preferentially. At the same time, the resource allocation is optimized in combination with the GPU node topology information to reduce resource fragmentation caused by static queue configuration, avoid the extension of the cycle of high-value tasks due to resource waiting, improve the matching degree of GPU resources and training tasks, and reduce the GPU idling rate.

[0026] Finally, in view of the problem of large preprocessing delay caused by large serialization overhead of Spark and PyTorch data interfaces, the scheme converts the mixed storage data processed by Spark into Arrow table format, uses the out-of-heap storage and columnar access characteristics of Arrow table to avoid JVM memory limitations, and realizes zero-copy conversion to PyTorch tensor through the Arrow memory direct mapping mechanism, bypassing the multi-step serialization and data replication process. At the same time, the large files and index files after merging are reused to reduce IO loss in the data reading process, and the topology of the preprocessing node and the GPU node is adapted in combination with the resource layout to further reduce the data transmission cost, thereby greatly reducing the data conversion time from Spark to PyTorch and breaking through the preprocessing delay bottleneck. In summary, the technical solution of the present application realizes the cooperative optimization of storage, resources and preprocessing in the whole process through precise correspondence with the technical problems of the prior art, and systematically solves the core problems that restrict the efficiency and resource utilization level of large model training.

[0027] The big model training resource optimization scheme based on the Hadoop ecology provided by the embodiments of the present application can also be executed by an electronic device, which can be a server, a server cluster, a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a special-purpose device (such as a special-purpose terminal device with a big model training resource optimization method based on the Hadoop ecology). These electronic devices can also be equipped with the chips or other hardware processing units introduced in the above embodiments. Alternatively, these electronic devices can also install a service program for executing the big model training resource optimization scheme based on the Hadoop ecology.

[0028] Figure 1 A flowchart of a big model training resource optimization method based on the Hadoop ecology provided by the embodiments of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0029] 101, collecting access features and content association features of a to-be-processed file;

[0030] 102, dividing data hotness levels based on the access features, the data hotness levels being divided into hot data, warm data, and cold data according to access frequencies;

[0031] 103, combining the content association features, configuring corresponding replica storage strategies or erasure code storage strategies for to-be-processed files of different hotness levels and different content combinations, and performing hybrid storage on the to-be-processed files;

[0032] 104, dividing the to-be-processed files into multiple small file clusters according to the content association features and performing a merging operation to obtain merged large files and index files, the index files being used to record positioning information of the small files in the merged large files;

[0033] 105, based on the access features and the data hotness levels, solving a resource layout with optimal communication cost in resource allocation to optimize GPU resource allocation in a big model distributed training process;

[0034] 106, converting the to-be-processed file stored in a hybrid manner after Spark processing into an Arrow table;

[0035] 107, invoking GPU resources through the resource layout, reusing the merged large files and the index files to complete data type adaptation, storage optimization, and preprocessing adaptation, and completing zero-copy conversion from the Arrow table to a PyTorch tensor based on an Arrow memory direct mapping mechanism.

[0036] ​In the embodiments of the present application, the to-be-processed file includes a plurality of small files. The small files are various types of fine-grained data files generated in the data storage, model running, and training monitoring in the large model training process, and the scope thereof closely matches the actual needs of large model training in the Hadoop ecology.

[0037] In terms of file types, these small files mainly include the following: first, training dataset related files, including fine-grained shards of original datasets (such as single picture files of ImageNet dataset split by category, single text sample files in NLP tasks), and intermediate feature files after dataset cleaning (such as single batch feature shards after normalization and tokenize processing). Second, model parameter and training state files, including incremental parameter files generated in the large model training process according to iteration steps (such as model layer parameter shards saved every 100 steps), checkpoint breakpoint training files (weight shards and optimizer state shards split by training stage). Third, training log and monitoring files, including real-time training log shards of each GPU node (split by time granularity such as every minute), and training index monitoring files (such as separate record files of each batch loss value and accuracy). Fourth, auxiliary configuration files, including dataset index shard files, single node configuration files of model training tasks, and node communication configuration shards of distributed training.

[0038] In terms of production scenarios, the large number of small files is generated due to the distributed characteristics and data processing needs of large model training. In the data preprocessing stage, when Spark performs parallel processing on large-scale datasets, the data is split into multiple small file shards according to the task parallelism (such as small files generated by Spark RDD partitioning) to adapt to the task scheduling logic of the distributed computing framework. In the model training stage, the distributed training framework (such as Horovod, DeepSpeed) splits the model parameters according to the number of GPU nodes, and each node only saves part of the parameter shards, while the checkpoint shards are saved in increments according to a fixed step size to avoid the storage and transmission pressure caused by large full parameter files. In the log and monitoring stage, to avoid read-write conflicts and low query efficiency caused by a single large log file, the training system splits the log into small files according to the node, time, or task dimension, facilitating real-time viewing and subsequent analysis.

[0039] From the core features, these small files have the characteristics of large quantity, small single file size, strong correlation and large access frequency difference. In terms of quantity, the number of small files to be processed in super large scale model training (such as trillion parameter model) can reach millions or even tens of millions, covering various files such as data set slices, parameter slices, and log slices. In terms of single file size, most small files are concentrated in the range of KB to MB (such as a single picture file of 100KB-5MB, a single batch feature file of 5MB-50MB), which is much smaller than the default block size of HDFS (such as 128MB-256MB). In terms of correlation, some small files have explicit content correlation (such as parameter slices of the same model layer and data set slices of the same batch), which need to be used together to complete the training task. In terms of access frequency, high-frequency accessed small files (such as data set slices of the current training batch and the latest parameter slice) and low-frequency accessed small files (such as early training logs and historical checkpoint slices) coexist, forming obvious access heat difference. These characteristics are also the core basis for the subsequent clustering and merging and mixed storage optimization of small files in this application. Through the optimization of the form and access characteristics of small files, the problems of low storage efficiency and large IO loss caused by small files are solved, supporting the efficient operation of large model training.

[0040] 101, collect access features and content association features of the to-be-processed files.

[0041] 102, divide data heat levels based on the access features.

[0042] 103, in combination with the content association features, configure corresponding replica storage strategies or erasure code storage strategies for to-be-processed files with different heat levels and different content combinations, and perform mixed storage on the to-be-processed files.

[0043] As an optional embodiment, step 101 mainly focuses on the collection of key features of to-be-processed files to provide data support for subsequent storage decisions. When collecting access features, relying on the audit log or NameNode access log of HDFS under the Hadoop ecosystem, the complete access record of the file within a preset period is obtained, and the access features reflecting the frequency of file access are obtained through statistical analysis. When collecting content association features, the content feature vector is extracted from the to-be-processed files (including model parameter files, training sample files, checkpoint slice files, and other large model training related files), and the similarity between different file feature vectors is calculated to obtain the content association features reflecting the closeness of file content association, ensuring that the subsequent data classification and storage strategy matching has objective basis. In practical applications, the to-be-processed files include model parameter files, training sample files, checkpoint slice files, and other large model training related files.

[0044] In step 102, the access frequency of the file in the preset period is taken as the core judgment index, two key parameters of high frequency threshold and low frequency threshold are set, and then the relationship between the actual access frequency of the file and the two threshold values is compared to complete the classification of the heat level. That is, the file with an access frequency higher than the high frequency threshold is classified as hot data, and such data is usually the key data frequently used in the current training process. The file with an access frequency between the high frequency threshold and the low frequency threshold is classified as warm data, and such data has a medium frequency of use and needs to balance the access efficiency and the storage cost. The file with an access frequency lower than the low frequency threshold is classified as cold data, and such data is mostly historical training data or backup data, and has a low requirement for access efficiency. The hierarchical mode lays a foundation for the development of subsequent differentiated storage strategies.

[0045] In step 103, the content association features obtained in 101 are combined to configure differentiated storage strategies for different combinations of data, and hybrid storage optimization is realized. For hot data, regardless of the content association features, a replica storage strategy is configured, the data read speed and reliability in the high-frequency access scenario are guaranteed through multiple replica redundancy, and the real-time access demand of the key data of the large model training is met. For warm data, if the content association features show that the files are closely associated (i.e., high similarity), an adaptive erasure code storage strategy is configured to control the storage cost while guaranteeing the cooperative access efficiency of the associated data, and if the content association features show that the files are loosely associated, an erasure code storage strategy that focuses more on cost control is selected. For cold data, a high fault tolerance and low storage cost erasure code storage strategy is uniformly configured, and through this hybrid storage mode, the access demand of data with different heat and different association degrees is met, and the overall storage overhead and metadata maintenance pressure are greatly reduced, which adapts to the storage demand of large model training massive data.

[0046] 104, according to the content association features, the to-be-processed file is divided into a plurality of small file clusters and a merging operation is performed to obtain a merged large file and an index file. In the embodiment of the application, the index file is used to record the positioning information of each small file in the merged large file.

[0047] Specifically, step 104 is a file form optimization link developed on the basis of the early feature collection and storage strategy configuration to solve the problem of large number of small files and large random IO loss in large model training. The core is to realize efficient management of small files through content association driven clustering and merging. In this step, first, the content association features collected in step 101 are used as the core basis to construct the association matrix of the files to be processed, and the matrix elements directly reflect the content association tightness between different small files. Then, a clustering algorithm is used to group all small files, ensuring that small files in the same cluster have high content association. These small files are mostly data that need to be used cooperatively in large model training (such as parameter fragments of the same model layer, training sample fragments of the same batch), avoiding the influence of unrelated small files on subsequent data access efficiency after merging. After clustering is completed, merging operation is performed on small files in each cluster to generate a unified large file. The size of the large file is adapted to the default block size of HDFS in the Hadoop ecosystem and the characteristics of the storage medium, which can reduce the metadata management pressure caused by the large number of small files, and convert multiple random IOs into a single continuous IO, improving the subsequent data reading speed.

[0048] At the same time of generating the merged large file, an index file is created synchronously, which is a key carrier to ensure accurate access to the merged small files. The positioning information recorded in the index file needs to completely cover the core position data of each original small file in the merged large file, ensuring that when accessing any original small file subsequently, the target data can be quickly located without loading the entire large file. The index file will be updated synchronously with subsequent data migration and storage strategy adjustment, always maintaining consistency with the small file positioning information in the large file, providing a basis for resource layout optimization and data preprocessing adaptation (such as super large scale data set block loading) in step 105, ensuring that after merging small files, both storage and access efficiency are improved, and the independent access ability of original small files is not lost.

[0049] As an optional embodiment, after the small files are divided into multiple clusters and the merging operation is performed to obtain the merged large file and the corresponding index file in step 104, a migration process can be started based on the condition of periodic triggering or change of the access feature. In the migration process, a mechanism is used to ensure data access continuity, and dynamic migration of hot and cold data is realized by combining data heat levels, updating the positioning information and metadata pointers of each small file in the index file, and dynamically switching the storage strategy of each small file.

[0050] Specifically, after completing the small file cluster merging, generating the merged large file and the index file in step 104, to cope with the dynamic changes in data access frequency during large model training and maintain the continuous adaptation of storage efficiency and access demand, a hot and cold data dynamic migration process can be further performed. The start of this process depends on two trigger conditions: one is periodic triggering, that is, starting the migration check according to the preset fixed period to ensure that the data storage strategy is adjusted in time as the training period advances. The other is access feature change triggering, which automatically starts the migration process when it is monitored that the access frequency of a certain type of file breaks through the threshold for dividing the hotness level in step 102 (such as the cold data access frequency continuously being higher than the low frequency threshold, and the hot data access frequency continuously being lower than the high frequency threshold), avoiding the waste of storage resources or access delay caused by changes in data hotness.

[0051] During the migration process, it is necessary to focus on ensuring the continuity of data access to avoid interrupting the normal data reading of large model training. Specifically, the migration operation will follow the logic of "write first, delete later, and metadata delayed update". If hot data needs to be migrated to cold data, a new file copy will be generated in the target storage location according to the corresponding erasure coding storage strategy of cold data. After the new file is completely written and verified to be error-free, the metadata pointing of HDFS NameNode is updated. During this period, all read requests for this data still point to the storage location of the original hot data. If cold data needs to be migrated to hot data, a new redundant copy is created according to the corresponding replica storage strategy of hot data. After ensuring that the new copy meets the reliability requirements of high-frequency access, the metadata update switching is completed to avoid data reading failure or delay during the migration process.

[0052] The migration operation will closely combine the data hotness level divided in step 102 to achieve dynamic switching of the storage strategy. When cold data is migrated to hot data, the storage strategy is switched from erasure coding storage to replica storage to improve the speed and reliability of high-frequency access. When hot data is migrated to cold data, the storage strategy is switched from replica storage to erasure coding storage to reduce the storage overhead of low-frequency data. At the same time, the index file generated in step 104 needs to be updated synchronously during the migration process. According to the change of data storage location, the positioning information of the corresponding small file in the index file is corrected to ensure that the data can be accurately positioned to the migrated storage location when accessed through the index file in the future. The metadata pointing is also updated synchronously after the migration is completed to ensure that the file management system in the Hadoop ecosystem can correctly identify the latest storage state of the data, and finally realize the dynamic matching of data hotness and storage strategy, taking into account the access efficiency and storage cost of large model training.

[0053] Further optionally, in one embodiment of the above step, a mechanism for ensuring data access continuity is adopted to realize dynamic migration of hot and cold data in combination with data heat level, including: when hot data is migrated to cold data, first write the hot data into a new file according to the corresponding erasure code storage strategy of cold data, then update the metadata of HDFS NameNode to point to the new file, and finally delete the original copy file of hot data; during the migration, all read requests still access the original copy file of hot data. When cold data is migrated to hot data, first create a new copy for cold data according to the corresponding copy storage strategy of hot data, update the NameNode metadata to point to the new copy after the new copy is completely generated; if there is a write request during the migration, use the copy-on-write mechanism to direct the write operation to the new copy, or temporarily block the write request until the migration is completed. The index file is updated synchronously during the migration process, the positioning information of the corresponding small file in the index file is corrected according to the change of the file storage location, and the file metadata pointing is updated, so that the storage strategy switching does not interrupt data access.

[0054] Specifically, in the process of realizing dynamic migration of hot and cold data in combination with data heat level, the mechanism for ensuring data access continuity realizes seamless connection of storage strategy switching and data access by designing differentiated processes for different migration directions and synchronously maintaining index consistency.

[0055] When hot data is migrated to cold data, the migration process follows the logic of orderly advancement of first storage, then switching, and finally cleaning up. First, according to the corresponding erasure code storage strategy of cold data, the hot data to be migrated is completely written into the target storage location to generate a new file, ensuring that the new file meets the format and fault tolerance requirements of low-cost storage. During this process, the metadata of HDFS NameNode still points to the original copy file of hot data, and all training-related read requests continue to access the original copy, avoiding access interruption or data errors caused by incomplete writing of the new file. After the new file is written and passes the integrity check, the metadata update operation is performed to switch the file pointing to the newly generated erasure code file, and finally the original hot data copy file is deleted to release redundant storage resources, and the entire process realizes continuous availability of read requests.

[0056] When cold data is migrated to hot data, a new redundant copy is first created for the cold data according to the copy storage strategy corresponding to the hot data, to ensure that the copy distribution meets the performance requirements of high-frequency access, and the pointing update of the NameNode metadata is started only after all new copies are completely generated and verified consistent. To address the write requests that may occur during migration, two mechanisms are used to ensure data consistency. When the copy-on-write mechanism is used, the write operation is directed to the newly created copy, and only the necessary data blocks are copied at the time of writing, avoiding version confusion caused by simultaneous modification of the source data and the new copy. If the write request is temporarily blocked, the blocking time needs to be controlled within a range that does not affect the training progress, and the write operation is resumed after the metadata switching is completed. Both methods can prevent data write conflicts.

[0057] During the migration process, the synchronization update of the index file is coordinated with the core migration steps. As the storage location of the file changes, the positioning information of the corresponding small file in the index file is corrected in real time to ensure that the offset, block location, and other data recorded in the index file accurately match the new storage address. At the same time, the file metadata pointing is also adjusted synchronously with the update of the NameNode, ensuring that the index file and the file management system of HDFS remain consistent. This linkage update mechanism of the index and metadata ensures that after the migration is completed, the training system can still quickly locate the target data through the index file, and there is no interruption in data access and no deviation in positioning during the switching process of the storage strategy from replication to erasure coding or vice versa.

[0058] 105, based on the access features and data hotness levels, solving the resource layout with optimal communication cost in resource allocation, optimizing GPU resource allocation in large model distributed training process.

[0059] As an optional embodiment, in step 105, the GPU node is configured with a dedicated resource tag to achieve physical isolation, an AI dedicated scheduling queue is divided, an initial priority of the AI dedicated scheduling queue is configured, and a range of accessible GPU nodes is configured; a priority calculation model is constructed based on GPU resource requirement characteristics and time constraint characteristics of the training tasks; a dynamic priority of each training task is generated through the priority calculation model, wherein the GPU resource requirement characteristics are used to reflect the demand size of each training task for GPU resources, and the time constraint characteristics are used to reflect the urgency of each training task; in the calculation process, the priority weight of a high-value training task associated with hot data is increased in combination with a data heat level; resource utilization and memory fragmentation rate of the GPU node are monitored in real time, when the resource utilization and the memory fragmentation rate jointly satisfy a preset preemption trigger condition, a low-value training task is selected and terminated based on a remaining life cycle evaluation result of the task to release the occupied GPU resources, the terminated low-value training task enters a retry queue and the dynamic priority after retrying is increased; topology information is reported on the configured GPU node; a GPU affinity matrix is constructed in combination with the topology information and the data heat level, and in resource allocation, a resource layout with optimal communication cost is solved through the GPU affinity matrix, and the adaptation degree between GPU resource allocation and hot data storage location of the distributed training is optimized.

[0060] Specifically, the GPU resource allocation optimization of large model distributed training in step 105 realizes the dual improvement of resource utilization and communication efficiency through multi-layer strategies such as basic isolation, dynamic scheduling, resource recycling, and topology adaptation, and connects the access features and data heat levels collected in the early stage. First, starting from the basic management of GPU resources, physical isolation is realized by configuring exclusive resource tags for GPU nodes, avoiding non-AI tasks from occupying core computing resources, and ensuring the exclusivity of GPU resources. On this basis, an AI exclusive scheduling queue is divided, and the initial priority of the queue and the accessible GPU node range are configured to ensure that large model training tasks can preferentially obtain adaptive hardware resources, reducing the delay caused by resource competition from the scheduling entrance. Subsequently, a priority calculation model is constructed to realize dynamic scheduling. The model takes the GPU resource demand characteristics and time constraint characteristics of the training task as the core input, the former reflects the demand size of the task for GPU resources, and the latter reflects the urgency of the task, and generates an initial priority through quantitative calculation. During the calculation process, weight adjustment is performed in combination with the data heat level, and the priority weight of high-value training tasks associated with hot data is increased, so that core training tasks can obtain resource support more quickly, adapting to the needs of high-frequency data access and task value difference in large model training. To solve the problem of resource fragmentation and idling, the system will monitor the resource utilization rate and memory fragmentation rate of the GPU node in real time. When these two indicators jointly meet the pre-set preemption trigger condition, the resource preemption process is started. By evaluating the remaining life cycle of the task, low-value training tasks are screened, and such tasks are terminated to release the occupied GPU resources. The terminated tasks will enter the retry queue, and the dynamic priority after retry will be correspondingly increased, which not only ensures the resource supply of high-value tasks, but also reduces the repeated loss of low-value tasks. Finally, relying on topology information and data heat, a resource adaptation mechanism is constructed. The configured GPU nodes will actively report topology information, including key data such as hardware ownership and interconnection relationship of GPU. Combined with these topology information and hot data storage location, the communication cost between GPUs and between GPU and data storage node is calculated, and a GPU affinity matrix is constructed. In the resource allocation process, the optimal resource layout with the lowest communication cost is solved through the matrix, and the GPU resources of the training task are allocated to the nodes that are adaptive to the hot data storage location, maximizing the reduction of data transmission and inter-node communication overhead, and realizing the precise matching of GPU resource allocation and data access demand.

[0061] Further optionally, in an optional embodiment of step 105, the strict physical isolation of the GPU nodes is first realized through tag configuration. A dedicated resource tag is configured for all GPU nodes, which is used to explicitly identify the GPU resource attributes of the nodes, and the core attribute of the tag is set to a specific value. Through this configuration, the permission boundary is drawn from the cluster resource scheduling level, ensuring that non-GPU tasks without access to the tag cannot occupy GPU node resources, avoiding the competition for hardware resources between ordinary computing tasks and large model training tasks, and ensuring the specificity and stability of GPU node resources. On this basis, the differential configuration of the scheduling queue is carried out, and a scheduling queue dedicated to large model training and related AI tasks is divided. The initial priority of this queue is set, and the highest priority of the ordinary task queue is limited. Through the hierarchical difference of priority, it is ensured that the tasks in the AI dedicated scheduling queue can obtain response priority when applying for resources, reducing the waiting time of core training tasks due to resource queuing, and improving the task startup efficiency.

[0062] In terms of resource allocation ratio, a fixed proportion of the total GPU resources of the cluster is allocated to the AI dedicated scheduling queue to ensure sufficient resource reserves for core AI tasks. At the same time, by configuring the limitation that this queue can only access GPU nodes with dedicated resource tags, closed-loop management of resource access is formed, avoiding the problem of mismatch between resource allocation and node attributes. In addition, a fixed proportion of elastic resources is reserved from the total GPU resources of the cluster. This part of resources is not directly allocated to the conventional allocation, but is specially used to cope with the sudden resource demand of the pre-storage optimization task or the subsequent data preprocessing task, ensuring the flexible and controllable resource supply of each link in the whole large model training process, and avoiding the whole process from being stuck due to local resource shortage.

[0063] Illustratively, the gpu-exclusive dedicated resource tag is configured for the GPU nodes, and the tag attribute is set to exclusive=true, which prohibits non-GPU tasks (such as ordinary CPU computing tasks) without the tag permission from occupying GPU node resources. The initial priority of the AI dedicated scheduling queue is set to 5, and the priority of the ordinary task queue is not more than 3, ensuring that the tasks in the AI dedicated scheduling queue can obtain resources in priority; 70% of the total GPU resources of the cluster is allocated to the AI dedicated scheduling queue, and it is limited that the queue can only access the GPU nodes configured with the gpu-exclusive tag, and 10% of the GPU resources of the cluster are reserved as elastic resources to cope with the sudden resource demand of the storage optimization task or the preprocessing task.

[0064] Further optionally, in an optional embodiment of step 105, when the resource utilization rate and the memory fragmentation rate jointly satisfy the preset preemption trigger condition, the low-value training tasks are screened and terminated based on the remaining life cycle evaluation result of the tasks to release the occupied GPU resources, including:

[0065] The resource utilization rate threshold and the memory fragmentation rate threshold of the GPU node are set, and when the actual resource utilization rate of the GPU node is lower than the resource utilization rate threshold and the actual memory fragmentation rate is higher than the memory fragmentation rate threshold, it is determined that the preset preemption trigger condition is met. Further, the current progress rate of the training task is calculated, the current progress rate and the historical average progress rate are weighted and fused by a smoothing algorithm, and the remaining step number of the task is divided by the fused progress rate to obtain the remaining life cycle of the task. The current progress rate of the training task is the quotient obtained by dividing the completed step number of the training task by the running time of the training task, and the running time of the training task is the difference obtained by subtracting the training task start time from the current time. Then, the training task with a remaining life cycle lower than the task remaining life cycle threshold is determined as a low-value training task; a termination process of the low-value training task is triggered to release the GPU resources occupied by the low-value training task, and the terminated low-value training task is included in a retry queue, and the dynamic priority of the low-value training task after retry is improved according to a preset rule.

[0066] It can be understood that in the resource monitoring and intelligent preemption implementation of step 105, the preemption trigger condition is first determined by setting double thresholds. The resource utilization rate threshold and the memory fragmentation rate threshold are set for the GPU node respectively, and the actual running indicators of the node are monitored in real time. Only when the actual resource utilization rate is lower than the set resource utilization rate threshold and the actual memory fragmentation rate is higher than the set memory fragmentation rate threshold, it is determined that the preset preemption trigger condition is met, avoiding false triggering caused by single indicator fluctuation and ensuring that resource preemption is started only when resource allocation needs to be optimized.

[0067] After the trigger condition is met, the calculation of the remaining life cycle of the task is entered. First, the current progress rate of the training task is calculated, which is obtained by dividing the completed step number of the training task by the running time of the task. The running time of the task is the difference between the current time and the task start time, which reflects the current advancement efficiency of the task. Then, the current progress rate and the historical average progress rate are weighted and fused by a smoothing algorithm to balance the influence of short-term fluctuations and long-term operation rules on rate evaluation. Finally, the remaining step number of the training task is divided by the fused progress rate to obtain the remaining life cycle of the task, which provides a quantitative basis for subsequent task value determination.

[0068] Finally, low-value training tasks are screened and processed based on the remaining life cycle. A task remaining life cycle threshold is set, and training tasks with a remaining life cycle lower than the threshold are determined as low-value training tasks, which usually have less impact on the overall training progress. The system then triggers the termination process of the low-value training tasks, releases the GPU resources occupied by the low-value training tasks, and supports the running of high-value tasks; at the same time, the terminated low-value training tasks are included in the retry queue, and the dynamic priority of these tasks after retry is improved according to the preset rules, which ensures that resources are tilted towards high-value tasks, avoids low-value tasks being long-term shelved due to frequent termination, and maintains the overall order of the training task queue.

[0069] Further optionally, in the process of calculating the task remaining life cycle and processing the retry of low-value tasks in step 105, the progress rate is first weighted and fused by an exponential smoothing method. In the calculation, the weight distribution of the current progress rate and the historical average progress rate is first determined, the weight of the current progress rate is set to a higher proportion, and the weight of the historical average progress rate is set to a lower proportion, so as to balance the influence of the recent running state and the long-term running law of the task on the rate evaluation. In the calculation, the current progress rate is first multiplied by its corresponding weight, then the historical average progress rate is multiplied by its corresponding weight, and finally the two products are added to obtain the fused progress rate, which provides a more accurate rate basis for subsequent remaining life cycle calculation.

[0070] In terms of setting and calculating the task remaining life cycle threshold, the total life cycle of the task is first determined as the preset completion time of the task, and then the task remaining life cycle threshold is set as a specific proportion of the total life cycle of the task, so as to define the determination standard of low-value tasks. In calculating the task remaining life cycle, the task remaining steps are taken as the numerator, the fused progress rate obtained by the exponential smoothing method is taken as the denominator, and the task remaining life cycle is obtained by dividing the task remaining steps by the fused progress rate. If the remaining life cycle is lower than the set threshold proportion, the task is determined as a low-value training task, and the subsequent termination and retry process is triggered.

[0071] For the retry rules of the terminated low-value training tasks, the system presets an explicit priority adjustment and number of times limit mechanism. When adjusting the retry priority, the original dynamic priority of the task before termination is taken as the basis, and the dynamic priority of the task after retry is improved by a fixed amplitude, so as to ensure that these tasks can obtain higher response priority when applying for resources in the future, and avoid long-term shelving. At the same time, the maximum number of retries of the retry queue is set, if the task is still not completed after continuous multiple retries, and the maximum number of retries is exceeded, the task is temporarily stored in the to-be-scheduled queue, and the administrator is automatically notified to check possible resource conflicts or task configuration problems, which not only ensures the orderliness of task retry, but also timely handles abnormal situations, and maintains the stability of resource scheduling.

[0072] It is worth noting that the criteria for determining low-value tasks are achieved through a three-layer logic of precise calculation of progress rate, quantification of remaining life cycle, and threshold comparison, and the determination is only triggered in specific resource scenarios, ensuring the objectivity and relevance of the criteria, as follows:

[0073] First, the prerequisite for the determination is that the GPU node meets the resource optimization requirements. Only when the actual resource utilization of the GPU node is lower than the preset resource utilization threshold and the actual memory fragmentation rate is higher than the preset memory fragmentation rate threshold, the determination process of low-value tasks is started. This premise ensures that the determination is not performed blindly, and is only triggered when there is idle waste or serious fragmentation of resources, and the resources need to be released to support high-value tasks, avoiding interference with tasks under normal resource conditions.

[0074] Second, the core basis of the determination is the quantitative calculation of the task's remaining life cycle, which relies on the precise integration of the progress rate. First, the current progress rate reflecting the recent efficiency of the task is obtained by dividing the number of completed steps by the running time of the task (the difference between the current time and the task start time). Then, the exponential smoothing method is used to weight and integrate the current progress rate with a higher weight and the historical average progress rate with a lower weight. This integration balances the short-term running fluctuations and long-term running regularity of the task, avoiding calculation bias caused by a single rate indicator, and ultimately obtaining a more accurate integrated progress rate. Then, the task's remaining life cycle is calculated by taking the remaining steps as the numerator and the integrated progress rate as the denominator. This result directly reflects the time needed to complete the task and is the core quantitative indicator for determining the value of the task.

[0075] Finally, the key threshold for the determination is that the task's remaining life cycle is less than a certain proportion of the total life cycle. First, the total life cycle of the task is set as the preset completion time of the task, and then the task's remaining life cycle threshold is set as a fixed proportion (such as 10%) of the total life cycle. If the remaining life cycle of a task is less than this proportion, it means that the task is very close to completion and the impact on the overall model training progress is minimal, and the released GPU resources can quickly support high-value tasks (such as related hot data and resource-demanding tasks), so the task is determined to be a low-value task. This threshold setting ensures minimal interference with the overall training progress and efficiently releases idle resources, optimizing resource utilization efficiency.

[0076] Further optionally, in an optional embodiment of step 105, a GPU affinity matrix is constructed in combination with the topology information and the data hotness level, and the resource layout with optimal communication cost is solved by the GPU affinity matrix in resource allocation, so as to optimize the adaptation degree between the GPU resource allocation of distributed training and the hot data storage location, including: extracting the topology information reported by the GPU node, the topology information including the hardware belonging information of each GPU and the high-speed interconnection relationship between the GPUs, and determining the physical distance and the communication bandwidth between different GPUs based on the topology information. Taking the hot data storage location in the data hotness level as a reference benchmark, the communication cost between any two GPUs is calculated in combination with the topology information of the GPUs, wherein the communication cost is positively correlated with the physical distance between the GPUs and is negatively correlated with the communication bandwidth, the communication cost is taken as a matrix element, and a GPU affinity matrix is constructed. Taking the weighted sum of the communication cost between the GPUs in the distributed training task and the distance from the GPU to the hot data storage location as an optimization objective, the GPU resource allocation combination is traversed based on the GPU affinity matrix, and the resource layout satisfying the optimization objective is selected. Through the optimal resource layout, the GPU resource of the distributed training task is allocated to the GPU node with the physical distance and the communication bandwidth reaching the preset condition.

[0077] Specifically, in the process of optimizing the GPU resource allocation in combination with the topology information and the data hotness level in step 105, first, the topology information is extracted from the GPU node and is converted into the basic data for resource adaptation. The GPU node actively reports the topology information of itself, which covers the hardware belonging relationship of each GPU and the high-speed interconnection relationship between the GPUs, and the system analyzes and determines the physical distance between different GPUs based on the information, and simultaneously calculates the communication bandwidth between the GPUs and between the GPU and the storage node, so as to provide the key parameters at the hardware level for subsequent communication cost calculation, and to ensure that the resource layout optimization has a clear hardware basis.

[0078] Subsequently, taking the hot data storage location in the data hotness level as a core reference benchmark, the communication cost between any two GPUs is calculated in combination with the GPU topology information determined in the previous stage. In the calculation logic, the communication cost is positively correlated with the physical distance between the GPUs, the farther the physical distance, the higher the communication cost in the data transmission process. At the same time, the communication cost is negatively correlated with the communication bandwidth, the higher the communication bandwidth, the higher the data transmission efficiency, and the lower the corresponding communication cost. Each pair of communication costs between the GPUs calculated is taken as a matrix element, and a GPU affinity matrix is constructed, which directly reflects the differences in communication efficiency of different GPU combinations and provides a quantitative reference for resource allocation.

[0079] Next, the optimization goal of resource layout is set, focusing on minimizing the weighted sum of two parts of the cost in distributed training tasks, one is the communication cost between GPUs, and the other is the distance cost from GPUs to hot data storage locations. Based on the constructed GPU affinity matrix, the system traverses all available GPU resource allocation combinations, calculates and compares the weighted sum corresponding to each combination, and selects the resource layout with the minimum weighted sum, i.e., the optimal communication cost, to ensure that this layout can simultaneously reduce the data interaction overhead between GPUs and the transmission overhead of GPU accessing hot data.

[0080] Finally, according to the selected optimal resource layout, GPU resource allocation is performed, and the required GPU resources for distributed training tasks are allocated to specific GPU nodes. These nodes need to meet the preset conditions of physical distance and communication bandwidth, i.e., the physical distance to the hot data storage location is within a reasonable range, and the communication bandwidth meets the efficient transmission requirements, finally realizing the precise adaptation of GPU resource allocation and hot data storage location, maximizing the reduction of data transmission delay in the training process, and improving the overall training efficiency.

[0081] Further optionally, in step 105, the physical distance preset condition is that the GPU node and the hot data storage node (NVMe SSD node storing hot data) are in the same rack or the same NUMA node, and if it exceeds this range, it is determined that the physical distance does not meet the condition; the communication bandwidth preset condition is that the network bandwidth between the GPU node and the hot data storage node is not less than 10 Gb / s, or the GPU supports NVLink high-speed interconnection (bandwidth not less than 300 GB / s); the weighted sum of the communication cost between GPUs and the distance from GPUs to hot data storage locations in distributed training tasks is minimized, where the weight of the communication cost between GPUs is set to 0.6, and the weight of the distance from GPUs to hot data storage locations is set to 0.4, to ensure balanced optimization of communication efficiency and data access efficiency.

[0082] It can be understood that in the process of allocating GPU resources by optimal resource layout in step 105, firstly, the dual preset conditions of physical distance and communication bandwidth are determined to ensure the adaptability of the GPU node and the hot data storage node. The physical distance preset condition is around the hot data storage node, which is the NVMe SSD node storing hot data. Only when the GPU node and such storage node are in the same rack or the same NUMA node, the physical distance is determined to meet the condition. If it exceeds the range, the physical distance does not meet the adaptability requirement of resource allocation. The communication bandwidth preset condition includes two scenarios. One is that the network bandwidth between the GPU node and the hot data storage node needs to reach a certain standard. The other is that the high-speed interconnection technology is supported between GPUs and the bandwidth reaches a certain standard. If either of the two scenarios is met, it is determined that the communication bandwidth meets the condition. Through the two conditions, the efficiency of data transmission is ensured, and transmission delay caused by insufficient bandwidth is avoided.

[0083] Further optionally, in step 105, the GPU resources of the distributed training task are allocated to the GPU nodes with short physical distance to the hot data storage location and high communication bandwidth, reducing the delay of hot data transmission from the storage location to the GPU in the training process, and realizing the dynamic adaptation of GPU resource allocation and hot data storage location.

[0084] It can be understood that, in order to balance the optimization of communication efficiency and data access efficiency, when setting the optimization goal of "minimizing the weighted sum of GPU intercommunication cost and GPU distance to hot data storage location in distributed training task", the weight distribution of the two costs is determined. That is, the weight of GPU intercommunication cost is set to a higher proportion, and the weight of GPU distance to hot data storage location is set to a lower proportion. This weight distribution not only values the efficiency of data interaction between GPU nodes in distributed training (avoiding node intercommunication as a bottleneck), but also takes into account the transmission efficiency of GPU accessing hot data (ensuring fast access to high-frequency data), ensuring that the optimization goal is not biased towards a single dimension.

[0085] Based on the above preset conditions and weight settings, when performing GPU resource allocation, the GPU resources of the distributed training task are preferentially allocated to the GPU nodes that meet both the physical distance and communication bandwidth conditions, i.e. the nodes with short physical distance and high communication bandwidth. Through such precise allocation, the delay of hot data transmission from the storage location to the GPU in the training process is greatly reduced, so that the GPU resource allocation can be dynamically adjusted according to the changes of hot data storage location and node hardware state, finally realizing the dynamic adaptation of GPU resource allocation and hot data storage location, and further consolidating the resource layout effect of optimal communication cost.

[0086] 106, converting the mixed storage file to be processed after Spark processing into an Arrow table.

[0087] Specifically, step 106 is mainly a key transitional link for connecting Spark data processing and subsequent GPU data interaction. This step is mainly to convert the mixed storage processed file converted by Spark into an Arrow table format. After Spark completes data cleaning, feature extraction and other preprocessing operations, the output file still maintains the mixed storage characteristics, and conversion to an Arrow table can fully utilize the advantages of Arrow columnar storage and off-heap memory storage. Both avoid the limitations of traditional JVM memory on large data processing, and improve data reading efficiency through columnar structure, lay a format foundation for efficient data interaction with PyTorch, and ensure that the data form can adapt to the zero-copy conversion requirements.

[0088] 107, call GPU resources through the resource layout, reuse and merge the large file and index file after merging to complete data type adaptation, storage optimization and preprocessing adaptation, and complete zero-copy conversion from Arrow table to PyTorch tensor based on Arrow memory direct mapping mechanism.

[0089] Specifically, in step 107, first, call GPU resources through the resource layout solved in step 105 to ensure the topology adaptation of GPU resources and data processing nodes. At the same time, reuse the merged large file and index file generated in step 104. The merged large file reduces the random IO loss during data reading, and the index file quickly locates the target data segment to be processed, both of which cooperate to provide efficient data input support for subsequent adaptation operations. On this basis, data type adaptation, storage optimization and preprocessing adaptation are completed in turn. Data type adaptation is aimed at data in the processed file that has compatibility differences with PyTorch tensors, and compatibility is achieved through type conversion mechanism. Storage optimization configures an adaptive serialization scheme for preprocessed data in combination with the mixed storage strategy in step 103. Preprocessing adaptation relies on the index file to realize data block processing, avoiding memory pressure caused by full loading.

[0090] As an optional embodiment, in step 107, zero-copy conversion from Arrow table to PyTorch tensor is completed based on Arrow memory direct mapping mechanism, including: converting target data in the Arrow table into an array view, and directly generating a PyTorch tensor based on the array view through a PyTorch interface to accelerate zero-copy conversion of the target data.

[0091] In the above embodiments, when the zero-copy conversion from the Arrow table to the PyTorch tensor is completed based on the Arrow memory direct mapping mechanism, the specific implementation process focuses on the direct reuse of the data form. Specifically, the target data to be converted in the Arrow table is first converted into an array view, which directly maps the out-of-heap memory data of the Arrow table without additional data copying. Then, the PyTorch tensor is directly generated based on the array view through the PyTorch interface. The entire process bypasses the multi-step serialization and data copying process in traditional data interaction, realizes zero-copy conversion, greatly shortens the time consumption from data processing completion to GPU availability, further connects the achievements of previous resource optimization and storage optimization, and ensures the efficiency of large model training data input.

[0092] It is worth understanding that the core of the zero-copy conversion process of the PyTorch tensor is the direct reuse of the memory address, which avoids data copying operations in the conversion link and realizes efficient mapping from the source data to the PyTorch tensor. First, rely on the memory characteristics of the source data (such as the Arrow table). The Arrow table uses a columnar storage structure, and the data exists in the form of a continuous block in memory, and is usually stored in out-of-heap memory (not limited by JVM memory management). This continuous and directly accessible memory layout provides a basis for zero-copy, and data can be directly referenced by external frameworks without reorganization.

[0093] Exemplarily, the first step of the conversion is to generate an array view. When extracting target data from the Arrow table, no data copying is performed, but an array view pointing to the data memory area is created. The array view is essentially a metadata description of the original memory, containing information such as data starting address, data type, and element number. Through this view, the original data in the Arrow table can be directly located and accessed, ensuring the accuracy of memory reference. Then, the tensor creation is completed through the underlying interface of PyTorch. PyTorch provides interfaces for constructing tensors directly from existing memory addresses. These interfaces receive parameters such as memory starting address, data type, and dimension information provided by the array view. When creating a tensor object internally, no new memory space is allocated, but the memory pointer of the tensor is directly pointed to the Arrow table data memory referenced by the array view. At this time, the PyTorch tensor and the Arrow table data share the same block of memory, and the read and write operations of the tensor directly act on the original memory area without going through the data copying link. In the above process, the data always resides in the original memory location, and the form conversion is completed only through the transmission of metadata and the mapping of pointers, which not only reduces memory usage (avoids repeated storage), but also eliminates the time overhead caused by data copying. Especially when processing massive data in large model training, it can reduce the delay from data preprocessing to model input, and become a key link to improve the efficiency of end-to-end training.

[0094] As an optional embodiment, before step 107, if the to-be-processed file is a super large-scale data set, the data block loading parameters are configured based on the index file, and the zero-copy conversion from the Arrow table to the PyTorch tensor is triggered in batches by using the stream processing mechanism.

[0095] Specifically, before step 107, for the scenario that the to-be-processed file is a super large-scale data set, the data block loading and stream processing are cooperated to lay a foundation for efficient data processing for the subsequent zero-copy conversion. This process takes the index file generated in step 104 as the core basis, configures the data block loading parameters based on the positioning information of the merged large file and the small file recorded in the index file. By clearly defining the data range and block size of each batch loading through the index, it is ensured that each data block can accurately correspond to a specific segment in the merged large file, avoiding memory overflow caused by too large data loading at a time, and relying on the index to quickly locate the target data to reduce the IO loss during data reading.

[0096] Subsequently, the stream processing mechanism is used to iteratively perform data processing and conversion triggering operations according to the configured block loading parameters. The stream processing mechanism does not perform one-time loading and processing of the full data, but advances in batches in units of blocks. After completing the loading of each batch of data, the batch of data is first converted into the Arrow table format generated in step 106, and then the zero-copy conversion based on the Arrow memory direct mapping mechanism is triggered to directly convert the batch of Arrow table data into the PyTorch tensor. This batch processing and conversion method can not only adapt to the processing needs of super large-scale data sets and control the memory occupancy of each batch, but also can realize the continuous advancement of data processing and conversion, avoid delays caused by the accumulation of full data, and at the same time, prepare data for subsequent step 107 by resource layout to call GPU resources, reuse and merge large files and index files, and ensure the efficiency and stability of data input to the GPU in the super large-scale data set scenario.

[0097] Further optionally, the data block loading parameters are configured based on the index file, each batch of blocks corresponds to 1-2 blocks in the merged large file, and the total number of records contained in each batch of blocks does not exceed 1 million, and when the data amount of a single record exceeds 1 MB, the number of records in each batch does not exceed 500,000. The stream processing mechanism uses Spark StructuredStreaming, sets the batch interval to 30 seconds, triggers the block loading and zero-copy conversion once every 30 seconds, automatically delays the triggering if the previous batch processing is not completed, avoids task accumulation. When loading the block, the hot data block marked in the index file is preferentially loaded, and the hot data block is cached to the off-heap memory. If the batch needs to reuse the block, the block is directly read from the off-heap memory and the zero-copy conversion is triggered, without the need to reload the merged large file.

[0098] It can be explained that for the block loading and streaming of super large scale data set, the configuration of data block loading parameters based on the index file needs to be completed first. The fixed number of blocks in the merged large file corresponding to each batch loading block is determined, and the upper limit of the total number of records contained in each batch block is limited. If the data volume of a single record exceeds a certain size, the upper limit of the number of records per batch is further reduced to ensure that the data volume of each batch loading is adapted to the memory carrying capacity, and to avoid memory overflow caused by too large single data volume. In the implementation of the streaming mechanism, Spark Structured Streaming is used as the core framework, and a fixed batch interval is set. The block loading and zero-copy conversion operation are triggered once every interval. If the processing task of the previous batch has not been completed, the trigger time of the current batch is automatically delayed, and the new batch task is not forcibly started, so as to avoid processing congestion caused by the accumulation of multiple batch tasks, and to ensure the smooth progress of streaming processing. In terms of efficiency optimization of block loading, the hot data blocks marked in the index file are preferentially read during the loading process, and these hot data blocks are cached in the off-heap memory after reading. If the hot data blocks need to be reused in the subsequent batch, there is no need to load data from the merged large file again, and the hot data blocks can be directly read from the off-heap memory and trigger zero-copy conversion, which greatly reduces the IO overhead of the data reading link, shortens the overall time consumption of data acquisition and conversion, and further adapts to the needs of efficient processing of super large scale data set.

[0099] As an optional embodiment, in step 107, the GPU resources are called through the resource layout to complete data type adaptation, storage optimization and preprocessing adaptation by multiplexing and merging the large file and the index file, including: locating the target data segment to be processed based on the index file, and multiplexing and merging the large file after merging as data input. For special data types in the to-be-processed file that have compatibility differences with PyTorch tensors, a type conversion mechanism is adopted to convert the special data types into intermediate compatible data types, and then the intermediate compatible data types are explicitly converted into target PyTorch tensor types through the PyTorch interface. In the adaptation process, the GPU resources adapted to the preprocessing node topology are called in combination with the resource layout. Based on the hybrid storage strategy, the corresponding serialization storage strategy is configured for the data in the preprocessing process. For preprocessing data associated with hot data, an efficient serialization protocol adapted to replica storage is adopted, and the efficient serialization protocol includes Kryo serialization; for preprocessing data associated with cold data or warm data, a compressed serialization scheme adapted to erasure code storage is adopted, and the compressed serialization scheme includes LZ4 compressed ApacheArrow serialization. And configure the cache eviction mechanism, when the disk usage is higher than the preset threshold, the cache of the cold data associated preprocessing data is preferentially deleted, and the memory and disk occupation is optimized. For large-scale data sets, based on the index file, data block loading parameters are configured, and a streaming processing mechanism is adopted to perform preprocessing operations in batches. Based on the data set size and task parallelism, memory configuration parameters are calculated, including executor memory and off-heap memory size. Finally, the data write-ahead log and periodic checkpoint mechanism are enabled to ensure the stability and fault recoverability of the preprocessing process. In the preprocessing process, the preprocessing nodes and GPU nodes are matched in topological affinity through the resource layout, reducing the communication cost of transferring the data after preprocessing to the GPU.

[0100] Specifically, in step 107, when calling GPU resources through the resource layout, first, the index file is taken as the core positioning basis to accurately find the target data segment to be processed from the merged large file, and the merged large file is directly reused as the data input source, avoiding the efficiency loss caused by re-reading scattered small files, and providing an efficient data basis for subsequent adaptation operations.

[0101] For special data types in the to-be-processed file that have compatibility differences with PyTorch tensors, a hierarchical type conversion mechanism is adopted. First, these special data types are uniformly converted into intermediate compatible data types to eliminate format differences, and then the intermediate compatible data types are explicitly converted into target PyTorch tensor types through the interface provided by PyTorch, ensuring that the data can be directly called by the GPU. In the entire adaptation process, the resource layout determined in step 105 is combined to preferentially call the GPU resources adapted to the topology of the preprocessing node, reducing the additional overhead of cross-node data transmission.

[0102] At the storage optimization level, the mixed storage strategy determined in advance is strictly followed to configure differentiated serialization storage strategies for the data in the preprocessing process. For preprocessing data associated with hot data, an efficient serialization protocol adapted to replica storage is used to improve the data reading and deserialization speed in high-frequency access scenarios. For preprocessing data associated with cold data or warm data, a compressed serialization scheme adapted to erasure code storage is used to control storage occupancy while ensuring data integrity. At the same time, a cache eviction mechanism is configured to monitor disk usage in real time. When the disk usage exceeds the preset threshold, the cache of cold data associated preprocessing data is preferentially deleted to balance the occupancy pressure of memory and disk and avoid insufficient storage resources affecting processing efficiency.

[0103] For super large-scale data sets, data block loading parameters are configured based on index files, and preprocessing operations are performed in batches step by step using a streaming processing mechanism to avoid memory overflow caused by full data loading. In terms of memory configuration, reasonable memory configuration parameters are calculated and set according to the overall size of the data set and the task parallelism, including executor memory and off-heap memory size, to ensure that the preprocessing task has sufficient and not excessive memory support.

[0104] To ensure the stability and fault recoverability of the preprocessing process, the system enables data write-ahead logging and regular checkpointing mechanisms. Write-ahead logging records the key information and data state of each preprocessing operation, and the checkpointing mechanism regularly stores the metadata and progress information of the preprocessing task to a reliable location. In the event of a failure, the system can quickly recover to the pre-failure state based on the log and checkpoint. At the same time, the preprocessing nodes and GPU nodes are matched according to topological affinity through resource layout, allowing the preprocessing nodes and GPU nodes to maintain a close, high-bandwidth adaptive relationship in terms of physical location or network connection, significantly reducing the communication cost of transferring data from preprocessing to GPU and further improving overall processing efficiency.

[0105] Further optionally, when calculating memory configuration parameters based on the size of the data set and the task parallelism, the calculation logic of executor memory is first determined, i.e., its value needs to be determined in combination with the single batch processing data volume of the data set and the task parallelism. The single batch processing data volume is obtained based on the previously configured block loading parameters. When calculating, the single batch data volume is first multiplied by a coefficient to reserve necessary buffer space, and then divided by the task parallelism to obtain the executor memory size that adapts to the task parallel processing requirements. The off-heap memory size is calculated based on the calculated executor memory size at a fixed ratio, and a minimum threshold for off-heap memory is set to ensure that even if the executor memory is small, the off-heap memory can still meet the basic data processing requirements, avoiding data overflow problems caused by insufficient memory.

[0106] In the configuration of the cache eviction mechanism, a preset threshold of disk usage is first set. When the system monitors that the disk usage exceeds the threshold, the cache eviction process is automatically started. The eviction order strictly follows the priority of data heat from low to high. First, the preprocessing data cache associated with cold data is evicted, then the preprocessing data cache associated with warm data is evicted, and finally the preprocessing data cache associated with hot data is processed. Only the hot data associated cache that has not been used for more than a certain time is evicted to prevent deleting high-frequency access key data. At the same time, the release of storage resources guarantees the accessibility of core data.

[0107] For the serialization configuration of different heat data associated preprocessing data, a differentiated scheme is adopted to balance efficiency and storage demand. For hot data associated preprocessing data, Kryo serialization is used with a fixed serialization compression ratio to ensure serialization speed while effectively reducing data volume. For cold or warm data associated preprocessing data, Apache Arrow serialization using LZ4 compression is used with a specific compression level that can achieve high compression efficiency to save storage space while avoiding low decompression speed caused by too high compression level, thereby balancing compression effect and data access efficiency.

[0108] For example, in an optional embodiment of BERT-Large model training optimization, the test environment configuration is built around the coordinated adaptation of hardware and software stacks to ensure that various optimization strategies can be effectively verified. The computing node uses 8 dual-channel Xeon 6348 servers, each equipped with 512GB DDR4 memory, and improves resource utilization through NUMA binding optimization. Each node is configured with 4 NVIDIA A100 80GB GPUs, which support NVLink high-speed interconnection and provide a hardware foundation for topology-aware scheduling. The network architecture uses 100Gbps RoCE double uplink, supports GPUDirect RDMA to accelerate data transmission. The storage system is a full flash architecture with a total NVMe SSD capacity of 1.5PB, which adapts to the mixed storage layering needs. On the software level, Hadoop 3.3.4, Spark 3.4.1, and PyTorch 2.0.1 are selected to achieve compatible adaptation of the full-stack version.

[0109] In the above example, the HDFS storage configuration implements a hierarchical strategy for different types of data characteristics. Hot data is the model parameter file, the storage path is set in the user model training parameter directory, the block size is 256 MB to match the GPU memory loading granularity, 3 copies are configured for storage strategy and physically stored in the NVMe SSD storage pool, monitored by real-time access counter, files with more than 10 access times in the last 7 days are automatically maintained in 3 copy state, ensuring the reliability and speed of high-frequency access. Warm data is the checkpoint file, the path is set in the user model checkpoint directory, the RS-6-3 erasure code strategy is adopted, the storage overhead is 1.5 times, the block size is 1024 KB to adapt to file characteristics, the automatic migration mechanism is configured, and files with less than 5 access times for 3 consecutive days are downgraded to cold data, balancing storage cost and access demand. Cold data is the training log, the path is set in the user model log directory, the RS-10-4 erasure code strategy is adopted, the storage overhead is 1.4 times, the block size is 1 GB to optimize large file sequential read and write, and it is stored in a high-density HDD array to reduce cost.

[0110] In the above example, the YARN scheduling configuration focuses on efficient use of GPU resources. A GPU exclusive node tag is created, and the tag attribute is set to allow only GPU tasks to occupy, achieving physical isolation of non-GPU tasks and GPU nodes. In terms of queue configuration, the AI exclusive queue allocates 70% of the cluster GPU resources, and the priority weight is set to 3.0 to ensure resource tilt through scheduling configuration file; when submitting training tasks, set appropriate priority and GPU allocation number per Executor to improve core task response speed. The preemption mechanism monitors GPU utilization and memory fragmentation rate in real time, and triggers preemption when GPU utilization is less than 50% for 5 minutes and memory fragmentation rate is higher than 20%. Based on the EMA algorithm, the remaining life cycle of the task is calculated by weighting the current progress rate and the historical average progress rate, and the low-value tasks with a remaining life cycle less than 10% are terminated and the resources are released. The terminated tasks enter the retry queue.

[0111] In the above example, Spark preprocessing optimization improves data input efficiency through process adjustment. Arrow acceleration configuration is enabled to improve data processing speed using Arrow columnar storage characteristics; direct conversion from Spark data frame to PyTorch tensor is realized through conversion tools, based on memory mapping mechanism to avoid data duplication, accelerating data input to GPU; configure data persistence strategy, use Kryo serialization to store preprocessing data, reduce memory occupancy by 40%, and register custom tensor class and its serializer in Kryo configuration to improve serialization efficiency.

[0112] In the above examples, hardware adaptation and performance comparison adjust optimization strategies for different hardware environments. For high-end configurations with 4 A100s per node, RS-10-4 erasure code is used, topology-aware scheduling is enabled to prefer NVLink interconnection, and Arrow and GPUDirect RDMA are combined to further accelerate data transmission. For mid-end configurations with 4 V100s per node, RS-6-3 erasure code is used, NUMA domain binding is implemented, and Arrow and memory mapping are optimized for performance. For edge configurations with 2 T4s per node, 3 copies plus Zstd compression strategy is used, preemption mechanism is disabled, and chunk loading and selective persistence are adapted to resource limits. Performance tests show that the optimization effect is significant, and the end-to-end training time is shortened from 18.5 hours to 10.2 hours in the A100 environment, an increase of 45%, and 14.7 hours in the V100 environment, an increase of 21%. Storage costs are reduced by 48% and 32%, GPU utilization peaks are increased to 89% and 78%, and preprocessing throughput is increased from 85 MB / s to 240 MB / s and 165 MB / s.

[0113] It is worth understanding that implementing best practices includes hardware adaptation, dynamic configuration, and elastic resource adjustment. In hardware adaptation, the erasure code strategy is dynamically determined according to the number of GPUs and network bandwidth. The calculation method is to take the smaller value of the integer part of the number of GPUs divided by 2 and 4, plus the integer part of the network bandwidth divided by 25 and 2, and the sum of the two as the basis for the strategy. Dynamic configuration templates automatically select strategies according to GPU types, such as enabling RS-10-4 erasure code and NVLink topology configuration in A100 environments and enabling RS-6-3 erasure code in V100 environments. In elastic resource adjustment, the AI queue resource occupancy ratio changes dynamically according to cluster utilization. When the cluster utilization is less than 80%, it remains at 70%, and when it is higher than 80%, it increases to 85%, balancing resource utilization and task demand.

[0114] In the technical solution of the present application, first, by collecting the access features and content association features of the files to be processed, combining the data heat level to configure the replica, erasure code hybrid storage strategy and clustering and merging small files, the storage efficiency and access speed of large model training data are improved, solving the technical problems of large model training data volume, high storage cost, and serious random IO loss caused by large number of small files, high data access delay. Secondly, by constructing the GPU affinity matrix based on the access features and data heat level, solving the communication cost optimal resource layout, optimizing the adaptation degree of GPU resource allocation and hot data storage location, improving the GPU resource utilization and resource acquisition priority of high value training task, solving the technical problems of high GPU resource fragmentation rate, low resource and data demand matching degree in large model distributed training, and high value hot data related task waiting for resources. Finally, by converting the data processed by Spark into Arrow table format, reusing and merging large files and index files to complete data type adaptation, storage optimization and preprocessing adaptation, and based on the Arrow memory direct mapping mechanism to realize zero-copy conversion to PyTorch tensor, the data preprocessing efficiency and the continuity of GPU data input are improved, solving the technical problems of high preprocessing delay caused by large model training data preprocessing process redundancy, serialization, and large data replication overhead, high communication cost of data transmission to GPU after preprocessing, and GPU idle waiting for data. The training efficiency and resource utilization level of the large model are improved.

[0115] In another embodiment of the present application, a large model training resource optimization system based on Hadoop ecology is also provided. Referring to Figure 2The system comprises the following units: a hybrid storage unit configured to collect access features and content association features of to-be-processed files; the to-be-processed files comprise a plurality of small files; data hotness levels are divided based on the access features, and the data hotness levels are divided into hot data, warm data and cold data according to access frequencies; in combination with the content association features, corresponding copy storage strategies or erasure code storage strategies are configured for to-be-processed files of different hotness levels and different content combinations, and the to-be-processed files are hybrid stored; the to-be-processed files are divided into a plurality of small file clusters and a merging operation is performed according to the content association features, to obtain merged large files and index files, the index files being used to record positioning information of each small file in the merged large files; a scheduling unit configured to solve a resource layout with optimal communication cost in resource allocation based on the access features and the data hotness levels, to optimize GPU resource allocation in a large model distributed training process; an acceleration unit configured to convert the to-be-processed files hybrid stored after Spark processing into Arrow tables; GPU resources are called through the resource layout, the merged large files and the index files are reused to complete data type adaptation, storage optimization and preprocessing adaptation, and zero-copy conversion from the Arrow tables to PyTorch tensors is completed based on an Arrow memory direct mapping mechanism. The above system can implement various steps in the method embodiments, which will not be expanded here.

[0116] Please refer to Figure 3 , Figure 3 The embodiment of the electronic device provided in the present application is shown in the embodiment schematic diagram of the electronic device. As shown in Figure 3 The present application provides an electronic device 500, which comprises a memory 510, a processor 520 and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the method for optimizing large model training resources based on Hadoop ecology is realized.

[0117] It is to be understood that the embodiments which have been described above are merely illustrative of the application and that modifications thereof can occur to those skilled in the art. Accordingly, it is not intended that the application be limited to the specific embodiments which have been described, and that the scope of the application should be determined not with reference to the above description but with reference to the claims appended hereto.

Claims

1. A method for optimizing large-scale model training resources based on the Hadoop ecosystem, characterized in that, The method includes: Collect access characteristics and content association characteristics of the files to be processed; the files to be processed include multiple small files. Based on the access characteristics, data popularity levels are divided into hot data, warm data, and cold data according to access frequency; combined with the content association characteristics, corresponding copy storage strategies or erasure coding storage strategies are configured for files to be processed with different popularity levels and different content combinations, and the files to be processed are stored in a mixed manner. Based on the content association features, the file to be processed is divided into multiple small file clusters and a merging operation is performed to obtain a merged large file and an index file. The index file is used to record the location information of each small file in the merged large file. Based on the access characteristics and data popularity levels, the resource allocation is optimized to achieve the optimal communication cost, thus improving GPU resource allocation during the distributed training of large models. This includes: configuring dedicated resource tags for GPU nodes to achieve physical isolation; dividing AI-specific scheduling queues; configuring the initial priority and accessible GPU node range of the AI-specific scheduling queues; constructing a priority calculation model based on the GPU resource demand characteristics and time constraint characteristics of training tasks; generating dynamic priorities for each training task through the priority calculation model, wherein the GPU resource demand characteristics reflect the scale of GPU resource demand for each training task, and the time constraint characteristics reflect the urgency of each training task; and combining data popularity during the calculation process. The system prioritizes high-value training tasks with associated hot data based on their ranking. It monitors GPU node resource utilization and memory fragmentation in real time. When both meet preset preemption trigger conditions, it filters and terminates low-value training tasks based on the remaining lifecycle assessment results to release occupied GPU resources. Terminated low-value training tasks enter a retry queue with dynamic priority increases after retry. It reports topology information through configured GPU nodes. A GPU affinity matrix is ​​constructed by combining topology information and data heat levels. In resource allocation, the GPU affinity matrix is ​​used to solve for the resource layout with optimal communication cost, optimizing the fit between GPU resource allocation and hot data storage locations in distributed training. The files to be processed after Spark processing and mixed storage are converted into Arrow tables; GPU resources are called through the resource layout to reuse the merged large files and index files to complete data type adaptation, storage optimization and preprocessing adaptation, and zero-copy conversion from Arrow tables to PyTorch tensors is completed based on the Arrow memory direct mapping mechanism.

2. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 1, characterized in that, After dividing the small files into multiple clusters based on the content association features and performing a merging operation to obtain the merged large file and the corresponding index file, the process further includes: The migration process is initiated based on conditions triggered periodically or by changes in the access characteristics. During the migration process, a mechanism to ensure data access continuity is adopted, and dynamic migration of hot and cold data is achieved by combining data popularity levels. The location information and metadata of each small file in the index file are updated, and the storage strategy corresponding to each small file is dynamically switched.

3. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 2, characterized in that, The mechanism for ensuring data access continuity, combined with data popularity levels, enables dynamic migration of hot and cold data, including: When hot data is migrated to cold data, the hot data is first written to a new file according to the erasure coding storage strategy corresponding to the cold data. Then, the metadata of the HDFS NameNode is updated to point to the new file. Finally, the original copy file of the hot data is deleted. During the migration, all read requests still access the original copy file of the hot data. When cold data is migrated to hot data, a new replica is first created for the cold data according to the replica storage strategy corresponding to the hot data. After the new replica is fully generated, the NameNode metadata is updated to point to the new replica. If there are write requests during the migration, the copy-on-write mechanism is used to direct the write operation to the new replica, or the write request is temporarily blocked until the migration is completed. During the migration process, the index file is updated synchronously. The location information of the corresponding small files in the index file is corrected according to the changes in the file storage location, and the file metadata is updated to ensure that the data access is not interrupted when the storage strategy is switched.

4. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 1, characterized in that, When resource utilization and memory fragmentation rate both meet the preset preemption trigger condition, low-value training tasks are selected and terminated based on the task's remaining lifecycle assessment results to release the occupied GPU resources, including: Set a resource utilization threshold and a memory fragmentation rate threshold for the GPU node. When the actual resource utilization of the GPU node is lower than the resource utilization threshold and the actual memory fragmentation rate is higher than the memory fragmentation rate threshold, it is determined that the preset preemption trigger condition is met. The current progress rate of the training task is calculated, and the current progress rate is weighted and fused with the historical average progress rate through a smoothing algorithm. The remaining lifespan of the task is obtained by dividing the remaining steps of the task by the fused progress rate. The current progress rate of the training task is the quotient obtained by dividing the number of steps completed by the training task by the running time of the training task. The running time of the training task is the difference obtained by subtracting the start time of the training task from the current time. Training tasks with remaining lifecycles below the task's remaining lifecycle threshold are identified as low-value training tasks. The termination process for the low-value training tasks is triggered to release the GPU resources occupied by the low-value training tasks. The terminated low-value training tasks are then included in the retry queue, and their dynamic priority is increased according to preset rules after retrying.

5. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 1, characterized in that, The process of constructing a GPU affinity matrix by combining topology information and data popularity levels, and then using this matrix to solve for the resource layout with optimal communication cost during resource allocation, optimizes the fit between GPU resource allocation and hot data storage locations in distributed training. This includes: Extract the topology information reported by the GPU nodes. The topology information includes the hardware ownership information of each GPU and the high-speed interconnection relationship between GPUs. Based on the topology information, determine the physical distance and communication bandwidth between different GPUs. Using the hot data storage location in the data heat level as a reference benchmark, and combining the GPU topology information, the communication cost between any two GPUs is calculated. The communication cost is positively correlated with the physical distance between GPUs and negatively correlated with the communication bandwidth. The communication cost is used as a matrix element to construct a GPU affinity matrix. With the goal of minimizing the weighted sum of the inter-GPU communication cost and the distance from the GPU to the hot data storage location in the distributed training task, the system iterates through the available GPU resource allocation combinations based on the GPU affinity matrix to select the resource layout that satisfies the optimization objective. The optimal resource layout allocates GPU resources for distributed training tasks to GPU nodes whose physical distance from the hot data storage location and communication bandwidth both meet preset conditions.

6. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 1, characterized in that, The zero-copy conversion from Arrow tables to PyTorch tensors based on the Arrow memory direct mapping mechanism includes: The target data in the Arrow table is converted into an array view, and PyTorch tensors are generated directly from the array view through the PyTorch interface, which accelerates the zero-copy conversion of the target data.

7. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 6, characterized in that, Before the zero-copy conversion from Arrow table to PyTorch tensor is completed based on the Arrow memory direct mapping mechanism, the following steps are also included: If the file to be processed is a very large dataset, the data block loading parameters are configured based on the index file, and a churn handling mechanism is used to trigger the zero-copy conversion from the Arrow table to PyTorch tensors in batches.

8. The method for optimizing large model training resources based on the Hadoop ecosystem according to claim 1, characterized in that, The process of calling GPU resources through the resource layout, reusing the merged large file and index file to complete data type adaptation, storage optimization, and preprocessing adaptation includes: The target data segment to be processed is located based on the index file, and the merged large file is reused as data input. For special data types in the files to be processed that have compatibility differences with PyTorch tensors, a type conversion mechanism is adopted to convert the special data types into intermediate compatible data types, and then explicitly convert them into the target PyTorch tensor type through the PyTorch interface; during the adaptation process, GPU resources that are compatible with the preprocessing node topology are called in combination with the resource layout. Based on a hybrid storage strategy, corresponding serialization storage strategies are configured for the data during the preprocessing process. For preprocessed data associated with hot data, an efficient serialization protocol adapted to replica storage is adopted, including Kryo serialization. For preprocessed data associated with cold or warm data, a compression serialization scheme adapted to erasure coding storage is adopted, including Apache Arrow serialization with LZ4 compression. A cache eviction mechanism is also configured so that when disk usage exceeds a preset threshold, the cache of preprocessed data associated with cold data is deleted first, optimizing memory and disk usage. For ultra-large-scale datasets, data block loading parameters are configured based on index files, and preprocessing operations are performed in batches using a streaming processing mechanism; memory configuration parameters are calculated based on the dataset size and task parallelism, including executor memory and off-heap memory size; Enabling a pre-write log and periodic checkpoint mechanism ensures the stability and fault recoverability of the preprocessing process. During preprocessing, the resource layout matches preprocessing nodes and GPU nodes according to topology affinity, reducing the communication cost of transmitting preprocessed data to the GPU.

9. A large-scale model training resource optimization system based on the Hadoop ecosystem, characterized in that, The system includes the following units, wherein, A hybrid storage unit is configured to collect access characteristics and content association characteristics of files to be processed; the files to be processed include multiple smaller files; based on the access characteristics, data popularity levels are divided into hot data, warm data, and cold data according to access frequency; combined with the content association characteristics, corresponding replica storage strategies or erasure coding storage strategies are configured for files to be processed with different popularity levels and different content combinations, and the files to be processed are stored in a hybrid manner; based on the content association characteristics, the files to be processed are divided into multiple smaller file clusters and a merging operation is performed to obtain a merged large file and an index file, the index file being used to record the location information of each smaller file in the merged large file; The scheduling unit is configured to solve for the resource layout with the optimal communication cost in resource allocation based on the access characteristics and data popularity level, and optimize the GPU resource allocation in the distributed training process of large models. Specifically, the scheduling unit is configured to: assign dedicated resource tags to GPU nodes for physical isolation; divide AI-specific scheduling queues; configure the initial priority and accessible GPU node range of the AI-specific scheduling queues; construct a priority calculation model based on the GPU resource requirement characteristics and time constraint characteristics of training tasks; generate dynamic priorities for each training task through the priority calculation model, wherein the GPU resource requirement characteristics reflect the scale of GPU resource demand for each training task, and the time constraint characteristics reflect the urgency of each training task; during the calculation process, increase the priority of high-value training tasks with associated hot data by combining data popularity levels. The system is designed to: monitor GPU node resource utilization and memory fragmentation rate in real time; when both resource utilization and memory fragmentation rate meet the preset preemption triggering conditions, select low-value training tasks based on the task's remaining lifecycle assessment results and terminate them to release occupied GPU resources; terminated low-value training tasks enter the retry queue and have their dynamic priority increased after retry; report topology information through configured GPU nodes; construct a GPU affinity matrix by combining topology information and data popularity level; solve for the resource layout with optimal communication cost in resource allocation using the GPU affinity matrix; and optimize the fit between GPU resource allocation and hot data storage location in distributed training. The acceleration unit is configured to convert the files to be processed after Spark processing and mixed storage into Arrow tables; it calls GPU resources through the resource layout, reuses the merged large files and index files to complete data type adaptation, storage optimization and preprocessing adaptation, and completes zero-copy conversion from Arrow tables to PyTorch tensors based on the Arrow memory direct mapping mechanism.

Images