A data storage device for supporting large model training and a method of use

Abstract

The application discloses a kind of data storage devices and use methods for supporting large model training, including system control hub, and system control hub is connected with: model construction module, storage module generation module, hierarchical architecture construction and strategy generation module and strategy verification module.The application is dynamically adjusted parameter by multidimensional data type, access frequency and security requirement of model construction module, avoid the single thinking that traditional storage only focuses on capacity;Storage module generation module is based on demand model, through multi-scene performance test and iterative correction, ensure that storage unit hardware performance and training demand are adapted;Different performance storage media are classified and utilized in hierarchical architecture, balance performance and cost, dynamic evolution of storage scheduling is realized by double-layer control strategy, adapt to training full-process load change;Strategy verification module avoids strategy deployment risk in advance through digital twin simulation and multi-scene optimization, guarantee system stable operation.

Description

Technical Field

[0001] This invention relates to the field of computer storage technology, and more specifically, to a data storage device and method for supporting the training of large models. Background Technology

[0002] Currently, large-scale model training has entered a stage of large-scale development, with training data exhibiting typical characteristics of "massive heterogeneity and dynamic load." In terms of data type, it encompasses structured model parameter tables, semi-structured XML annotation data, and unstructured image / long text materials. Different types of data have significantly different requirements for storage read / write latency, capacity, and security. For example, model parameters require high-frequency random access, while unstructured image data requires large-capacity storage. In terms of load characteristics, the data access frequency fluctuates dramatically across different stages of large-scale model training, fine-tuning, and inference. The pre-training stage involves frequent calls to sample feature vectors, while the fine-tuning stage focuses on writing intermediate parameters. This dynamically changing load places extremely high demands on the real-time adaptability of the storage system. Against this backdrop, the technical limitations of traditional storage architectures are becoming increasingly apparent, becoming a core bottleneck restricting the efficiency of large-scale model training, and targeted breakthroughs are urgently needed to meet the storage requirements for efficient large-scale model training.

[0003] Traditional storage architectures generally adopt a static resource allocation + single fixed strategy model, with storage capacity as the core configuration basis. This lacks the ability to deeply adapt to the multi-dimensional characteristics of large model training data and cannot accurately match storage media and security strategies according to data characteristics. This results in high-frequency data being stored on low-performance media, causing excessive access latency and insufficient protection of sensitive data. Furthermore, it lacks dynamic scheduling capabilities, failing to capture the load change trends of large models at different training stages. It cannot pre-adjust resource allocation or optimize scheduling strategies in real time, leading to uneven storage resource utilization, low system response efficiency, and even problems such as insufficient concurrent write bandwidth and cache jitter due to rigid strategies, making it difficult to support efficient and stable training of large models. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a data storage device and a method for supporting the training of large models, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data storage device for supporting large model training, comprising a system control center, wherein the system control center is communicatively connected to: The model building module is used to obtain the data types, access frequencies, and security requirements parameters in the training of large models, and to build a data storage requirement model through calculation. The storage module generation module is used to determine storage indicators based on the data storage requirement model and generate preliminary storage units; after performance testing in multiple scenarios, it is corrected to generate the final storage units. The layered architecture construction and strategy generation module is used to build a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, it generates basic control strategies through dynamic resource allocation, introduces adaptive learning by combining historical data access records, and updates and optimizes resource allocation parameters to form dynamic control strategies. The strategy verification module is used to build dynamic control strategies for the simulation platform and optimize them to obtain target control strategies, thereby controlling the hierarchical system architecture to perform storage scheduling.

[0006] Optionally, constructing a data storage requirement model specifically includes: Obtain the proportions of structured, semi-structured, and unstructured data used in large model training, compare them with preset data storage standards for each type, and determine the priority storage weights for each type of data based on the deviation values. The access frequency of data in different training stages is statistically analyzed. Data with high access frequency is marked as high-frequency access data, and data with low access frequency is marked as low-frequency access data. The storage resource allocation ratio is calculated based on the ratio of high-frequency to low-frequency data. Security requirement parameters are determined based on data sensitivity levels, which include public, internal, and confidential levels. Priority storage weights, storage resource allocation ratios, and security requirement parameters are integrated into a multi-dimensional feature set. Through statistical analysis and training scenario matching tests, parameter weights are adjusted to construct a data storage requirement model.

[0007] Optionally, determining security requirement parameters based on data sensitivity levels specifically includes: Obtain the source identification, scope of use, and encryption requirements of various types of data, and establish a data sensitivity level assessment index system; The Analytic Hierarchy Process (AHP) was used to assign weights to the various indicators in the data sensitivity assessment index system, and the comprehensive data sensitivity score was calculated. The overall data sensitivity score is compared with the preset sensitivity level threshold to determine the sensitivity level of the data. Configure corresponding security storage policy parameters according to different sensitivity levels, including encryption algorithm strength, access permission level and data transmission encryption requirements, to form security requirement parameters.

[0008] Optionally, generating the initial storage unit specifically includes: The storage capacity requirement, read / write speed requirement, data reliability requirement, and scalability requirement are extracted from the data storage requirement model as core storage indicators. Based on core storage metrics, select suitable storage media types, including solid-state drives, hard disk drives, and distributed storage nodes; Based on the priority storage weights of various types of data and the storage resource allocation ratio, calculate the configuration quantity and capacity allocation scheme of different storage media. The selected storage media are logically combined according to the capacity allocation scheme to form preliminary storage units.

[0009] Optionally, generating the final storage unit specifically includes: Construct simulation environments for various training scenarios, including batch data loading scenarios, high-frequency random access scenarios, and concurrent write scenarios; Run the initial storage unit in various simulation environments and collect storage response time, data transfer rate and error rate as performance test data; Analysis of variance was used to process the performance test data and identify performance bottleneck parameters. Based on performance bottleneck parameters, the storage medium type and capacity allocation scheme in the initial storage unit are adjusted to obtain the modified storage unit; Repeat the performance testing steps until the performance test data of the corrected storage unit in each simulation environment meets the preset performance standards, and then output it as the final storage unit.

[0010] Optionally, constructing a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit specifically includes: Based on the data access frequency and real-time requirements, the storage medium with the fastest response speed in the final storage unit is divided into a data cache subsystem, which is used to store frequently accessed data. The storage medium with large capacity and high stability is divided into a core storage subsystem, which is used to store the core dataset for training large models. Select storage media with redundant backup capabilities to build a backup storage subsystem for disaster recovery backup of core data; A high-speed data bus is used to establish communication connections between the three subsystems, configure data synchronization mechanisms and access priority rules, and form a complete layered system architecture.

[0011] Optionally, generating the basic control strategy specifically includes: The real-time load rate, data access request volume, and remaining storage space of each subsystem are used as input parameters. A mathematical model for resource allocation is constructed with the optimization objectives of maximizing system response efficiency and balancing resource utilization. A genetic algorithm is used to solve the mathematical model of resource allocation, resulting in data caching strategies, storage resource allocation schemes, and data migration rules. Integrate data caching strategies, storage resource allocation schemes, and data migration rules into a basic control strategy; The basic control strategy is run in a layered system architecture, the data on the effect of the strategy execution is collected, compared with the preset optimization target, and the parameters in the basic control strategy are fine-tuned.

[0012] Optionally, forming a dynamic control strategy specifically includes: Collect historical data access records from different training phases, including access time, data type, and access frequency characteristics; A data access trend prediction model is constructed by training a long short-term memory network on historical data access records. The data access trend prediction model is used to predict data access demand in future time periods, and the prediction results are obtained. Based on the prediction results, the resource allocation parameters in the basic control strategy are pre-adjusted to obtain the pre-adjustment strategy. In a layered system architecture, a pre-adjustment strategy is deployed, the execution effect of the strategy is monitored in real time, and a reinforcement learning algorithm is used to continuously optimize the pre-adjustment strategy to form a dynamic control strategy.

[0013] Optionally, the target control strategy specifically includes: A simulation platform with a layered system architecture is constructed using digital twin technology. The simulation platform includes a storage device simulation layer, a data interaction simulation layer, and a policy execution simulation layer. Import various large-scale model training task scenarios into the simulation platform, run dynamic control strategies, and collect system response time, resource utilization, and data security indicators. The collected indicators are compared with preset thresholds to determine whether the dynamic control strategy meets the requirements. When the indicator fails to reach the preset threshold, the weak link in the strategy is identified, and the particle swarm optimization algorithm is used to adjust the dynamic control strategy. Repeat the strategy execution and optimization steps until all indicators in the simulation platform reach the preset thresholds, and output the final dynamic control strategy as the target control strategy.

[0014] A method of using a data storage device to support the training of large models includes the following steps: S1: Obtain the data types, access frequencies, and security requirements parameters in the training of large models, and construct a data storage requirement model through calculation; S2: Based on the data storage requirement model, determine storage metrics and generate preliminary storage units; after performance testing in multiple scenarios, refine and generate the final storage units. S3: Construct a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, generate basic control strategies through dynamic resource allocation, introduce adaptive learning by combining historical data access records, and update and optimize resource allocation parameters to form dynamic control strategies. S4: Construct a simulation platform to run dynamic control strategies and optimize them to obtain the target control strategy, thereby controlling the hierarchical system architecture to execute storage scheduling.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. The layered architecture construction and strategy generation module aims at system response efficiency and resource balance. It generates basic control strategies adapted to the layered architecture through genetic algorithms, combines historical data access records, and uses a long short-term memory network to build a data access trend prediction model. After pre-adjusting the basic strategy, it continuously optimizes it through deep reinforcement learning to form a dynamic control strategy. This module breaks through the limitation of traditional storage architectures that cannot adapt to the dynamic load of large model training. It can not only ensure the stable operation baseline of the storage system through basic control strategies, but also capture the data access patterns of different training stages in real time through adaptive learning, pre-adjust resource allocation parameters and dynamically optimize them, so that storage scheduling always matches the training load demand, reduces high-frequency data access latency, avoids resource idleness or overload, and improves the adaptability of the storage system to the dynamic changes in the demand of large model training. 2. This invention forms a closed-loop technology through the coordinated operation of four modules in a system control center: "data requirement modeling - storage unit generation - hierarchical architecture and dynamic strategy construction - strategy verification and optimization." The model construction module collects data types, access frequencies, and security requirements from multiple dimensions, dynamically adjusting parameters based on training scenarios. This avoids the single-minded focus on capacity found in traditional storage, providing precise data support for subsequent stages. The storage module generation module, based on the requirement model, ensures that the hardware performance of the storage units matches the training requirements through multi-scenario performance testing and iterative correction, solving the problem of blind hardware configuration in traditional systems. The hierarchical architecture utilizes storage media with different performance levels, balancing performance and cost. A two-layer control strategy enables dynamic evolution of storage scheduling, adapting to load changes throughout the training process. The strategy verification module, through digital twin simulation and multi-scenario optimization, proactively mitigates strategy deployment risks, ensuring stable system operation. This invention addresses the shortcomings of traditional storage architectures, such as static resource allocation and single-strategy control, improving the storage efficiency and scheduling flexibility of large model training data, reducing data access latency and storage resource waste. Simultaneously, it ensures data security through sensitive data hierarchical protection, providing storage support for large model training. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1A flowchart of Embodiment 1 provided by the present invention; Figure 2 The flowchart is for Embodiment 2 of the present invention. Detailed Implementation

[0018] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1 During large-scale model training, the storage and efficient scheduling of massive amounts of heterogeneous data place extremely high demands on system performance. Traditional storage architectures, due to static resource allocation and single-policy control, are difficult to adapt to the dynamically changing load demands during the training phase. Therefore, one embodiment of this invention provides a data storage device to support large-scale model training. For details, please refer to the appendix to the specification. Figure 1 It includes the system control center, which has communication connections to: the model building module, the storage module generation module, the layered architecture building and strategy generation module, and the strategy verification module.

[0021] The model building module is used to obtain the data types, access frequencies, and security requirements parameters in the training of large models, and to build a data storage requirement model through calculation.

[0022] In one embodiment, constructing a data storage requirement model includes the following steps: First, obtain the proportions of structured, semi-structured, and unstructured data used in the training of the large model, compare them with the preset data storage standards for each type, and determine the priority storage weight of each type of data based on the deviation value. Structured data includes data with fixed formats such as training logs and model parameter tables; semi-structured data includes JSON configuration files, XML labeled data, etc.; unstructured data includes training materials such as images, audio, and long text. By using data classification tools to statistically analyze the training data, if unstructured image data accounts for 70% in the training of a large computer vision model, exceeding the preset standard of 20%, then its priority storage weight is increased from 0.5 to 0.7 according to the deviation coefficient method; if structured data accounts for 10%, which is lower than the standard of 10%, its weight is reduced to 0.3.

[0023] Secondly, the access frequency of data in different training stages is statistically analyzed. Data with high access frequency is marked as high-frequency access data, and data with low access frequency is marked as low-frequency access data. The storage resource allocation ratio is calculated according to the ratio of high frequency to low frequency. The sliding window algorithm is used to statistically analyze the data access frequency within 24 hours. More than 10 accesses per minute are defined as high-frequency range, and less than 1 access per minute is defined as low-frequency range.

[0024] Then, security requirement parameters are determined based on data sensitivity levels, which include public, internal, and confidential levels. The source identifiers, usage scope, and encryption requirements of various types of data are obtained, and a data sensitivity level assessment index system is established. The Analytic Hierarchy Process (AHP) was used to assign weights to each indicator in the indicator system, and the comprehensive data sensitivity score was calculated. The overall data sensitivity score is compared with the preset sensitivity level threshold to determine the sensitivity level of the data. Based on different sensitivity levels, corresponding security storage policy parameters are configured, including encryption algorithm strength, access permission levels, and data transmission encryption requirements, forming security requirement parameters. For example, training data containing user privacy information, after evaluation, scores 90 points and is classified as confidential, so AES-256 encryption algorithm and multi-factor authentication are configured; public datasets score 30 points, use the SM4 standard encryption algorithm, and allow anonymous access.

[0025] Finally, the priority storage weights, storage resource allocation ratios, and security requirement parameters are integrated into a multi-dimensional feature set. Through statistical analysis and training scenario matching tests, the parameter weights are adjusted to construct a data storage requirement model. The K-means clustering algorithm is used to divide the feature set into three scenarios: high, medium, and low storage requirements. In natural language processing training scenarios, semi-structured text data has a higher access frequency, so its priority storage weight is adjusted from 0.4 to 0.6. In multimodal training scenarios, unstructured data has a larger capacity requirement, so the proportion of low-frequency storage resources is increased from 10% to 15%, ensuring the model adapts to different training scenarios.

[0026] Specifically, for the priority storage weight calculation of different types of data, taking a large model training project as an example: although the model parameter table accounts for only 5% of the structured data, it directly affects the training iteration efficiency, so it is given an additional correction weight of 0.2, and the final weight is 0.3 + 0.2 = 0.5; when counting access frequency, the mean μ and standard deviation σ of the access frequency are calculated, and access data exceeding the range of [μ-3σ, μ+3σ] are identified as outliers and removed. For example, 50 access data per minute generated by engineers debugging during a certain period were identified and removed to ensure the accuracy of the statistical results; in the configuration of security requirements parameters, the access permissions for internal data such as unpublished model structure documents are limited to training team members, and the log of each access operation is recorded for easy traceability.

[0027] The model building module collects data on data types, access frequencies, and security requirements from multiple dimensions, avoiding the single-minded focus on capacity found in traditional storage. By dynamically adjusting parameter weights in conjunction with training scenarios, the storage requirement model is made to better match actual training needs, laying a data foundation for subsequent storage unit generation and strategy formulation.

[0028] The storage module generation module is used to determine storage indicators based on the data storage requirement model and generate preliminary storage units; after performance testing in multiple scenarios, it is corrected to generate the final storage units.

[0029] In one embodiment, generating a preliminary storage unit includes the following steps: The core storage metrics extracted from the data storage requirement model are storage capacity requirement, read / write speed requirement, data reliability requirement, and scalability requirement. Storage capacity requirement is calculated using the formula: "Total capacity = (Current data volume × (1 + Monthly growth rate × Reserved months)) × 1.3". A 30% redundancy space is reserved to handle data growth and backup needs. For example, if the current training data volume is 50TB, the monthly growth rate is 10%, and 6 months are reserved, the total capacity requirement is 50 × (1 + 0.1 × 6) × 1.3 = 104TB. Read / write speed requirement is determined based on the training batch size. When processing 10GB of data per batch, a continuous read speed of 2GB per second must be guaranteed. Data reliability requirement requires an annual failure rate of less than 0.5%. Scalability requirement supports single-node capacity expansion to 20TB without service interruption during the expansion process.

[0030] It is worth noting that the total capacity of the media configuration is 210TB, of which 104TB is the basic requirement (including 30% redundancy), and the remaining 106TB is reserved for expansion to cope with the growth of training data in the next 6-12 months.

[0031] The appropriate storage media types are selected based on core storage metrics, including solid-state drives (including NVMe SSDs and SATA SSDs), hard disk drives, and distributed storage nodes.

[0032] Among them, the NVMeSSD has a random read latency of 20-80μs and an IOPS of 500,000-1,500,000, making it suitable for high-frequency data storage; SATASSD latency 50-150μs, IOPS 50000-100000, used for mid-frequency data; Mechanical hard drives have a latency of 5-20ms and IOPS of 50-200, making them suitable for low-frequency data.

[0033] Distributed storage nodes provide tens of gigabytes per second of throughput through multiple nodes operating in parallel, supporting large-scale concurrent access. Based on this, NVMe SSDs are selected for high-frequency data, SATA SSDs for mid-frequency data, and mechanical hard drives for low-frequency data, supplementing the distributed storage nodes to meet concurrent demands.

[0034] Based on the priority storage weights of various types of data and the storage resource allocation ratio, the configuration quantity and capacity allocation scheme of different storage media are calculated.

[0035] Specifically, high-frequency data accounts for 30%, with a priority storage weight of 0.7, configured with 5 x 2TB NVMe SSDs, for a total capacity of 10TB; mid-frequency data accounts for 20%, configured with 10 x 4TB SATA SSDs, for a total capacity of 40TB; and low-frequency data accounts for 50%, configured with 20 x 8TB HDDs, for a total capacity of 160TB. Considering RAID configuration losses, the capacity utilization rate when using RAID5 is (n-1) / n (where n is the number of disks in a single group, which must be ≥3). In this embodiment, the 20 HDDs are divided into 3 groups (6 disks / 6 disks / 8 disks), each group independently configured with RAID5, resulting in a total usable capacity of (5×8+5×8+7×8) = 136TB. An additional 8TB is reserved to handle single-disk failure rebuilding needs, resulting in a final usable capacity of 128TB. This extra space is reserved when calculating the configuration quantity.

[0036] According to the capacity allocation scheme, the selected storage media are logically combined to form preliminary storage units. Using LVM storage management software, 5 NVMe SSDs are combined into a "high-frequency storage pool" and a logical volume is allocated for storing sample feature vectors; 10 SATA SSDs are combined into a "mid-frequency storage pool" and a logical volume is allocated for storing training configuration files; 20 mechanical hard drives are combined into a "low-frequency storage pool" and a logical volume is allocated for storing original sample backups, thus achieving partitioned storage of different types of data.

[0037] In one embodiment, generating the final storage unit includes the following steps: Multiple training scenario simulation environments were constructed, including batch data loading, high-frequency random access, and concurrent write scenarios. The batch data loading scenario simulates the initial data import, loading a 100GB mixed dataset in a single run; the high-frequency random access scenario simulates parameter queries during training, initiating 1000 random requests per second; and the concurrent write scenario simulates writing intermediate training results across multiple nodes, with 10 nodes writing simultaneously at a single node speed of 500MB / s.

[0038] Initial storage units were run in various simulation environments, and tools such as iostat and fio were used to collect storage response time, data transfer rate, and error rate as performance test data. Response time was recorded as average latency and 95th percentile latency; data transfer rate was calculated as a 5-minute average; and error rate was recorded as CRC check errors and IO timeouts. The sampling rate was set to 10kHz to ensure data accuracy.

[0039] Analysis of variance was used to process the performance test data and identify performance bottleneck parameters. The between-group and within-group variances of performance indicators under different scenarios were calculated, and the significance of the differences was judged by the F test. If the 95th percentile latency in the high-frequency random access scenario reached 200ms, which far exceeded the preset threshold of 50ms, and the F value of 12.8 was greater than the critical value of 3.48, the insufficient number of NVMeSSDs was identified as the main bottleneck.

[0040] Based on performance bottleneck parameters, the storage media type and capacity allocation scheme in the initial storage unit were adjusted to obtain the modified storage unit. To address the insufficient NVMeSSD issue, three 2TB NVMeSSDs were added and the RAID configuration was changed from RAID0 to RAID10; to address the insufficient concurrent write bandwidth, two distributed storage nodes were added and the transmission protocol was optimized to RDMA.

[0041] Repeat the performance testing steps until the performance test data of the corrected storage unit in each simulation environment meets the preset performance standards, and then output it as the final storage unit.

[0042] Specifically, during performance testing, the initial transfer rate for batch data loading scenarios was only 800MB / s, lower than the preset standard of 1GB / s. Analysis revealed that the mechanical hard drive's read and write speed was insufficient. Some low-frequency data was migrated to the SATA SSD, and the number of data loading threads was optimized, ultimately increasing the speed to 1.2GB / s. During error rate monitoring, a mechanical hard drive exhibited a CRC check error. It was promptly replaced, and the RAID configuration was adjusted to ensure data reliability.

[0043] By generating storage modules, from extracting core metrics to optimizing performance across multiple scenarios, every step is based on the data storage requirement model, avoiding the blind spots of traditional storage hardware configuration. Iterative performance testing and correction ensure that the final storage unit can stably handle various load scenarios in large model training, providing reliable hardware support for data storage.

[0044] The layered architecture construction and strategy generation module is used to build a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, it generates basic control strategies through dynamic resource allocation, introduces adaptive learning by combining historical data access records, and updates and optimizes resource allocation parameters to form dynamic control strategies.

[0045] In one embodiment, constructing a layered system architecture includes the following steps: Based on data access frequency and real-time requirements, the fastest-responding storage medium in the final storage unit is designated as a data cache subsystem for storing frequently accessed data. This subsystem is constructed using PCIe 4.0 NVMe SSDs with a total capacity of 16TB and latency controlled within 50μs. It stores data accessed more than 10 times per minute in the last 24 hours, such as sample features and intermediate parameters of the current training batch.

[0046] The high-capacity and highly stable storage media is divided into a core storage subsystem for storing the core datasets used for large model training. A hybrid architecture of SATA SSDs and distributed storage is adopted, with a total capacity of 50TB. SATA SSDs provide support for medium- and high-frequency access, while three distributed storage nodes (20TB each) provide capacity expansion, storing the pre-trained corpus, feature library, and model checkpoint files. A multi-replica mechanism (3 replicas) ensures data reliability, meeting the 99.99% availability requirement.

[0047] A backup storage subsystem is constructed using storage media with redundant backup capabilities for disaster recovery backup of core data. A RAID 5 array is built using ten 16TB hard drives, with a total capacity of 144TB. Combined with an off-site backup solution (backup nodes are more than 10km away from the master node), it supports hourly incremental backups and daily full backups, with a 30-day data retention period to meet data recovery needs under extreme failure scenarios.

[0048] A high-speed data bus establishes communication connections between the three subsystems, configuring data synchronization mechanisms and access priority rules to form a complete layered system architecture. The data caching subsystem and the core storage subsystem are connected via a PCIe 4.0 bus with a bandwidth of 8GB / s; the core storage subsystem and the backup storage subsystem are connected via an InfiniBand network with a bandwidth of 200GB / s, with redundant links configured to avoid single points of failure. For data synchronization, the cache and core storage perform real-time incremental synchronization, while the core storage and backup storage perform periodic synchronization. Access priority rules are set with a weight of 0.6 for the cache subsystem, 0.3 for the core storage, and 0.1 for the backup storage, prioritizing responses to higher-priority requests.

[0049] In one embodiment, generating a basic control policy includes the following steps: The input parameters are the real-time load rate, data access request volume, and remaining storage space of each subsystem. The real-time load rate includes CPU utilization and I / O utilization, the data access request volume includes requests per second and average request size, and the remaining storage space is presented as a percentage.

[0050] The optimization objectives are to maximize system response efficiency and balance resource utilization. A mathematical model for resource allocation is constructed. The objective function is Maximize(Σ(number of requests processed / response time)) - λ × Var(resource utilization), where λ is the balancing coefficient with a value of 0.3. The constraints include: each subsystem having ≥10% remaining storage space, ≤90% IO utilization, and ≤1 second data synchronization latency.

[0051] A genetic algorithm was used to solve the mathematical model of resource allocation, resulting in a data caching strategy, storage resource allocation scheme, and data migration rules. The algorithm parameters were set to a population size of 50, an iteration count of 100, a crossover probability of 0.8, and a mutation probability of 0.1. Through selection, crossover, and mutation operations, the following were obtained: an LRU cache eviction strategy (eviction triggered when space utilization reaches 75%), an IO resource allocation scheme (40% cache, 50% core, 10% backup), and data migration rules (core storage data accessed more than 10 times / minute for 3 consecutive hours is migrated to the cache; cached data not accessed for 24 hours is migrated to the core).

[0052] Data caching strategies, storage resource allocation schemes, and data migration rules are integrated into a basic control strategy. This basic control strategy is then run within a layered system architecture. Data on the strategy execution results is collected, compared with preset optimization targets, and the parameters in the basic control strategy are fine-tuned.

[0053] In one embodiment, forming a dynamic control strategy includes the following steps: Historical data access records from different training phases were collected, including access time, data type, and access frequency characteristics. A time-series dataset was constructed using the ELKStack log collection tool. The data was preprocessed to remove abnormal records with access latency exceeding 1000ms, and missing values ​​were filled in using linear interpolation. The access frequency was standardized to the 0-1 range.

[0054] A data access trend prediction model is constructed by training a Long Short-Term Memory (LSTM) network on historical data access records. The model structure is as follows: input layer (feature dimension 8) → 3-layer LSTM unit (hidden layer dimension 64) → fully connected layer (output dimension 4). The Adam optimizer (learning rate 0.001), MAE loss function, and 50 iterations are used. Early stopping is employed to avoid overfitting.

[0055] It is worth noting that the input layer features have 8 dimensions, specifically including: access time (hours), data type (one-hot encoding 3-dimensional), historical access frequency (last 1 hour / 6 hours / 12 hours), data size (after standardization), and number of access nodes, for a total of 8 dimensions.

[0056] The data access trend prediction model is used to predict data access demand for future periods, and the prediction results are obtained.

[0057] Based on the prediction results, the resource allocation parameters in the basic control strategy are pre-adjusted to obtain the pre-adjustment strategy. The pre-adjustment strategy is deployed in the hierarchical system architecture, the execution effect of the strategy is monitored in real time, and the pre-adjustment strategy is continuously optimized by deep reinforcement learning algorithm to form a dynamic control strategy.

[0058] Through the layered architecture construction and policy generation module, the layered architecture classifies and utilizes storage media with different performance levels, balancing performance and cost. The evolution from basic policies to dynamic policies enables storage scheduling to adapt to the changing needs of different training stages of large models, avoiding the limitations of traditional fixed policies and improving the operating efficiency and adaptability of the storage system.

[0059] The strategy verification module is used to build dynamic control strategies for the simulation platform and optimize them to obtain the target control strategy, thereby controlling the hierarchical system architecture to perform storage scheduling.

[0060] In one embodiment, obtaining the target control strategy includes the following steps: A layered system architecture simulation platform is constructed using digital twin technology. The platform comprises a storage device simulation layer, a data interaction simulation layer, and a policy execution simulation layer. The storage device simulation layer uses Flexsim to build a virtual model, simulating the performance characteristics of NVMe SSDs (latency 20-80μs, IOPS 1,000,000) and mechanical hard drives (seek time 5-10ms), and adds a fault injection module to simulate IO timeouts and disk failures. The data interaction simulation layer uses OMNeT++ to build a bus model, simulating the communication characteristics of PCIe 4.0 (bandwidth 8GB / s, latency 1-2μs) and InfiniBand (concurrency 1000+ nodes), and uses a flow control module to simulate peak transmission speeds. The policy execution simulation layer uses Python to translate dynamic control policies into code logic, and adds a metric acquisition module to record data such as response time and resource utilization.

[0061] Various large-scale model training scenarios were imported into the simulation platform, dynamic control strategies were implemented, and system response time, resource utilization, and data security metrics were collected. The natural language processing pre-training scenario input a 10TB text dataset, trained for 72 hours, and tested high-concurrency small file access. The computer vision fine-tuning scenario input a 5TB image dataset, trained for 24 hours, and tested peak large file read / write and access times. The multimodal training scenario input an 8TB mixed dataset, trained for 48 hours, and tested heterogeneous data access. Collected metrics included average latency of the caching subsystem (target <100μs), core storage utilization (target 60%–70%), and backup integrity (target 100%).

[0062] The collected metrics are compared with preset thresholds to determine whether the dynamic control strategy meets the requirements. If the 95th percentile latency of the caching subsystem in the multimodal training scenario reaches 2.5ms, exceeding the 2ms threshold, analysis reveals that mixed access to heterogeneous data leads to a low cache hit rate (75%), which is identified as a weak link in the "caching strategy does not distinguish data types". If the backup integrity is only 98%, it is located as a problem of "small file loss due to unprocessed backup synchronization".

[0063] When the metric fails to reach the preset threshold, weak links in the strategy are identified, and particle swarm optimization (PSO) is used to adjust the dynamic control strategy. Parameters such as cache eviction threshold and migration trigger frequency are used as particle position vectors, with the "metric achievement rate" as the fitness function. The particle count is set to 30, the inertia weight to 0.7, and the learning factor to 1.5, with 50 iterations. To address cache hit issues, the eviction threshold is adjusted from 70% to 65%, and a "set cache priority by data type" rule is added. To address backup issues, the synchronization protocol is optimized, and a small file merging and transmission mechanism is added.

[0064] Repeat the strategy execution and optimization steps until all indicators in the simulation platform reach the preset thresholds, and output the final dynamic control strategy as the target control strategy.

[0065] The strategy verification module enables the testing and optimization of dynamic control strategies: the digital twin simulation platform recreates the actual storage environment, and multi-scenario verification and iterative optimization ensure that the target control strategy can cope with various complex situations in large model training, providing a reliable guarantee for the stable operation of the storage system.

[0066] This invention forms a closed-loop technology encompassing four modules through a system control center: data requirement modeling, storage unit generation, hierarchical architecture and dynamic strategy construction, and strategy verification and optimization. The model construction module collects data types, access frequencies, and security requirements from multiple dimensions, dynamically adjusting parameters based on training scenarios. This avoids the single-minded focus on capacity found in traditional storage, providing precise data support for subsequent stages. The storage module generation module, based on the requirement model, ensures that the hardware performance of storage units matches training requirements through multi-scenario performance testing and iterative correction, resolving the problem of blind hardware configuration in traditional systems. The hierarchical architecture utilizes storage media with different performance levels, balancing performance and cost. A two-layer control strategy enables dynamic evolution of storage scheduling, adapting to load changes throughout the training process. The strategy verification module, through digital twin simulation and multi-scenario optimization, proactively mitigates strategy deployment risks, ensuring stable system operation. This invention addresses the shortcomings of traditional storage architectures with static resource allocation and single-strategy control, improving storage efficiency and scheduling flexibility for large model training data, reducing data access latency and storage resource waste. Simultaneously, it ensures data security through sensitive data tiered protection, providing storage support for large model training.

[0067] Example 2 See attached document Figure 2 The method of using a data storage device to support the training of large models according to this embodiment includes the following steps: S1: Obtain the data types, access frequencies, and security requirements parameters in the training of large models, and construct a data storage requirement model through calculation; S2: Based on the data storage requirement model, determine storage metrics and generate preliminary storage units; after performance testing in multiple scenarios, refine and generate the final storage units. S3: Construct a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, generate basic control strategies through dynamic resource allocation, introduce adaptive learning by combining historical data access records, and update and optimize resource allocation parameters to form dynamic control strategies. S4: Construct a simulation platform to run dynamic control strategies and optimize them to obtain the target control strategy, thereby controlling the hierarchical system architecture to execute storage scheduling.

[0068] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data storage device for supporting large model training, comprising a system control center, characterized in that, The system control center has the following communication connections: The model building module is used to obtain the data types, access frequencies, and security requirements parameters in the training of large models, and to build a data storage requirement model through calculation. Constructing the data storage requirement model specifically includes: Obtain the proportions of structured, semi-structured, and unstructured data used in large model training, compare them with preset data storage standards for each type, and determine the priority storage weights for each type of data based on the deviation values. The access frequency of data in different training stages is statistically analyzed. Data with high access frequency is marked as high-frequency access data, and data with low access frequency is marked as low-frequency access data. The storage resource allocation ratio is calculated based on the ratio of high-frequency to low-frequency data. Security requirement parameters are determined based on data sensitivity levels, which include public, internal, and confidential levels. The priority storage weight, storage resource allocation ratio and security requirement parameters are integrated into a multi-dimensional feature set. Through statistical analysis and training scenario matching tests, the parameter weights are adjusted to construct a data storage requirement model. The storage module generation module is used to determine storage indicators based on the data storage requirement model and generate preliminary storage units; after multi-scenario performance testing, it is corrected to generate final storage units. The layered architecture construction and strategy generation module is used to construct a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, it generates basic control strategies through dynamic resource allocation, introduces adaptive learning by combining historical data access records, and updates and optimizes resource allocation parameters to form dynamic control strategies. The generation of the basic control strategy specifically includes: The real-time load rate, data access request volume, and remaining storage space of each subsystem are used as input parameters. A mathematical model for resource allocation is constructed with the optimization objectives of maximizing system response efficiency and balancing resource utilization. A genetic algorithm is used to solve the resource allocation mathematical model to obtain the data caching strategy, storage resource allocation scheme and data migration rules; The data caching strategy, storage resource allocation scheme, and data migration rules are integrated into a basic control strategy. The basic control strategy is run in the layered system architecture, the strategy execution effect data is collected, compared with the preset optimization target, and the parameters in the basic control strategy are fine-tuned. The dynamic control strategy is specifically formed by: Collect historical data access records from different training phases, including access time, data type, and access frequency characteristics; A data access trend prediction model is constructed by training the historical data access records using a long short-term memory network. The data access trend prediction model is used to predict data access demand in future time periods, and the prediction results are obtained. Based on the prediction results, the resource allocation parameters in the basic control strategy are pre-adjusted to obtain the pre-adjustment strategy. The pre-adjustment strategy is deployed in the hierarchical system architecture, the execution effect of the strategy is monitored in real time, and the pre-adjustment strategy is continuously optimized by reinforcement learning algorithm to form a dynamic control strategy. The strategy verification module is used to build a simulation platform to run the dynamic control strategy, optimize the target control strategy, and control the hierarchical system architecture to perform storage scheduling.

2. The data storage device for supporting large model training according to claim 1, characterized in that: The determination of security requirement parameters based on data sensitivity levels specifically includes: Obtain the source identification, scope of use, and encryption requirements of various types of data, and establish a data sensitivity level assessment index system; The Analytic Hierarchy Process (AHP) was used to assign weights to each indicator in the data sensitivity assessment index system, and the comprehensive data sensitivity score was calculated. The overall data sensitivity score is compared with the preset sensitivity level threshold to determine the sensitivity level of the data. Configure corresponding security storage policy parameters according to different sensitivity levels, including encryption algorithm strength, access permission level and data transmission encryption requirements, to form security requirement parameters.

3. The data storage device for supporting large model training according to claim 2, characterized in that: The generation of the initial storage unit specifically includes: The storage capacity requirement, read / write speed requirement, data reliability requirement, and scalability requirement are extracted from the data storage requirement model as core storage indicators. Based on the core storage metrics, suitable storage media types are selected, including solid-state drives, hard disk drives, and distributed storage nodes; Based on the priority storage weights of various types of data and the storage resource allocation ratio, calculate the configuration quantity and capacity allocation scheme of different storage media. The selected storage media are logically combined according to the capacity allocation scheme to form preliminary storage units.

4. A data storage device for supporting large model training according to claim 3, characterized in that: The generation of the final storage unit specifically includes: Construct simulation environments for various training scenarios, including batch data loading scenarios, high-frequency random access scenarios, and concurrent write scenarios; Run the initial storage unit in various simulation environments and collect storage response time, data transfer rate and error rate as performance test data; Analysis of variance was used to process the performance test data and identify performance bottleneck parameters. Based on the aforementioned performance bottleneck parameters, the storage medium type and capacity allocation scheme in the initial storage unit are adjusted to obtain the modified storage unit; Repeat the performance testing steps until the performance test data of the corrected storage unit in each simulation environment meets the preset performance standards, and then output it as the final storage unit.

5. A data storage device for supporting large model training according to claim 4, characterized in that: The hierarchical system architecture constructed based on the final storage unit, consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem, specifically includes: Based on the data access frequency and real-time requirements, the storage medium with the fastest response speed in the final storage unit is divided into a data cache subsystem, which is used to store frequently accessed data. The storage medium with large capacity and high stability is divided into a core storage subsystem, which is used to store the core dataset for training large models. Select storage media with redundant backup capabilities to build a backup storage subsystem for disaster recovery backup of core data; A high-speed data bus is used to establish communication connections between the three subsystems, configure data synchronization mechanisms and access priority rules, and form a complete layered system architecture.

6. A data storage device for supporting large model training according to claim 5, characterized in that: The specific steps to obtain the target control strategy include: A simulation platform for the layered system architecture is constructed using digital twin technology. The simulation platform includes a storage device simulation layer, a data interaction simulation layer, and a policy execution simulation layer. The simulation platform is used to import various large-scale model training scenarios, run dynamic control strategies, and collect system response time, resource utilization and data security indicators. The collected indicators are compared with preset thresholds to determine whether the dynamic control strategy meets the requirements. When the indicator fails to reach the preset threshold, the weak link in the strategy is identified, and the particle swarm optimization algorithm is used to adjust the dynamic control strategy. Repeat the strategy execution and optimization steps until all indicators in the simulation platform reach the preset thresholds, and output the final dynamic control strategy as the target control strategy.

7. A method of using a data storage device for supporting large model training according to claim 1, characterized in that, Includes the following steps: S1: Obtain the data types, access frequencies, and security requirements parameters in the training of large models, and construct a data storage requirement model through calculation; S2: Determine storage indicators based on the data storage requirement model and generate preliminary storage units; The final storage unit was generated after performance testing and adjustments in multiple scenarios. S3: Construct a layered system architecture consisting of a data caching subsystem, a core storage subsystem, and a backup storage subsystem based on the final storage unit. At the same time, generate a basic control strategy through dynamic resource allocation, introduce adaptive learning by combining historical data access records, and update and optimize resource allocation parameters to form a dynamic control strategy. S4: Build a simulation platform to run the dynamic control strategy and optimize it to obtain the target control strategy, and control the hierarchical system architecture to perform storage scheduling.

Images