Storage resource scheduling method, device, storage medium and electronic device

Abstract

The present specification discloses a storage resource scheduling method, device, storage medium and electronic device. The method can execute the training task for the target model through a pre-built data storage system including multiple levels of layers. In the process of executing the training task for the target model, the method can schedule and manage different heterogeneous storage resources arranged in different layers based on the predetermined data quality parameters of different training samples and the target model's requirements for training sample data with different data qualities under different state parameters of different training cycles, so as to improve the data access efficiency in the horizontal direction, thereby comprehensively improving the data access efficiency from both horizontal and vertical aspects, and further improving the model training efficiency.

Description

Technical Field

[0001] The present specification relates to the technical field of computer system structure, and in particular to a storage resource scheduling method, device, storage medium and electronic device. Background Art

[0002] In the field of artificial intelligence (AI), big models have become a focal technology and an important driving force for industrial revolution. However, the performance of big models mainly depends on the huge number of big model parameters and the size of training data sets. As the number of big model parameters and the size of training data sets continue to grow, the requirements for storage resources for big model training have also increased significantly.

[0003] However, the storage technology currently used for cloud computing, high-performance computing and other scenarios has not been optimized for the characteristics of large model training. As a result, it is impossible to fully coordinate software and hardware resources during the training of large models, which reduces the efficiency of model training, and further causes problems such as long model training time, high energy consumption, and high cost.

[0004] Therefore, how to improve the efficiency of model training is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a storage resource scheduling method, device, storage medium and electronic device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] A storage resource scheduling method, the method is applied to a data storage system, the data storage system includes various heterogeneous storage resources, the various heterogeneous storage resources include: a distributed memory cache module and a persistent cache module arranged in a computing node, and a parallel file system arranged in a storage node;

[0008] In the process of executing the training task for the target model, the state parameters of the target model in the current training cycle are obtained, and the performance parameters of each preset heterogeneous storage resource are determined as reference performance parameters;

[0009] According to the preset demand relationship between the target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model, the training sample data required by the target model in the next training cycle are determined as the target training sample data;

[0010] Determine, according to the reference performance parameters, a storage relationship between the target training sample data in each heterogeneous storage resource, and determine an estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship;

[0011] With minimizing the estimated remaining training time as the optimization goal, the storage relationship is adjusted to obtain an adjusted storage relationship, and the heterogeneous storage resources are scheduled according to the adjusted storage relationship to continue executing the training task through the scheduled data storage system.

[0012] Optionally, the data storage system is provided with various computing nodes, and the training task for the target model includes: various sub-training tasks;

[0013] Determining the storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter specifically includes:

[0014] For each computing node, determining a sub-training task among the sub-training tasks that is assigned to the computing node for execution as a target sub-training task;

[0015] Determine the target training sample data required for executing the target sub-training task contained in each target training sample data as the target training sample data corresponding to the computing node;

[0016] According to the target training sample data corresponding to each computing node and the reference performance parameters, the storage relationship of the target training sample data in each heterogeneous storage resource set in each computing node is determined.

[0017] Optionally, determining the storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter specifically includes:

[0018] For each target training sample data, determining the similarity between the target training sample data and other target training sample data;

[0019] According to the similarity between each target training sample data and other target training sample data, the target training sample data are divided into target training sample data sets, wherein, for each target training sample data set, the similarity between the target training sample data contained in the target training sample data set is higher than a preset threshold;

[0020] According to the reference performance parameters, the storage relationship of the target training sample data sets in each heterogeneous storage resource is determined.

[0021] Optionally, the distributed memory cache module includes at least one high-performance storage device, wherein when the distributed memory cache module includes more than one high-performance storage device, the high-performance storage devices in the distributed memory cache module are interconnected at high speed, the persistent cache module includes at least one non-volatile storage device, and the parallel file system includes at least one large-capacity storage device;

[0022] Determine the preset performance parameters of each heterogeneous storage resource as reference performance parameters, including:

[0023] The performance parameters of the high-performance memory device, the performance parameters of the non-volatile memory device, and the performance parameters of the large-capacity memory device are determined as reference performance parameters.

[0024] Optionally, according to the reference performance parameters, determining the storage relationship of the target training sample data in each heterogeneous storage resource, and determining the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship, specifically includes:

[0025] Determining, according to the reference performance parameters, a storage relationship of the target training sample data in each heterogeneous storage resource; and

[0026] Determining the storage relationship of the model parameters of the target model in each heterogeneous storage resource according to the reference performance parameters and the preset dependency relationship between the model parameters of the target model in different training cycles;

[0027] Determine the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship of the target training sample data in each heterogeneous storage resource and the storage relationship of the model parameters of the target model in each heterogeneous storage resource.

[0028] Optionally, scheduling the heterogeneous storage resources according to the adjusted storage relationship specifically includes:

[0029] Determining configuration parameters of each heterogeneous storage resource according to the adjusted storage relationship;

[0030] For each heterogeneous storage resource, the heterogeneous storage resource is configured according to the configuration parameters of the heterogeneous storage resource to obtain the configured heterogeneous storage resource corresponding to the heterogeneous storage resource, and according to the adjusted storage relationship, the data in the heterogeneous storage resource is unloaded to other heterogeneous storage resources or data is read from other heterogeneous storage resources to the heterogeneous storage resource.

[0031] Optionally, the persistent cache module is further provided with a data generation model;

[0032] Continuing to execute the training task through the scheduled data storage system specifically includes:

[0033] In the process of continuing to execute the training task through the scheduled data storage system, if it is determined that the number of iterations of the target model does not exceed a preset number threshold, then obtaining the used training sample data in the training sample set corresponding to the target model, and adjusting the model parameters of the data generation model according to the used training sample data to obtain an adjusted data generation model;

[0034] If it is determined that the number of iterations of the target model exceeds a preset threshold, then in the process of responding to the training sample data acquisition request sent by the computing node, if it is determined that there is no data matching the training sample data acquisition request in the distributed memory cache module and the persistent cache module, the training sample data acquisition request is input into the adjusted data generation model to generate supplementary training sample data through the adjusted data generation model, and the training task for the target model is performed based on the supplementary training sample data.

[0035] This specification provides a data storage device, the device comprising:

[0036] An acquisition module is used to obtain the state parameters of the target model in the current training cycle during the execution of the training task for the target model, and determine the performance parameters of each preset heterogeneous storage resource as reference performance parameters;

[0037] A first determination module is used to determine the training sample data required by the target model in the next training cycle as each target training sample data according to the preset demand relationship between the target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model;

[0038] A second determination module is used to determine the storage relationship of each target training sample data in each heterogeneous storage resource according to the reference performance parameters, and determine the estimated remaining training time of the target model after scheduling each heterogeneous storage resource according to the storage relationship;

[0039] An adjustment module is used to adjust the storage relationship with the optimization goal of minimizing the estimated remaining training time, obtain an adjusted storage relationship, and schedule the heterogeneous storage resources according to the adjusted storage relationship to continue executing the training task through the scheduled data storage system.

[0040] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above storage resource scheduling method is implemented.

[0041] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned storage resource scheduling method when executing the program.

[0042] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0043] In the method provided in the present specification, in the process of executing the training task for the target model, the state parameters of the target model in the current training cycle are obtained, and the preset performance parameters of each heterogeneous storage resource are determined as the reference performance parameters. According to the preset demand relationship between the target model in different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle, the data quality parameters of the unused training sample data in the training sample set corresponding to the target model, the training sample data required by the target model in the next training cycle are determined as the target training sample data. According to the reference performance parameters, the storage relationship of each target training sample data in each heterogeneous storage resource is determined, and the estimated remaining training time of the target model after scheduling each heterogeneous storage resource according to the storage relationship is determined. With minimizing the estimated remaining training time as the optimization goal, the storage relationship is adjusted to obtain the adjusted storage relationship, and each heterogeneous storage resource is scheduled according to the adjusted storage relationship.

[0044] It can be seen from the above method that the training task for the target model can be executed through a pre-built data storage system containing multiple levels of layers. In the process of executing the training task for the target model, based on the predetermined data quality parameters of different training samples and the target model's requirements for training sample data with different data qualities under different state parameters of different training cycles, different heterogeneous storage resources set in different layers can be scheduled and managed to improve the data access efficiency in the horizontal direction, thereby comprehensively improving the data access efficiency from both horizontal and vertical aspects, and further improving the model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0046] Figure 1 A flowchart of a storage resource scheduling method provided in this specification;

[0047] Figure 2 A schematic diagram of a data storage system provided in this specification;

[0048] Figure 3 A schematic diagram of the Bayesian optimization process provided in this specification;

[0049] Figure 4 A schematic diagram of a data storage device provided in this specification;

[0050] Figure 5 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0052] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0053] Figure 1 A flowchart of a storage resource scheduling method provided in this specification includes the following steps:

[0054] S101: In the process of executing a training task for a target model, state parameters of the target model in a current training cycle are obtained, and performance parameters of each preset heterogeneous storage resource are determined as reference performance parameters.

[0055] In this specification, the data storage system may include various heterogeneous storage resources, wherein each heterogeneous storage resource includes: a distributed memory cache module and a persistent cache module set in a computing node, a parallel file system set in a storage node, and the like. Figure 2 shown.

[0056] Figure 2A schematic diagram of a data storage system provided in this specification.

[0057] Combination Figure 2 It can be seen that the data storage system can be divided into three layers of storage modules, among which the first layer of storage modules is the distributed memory cache module, which is mainly composed of high-performance storage devices such as dynamic random access memory (DRAM) and high bandwidth memory (HBM). The second layer of storage module is the persistent cache module, which is mainly composed of non-volatile storage devices such as (Non-Volatile Memory Express Solid State Drive, NVMe SSD)). The third layer of storage module is the parallel file system, which is mainly composed of conventional large-capacity storage devices such as Serial ATA Solid State Drive (SATA SSD), Serial Attached SCSI Solid State Drive (SAS SSD), and Hard Disk Drive (HDD).

[0058] It should be noted that the above-mentioned distributed memory cache module and persistent cache module can be set in the computing node, and the above-mentioned parallel file system can be set in the storage node. Since the above-mentioned high-performance memory has the characteristics of high-speed reading and writing performance and high number of input and output operations per second, and the above-mentioned high-performance memory can be set inside the computing node, that is, the high-performance memory and the computing cores such as the central processing unit (CPU) core and the graphics processing unit (GPU) core set in the computing node are on the same device, so that the computing node can read and write data from the distributed memory cache module at high speed, and the same data usually only needs to be stored in one copy, maximizing data sharing and storage space utilization, and high-speed interconnection technology (such as Compute Express) can be used between each high-performance storage device. Link) to build connections, and to form memory pools through high-performance storage devices such as DRAM and HBM, so that high-quality training data, model parameters, intermediate results and other data required for training can be stored in distributed memory caches to reduce the memory usage of a single computing node and reduce the data sharing overhead between multiple computing nodes. It also reduces memory fragmentation through memory optimization and high-quality data reuse, as well as reduces frequent memory allocation and release operations, thereby improving the efficiency of model training.

[0059] However, due to the problem that the data in the above-mentioned high-performance memory cannot be retained in the event of a data storage system failure such as a power outage in the data storage system, and the fact that the computing nodes and storage nodes may be deployed on different devices, when the data storage system reads data from the storage node, it is necessary to transmit data through the local area network, which in turn causes the data storage system to take a long time to retrieve data from the storage node to the distributed memory cache, resulting in the computing node acquiring data more efficiently. Therefore, the above-mentioned data storage system is also provided with the above-mentioned persistent cache module, and the above-mentioned persistent cache module can be used to pre-retain frequently accessed data (such as model parameters, weight matrices, etc.) in the persistent cache module, thereby improving the overall data transmission rate and reducing the number of times data is obtained from the storage node, so as to improve the efficiency of the computing node in acquiring data.

[0060] To sum up, when the computing node performs a training task for the target model, the efficiency of reading data from the storage modules of different layers mentioned above is different, that is, the upper storage module can be used as a cache for the lower storage module, and the storage modules of different layers mentioned above can belong to the same namespace, that is, they maintain a unified directory tree view through the metadata server, and build a unified addressing space between different storage levels, thereby improving the data sharing efficiency of storage modules of different levels.

[0061] Furthermore, the above-mentioned data storage system may also include an acceleration module, where the acceleration module is used to interact with computing nodes and storage nodes. Specifically, when the computing node is performing a training task for the target model, it can send a data acquisition request to the acceleration module, so that the acceleration module obtains data from the distributed memory cache module according to the data acquisition request. If there is no data matching the data acquisition request in the distributed cache module, the acceleration module can determine whether there is data matching the data acquisition request in the persistent cache module. If there is data matching the data acquisition request in the persistent cache module, the data matching the data acquisition request in the persistent cache module can be retrieved into the distributed memory cache module.

[0062] Of course, when the data matching the data acquisition request mentioned above is a large block of data (for example: image data, audio data, video data, etc.), at this time, if the data matching the data acquisition request in the persistent cache module is retrieved to the distributed memory cache module and then transmitted to the computing node, the time and bandwidth consumption required for the process are relatively high. Therefore, the acceleration module can also directly read the data matching the data acquisition request from the persistent cache module and transmit it to the computing node.

[0063] Furthermore, if there is no data matching the data acquisition request in the persistent cache module, at this time, if it is determined that the number of iterations of the target model does not exceed the preset threshold, the data matching the data acquisition request can be obtained from the storage node and transmitted to the computing node, and the model parameters of the data generation model pre-set in the persistent cache module can be fine-tuned based on the data matching the data acquisition request obtained from the storage node to obtain an adjusted data generation model.

[0064] Of course, if it is determined that the number of iterations of the target model exceeds the preset threshold, in order to reduce the access overhead of the parallel file system data and improve the model training efficiency, at this time, the training sample data acquisition request can be input into the adjusted data generation model to generate supplementary training sample data through the adjusted data generation model, and the supplementary training sample data can be transmitted to the computing node to perform the training task for the target model based on the supplementary training sample data.

[0065] Among them, the above-mentioned method of fine-tuning the model parameters of the data generation model pre-set in the persistent cache module according to the data matching the data acquisition request obtained from the storage node to obtain the adjusted data generation model can be, inputting the data acquisition request into the data generation model, so that the data generation model generates candidate supplementary training sample data according to the data acquisition request, and then determining the perplexity corresponding to the candidate supplementary training sample data according to the deviation between the candidate supplementary training sample data and the training sample data actually obtained from the storage node according to the data acquisition request, and taking minimizing the perplexity corresponding to the candidate supplementary training sample data as the optimization goal, fine-tuning the model parameters of the data generation model until the adjustment is stopped when it is determined that the perplexity corresponding to the candidate supplementary training sample data is less than the preset perplexity threshold, so as to obtain the adjusted data generation model.

[0066] In the above content, the deviation between the candidate supplementary training sample data and the training sample data actually obtained from the storage node according to the data acquisition request is positively correlated with the perplexity corresponding to the candidate supplementary training sample data.

[0067] It can be seen from the above content that the data storage system with the above structure can simplify the interaction between computing nodes and data storage systems, that is, for all computing nodes, all data comes from the acceleration module, and for all storage nodes, it is only necessary to serve the data access requests sent by the acceleration module.

[0068] In addition, in actual application scenarios, the data interaction behaviors of the computing nodes in different training cycles for training the target model, such as the training sample data, model parameters, gradients, optimizer states, activation values, etc., that need to be obtained from the data storage system are also different.

[0069] For example: in different training cycles for training the target model, the relationship between the degree of improvement in the training effect of the target model and the training sample data obtained by the computing node from the data storage system is different, that is, in the early stage of the target model training, the target model needs to use a large amount of training sample data, and the quality requirements for the training sample data are low. As the number of training cycles for training the target model increases, the computing node needs to obtain higher-quality training sample data from the data storage system in order to continue to improve the training effect of the target model. Otherwise, lower-quality training sample data may cause the training effect of the target model to decline. At this time, the data storage system can adjust the training sample data stored in each heterogeneous storage resource according to the above rules, that is, for different training cycles of the target model, select the data that has the greatest improvement on the model training effect in the current training cycle and store it in the distributed memory cache module and the persistent cache module to reduce the access overhead of the parallel file system layer data and improve the model training efficiency.

[0070] Another example: when training the target model, the target model has two stages: forward propagation and back propagation. During the forward propagation of the target model, the model's gradient, optimizer state, activation value and other data will not be used. At this time, the data storage system can unload the model's gradient, optimizer state, activation value and other data from the distributed memory cache module to the persistent cache module when it determines that the target model is undergoing forward propagation in the current training cycle, thereby improving the utilization rate of the high-performance cache module and further improving the efficiency of model training.

[0071] Based on this, the business platform can obtain the state parameters of the target model in the current training cycle and determine the preset performance parameters of each heterogeneous storage resource as reference performance parameters during the execution of training tasks for the target model. Then, based on the above-mentioned state parameters of the target model in the current training cycle and the above-mentioned reference performance parameters, the distribution of data required by the computing nodes in the process of executing training tasks for the target model in various heterogeneous storage resources and the configuration parameters of each heterogeneous storage resource can be adjusted in real time to improve the utilization rate of each heterogeneous storage resource in the process of executing training tasks for the target model, thereby improving the training efficiency of the target model.

[0072] Among them, the above-mentioned status parameters can be used to reflect the current training progress of the target model, such as: number of iterations, loss value, accuracy, recall rate, etc.

[0073] The performance parameters of each heterogeneous storage resource mentioned above can be used to reflect the applicability of different heterogeneous storage resources in different application scenarios, such as: DRAM access latency, bandwidth, capacity, persistence, and cost.

[0074] Another example: SSD access latency, bandwidth, capacity, persistence, cost, IOPS (input and output operations per second), durability, etc.

[0075] Another example: NVM access latency, bandwidth, capacity, persistence, cost, IOPS (input and output operations per second), durability, etc.

[0076] In this specification, the execution entity used to implement the storage resource scheduling method can be a designated device such as a server set up on the business platform, or it can be a terminal device set up in the data storage system for managing various heterogeneous storage resources. For the sake of ease of description, this specification only takes the server as the execution entity as an example to illustrate the storage resource scheduling method provided in this specification.

[0077] S102: Determine the training sample data required by the target model in the next training cycle as the target training sample data based on the demand relationship between the preset target models under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model.

[0078] Furthermore, the server can determine the training sample data required by the target model in the next training cycle as the target training sample data based on the demand relationship between the preset target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle, and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model.

[0079] Among them, the demand relationship between the target model under different state parameters and the training sample data with different data quality parameters can be determined in advance based on the rules presented by the data interaction behavior of the computing node obtaining training sample data, model parameters, gradients, optimizer states, activation values ​​and other data from the data storage system during the training of the target model. The data quality parameters here are used to reflect the size of the training sample data, the type of the training sample data, and the contribution of the training sample data to the execution of the training task for the target model.

[0080] For example: in the early stage of training the target model, the target model needs to use a large amount of training sample data, but the quality requirements for the training sample data are relatively low. At this time, when the number of iterations of the target model is determined to be lower than the preset iteration number threshold according to the above-mentioned state parameters, it can be determined that the size of each training sample data required by the target model in the next training cycle needs to be greater than the preset data scale threshold, and the contribution of each training sample data required by the target model in the next training cycle to the execution of the training task for the target model is greater than the preset minimum threshold.

[0081] S103: Determine, based on the reference performance parameters, the storage relationship of the target training sample data in each heterogeneous storage resource, and determine the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship.

[0082] S404: With minimizing the estimated remaining training time as the optimization goal, the storage relationship is adjusted to obtain an adjusted storage relationship, and the heterogeneous storage resources are scheduled according to the adjusted storage relationship to continue executing the training task through the scheduled data storage system.

[0083] Furthermore, the server can determine the storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter, and determine the estimated remaining training time of the target model after scheduling each heterogeneous storage resource according to the storage relationship. Then, the storage relationship can be adjusted with the optimization goal of minimizing the estimated remaining training time of the target model after scheduling each heterogeneous storage resource according to the storage relationship to obtain the adjusted storage relationship, and each heterogeneous storage resource can be scheduled according to the adjusted storage relationship, so as to continue to execute the training task for the next training cycle of the target model through the scheduled data storage system.

[0084] The method for the server to determine the storage relationship of each target training sample data in each heterogeneous storage resource and adjust the above storage relationship according to each reference performance parameter may be that the server determines the storage relationship of each target training sample data in each heterogeneous storage resource and adjusts the above storage relationship according to each reference performance parameter through a Bayesian optimization algorithm, specifically as follows: Figure 3 shown.

[0085] Figure 3 A schematic diagram of the Bayesian optimization process provided in this specification.

[0086] Combination Figure 3It can be seen that the server can determine the initial storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter, and then select a proxy model (such as a Gaussian process) to simulate the impact of resource configuration on model training performance, and can select acquisition functions, such as: Upper Confidence Bound (UCB), Lower Confidence Bound (LCB), Expected Improvement (EI), Probability of Improvement (PI), etc., to calculate the estimated remaining training time of the target model, so that the storage relationship can be adjusted with minimizing the estimated remaining training time as the optimization goal until it is determined that the preset termination goal is met (for example, the number of iterations reaches a preset threshold, or the determined estimated remaining training time is less than a preset threshold, etc.). At this time, the adjusted storage relationship is obtained, and the heterogeneous storage resources are scheduled according to the adjusted storage relationship.

[0087] Specifically, the server can determine the storage relationship of each target training sample data in each heterogeneous storage resource based on each reference performance parameter, and can determine the storage relationship of the model parameters of the target model in each heterogeneous storage resource based on each reference performance parameter and the preset dependency relationship between each model parameter of the target model in different training cycles.

[0088] The above-mentioned dependency relationship between the model parameters of the target model in different training cycles may refer to the dependency relationship of the weight update of the target model in different training cycles.

[0089] For example: In the process of gradient propagation, the update of each parameter depends not only on its own gradient, but also on the gradient of the parameters of the previous network layer and the gradient of the parameters of the next network layer. For example: In a neural network, the gradient of the weight W1 of the lth network layer is not only affected by the weight W2 of the l+1th network layer, but also by the input data and the weight W0 of the previous network layer l-1.

[0090] It should be noted that the server adjusts the storage relationship with minimizing the estimated remaining training time as the optimization goal. The method for obtaining the adjusted storage relationship can be that the server determines the target loss value based on the determined estimated remaining training time, and then adjusts the storage relationship with minimizing the target loss value as the optimization goal to obtain the adjusted storage relationship, and the above target loss value is positively correlated with the determined estimated remaining training time.

[0091] In addition, in actual application scenarios, since the training tasks for the target model are often split into sub-training tasks and executed in parallel by various computing nodes, different computing nodes require different data when executing different sub-training tasks. In addition, since the efficiency of data acquisition across computing nodes is often low, the distance between each computing node and each heterogeneous storage resource needs to be considered when determining the storage relationship of each target training sample data in each heterogeneous storage resource set in each computing node.

[0092] Based on this, the server can also determine, for each computing node, the sub-training tasks assigned to the computing node in each sub-training task as the target sub-training task, and then determine the target training sample data required to execute the target sub-training task in each target training sample data as the target training sample data corresponding to the computing node, so that the storage relationship of each target training sample data in each heterogeneous storage resource set in each computing node can be determined according to the target training sample data corresponding to each computing node and each reference performance parameter.

[0093] Specifically, the server can determine the storage relationship of each target training sample data in each heterogeneous storage resource set in each computing node based on the target training sample data corresponding to each computing node, various reference performance parameters, and the communication delay and / or the length of the data transmission path between each computing node and each heterogeneous storage resource.

[0094] Among them, the above-mentioned communication delay may refer to the time required for data to be transmitted from various heterogeneous storage resources to the computing node. The length of the above-mentioned data transmission path may refer to the actual physical distance between the computing node and the various heterogeneous storage resources, or it may refer to the network path that the data passes through in the process of reaching the various heterogeneous storage resources from the computing node. The more intermediate nodes included in the network path here, the higher the transmission delay.

[0095] In addition, the server can also determine the similarity between each target training sample data and other training sample data, and then divide each target training sample data into target training sample data sets according to the similarity between each target training sample data and other training sample data, and then determine the storage relationship of each target training sample data set in each heterogeneous storage resource according to each reference performance parameter.

[0096] Wherein, for each target training sample data set, the similarity between each target training sample data included in the target training sample data set is higher than a preset threshold.

[0097] Furthermore, the server can determine the configuration parameters of each heterogeneous storage resource based on the adjusted storage relationship, and for each heterogeneous storage resource, configure the heterogeneous storage resource according to the configuration parameters of the heterogeneous storage resource to obtain the configured heterogeneous storage resource corresponding to the heterogeneous storage resource, and according to the adjusted storage relationship, unload the data in the heterogeneous storage resource to other heterogeneous storage resources or read data from other heterogeneous storage resources to obtain the scheduled data storage system, and continue to execute the training task for the next training cycle of the target model through the scheduled data storage system.

[0098] From the above content, it can be seen that the server can solve the problems of complex multi-level storage data management, frequent data copying and migration, and waste of storage resources through the above data quality-driven model data access expansion method. Through unified namespace and metadata management, an efficient and hierarchical storage system is built. The system uses hierarchical storage and training data generation technology to accelerate the efficiency of data access in the vertical direction, and adopts strategies such as data unloading and memory pooling to improve the efficiency of data access in the horizontal direction, thereby comprehensively improving the efficient access of data from both horizontal and vertical aspects.

[0099] In addition, the server can also use the above-mentioned data quality feature abstraction and storage resource combination optimization technology. In order to solve the problems of different data qualities, model training characteristics and heterogeneous resources, the collaborative solution space is complex, the multiple targets lead to difficult search, and the dynamic changes of system resources lead to poor timeliness of configuration solutions, the characteristics of heterogeneous storage resources are combined with different data qualities to construct a collaborative solution space for data quality partitioning and dynamic resource scheduling. A multi-objective optimization strategy based on Bayesian optimization is adopted to achieve efficient matching of data and resources and load balancing configuration optimization and efficient scheduling methods. This comprehensive approach not only improves the efficiency of data access, but also ensures the flexibility of the data storage system and the optimization of resource utilization.

[0100] The above are one or more storage resource scheduling methods implemented in this specification. Based on the same idea, this specification also provides a corresponding storage resource scheduling device, such as Figure 4 shown.

[0101] Figure 4 A schematic diagram of a data storage device provided in this specification includes:

[0102] The acquisition module 401 is used to acquire the state parameters of the target model in the current training cycle during the execution of the training task for the target model, and determine the preset performance parameters of each heterogeneous storage resource as reference performance parameters;

[0103] A first determination module 402 is used to determine the training sample data required by the target model in the next training cycle as each target training sample data according to the preset demand relationship between the target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model;

[0104] The second determination module 403 is used to determine the storage relationship of each target training sample data in each heterogeneous storage resource according to the reference performance parameters, and determine the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship;

[0105] The adjustment module 404 is used to adjust the storage relationship with the optimization goal of minimizing the estimated remaining training time, obtain the adjusted storage relationship, and schedule the heterogeneous storage resources according to the adjusted storage relationship to continue to execute the training task through the scheduled data storage system.

[0106] Optionally, the data storage system is provided with various computing nodes, and the training task for the target model includes: various sub-training tasks;

[0107] The second determination module 403 is specifically used to, for each computing node, determine the sub-training task assigned to the computing node in the sub-training tasks as the target sub-training task; determine the target training sample data required to execute the target sub-training task contained in the target training sample data as the target training sample data corresponding to the computing node; and determine the storage relationship of the target training sample data in each heterogeneous storage resource set in each computing node according to the target training sample data corresponding to each computing node and the reference performance parameters.

[0108] Optionally, the second determination module 403 is specifically used to determine, for each target training sample data, the similarity between the target training sample data and other target training sample data; divide the target training sample data into target training sample data sets according to the similarity between each target training sample data set and other target training sample data, wherein, for each target training sample data set, the similarity between the target training sample data sets contained in the target training sample data set is higher than a preset threshold; and determine, according to the reference performance parameters, the storage relationship of the target training sample data sets in each heterogeneous storage resource.

[0109] Optionally, the distributed memory cache module includes at least one high-performance storage device, wherein when the distributed memory cache module includes more than one high-performance storage device, the high-performance storage devices in the distributed memory cache module are interconnected at high speed, the persistent cache module includes at least one non-volatile storage device, and the parallel file system includes at least one large-capacity storage device;

[0110] The acquisition module 401 is specifically used to determine the performance parameters of the high-performance storage device, the performance parameters of the non-volatile storage device, and the performance parameters of the large-capacity storage device as reference performance parameters.

[0111] Optionally, the second determination module 403 is specifically used to determine the storage relationship of the target training sample data in each heterogeneous storage resource according to the reference performance parameters; and determine the storage relationship of the model parameters of the target model in each heterogeneous storage resource according to the reference performance parameters and the preset dependency relationship between the model parameters of the target model in different training cycles; determine the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship of the target training sample data in each heterogeneous storage resource and the storage relationship of the model parameters of the target model in each heterogeneous storage resource.

[0112] Optionally, the adjustment module 404 is specifically used to determine the configuration parameters of each heterogeneous storage resource according to the adjusted storage relationship; for each heterogeneous storage resource, configure the heterogeneous storage resource according to the configuration parameters of the heterogeneous storage resource to obtain the configured heterogeneous storage resource corresponding to the heterogeneous storage resource; and according to the adjusted storage relationship, unload the data in the heterogeneous storage resource to other heterogeneous storage resources or read data from other heterogeneous storage resources to the heterogeneous storage resource.

[0113] Optionally, the persistent cache module is further provided with a data generation model;

[0114] The adjustment module 404 is specifically used to, during the process of continuing to execute the training task through the scheduled data storage system, if it is determined that the number of iterations of the target model does not exceed the preset number threshold, then obtain the used training sample data in the training sample set corresponding to the target model, and adjust the model parameters of the data generation model according to the used training sample data to obtain the adjusted data generation model; if it is determined that the number of iterations of the target model exceeds the preset number threshold, then in the process of responding to the training sample data acquisition request sent by the computing node, if it is determined that there is no data matching the training sample data acquisition request in the distributed memory cache module and the persistent cache module, then input the training sample data acquisition request into the adjusted data generation model to generate supplementary training sample data through the adjusted data generation model, and execute the training task for the target model according to the supplementary training sample data.

[0115] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A data storage method is provided.

[0116] This manual also provides Figure 5 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0117] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0118] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0119] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0120] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0121] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0129] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0131] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0132] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A storage resource scheduling method, characterized in that: The method is applied to a data storage system, wherein the data storage system includes various heterogeneous storage resources, wherein the various heterogeneous storage resources include: a distributed memory cache module and a persistent cache module arranged in a computing node, and a parallel file system arranged in a storage node; In the process of executing the training task for the target model, the state parameters of the target model in the current training cycle are obtained, and the performance parameters of each preset heterogeneous storage resource are determined as reference performance parameters; According to the preset demand relationship between the target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model, the training sample data required by the target model in the next training cycle are determined as the target training sample data; Determine, according to the reference performance parameters, a storage relationship between the target training sample data in each heterogeneous storage resource, and determine an estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship; With minimizing the estimated remaining training time as the optimization goal, the storage relationship is adjusted to obtain an adjusted storage relationship, and the heterogeneous storage resources are scheduled according to the adjusted storage relationship to continue executing the training task through the scheduled data storage system.

2. The method according to claim 1, characterized in that The data storage system is provided with various computing nodes, and the training task for the target model includes: various sub-training tasks; Determining the storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter specifically includes: For each computing node, determining a sub-training task among the sub-training tasks that is assigned to the computing node for execution as a target sub-training task; Determine the target training sample data required for executing the target sub-training task contained in each target training sample data as the target training sample data corresponding to the computing node; According to the target training sample data corresponding to each computing node and the reference performance parameters, the storage relationship of the target training sample data in each heterogeneous storage resource set in each computing node is determined.

3. The method according to claim 1, characterized in that Determining the storage relationship of each target training sample data in each heterogeneous storage resource according to each reference performance parameter specifically includes: For each target training sample data, determining the similarity between the target training sample data and other target training sample data; According to the similarity between each target training sample data and other target training sample data, the target training sample data are divided into target training sample data sets, wherein, for each target training sample data set, the similarity between the target training sample data contained in the target training sample data set is higher than a preset threshold; According to the reference performance parameters, the storage relationship of the target training sample data sets in each heterogeneous storage resource is determined.

4. The method according to claim 1, characterized in that The distributed memory cache module includes at least one high-performance storage device, wherein when the distributed memory cache module includes more than one high-performance storage device, the high-performance storage devices in the distributed memory cache module are interconnected at high speed, the persistent cache module includes at least one non-volatile storage device, and the parallel file system includes at least one large-capacity storage device; Determine the preset performance parameters of each heterogeneous storage resource as reference performance parameters, including: The performance parameters of the high-performance memory device, the performance parameters of the non-volatile memory device, and the performance parameters of the large-capacity memory device are determined as reference performance parameters.

5. The method according to claim 1, characterized in that Determining the storage relationship of each target training sample data in each heterogeneous storage resource according to the reference performance parameters, and determining the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship, specifically includes: Determining, according to the reference performance parameters, a storage relationship of the target training sample data in each heterogeneous storage resource; and Determining the storage relationship of the model parameters of the target model in each heterogeneous storage resource according to the reference performance parameters and the preset dependency relationship between the model parameters of the target model in different training cycles; Determine the estimated remaining training time of the target model after scheduling the heterogeneous storage resources according to the storage relationship of the target training sample data in each heterogeneous storage resource and the storage relationship of the model parameters of the target model in each heterogeneous storage resource.

6. The method according to claim 1, characterized in that Scheduling the heterogeneous storage resources according to the adjusted storage relationship specifically includes: Determining configuration parameters of each heterogeneous storage resource according to the adjusted storage relationship; For each heterogeneous storage resource, the heterogeneous storage resource is configured according to the configuration parameters of the heterogeneous storage resource to obtain the configured heterogeneous storage resource corresponding to the heterogeneous storage resource, and according to the adjusted storage relationship, the data in the heterogeneous storage resource is unloaded to other heterogeneous storage resources or data is read from other heterogeneous storage resources to the heterogeneous storage resource.

7. The method according to claim 6, characterized in that The persistent cache module is also provided with a data generation model; Continuing to execute the training task through the scheduled data storage system specifically includes: In the process of continuing to execute the training task through the scheduled data storage system, if it is determined that the number of iterations of the target model does not exceed a preset number threshold, then obtaining the used training sample data in the training sample set corresponding to the target model, and adjusting the model parameters of the data generation model according to the used training sample data to obtain an adjusted data generation model; If it is determined that the number of iterations of the target model exceeds a preset threshold, then in the process of responding to the training sample data acquisition request sent by the computing node, if it is determined that there is no data matching the training sample data acquisition request in the distributed memory cache module and the persistent cache module, the training sample data acquisition request is input into the adjusted data generation model to generate supplementary training sample data through the adjusted data generation model, and the training task for the target model is performed based on the supplementary training sample data.

8. A data storage device, characterized in that: The device comprises: An acquisition module is used to obtain the state parameters of the target model in the current training cycle during the execution of the training task for the target model, and determine the performance parameters of each preset heterogeneous storage resource as reference performance parameters; A first determination module is used to determine the training sample data required by the target model in the next training cycle as each target training sample data according to the preset demand relationship between the target model under different state parameters and the training sample data with different data quality parameters, as well as the state parameters of the target model in the current training cycle and the data quality parameters of the unused training sample data in the training sample set corresponding to the target model; A second determination module is used to determine the storage relationship of each target training sample data in each heterogeneous storage resource according to the reference performance parameters, and determine the estimated remaining training time of the target model after scheduling each heterogeneous storage resource according to the storage relationship; An adjustment module is used to adjust the storage relationship with the optimization goal of minimizing the estimated remaining training time, obtain an adjusted storage relationship, and schedule the heterogeneous storage resources according to the adjusted storage relationship to continue executing the training task through the scheduled data storage system.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images