Model training method and device

Abstract

The present application provides a model training method, which is applied to computing nodes in a cloud service cluster, wherein a model to be trained is deployed in the computing nodes. The cloud service cluster also includes storage nodes, wherein a training data set is stored in the storage nodes. The method comprises: determining the amount of training data required for the current training cycle; downloading the training data required for the current training cycle from the storage nodes based on the amount of training data required for the current training cycle; preprocessing the training data; and inputting the preprocessed training data into the model to be trained to perform model training. The model training method provided in the present application downloads training data based on the amount of data required for the current training cycle, achieving on-demand downloading, reducing disk space requirements for training nodes, shortening download waiting time, and improving user experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and device. Background Art

[0002] With the development of artificial intelligence (AI) technology, artificial intelligence (AI) models, represented by deep learning, are widely used in various fields, such as image classification, object detection, and natural language processing. Training the initial AI model is a critical process. Training refers to the process of inputting data from the training dataset into the initial AI model, having the initial AI model perform calculations, and updating the parameters of the initial AI model through the computational structure, ultimately obtaining an AI model with certain capabilities (for example, image classification, object detection, natural language recognition, etc.).

[0003] Due to the large size of the training data set and the increasingly complex model structure, model training places high demands on the computing power and storage of the training equipment, which are generally difficult to meet with local computing devices. Therefore, cloud training that uses cloud resources for model training is the current development trend. However, in the cloud training scenario, the training node will not start training until the training data set stored in the storage node is fully downloaded. This will lead to a series of problems. For example, for extremely large data sets on the cloud, the disk space of the training node cannot store the full training data set, resulting in the inability to conduct normal training; and the download time of the full training data set is long, and it is necessary to wait for the training data set to be downloaded (usually 10 minutes to 10 hours) before starting training, which results in a poor user experience. Summary of the Invention

[0004] The embodiments of the present application provide a model training method and device, which downloads training data according to the amount of data required for the current training cycle, realizes on-demand downloading, reduces the disk space requirements for training nodes, reduces waiting time for downloading, and improves user experience.

[0005] In the first aspect, the present application provides a model training method, which is applied to the computing nodes in a cloud service cluster, where the model to be trained is deployed in the computing nodes. The cloud service cluster also includes a storage node, which stores a training data set. The method includes determining the amount of training data required for the current training cycle; based on the amount of training data required for the current training cycle, downloading the training data required for the current training cycle from the storage node; preprocessing the training data; and inputting the preprocessed training data into the model to be trained to perform model training.

[0006] The model training method provided in this application downloads training data according to the amount of data required for the current training cycle, realizes on-demand downloading, reduces the disk space requirements for training nodes, reduces waiting time for downloading, and improves user experience.

[0007] In one possible implementation, the computing node includes multiple training sub-nodes, the model to be trained includes multiple sub-models to be trained, and the multiple training sub-nodes are respectively used to train the multiple sub-models to be trained. For example, the multiple training sub-nodes are respectively used to train different sub-models to be trained in the multiple sub-models to be trained.

[0008] In another possible implementation, the above method also includes: obtaining the preprocessing status of the training data corresponding to each training sub-node in the multiple training sub-nodes; inputting the preprocessed training data into the model to be trained to perform model training, including: determining that the number of preprocessed training data corresponding to each training sub-node in the multiple training sub-nodes reaches a first preset threshold, and then controlling the multiple training sub-nodes to synchronously train the sub-model to be trained.

[0009] In another possible implementation, the above method also includes: obtaining the download status corresponding to multiple training sub-nodes, where the download status indicates the status of the training sub-node downloading the training data; and preprocessing the training data, including: determining that the amount of training data downloaded by the multiple training sub-nodes reaches a second preset threshold, and then controlling the multiple training sub-nodes to synchronously preprocess the downloaded training data.

[0010] In another possible implementation, based on the amount of training data required for the current training cycle, the training data required for the current training cycle is downloaded from the storage node, including: based on the amount of training data required for the current training cycle of multiple training sub-nodes, multiple data shards are divided from the training data set, and the multiple data shards include multiple training sample data; a download sub-thread is established for each of the multiple training sub-nodes to download the data shards corresponding to each training sub-node; and the download status is synchronized between the download sub-threads corresponding to each training sub-node to synchronize the download of the download sub-threads corresponding to each training sub-node.

[0011] In another possible implementation, multiple training sub-nodes download training data in a balanced manner.

[0012] In another possible implementation, the downloading method of the training data required for the current training cycle from the storage node is asynchronous downloading.

[0013] In another possible implementation, before preprocessing the training data, the method further includes: performing a shuffling operation on the training data set.

[0014] In another possible implementation, the training data set includes multiple training files, each training file includes multiple training sample data; the shuffling operation on the training data set includes: shuffling the multiple training files; and shuffling the multiple sample data in the downloaded training files.

[0015] In a second aspect, the present application provides a model training device, which is applied to a computing node in a cloud service cluster, wherein a model to be trained is deployed in the computing node, and the cloud service cluster further includes a storage node, wherein a training data set is stored in the storage node; the device comprises:

[0016] A determination module is used to determine the amount of training data required for the current training cycle;

[0017] A download module, configured to download the training data required for the current training cycle from the storage node based on the amount of training data required for the current training cycle;

[0018] A preprocessing module, used for preprocessing the training data;

[0019] The training module is used to input the preprocessed training data into the model to be trained to perform model training.

[0020] In one possible implementation, the computing node includes a plurality of training sub-nodes, the model to be trained includes a plurality of sub-models to be trained, and the plurality of training sub-nodes are respectively used to train the plurality of sub-models to be trained.

[0021] In another possible implementation, the apparatus further includes a synchronization module configured to obtain a preprocessing status of training data corresponding to each of the plurality of training sub-nodes;

[0022] The training module is specifically used to determine that the number of pre-processed training data corresponding to each training sub-node in the multiple training sub-nodes reaches a first preset threshold, and then control the multiple training sub-nodes to synchronously train the sub-model to be trained.

[0023] In another possible implementation, the synchronization module is further configured to obtain a download status corresponding to the multiple training sub-nodes, where the download status indicates a status of the training sub-node downloading the training data;

[0024] The preprocessing module is specifically configured to, when determining that the amount of training data downloaded by the plurality of training sub-nodes reaches a second preset threshold, control the plurality of training sub-nodes to synchronously preprocess the downloaded training data.

[0025] In another possible implementation, the download module is specifically configured to divide the training data set into a plurality of data shards based on an amount of training data required for a current training cycle of the plurality of training sub-nodes, the plurality of data shards including a plurality of training sample data;

[0026] Establishing a download sub-thread for each training sub-node among the multiple training sub-nodes to download the data slice corresponding to each training sub-node;

[0027] The downloading status of the downloading sub-threads corresponding to the respective training sub-nodes is synchronized so that the downloading sub-threads corresponding to the respective training sub-nodes can download in synchronization.

[0028] In another possible implementation, the multiple training sub-nodes download the training data in a balanced manner.

[0029] In another possible implementation, the download module is specifically configured to download the training data required for the current training cycle from the storage node using an asynchronous download method.

[0030] In another possible implementation, a shuffling module is further included, configured to perform a shuffling operation on the training data set.

[0031] In another possible implementation, the training data set includes a plurality of training files, each of which includes a plurality of training sample data;

[0032] The shuffling module is specifically configured to perform a shuffling operation on the plurality of training files;

[0033] The multiple training sample data in the downloaded training file are shuffled.

[0034] In a third aspect, the present application further provides a computing device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, enables the method described in the first aspect to be implemented.

[0036] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product, which includes instructions that, when executed, cause a computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the process of model training;

[0038] Figure 2 A schematic diagram of using a local training machine for model training;

[0039] Figure 3 A diagram showing a training scenario for a model on the cloud.

[0040] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0041] Figure 5 Schematic diagram of computing nodes asynchronously downloading training data from cloud datasets;

[0042] Figure 6 Schematic diagram of the download synchronization and training synchronization mechanism for multiple training nodes;

[0043] Figure 7 Schematic diagram of the model training process for multiple training nodes after adding a synchronization mechanism;

[0044] Figure 8 A schematic diagram of the structure of a computing node provided in an embodiment of the present application;

[0045] Figure 9 Schematic diagram of asynchronous download and synchronous download status of multiple training nodes;

[0046] Figure 10 Schematic diagram of balanced downloading for multiple training nodes;

[0047] Figure 11 This is a schematic diagram of achieving balanced downloading on multiple training nodes when the maximum concurrent number of computing nodes is 2;

[0048] Figure 12 A schematic diagram of the interaction between various modules in the computing node;

[0049] Figure 13 Schematic diagram of data preprocessing synchronization for multiple training nodes;

[0050] Figure 14 This is a diagram of synchronization of training of multiple training nodes;

[0051] Figure 15 Schematic diagram of two-level shuffling of the training data set;

[0052] Figure 16 This is a schematic diagram of the specific implementation process of two-level shuffling of the training data set;

[0053] Figure 17 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0054] Figure 18 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.

[0056] In order to facilitate understanding of the solutions provided below in the embodiments of the present application, the following introduces the professional terms that may be involved in the solutions provided in the embodiments of the present application.

[0057] Data set: It is a collection of data, which can be an image, a piece of text, an audio clip, a video clip, etc. These can be collectively referred to as samples, and the collection of these samples is a data set.

[0058] The dataset used for model training is called a training dataset. It consists of multiple training samples and is the original input for machine learning model training. However, these training samples generally require data preprocessing before they can be used for further model training. For simplicity, the dataset below refers to the training dataset.

[0059] Data preprocessing: The input of data preprocessing can be a dataset or a single sample (i.e., a training sample). The dataset / sample will be processed by the data preprocessing module. Processing examples include: image - decoding, cutout, scaling, normalization, channel conversion, batching, etc.; text - conversion to lexicographic order, mask generation, padding, batching, etc. The processed results will be passed to the model training for machine learning.

[0060] Model training: Receive training data from data preprocessing and train the model through operations such as convolution, ReLU, and pooling to obtain a model. This model can be used for reasoning and prediction in various scenarios.

[0061] Training Cycle (Epoch): Generally, model training goes through multiple epochs. Each epoch refers to training the entire dataset. Model training generally involves multiple epochs. However, in certain scenarios, such as when the dataset is large, a portion of the dataset may be used as training data for a single training epoch.

[0062] Steps of training the model: In a training cycle, there are generally multiple steps of training. Generally, the dataset of a training cycle is divided into multiple batches of training data according to the training steps. One batch of training data is used for one training step.

[0063] Training machine: A computing device with a certain level of computing power that can be used for model training, for example, a server / virtual machine with a CPU, memory, and disk to form an operating environment.

[0064] Training node: refers to a server / virtual machine that can be equipped with multiple GPUs / NPU devices. Typically, a server can be equipped with 1, 2, 4, or 8 GPUs / NPUs. GPUs / NPUs are computing devices used for model training. In other words, multiple training nodes can be deployed on a training machine. For example, if 8 NPUs are deployed on a server, the server has 8 training nodes.

[0065] Figure 1 Figure 2 shows a schematic diagram of the model training process. Figure 1 As shown in the figure, the training data set will be preprocessed before being input into the model for training. The preprocessed model will be segmented into multiple batches according to the number of training steps of the model. After multiple batches of training data are used up, one cycle of training is completed, and the model training of the next cycle will continue until the model meets the conditions for training completion, such as reaching the preset number of cycles, or reaching the preset time, or the model converges.

[0066] When the model structure is relatively simple, the data set is not very large, and the configuration requirements for the training machine are not very high, the model can be trained on a local training machine, that is, using a local computing device as a training machine, such as a local server or computing terminal, for model training.

[0067] Figure 2 Figure 1 shows a schematic diagram of model training using a local training machine. Figure 2 As shown, when performing machine learning locally, it can be a distributed training scenario with multiple training machines, such as 2 training machines (e.g. Figure 2 The user first needs to place the dataset on the disk of all machines. After the training task is started, data preprocessing will load the dataset from the disk into the memory for processing and use it for training.

[0068] When the model structure is complex, the data set is large, and the training machine requirements are high, you can choose cloud model training that uses cloud resources for model training.

[0069] Figure 3 A training diagram of a model on the cloud training scenario is shown, Figure 3As shown in the figure, compared with local machine training, cloud training is a typical machine learning scenario. In this scenario, users generally store datasets in cloud object storage (for example, Huawei Cloud: OBS, Amazon: S3, Alibaba Cloud: OSS), and then use cloud computing resources (for example, Huawei Cloud: ModelArts, Amazon: SageMaker, Alibaba Cloud: PAI) for training. After the training starts, the user needs to download the dataset to the training cluster first and preprocess the data before passing it to the computing network for training.

[0070] Cloud-based model training eliminates the need for local computing devices or data sets to be downloaded locally. Instead, cloud resources (including storage and computing resources) can be used to complete model training. However, the cloud-based training process presents challenges, such as efficiently downloading cloud data sets, efficiently shuffling data (disrupting data samples without disrupting correlations, which is beneficial for model training), and coordinating data loading with training.

[0071] To address the above issues, this application proposes a model training method that is applied to cloud model training scenarios and to computing nodes in a cloud service cluster. The computing nodes have models to be trained deployed in them. The cloud service cluster also includes storage nodes, which store data sets. The method includes first determining the amount of training data required for the current training cycle of the model, downloading the training data required for the current training cycle from the storage nodes, preprocessing the training data, and then training the model based on the preprocessed training data. This method enables on-demand downloading of data sets, allowing training to begin without downloading the entire data set, reducing data set download time and improving the user experience.

[0072] Figure 4 This is a flowchart of a model training method provided in an embodiment of the present application. This method can be applied to a model training scenario on the cloud and executed by a computing node in a cloud service cluster. The computing node stores the model to be trained, that is, the computing node serves as a training machine. The computing node can be a cloud server or a virtual machine, on which are deployed several NPUs / GPUs for model training, that is, several training nodes are deployed on the computing node. Figure 4 As shown, the method includes steps S401-S404.

[0073] In step S401 , the amount of training data required for the current training cycle is determined.

[0074] The amount of training data required for the current training cycle can be set by the user or by default.

[0075] The amount of training data required for the current training cycle is user-configured. For example, the user can set the amount of training data required for the current training cycle to 500 samples. The method for determining the amount of training data required for the current training cycle is to obtain the user-configured amount of training data required for the current training cycle, such as 500 samples, and then determine the amount of training data required for the current training cycle based on the user's settings. In other words, the amount of training data required by the compute node for the current training cycle can be flexibly selected, and the compute node can download the required amount of training data for the current cycle on demand.

[0076] For example, the computing node receives information about the amount of training data required for the current training cycle set by the user, and determines the amount of training data required for the current training cycle based on the information about the amount of training data.

[0077] Alternatively, the training task start instruction carries the information on the amount of training data required for the current training cycle set by the user. The computing node obtains the information on the amount of training data required for the current training cycle set by the user from the received training task start instruction, and determines the amount of training data required for the current training cycle based on the information on the amount of training data required for the current training cycle set by the user.

[0078] The amount of training data required for the current training cycle can also be a default. For example, if the default amount of training data is 1,000 sample data, the computing node directly uses the default amount of training data required for the current cycle (for example, 1,000 sample data) as the amount of training data required for the current training cycle.

[0079] In step S402, based on the amount of training data required for the current training cycle, the training data required for the current training cycle is downloaded from the storage node.

[0080] That is, the computing node downloads training data from the storage node to the disk on demand, without downloading the entire dataset. It only needs to download the part of the data required for the current cycle from the dataset stored in the storage node to the computing node, avoiding the large amount of time consumed by downloading the full dataset, which increases the time of the entire model training process.

[0081] In step S403, the training data is preprocessed.

[0082] After downloading the training data required for the current cycle, the compute node loads the training data into memory for preprocessing. For example, if the training data is image data, preprocessing includes image decoding, image cutout, scaling, normalization, channel conversion, and batching. If the training data is text data, preprocessing includes text conversion to lexicographic order, mask generation, padding, and batching.

[0083] In step S404, the preprocessed training data is input into the model to be trained to perform model training.

[0084] The preprocessed training data is fed to the training model, the training model performs calculations, and the parameters of the training model are modulated according to the calculation results.

[0085] The model training method provided in this application coordinates data download and training tasks, flexibly selects the amount of training data required for the current training cycle of the computing node, and realizes on-demand download. Compared with full download, training can start as soon as a small amount of sample data is ready. On the one hand, it reduces the disk space requirements for the training node, and on the other hand, it reduces the waiting time for downloading, thereby reducing the time required for the entire model training process and increasing the efficiency of model training.

[0086] In another example, see Figure 5 ,The download method of the computing node to download the training data from the ,cloud storage node is asynchronous download, which can ,be downloading while preprocessing the data, further increasing the download ,efficiency, and hiding the download time in the training process, thereby ,improving the overall model training efficiency.

[0087] Optionally, the cloud service cluster includes multiple computing nodes for model training, and the multiple computing nodes serve as multiple training machines for model training. The multiple computing nodes download training data from the cloud storage node asynchronously.

[0088] Optionally, multiple training nodes (i.e., training subnodes) can be deployed in each compute node. For example, if four NPUs / GPUs are deployed in each compute node, each compute node can have four training nodes, each of which can execute training tasks. The compute node creates a download task for each training node to asynchronously download training data.

[0089] That is to say, the computing node includes multiple training nodes, which can split the model to be trained into multiple sub-models to be trained. Each training node stores a sub-model to be trained and is responsible for the training task of a sub-model to be trained. When all sub-models to be trained are completed, the trained model is finally obtained based on the multiple sub-models that have been trained.

[0090] In another example, when there are multiple training nodes, in order to ensure smooth training and no errors during the training of multiple sub-models to be trained, the download synchronization and data preprocessing synchronization of multiple training nodes are synchronized to ensure training synchronization, and a synchronization mechanism is added, including a download synchronization mechanism, a loading synchronization mechanism (i.e., data preprocessing synchronization) and a training synchronization mechanism. Figure 6 The download synchronization and training synchronization mechanism of multiple training nodes is shown. The download status of multiple training nodes is synchronized between nodes to ensure the synchronization of training data downloaded between nodes. The status of the dataset is synchronized with the training task to ensure the synchronization of training tasks.

[0091] Figure 7The figure shows the model training process of multiple training nodes after adding a synchronization mechanism.

[0092] In order to implement the above synchronization mechanism, Figure 8 A schematic diagram of the structure of a computing node is shown in FIG. Figure 7 As shown, the computing node includes a data preprocessing module, a data download module, a state synchronization module and a training module.

[0093] The data download module asynchronously downloads dataset files from cloud storage to the computation module. The data preprocessing module is responsible for loading and processing the downloaded dataset. The state synchronization module synchronizes the data download, data loading, and training task status of each training node. The training task module is responsible for training using preprocessed data and coordinating data processing and training on each training node.

[0094] In order to implement the above-mentioned synchronization mechanism, the model training method provided in the embodiment of the present application also includes the steps of data download synchronization, data loading (preprocessing) synchronization and training synchronization.

[0095] In one example, the data download synchronization step is specifically implemented as follows: based on the amount of training data required for the current training cycle of multiple training sub-nodes, multiple data shards are divided from the training data set, and the multiple data shards include multiple training sample data; a download sub-thread is established for each of the multiple training sub-nodes to download the data shards corresponding to each training sub-node; the download status of the download sub-threads corresponding to each training sub-node is synchronized so that the download sub-threads corresponding to each training sub-node can download synchronously.

[0096] Specifically, the data set includes multiple training files, each of which includes multiple training sample data. The computing node divides the multiple training files into data shards according to the amount of training data required by each training node. Each training node downloads the corresponding data shard to meet the amount of training data required for the current training cycle.

[0097] like Figure 9As shown, the training main process of each training node starts the download sub-thread and establishes communication between the main process and the sub-thread; the main process takes the training data required for the current training cycle of each training node as a download task and sends it to the corresponding download sub-thread in sequence. The download sub-thread receives the request and determines whether the disk space usage has reached the threshold, otherwise deletes the old training file (used by the previous data iteration). Download the training files in sequence, and verify them after the download is complete. Update the training file download to the completed status and synchronize it back to the main process. The download status is synchronized between all training nodes. When the training volume (i.e., the number of training samples) of the training data downloaded by all training nodes reaches the preset threshold, all training nodes start distributed training, that is, training synchronously. After completing a round of training, repeat the above steps.

[0098] In another example, Figure 10 As shown in the figure, multiple training nodes download training data in a balanced manner. When training with multiple training nodes, considering the pressure of high concurrent downloads on the cloud storage service, it is necessary to control the download flow of multiple training nodes to ensure that each training node downloads the expected amount of training data in the shortest possible time. For example, assuming there are 4 training nodes, training node 1 needs to download data shard 0, data shard g, data shard x, and data shard l; training node 2 needs to download data shard b, data shard r, data shard e, and data shard p; training node 3 needs to download data shard z, data shard w, data shard n, and data shard d; training node 4 needs to download data shard f, data shard i, data shard j, and data shard k; balanced downloading requires ensuring that all training nodes have balanced resources to download data shard 0, data shard b, data shard z, and data shard f first, and then data shard g, data shard r, data shard w, and data shard i. It is not allowed to download only data shard 0, data shard g, data shard x, and data shard l on training node 1, while training nodes 2, training nodes 3, and training nodes 4 cannot compete for download resources.

[0099] For example, Figure 11 As shown in the figure, assuming that the maximum concurrent number of computing nodes is 2, the multi-node download process is as follows:

[0100] 1. Multiple training node processes start download threads separately;

[0101] 2. The download thread of training node 1 executes download task 1 and synchronizes the status to other training nodes;

[0102] 3. The download thread of training node 2 executes download task 2 and synchronizes the status to other training nodes;

[0103] 4. Training node 3 is ready to download. Due to the maximum concurrency limit, training node 3 enters the waiting queue;

[0104] 5. After training node 1 completes download task 1, it synchronizes the status to other training nodes. Training node 3 exits the waiting queue and starts downloading tasks.

[0105] 6. Wait for training node 2 to complete downloading task 2, synchronize the status to other training nodes, wait for the head of the queue to be dequeued, and the corresponding node starts downloading the task;

[0106] 7. When all download sub-threads complete their download tasks and the waiting queue is empty, all download tasks for this round of data iteration are completed.

[0107] In another example, the specific implementation of the data loading (preprocessing) synchronization step is: obtaining the download status corresponding to multiple training nodes, the download status indicates the status of the training node downloading the training data, such as the download progress, including the amount of data of the downloaded training data, such as the number of downloaded data shards, or the number of training samples; determining that the number of training data downloaded by multiple training sub-nodes reaches a preset threshold (i.e., a second preset threshold), for example, 100 sample data, and then controlling multiple training nodes to synchronously preprocess the downloaded training data.

[0108] In other words, ensuring that multiple training nodes start preprocessing of training data synchronously lays the foundation for synchronous training.

[0109] In another example, the specific implementation of the training synchronization step is: obtaining the preprocessing status of the training data corresponding to each training node in the multiple training nodes, determining that the number of preprocessed training data corresponding to each training node in the multiple training nodes reaches a first preset threshold, and then controlling the multiple training nodes to synchronously train the sub-model to be trained.

[0110] Figure 12 The following illustrates the interaction between modules in a computing node to implement the model training method provided in an embodiment of the present application, so as to achieve on-demand loading and training synchronization:

[0111] S1. The data preprocessing module of each training node notifies the data download model to implement the training data download operation;

[0112] S2. After the data download model of each training node completes downloading the training data, it synchronizes the downloaded sample data to the state synchronization module in an incremental manner;

[0113] S3. The data preprocessing module of each training node obtains the sample status information of other training nodes from the state synchronization model and determines whether to start the data preprocessing operation. For example, when all training nodes have completed downloading the training data, the data preprocessing operation is started; ensuring that the data preprocessing of all training nodes is synchronized;

[0114] S4. The data preprocessing module of each training node provides the processed sample data to the training module of the training node;

[0115] S5. Each training module also uses the status synchronization module to determine the sample preparation status of all training nodes. When the number of samples prepared by all training nodes reaches the specified value, the current round of training begins.

[0116] Through steps S2 and S3, multiple training nodes can process samples with the same sequence number at the same time or with a small time difference, ensuring data preprocessing synchronization. That is, within the preset time, multiple training nodes can process the same amount of sample data (see Figure 13 ).

[0117] Through steps S2 and S5, multiple training tasks can be started synchronously among multiple training nodes, that is, the start of training can be controlled according to the number of samples prepared by all training nodes. For example, when the number of samples pre-processed by each training node is 100, all training nodes are controlled to start training tasks (see Figure 14 ).

[0118] In another example, a model training method provided by an embodiment of the present application further includes, before step S403, performing a shuffling operation on the training data set.

[0119] The training data set includes multiple training files, each of which includes multiple training sample data. The training files can be stored in the cloud storage node in a variety of storage methods, such as block storage, file storage, and object storage. The embodiment of the present application stores the training files in the cloud storage node in the form of block storage, and introduces the shuffling operation of the training data set.

[0120] The training files are stored in the cloud storage node in the form of block storage. Figure 15 As shown in the figure, the block file is the training file. The embodiment of the present application performs a two-level shuffling operation on the data set. First, file-granularity shuffling is implemented based on the block file. All block files are sharded to different training nodes. After the block file is downloaded locally, sample-granularity shuffling is implemented within the block file. The two-level shuffling operation is repeated in the next training cycle.

[0121] Figure 16 The specific process of two-stage shuffling is shown in FIG. The two-stage shuffling process is as follows:

[0122] 1) Read the metadata of each block file in advance. The metadata includes the number of samples in each block file. The number of samples allocated to each training node in each round of data iteration is calculated according to the number of training nodes.

[0123] 2) At the beginning of each round of data iteration, each training node uses the same seed (i.e. the same shuffling rule) to shuffle the file name of the dataset to ensure that the shuffling results of each training node are the same. Each training node is assigned a corresponding block file, such as Figure 13 The shuffling results shown are block file 3, block file 6, block file 1, block file 8, block file 4, block file 5, block file 2, and block file 7;

[0124] 3) Assign the shuffled data set files to different training nodes, such as assigning block files 3 and 6 to training node 1, block files 1 and 8 to training node 2, block files 4 and 5 to training node 3, and block files 2 and 7 to training node 4;

[0125] 4) Continue shuffling within the file. Taking training node 1 as an example, shuffle all samples in block file 3 and output samples i, c, h, ..., z, o, and g. After data processing, convert them into Tensors and input them into the training task. Perform the same operation on block file 6.

[0126] 5) Jump to step (2) to process the next round of data iteration.

[0127] It is understandable that the number of samples in each block file may be the same or different, and this application does not impose any limitation on this.

[0128] The model training method provided in the embodiments of this application is mainly applicable to the typical scenario of training ultra-large datasets on the cloud. By downloading on demand, it solves the problem that ultra-large datasets are stored on the cloud, are difficult to download in full to the training machine (i.e., the computing node mentioned above), or take a long time to download the full dataset. This saves training resources and improves training efficiency from the perspective of the entire training process.

[0129] Through the on-demand loading and training synchronization mechanism, the data loading of each training cycle and the amount of training data required to start training are controlled. Even if a small amount of data is prepared, training can be started, which is as close as possible to the effect of data downloading and training.

[0130] Through balanced downloading across multiple training nodes, multiple training nodes are based on a download balancing mechanism. Each training node synchronously downloads data at the granularity of files or training steps, ultimately achieving consistent download status of training data across multiple training nodes, thereby ensuring efficient training.

[0131] By adopting multi-level shuffling (such as two-level shuffling) to shuffle the training data set, the shuffling effect is guaranteed, and the shuffling effect is made close to the global shuffling of the training data set, thereby ensuring the model training effect.

[0132] Through asynchronous download, that is, multiple training nodes all use asynchronous download to download training data from the cloud dataset, achieving the lowest data download logic overhead.

[0133] Based on the same concept as the above method embodiment, the present application embodiment also provides a model training device 1700, which includes a method for implementing Figure 4-16 The units or modules for each step in the model training method shown.

[0134] Figure 17 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. The device can be deployed in a computing node for model training in a cloud service cluster, where the model to be trained is deployed. The cloud service cluster also includes a storage node, where the training data set is stored; Figure 17 As shown, the model training device 1700 includes at least:

[0135] Determination module 1701, used to determine the amount of training data required for the current training cycle;

[0136] A downloading module 1702 is configured to download the training data required for the current training cycle from the storage node based on the amount of training data required for the current training cycle;

[0137] A preprocessing module 1703 is used to preprocess the training data;

[0138] The training module 1704 is used to input the pre-processed training data into the model to be trained to perform model training.

[0139] In one possible implementation, the computing node includes a plurality of training sub-nodes, the model to be trained includes a plurality of sub-models to be trained, and the plurality of training sub-nodes are respectively used to train the plurality of sub-models to be trained.

[0140] In another possible implementation, the apparatus 1700 further includes a synchronization module 1705, configured to obtain a pre-processing status of the training data corresponding to each of the plurality of training sub-nodes;

[0141] The training module 1704 is specifically used to determine that the number of pre-processed training data corresponding to each training sub-node in the multiple training sub-nodes reaches a first preset threshold, and then control the multiple training sub-nodes to synchronously train the sub-model to be trained.

[0142] In another possible implementation, the synchronization module 1705 is further configured to obtain a download status corresponding to the plurality of training sub-nodes, where the download status indicates a status of the training sub-node downloading the training data;

[0143] The preprocessing module 1703 is specifically configured to, upon determining that the amount of training data downloaded by the plurality of training sub-nodes reaches a second preset threshold, control the plurality of training sub-nodes to synchronously preprocess the downloaded training data.

[0144] In another possible implementation, the downloading module 1702 is specifically configured to divide the training data set into a plurality of data shards based on the amount of training data required for the current training cycle of the plurality of training sub-nodes, the plurality of data shards including a plurality of training sample data;

[0145] Establishing a download sub-thread for each training sub-node among the multiple training sub-nodes to download the data slice corresponding to each training sub-node;

[0146] The downloading status of the downloading sub-threads corresponding to the respective training sub-nodes is synchronized so that the downloading sub-threads corresponding to the respective training sub-nodes can download in synchronization.

[0147] In another possible implementation, the multiple training sub-nodes download the training data in a balanced manner.

[0148] In another possible implementation, the download module 1702 is specifically configured to download the training data required for the current training cycle from the storage node using an asynchronous download method.

[0149] In another possible implementation, a shuffling module 1706 is further included, configured to perform a shuffling operation on the training data set.

[0150] In another possible implementation, the training data set includes a plurality of training files, each of which includes a plurality of training sample data;

[0151] The shuffling module 1706 is specifically configured to perform a shuffling operation on the plurality of training files;

[0152] The multiple sample data in the downloaded training file are shuffled.

[0153] The model training device 1700 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each module in the model training device 1700 are respectively to achieve Figure 4-16 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0154] The present application also provides a computing device comprising at least one processor, a memory, an NPU / GPU, and a communication interface, wherein the processor is configured to execute Figure 4-16The computing device may be a computing node in a cloud service cluster used for model training.

[0155] Figure 18 A schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0156] like Figure 18 As shown, the computing device 1800 includes at least one processor 1801, a memory 1802, a communication interface 803 and an NPU / GPU 1804. The processor 1801 and the memory 1802, the communication interface 1803 and the NPU / GPU 1804 are communicatively connected, and the communication connection can be achieved wirelessly or by wire. The communication interface 1803 is used to receive data sent by other devices (such as information sent by other computing nodes); the NPU / GPU 1804 is used to train the model; the memory 1802 stores computer instructions, and the processor 1801 executes the computer instructions to execute the model training method in the aforementioned method embodiment.

[0157] It should be understood that in the embodiment of the present application, the processor 1801 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0158] The memory 1802 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1801. The memory 1802 may also include a nonvolatile random access memory.

[0159] The memory 1802 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0160] It should be understood that the computing device 1800 according to the embodiment of the present application can execute the implementation of the embodiment of the present application. Figure 4-16 The method shown in the figure is described in detail above and will not be repeated here for the sake of brevity.

[0161] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer instructions are executed by a processor, the above-mentioned model training method is implemented.

[0162] An embodiment of the present application provides a computer program or a computer program product, which includes instructions that, when executed, cause a computer to execute the model training method mentioned above.

[0163] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0165] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A model training method, characterized in that: A computing node in a cloud service cluster is applied, wherein a model to be trained is deployed in the computing node, the computing node includes multiple training sub-nodes, the model to be trained includes multiple sub-models to be trained, and the multiple training sub-nodes are respectively used to train the multiple sub-models to be trained. The cloud service cluster also includes a storage node, and the storage node stores a training data set. The method includes: Determine the amount of training data required for the current training cycle; Downloading the training data required for the current training cycle from the storage node based on the amount of training data required for the current training cycle; Preprocessing the training data; Obtaining a preprocessing status of training data corresponding to each training sub-node among the multiple training sub-nodes; When it is determined that the amount of pre-processed training data corresponding to each of the multiple training sub-nodes reaches a first preset threshold, the multiple training sub-nodes are controlled to synchronously train the sub-model to be trained.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining download statuses corresponding to the multiple training sub-nodes, where the download statuses indicate a status of the training sub-nodes downloading the training data; The preprocessing of the training data includes: If it is determined that the amount of the training data downloaded by the multiple training sub-nodes reaches a second preset threshold, the multiple training sub-nodes are controlled to synchronously pre-process the downloaded training data.

3. The method according to claim 1 or 2, characterized in that The downloading, from the storage node, the training data required for the current training cycle based on the amount of training data required for the current training cycle includes: Based on the amount of training data required for the current training cycle of the multiple training sub-nodes, divide the training data set into multiple data shards, the multiple data shards including multiple training sample data; Establishing a download sub-thread for each training sub-node among the multiple training sub-nodes to download the data slice corresponding to each training sub-node; The downloading statuses of the downloading sub-threads corresponding to the respective training sub-nodes are synchronized, so that the downloading sub-threads corresponding to the respective training sub-nodes can download in synchronization.

4. The method according to any one of claims 1 to 3, characterized in that The multiple training sub-nodes download the training data in a balanced manner.

5. The method according to any one of claims 1 to 4, characterized in that The downloading method of downloading the training data required for the current training cycle from the storage node is asynchronous downloading.

6. The method according to any one of claims 1 to 5, characterized in that Before preprocessing the training data, the method further includes: A shuffling operation is performed on the training data set.

7. The method according to claim 6, characterized in that The training data set includes a plurality of training files, each of which includes a plurality of training sample data; The shuffling operation on the training data set includes: performing a shuffling operation on the plurality of training files; The multiple training sample data in the downloaded training file are shuffled.

8. A model training device, characterized in that: A computing node applied to a cloud service cluster, wherein a model to be trained is deployed in the computing node, the computing node includes multiple training sub-nodes, the model to be trained includes multiple sub-models to be trained, and the multiple training sub-nodes are respectively used to train the multiple sub-models to be trained. The cloud service cluster also includes a storage node, and the storage node stores a training data set; The device comprises: A determination module is used to determine the amount of training data required for the current training cycle; A download module, configured to download the training data required for the current training cycle from the storage node based on the amount of training data required for the current training cycle; A preprocessing module, used for preprocessing the training data; a synchronization module, configured to obtain a preprocessing status of training data corresponding to each of the plurality of training sub-nodes; The training module is used to determine that the number of pre-processed training data corresponding to each training sub-node in the multiple training sub-nodes reaches a first preset threshold, and then control the multiple training sub-nodes to synchronously train the sub-model to be trained.

9. The device according to claim 8, characterized in that The synchronization module is further configured to obtain a download status corresponding to the plurality of training sub-nodes, wherein the download status indicates a status of the training sub-node downloading the training data; The preprocessing module is specifically configured to, when determining that the amount of training data downloaded by the plurality of training sub-nodes reaches a second preset threshold, control the plurality of training sub-nodes to synchronously preprocess the downloaded training data.

10. The device according to claim 8 or 9, characterized in that The download module is specifically configured to divide the training data set into a plurality of data slices based on the amount of training data required for the current training cycle of the plurality of training sub-nodes, wherein the plurality of data slices include a plurality of training sample data; Establishing a download sub-thread for each training sub-node among the multiple training sub-nodes to download the data slice corresponding to each training sub-node; The downloading status of the downloading sub-threads corresponding to the respective training sub-nodes is synchronized so that the downloading sub-threads corresponding to the respective training sub-nodes can download in synchronization.

11. The device according to any one of claims 8 to 10, characterized in that: The multiple training sub-nodes download the training data in a balanced manner.

12. The device according to any one of claims 8 to 11, characterized in that The download module is specifically configured to download the training data required for the current training cycle from the storage node in an asynchronous download manner.

13. The device according to any one of claims 8 to 12, characterized in that It also includes a shuffling module for performing a shuffling operation on the training data set.

14. The device according to claim 13, characterized in that The training data set includes a plurality of training files, each of which includes a plurality of training sample data; The shuffling module is specifically configured to perform a shuffling operation on the plurality of training files; The multiple training sample data in the downloaded training file are shuffled.

15. A computing device comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images