Model training method and device, computer program product and storage medium

Abstract

The embodiment of the invention provides a model training method and device, a computer program product and a computer readable storage medium. According to the method provided by the embodiment of the invention, for a to-be-trained model comprising a plurality of sub-models, a first sub-model which does not participate in training and a second sub-model which participates in training and is located behind the first sub-model are determined, and data pre-loading for a training data set is completed by utilizing the first sub-model before model training; and the data pre-loading result can be directly called in model training to execute parameter updating of the second sub-model. According to the method provided by the embodiment of the invention, repeated calculation of sub-models which do not participate in training in different training periods can be avoided through data preloading, a large amount of calculation power is saved, and the training efficiency of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a model training method, device, computer program product, and storage medium. Background Art

[0002] With the rapid development of deep learning technology, various models continue to emerge, and these models are increasingly evolving in the direction of generative and multimodal. The emergence of multimodal models enables the comprehensive use of different types of data (such as text, images, and audio), providing a new perspective for improving the model's cognitive and decision-making capabilities, thereby enhancing the model's expressive power and prediction accuracy, and improving the model's ability to solve complex problems. For example, in video understanding tasks, the model can more accurately identify and understand the scenes and emotions in the video by combining the video's visual information and audio information.

[0003] In multimodal models, the phenomenon of model combination is becoming increasingly common, that is, a multimodal model may be composed of multiple sub-models. Usually, the entire training process of a multimodal model can involve multiple sub-models, but only some of them may participate in the training, while other models do not participate in the training, that is, keep the parameters unchanged. This situation is common in scenarios such as transfer learning or incremental learning, where the weights of some models are not updated with the training process, becoming "frozen models" in training. Summary of the invention

[0004] Embodiments of the present disclosure provide a model training method, apparatus, computer program product, and computer-readable storage medium.

[0005] An embodiment of the present disclosure provides a model training method, including: determining a first sub-model and a second sub-model in a model to be trained, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, and the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model; before the training of the model to be trained, using the first sub-model to generate an intermediate training data set based on at least part of the training data in the training data set in a video memory of a computing device, and caching the generated intermediate training data set; in the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device without training the first sub-model, so as to update the parameters of the second sub-model.

[0006] An embodiment of the present disclosure provides a model training device, comprising: a model determination module, configured to determine a first sub-model and a second sub-model in a model to be trained, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, and the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model; a data preloading module, configured to generate an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in a video memory of a computing device before the training of the model to be trained, and cache the generated intermediate training data set; and a model training module, configured to train only the second sub-model in the video memory of the computing device based on the cached intermediate training data set during the training of the model to be trained, but not the first sub-model, so as to update the parameters of the second sub-model.

[0007] An embodiment of the present disclosure provides a model training device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the model training method described above is performed.

[0008] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the model training method as described above.

[0009] The embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the model training method according to the embodiments of the present disclosure.

[0010] The method provided by the embodiment of the present disclosure is for a model to be trained including multiple sub-models, determines a first sub-model that does not participate in the training, and a second sub-model that participates in the training and is located after the first sub-model, and uses the first sub-model to complete data preloading for the training data set before model training, so that the result of the data preloading can be directly called in the model training to perform parameter updates for the second sub-model. The method of the embodiment of the present disclosure can avoid repeated calculations of sub-models that do not participate in the training between different training cycles through the above-mentioned data preloading, saving a lot of computing power and improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0012] Figure 1 is a schematic diagram showing a training process for a to-be-trained model including a plurality of sub-models;

[0013] Figure 2 is a flow chart showing a model training method according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram showing a model training method according to an embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram showing data caching using a data set category and a data loader according to an embodiment of the present disclosure;

[0016] Figure 5 is a schematic flow chart showing the processing of video memory optimization in the model training method according to an embodiment of the present disclosure;

[0017] Figure 6 is a schematic flowchart showing the application of a model training method according to an embodiment of the present disclosure to the training of a model including a frozen layer and a training layer;

[0018] Figure 7 is a schematic diagram showing a model training device according to an embodiment of the present disclosure;

[0019] Figure 8 A schematic diagram of a model training device according to an embodiment of the present disclosure is shown;

[0020] Fig. 9 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0022] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0023] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0025] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0026] The model training method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0027] The model training method disclosed in the present invention can be based on the training of a deep learning model. A deep learning model is a machine learning model based on an artificial neural network, with particular emphasis on the use of a multi-layer ("deep") structure. Deep learning is a subfield of machine learning that mainly uses large amounts of data and computing resources to automatically extract features to achieve tasks such as pattern recognition, classification, and generation. The training of a deep learning model mainly involves using training data to adjust the parameters of the model so that it can better accomplish the target task. The following is an introduction to some of the terms that may be used in training.

[0028] Epoch: A complete training of the model using all the data in the training dataset.

[0029] Training batch: Use a part of the samples in the training data set to train the model once;

[0030] Training data sample (Sample): a sample data in the training data set;

[0031] Datasets: Encapsulation of datasets in deep learning, which specifies the operations required to extract a sample data;

[0032] Dataloader: A module for loading data in deep learning, which supports out-of-order loading, multi-threaded data asynchronous loading, and other functions.

[0033] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence and deep learning model training. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0034] As mentioned above, a multimodal model may generally include multiple sub-models, so the training of a multimodal model may involve the training of multiple sub-models, but only some of the models may participate in the training while other models do not participate in the training, that is, the parameters are kept unchanged.

[0035] Figure 1 1 is a schematic diagram showing a training process 100 for a model to be trained including multiple sub-models. As an example, Figure 1 As shown, the model to be trained can consist of three sub-models, such as Figure 1 Model 0, Model 1 and Model 2 in the training of the model to be trained, where Model 0 and Model 1 do not participate in the training, that is, they are only calculated in the forward propagation stage, while Model 2 needs to update its parameters through back propagation.

[0036] exist Figure 1 In the example, the input sample data for this training is Input, which can be split into Input_0 and Input_1 for processing by different sub-models. For example, in the training of a multimodal model, the input Input can be divided into data of different modalities (for example, extracting text data from Input as Input_0, extracting image data from Input as Input_1, etc.) to provide to different sub-models. Therefore, based on different inputs, Model 0 and Model 1 can generate corresponding outputs Output_0 and Output_1 respectively in the forward propagation stage.

[0037] like Figure 1As shown, the outputs Output_0 and Output_1 of Model 0 and Model 1 can be used as the input of Model 2 executed thereafter, so that Model 2 can obtain the output Output_2 based on the input. The output Output_2 can be calculated with the target value (target) to obtain the loss, and the parameters of Model 2 can be updated through back propagation based on the obtained loss. Among them, the back propagation is only performed on Model 2, because the parameters of Model 0 and Model 1 do not need to be updated.

[0038] Based on this, the model training for the input sample data can be completed. For one training cycle (epoch), each sample data in the training data set can be traversed to complete one round of training.

[0039] As mentioned above, the training of deep learning models usually requires multiple epochs, and each epoch will traverse the same data set. During this process, although the input data of the model will be reordered for model learning, since the parameters of the models not participating in the training are not updated during the training, a large number of repeated calculations will be generated for the models not participating in the training between different epochs. These repeated calculations not only reduce the computing efficiency and waste computing power, but also affect the overall training time of the model. As the scale of data continues to expand, the problem of resource waste caused by repeated calculations has become more and more significant and needs to be effectively solved.

[0040] In addition, there is another problem that cannot be ignored, that is, although the parameters of the models that are not involved in the training do not change during the training process, they still need to be calculated in each epoch. This means that in order to achieve accelerated computing, the models that are not involved in the training must be loaded onto devices (such as graphics processing units (GPUs)) to utilize their parallel computing capabilities to accelerate the training and reasoning processes of the models. However, the video memory capacity of existing computing devices is limited, and loading multiple models will increase the pressure on the device's video memory. In this case, if the device video memory is insufficient, it will lead to incomplete model loading or inefficient training process, which will affect the smooth progress of the entire training process.

[0041] Therefore, an effective model training method is needed to effectively reduce repeated calculations, optimize the use of video memory, and improve computing efficiency in the training of deep learning models.

[0042] In order to solve the above problems, the present disclosure provides a model training method, which determines a first sub-model that does not participate in the training and a second sub-model that participates in the training and is located after the first sub-model for a model to be trained including multiple sub-models, and uses the first sub-model to complete data preloading for the training data set before model training, so that the result of the data preloading can be directly called in the model training to perform parameter updates for the second sub-model. The method disclosed in the present disclosure can avoid repeated calculations of sub-models that do not participate in the training between different training cycles through the above data preloading, saving a lot of computing power and improving the training efficiency of the model.

[0043] Figure 2 is a flowchart illustrating a model training method 200 according to an embodiment of the present disclosure. Figure 3 is a schematic diagram illustrating a model training method according to an embodiment of the present disclosure.

[0044] In step S201, a first sub-model and a second sub-model in the model to be trained can be determined, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, and the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model.

[0045] Optionally, for the model to be trained, a first sub-model that does not participate in the training can be determined. The first sub-model may include one or more sub-models whose parameters do not need to be updated during training. These sub-models may have stable capabilities (for example, feature extraction, etc.) in advance, thereby providing reliable basic data for the subsequent second sub-model. Similarly, the second sub-model may also include one or more sub-models whose parameters need to be updated during training. It should be noted that all sub-models in the first sub-model should be located before all sub-models in the second sub-model, that is, they should be run before the second sub-model in the forward propagation to ensure the data consistency of their calculation results. For example, Figure 1 As shown, since Model 0 and Model 1 that do not participate in the training are both located before Model 2 that participates in the training, Model 0 and Model 1 can be determined as the first sub-model, and Model 2 can be determined as the second sub-model. In addition, if Model 1 that does not participate in the training is located after Model 2, Model 1 is not determined as the first sub-model, that is, in this case, the first sub-model only includes Model 0.

[0046] Therefore, through the operation of step S201, the original model to be trained can be divided into a frozen model (evalmodel) and a training model (train model), wherein the frozen model corresponds to the above-mentioned first sub-model, that is, the sub-model without parameter update, and the training model corresponds to the above-mentioned second sub-model, that is, the sub-model with parameter update.

[0047] like Figure 3 As shown in 300, the model training method of the present disclosure may include two-stage processing, pre-training 310 and training 320, wherein the pre-training 310 stage may preload data based on the frozen model, wherein the preloaded data may be cached as intermediate training data, and the training 320 stage may train the training model based on the preloaded data.

[0048] The following will refer to steps S202 and S203 to introduce the processing of the pre-training 310 stage and the training 320 stage respectively.

[0049] In step S202, before training the model to be trained, an intermediate training data set may be generated in a video memory of a computing device using the first sub-model based on at least part of the training data in the training data set, and the generated intermediate training data set may be cached. As an example, the computing device in the present disclosure may be various devices capable of model training, including but not limited to a GPU board, etc.

[0050] like Figure 3 As shown, in the pre-training stage 310, the training dataset can be traversed to generate an intermediate training dataset using the frozen model. Specifically, the training dataset can be divided into multiple batches of training data (e.g., Figure 3 In the example, the training data of batches 1 to batch n in the training dataset are obtained, and for each batch of training data, such as the training data of batch i, a forward propagation calculation on the frozen model can be performed, that is, the frozen model (for example, including model 0 and model 1) is used to generate a corresponding output, such as intermediate training dataset_i, which can represent the i-th subset of the intermediate training dataset corresponding to the training dataset. Therefore, based on n batches of training data, n subsets of the intermediate training dataset can be generated, and these subsets can constitute the intermediate training dataset.

[0051] Although in each round of model training, the order of training data in different batches can be exchanged to perform forward propagation calculations on the training model, in fact, the training data in each batch is fixed rather than randomly arranged, and this fixity may lead to potential biases and limitations in the model learning process. Specifically, when the training data is submitted to the model in a relatively consistent order, the model may rely too much on this specific order, making it less sensitive to the variability and diversity of data within the batch. This situation may cause the model to perform poorly when dealing with real data with high variability or uncertainty. In addition, fixed-order batch training may also affect the effect of gradient updates, restricting the model's convergence speed and ultimate generalization ability.

[0052] Therefore, considering the above problems, the data processing method disclosed in the present invention proposes a method of data caching using dataset categories (Datasets) and data loaders (Dataloader). In the field of deep learning, Datasets, as a mature and widely used data loading method, provides researchers with a convenient way to process and organize data. By packaging the data into Datasets classes, model training can focus on the loading and pre-processing of a single training sample, thereby improving the efficiency of data management. The subsequent data transmission can be handled by the Dataloader, which can flexibly combine different training samples to form different sample batches.

[0053] Specifically, in the pre-training stage 310, according to an embodiment of the present disclosure, before the training of the model to be trained, generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in the video memory of the computing device, and caching the generated intermediate training data set may include: constructing a data set category, the data set category including a data preprocessing module and a data caching module; and for each training data in the at least part of the training data: using the data preprocessing module, preprocessing the training data through a data processing function to generate intermediate training data corresponding to the training data, wherein the data processing function includes a calling function for the first sub-model; caching the generated intermediate training data in the data caching module in the form of a key-value pair, wherein the intermediate training data serves as a value, and the index of the training data corresponding to the intermediate training data in the training data set serves as a key.

[0054] In an embodiment of the present disclosure, the cache of each batch of training data can be converted into the cache of each sample data to further improve the efficiency of data processing. Optionally, based on the existing Datasets framework, the data set category and data loader can be used for data caching.

[0055] Figure 4 FIG. 1 is a schematic diagram showing data caching using a data set category and a data loader according to an embodiment of the present disclosure. Figure 4 As shown in 400, in an embodiment of the present disclosure, two modules can be added to Datasets: a data preprocessing module and a data cache module. The data preprocessing module can store a user-defined data processing function, which can further process the results after conventional data preprocessing. As an example, the execution operation of the frozen model can be completed in the data processing function, for example, by defining an extended processing logic to call the first sub-model to perform calculations on the frozen models (e.g., model 0 and model 1).

[0056] Next, the calculated outputs of the frozen models (e.g., Model 0 and Model 1) are not directly discarded, but cached in the data cache module. Optionally, each intermediate training data may be associated with an index, where the index serves as the key of the intermediate training data, which may correspond to the index of the training sample corresponding to the intermediate training data in the training data set, and the intermediate training data may serve as the value.

[0057] Based on the above caching method, in the subsequent training 320 stage, when Dataloader reads sample data from Datasets, it can query based on the index of the training sample, and if it is found that the intermediate training data corresponding to the index already exists in the data cache module, the data can be directly read from the data cache module without recalculating the calculation results of the frozen model. This method not only effectively avoids redundant calculations for the frozen model in each training cycle, but also maintains the random shuffling function of Dataloader during data processing, so as to make full use of computing resources, improve the overall efficiency of data loading and processing, and prompt the training effect of the model.

[0058] return Figure 2 In step S203, during the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model may be trained in the video memory of the computing device without training the first sub-model, so as to update the parameters of the second sub-model.

[0059] In the training stage 320, according to an embodiment of the present disclosure, the training of the model to be trained may include multiple rounds of training, wherein each round of training may adopt a different training data subset of the training data set.

[0060] In each training cycle of the pre-training stage and the training stage of the model training, although the theoretical goal is to traverse the entire training data set to ensure that the model fully learns and optimizes each training sample, in the actual training process, due to the influence of various factors, it is often impossible to achieve a comprehensive traversal of all training samples. First, the size of the data set may have a significant impact on the training efficiency, especially when processing large training data sets. Due to memory limitations and computing power constraints, the model may only be able to process part of the samples in batches, resulting in some samples being ignored in a specific cycle. Secondly, the random sampling or mini-batch processing strategy used in the training process, although it can accelerate the training process and improve the generalization ability of the model, it also makes the sample traversal in each cycle incomplete. In addition, the adjustment of training parameters, the change of learning rate, and the implementation of early stopping strategies may limit the model's coverage of the entire data set. Therefore, in the model training method disclosed in the present invention, a supplementary caching mechanism for training samples in the training data set that have not been previously preloaded or covered in training can be added in the training 320 stage.

[0061] According to an embodiment of the present disclosure, in the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device, and the first sub-model is not trained, so as to update the parameters of the second sub-model, which may include: for each round of training of the model to be trained, for each training data in the training data subset used for the round of training: determining whether there is intermediate training data corresponding to the training data in the intermediate training data set; in the case of determining that there is intermediate training data corresponding to the training data in the intermediate training data set, using the intermediate training data as input of the second sub-model to train the second sub-model; in the case of determining that there is no intermediate training data corresponding to the training data in the intermediate training data set, generating intermediate training data corresponding to the training data based on the training data using the first sub-model, and adding the intermediate training data to the intermediate training data set.

[0062] Optionally, in order to improve the training efficiency, a supplementary caching mechanism can be introduced to generate intermediate training data for samples that were not fully utilized in the previous training process and cache them for direct use in the subsequent training process. For example, for each round of training, a judgment can be made for each training sample used in that round of training to determine whether there is intermediate training data corresponding to it in the current intermediate training data set. When it is confirmed that there is corresponding intermediate training data, the intermediate training data can be directly called as the input of the second sub-model for training. When there is no relevant data in the intermediate training data set, this means that the sample was not used for calculation in the previous preloading or training process, resulting in the absence of relevant data in the intermediate training data set. Therefore, the first sub-model can be used to generate corresponding intermediate training data for the training sample, and the newly generated data can be incorporated into the intermediate training data set. This flexible supplementary caching mechanism can enhance the reliability of the entire training process and improve the efficiency of model training.

[0063] According to an embodiment of the present disclosure, in the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device, and the first sub-model is not trained, so as to update the parameters of the second sub-model, which may include: for each round of training of the model to be trained: using a data loader to determine the index of each training data in the training data subset used for that round of training; using the data loader to obtain the intermediate training data corresponding to the training data from the data set category based on the index of the training data; and using the returned intermediate training data as the input of the second sub-model to train the second sub-model.

[0064] Optionally, when using dataset categories and data loaders for data caching, for each round of training, the data loader can be used to randomly shuffle the training data in the training data subset used for each round of training at the beginning of each round of training, so that the order of training data in each batch is different, thereby improving the generalization ability of the model.

[0065] Optionally, after randomly shuffling the order of the training data, the data loader may be used to determine the index of each training sample to be used for the round of training, and use the index of the training sample to read the intermediate training data corresponding to the index from the dataset category.

[0066] According to an embodiment of the present disclosure, using the data loader, obtaining intermediate training data corresponding to the training data from the data set category based on the index of the training data may include: in a case where the intermediate training data is obtained from the data cache module, returning the intermediate training data; in a case where the intermediate training data is not obtained from the data cache module, generating intermediate training data corresponding to the training data using the data preprocessing module, returning the intermediate training data, and caching the intermediate training data in the data cache module in the form of key-value pairs.

[0067] Optionally, the above supplementary cache mechanism can be expressed as follows Figure 4 The processing shown. Specifically, in the process of using the index of the training sample to read the intermediate training data corresponding to the index from the data set category, it can be determined based on the index whether the intermediate training data corresponding to the index exists in the data cache module. When it is confirmed that the corresponding intermediate training data exists, the intermediate training data can be directly called as the input of the second sub-model for training, and when there is no relevant data in the intermediate training data set, therefore, the data preprocessing module can be used to generate the corresponding intermediate training data for the training sample by calling the data processing function of the first sub-model, so as to return the generated intermediate training data as the reading result to the data loader, and supplement the intermediate training data to the data cache module, for example, the intermediate training data is used as the value, and the index of the training sample corresponding to the intermediate training data is used as the key.

[0068] By referring to the above Figure 2-Figure 4 The described method can, for a model to be trained including multiple sub-models, determine a first sub-model that does not participate in the training, and a second sub-model that participates in the training and is located after the first sub-model, and use the first sub-model to complete data preloading for the training data set before model training, so that the result of the data preloading can be directly called in the model training to perform parameter updates for the second sub-model. The above-mentioned data preloading avoids repeated calculations of sub-models that do not participate in the training between different training cycles, saves a lot of computing power, and improves the training efficiency of the model.

[0069] As described above, the model training method disclosed herein aims to propose an effective redundancy elimination solution for the common redundant calculation problem in current model training. The core of the method is to minimize unnecessary resource consumption by reasonably arranging the calculation process. The redundancy elimination process disclosed herein may include: removing redundant calculations and removing redundant video memory consumption. Figure 2-Figure 4 The described processing can realize the calculation of removing redundancy. The operation of video memory consumption for removing redundancy will be described in the following additional steps.

[0070] Figure 5 is a schematic flow chart showing the processing of video memory optimization in the model training method according to an embodiment of the present disclosure. Figure 5 As shown in process 500, it depicts the processing steps of the model training method of the present disclosure regarding video memory optimization. Among them, step 501 can correspond to the above-mentioned step S202, that is, constructing a data set category, and using the first sub-model to perform forward propagation calculations to preload intermediate training data. In one embodiment of the present disclosure, the intermediate training data can be cached in the video memory through a data cache module for subsequent training, wherein the data in the data cache module can be stored in the video memory, so that the intermediate training data written into the data cache module can be directly cached in the video memory.

[0071] According to an embodiment of the present disclosure, before the training of the model to be trained, generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in the video memory of the computing device, and caching the generated intermediate training data set may include: after generating the intermediate training data set based on at least part of the training data in the training data set using the first sub-model, caching the generated intermediate training data set in the video memory of the computing device, and releasing the first sub-model from the video memory of the computing device. And according to an embodiment of the present disclosure, the model training method of the present disclosure may also include: in the training of the model to be trained, loading the second sub-model into the video memory of the computing device to train the second sub-model based on the intermediate training data set cached in the video memory of the computing device.

[0072] Optionally, as shown in step 502, after the calculation and caching of the intermediate training data set are completed, the frozen model can be released from the video memory of the computing device to significantly reduce the consumption of the video memory.

[0073] According to an embodiment of the present disclosure, the model training method of the present disclosure may further include: when the available space of the video memory of the computing device meets a predetermined condition, flattening the storage space of the intermediate training data set to merge the storage space of the intermediate training data in the intermediate training data set into a continuous storage space, and releasing the video memory fragments between the storage space of the intermediate training data in the intermediate training data set; when the available space of the video memory of the computing device does not meet the predetermined condition, transferring the intermediate training data set cached in the video memory of the computing device to a central processing unit, and in the training of the model to be trained, loading the required intermediate training data in the intermediate training data set from the central processing unit into the video memory of the computing device.

[0074] Next, during the model training process, the optimization operation may be determined according to the availability of the real-time video memory in step 503. Optionally, it is determined whether the current video memory meets a predetermined condition, for example, whether the available space of the video memory meets a threshold condition.

[0075] Optionally, when the video memory satisfies a predetermined condition, the storage of the cached intermediate training data can be flattened, and the caches can be spliced ​​into a complete video memory space to prevent the video memory from being insufficient due to fragmentation. As an example, the intermediate training data in the intermediate training data set can be stored in batches of training data, for example, the intermediate training data corresponding to a batch can be stored as a tensor data, so the storage video memory of the tensor of the intermediate training data corresponding to each batch can be flattened, that is, the caches of these tensors are spliced ​​into a complete video memory space.

[0076] Optionally, when the video memory does not meet the predetermined conditions, all intermediate training data can be completely unloaded to the CPU, and the cached data can be loaded from the CPU to the video memory when it is needed. For example, in one embodiment of the present disclosure, the above-mentioned data cache module can be transferred to the CPU for subsequent loading of the corresponding intermediate training data from the CPU. Using the asynchronous data loading capability of the Dataloader, data loading can be performed asynchronously, and the overhead of H2D (from host to device) will also be minimized.

[0077] Optionally, in step 504, cache fragments generated by the training process can be cleared by calling a specific instruction of the device (for example, an emptycache instruction) to clear video memory fragments caused by small tensors during the running process.

[0078] Next, in step 505, the intermediate training data (eg, the intermediate training data of batch i) may be read, and the training model (the second sub-model) may be loaded into the video memory to train the training model using the intermediate training data as input.

[0079] Through the above video memory optimization strategy, the video memory occupancy can be effectively reduced and the rational use of resources can be ensured.

[0080] The above describes a training method for a model to be trained that includes multiple sub-models, where the frozen model is an independent model. However, it should be understood that the model training method disclosed in the present invention can also be applied to the training of other types of models, such as models that include multiple sub-modules (e.g., layer structures), but the first several sub-modules do not participate in the training and only the models of the latter several sub-modules need to be trained.

[0081] According to an embodiment of the present disclosure, the first sub-model may include one or more first-layer structures whose parameters will not be updated during the training of the model to be trained, and the second sub-model may include one or more second-layer structures whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the one or more first-layer structures are located before the one or more second-layer structures. That is, in this embodiment, the frozen model (first sub-model) and the training model (second sub-model) are no longer independent models, but layer structures belonging to the same model. In this scenario, the frozen model and the training model can be described as a frozen layer and a training layer, respectively.

[0082] Reference below Figure 6 An example of training for such a scenario is given.

[0083] Figure 6 is a schematic flowchart showing the application of a model training method according to an embodiment of the present disclosure to the training of a model including a frozen layer and a training layer.

[0084] like Figure 6 As shown in the process 600, the model to be trained may include a 5-layer structure, for example, layer 0 to layer 4, wherein layer 0 and layer 1 are frozen layers that do not participate in training, and layer 2, layer 3, and layer 4 are training layers that participate in training. This situation often occurs in deep learning, for example, in a scenario where parameters are frozen layer by layer and only the parameters of the subsequent layers are trained.

[0085] Therefore, for a model including frozen layers and training layers, its training may also include a pre-training 610 stage and a training 620 stage, wherein the processing in the pre-training 610 stage and the training 620 stage is similar to the processing in the above-mentioned reference pre-training 310 stage and training 320 stage.

[0086] Optionally, in the pre-training stage 610, the training data set may be traversed to generate an intermediate training data set using the frozen layer. In the training stage 620, the training layer may be trained based on the intermediate training data set to update the parameters of the training layer.

[0087] Figure 6 The memory optimization process for training a model including frozen layers and training layers is also shown, which is consistent with the reference Figure 5 The described processes are similar and will not be described again in detail in this disclosure.

[0088] Figure 7 is a schematic diagram showing a model training device 700 according to an embodiment of the present disclosure.

[0089] According to an embodiment of the present disclosure, the model training device 700 may include a model determination module 701 , a data preloading module 702 , and a model training module 703 .

[0090] The model determination module 701 may be configured to determine a first sub-model and a second sub-model in the model to be trained, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model. Optionally, the model determination module 701 may perform the operations described above with reference to step S201.

[0091] Optionally, for the model to be trained, a first sub-model that does not participate in the training can be determined. The first sub-model may include one or more sub-models whose parameters do not need to be updated during training. These sub-models may have stable capabilities (e.g., feature extraction, etc.) in advance, thereby providing reliable basic data for the subsequent second sub-model. Similarly, the second sub-model may also include one or more sub-models whose parameters need to be updated during training. It should be noted that all sub-models in the first sub-model should be located before all sub-models in the second sub-model, that is, run before the second sub-model in the forward propagation to ensure the data consistency of its calculation results.

[0092] The model determination module 701 can divide the original model to be trained into a frozen model and a training model, wherein the frozen model corresponds to the first sub-model mentioned above, i.e., the sub-model without parameter update, and the training model corresponds to the second sub-model mentioned above, i.e., the sub-model with parameter update.

[0093] The data preloading module 702 may be configured to generate an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in the video memory of the computing device before training the model to be trained, and cache the generated intermediate training data set. Optionally, the data preloading module 702 may perform the operations described above with reference to step S202.

[0094] Optionally, the data preloading module 702 can traverse the training data set to generate an intermediate training data set using the frozen model. Specifically, the training data set can be divided into multiple batches of training data, and for each batch of training data, such as batch i of training data, a forward propagation calculation on the frozen model can be performed, that is, the frozen model is used to generate a corresponding output, such as an intermediate training data set_i, which can represent the i-th subset of the intermediate training data set corresponding to the training data set. Therefore, based on n batches of training data, n subsets of the intermediate training data set can be generated, and these subsets can constitute the intermediate training data set.

[0095] Optionally, the cache of training data for each batch can be converted to a cache of each sample data to further improve the efficiency of data processing. Optionally, data can be cached using dataset categories and data loaders based on the existing Datasets framework.

[0096] Optionally, the output of the calculation of the frozen model is not directly discarded, but cached in the data cache module. Optionally, each intermediate training data can be associated with an index, wherein the index serves as a key of the intermediate training data, which can correspond to the index of the training sample corresponding to the intermediate training data in the training data set, and the intermediate training data can serve as a value.

[0097] The model training module 703 may be configured to train only the second sub-model in the computing device video memory, but not the first sub-model, in the training of the model to be trained, based on the cached intermediate training data set, so as to update the parameters of the second sub-model. Optionally, the model training module 703 may perform the operations described above with reference to step S203.

[0098] Optionally, the model training module 703 may include a supplementary caching mechanism for training samples in the training data set that have not been covered in previous preloading or training.

[0099] For example, in order to improve training efficiency, intermediate training data can be generated and cached for samples that were not fully utilized in the previous training process in the current training for direct use in the subsequent training process. For example, for each round of training, a judgment can be made for each training sample used in that round of training to determine whether there is intermediate training data corresponding to it in the current intermediate training data set. When it is confirmed that there is corresponding intermediate training data, the intermediate training data can be directly called as the input of the second sub-model for training. When there is no relevant data in the intermediate training data set, this means that the sample was not used for calculation in the previous preloading or training process, resulting in the absence of relevant data in the intermediate training data set. Therefore, the first sub-model can be used to generate corresponding intermediate training data for the training sample, and the newly generated data can be incorporated into the intermediate training data set. This flexible supplementary caching mechanism can enhance the reliability of the entire training process and improve the efficiency of model training.

[0100] Optionally, when using dataset categories and data loaders for data caching, for each round of training, the data loader can be used to randomly shuffle the training data in the training data subset used for each round of training at the beginning of each round of training, so that the order of training data in each batch is different, thereby improving the generalization ability of the model.

[0101] Optionally, after randomly shuffling the order of the training data, the data loader may be used to determine the index of each training sample to be used for the round of training, and use the index of the training sample to read the intermediate training data corresponding to the index from the dataset category.

[0102] Optionally, in the process of using the index of the training sample to read the intermediate training data corresponding to the index from the data set category, it can be determined based on the index whether the intermediate training data corresponding to the index exists in the data cache module. In the case of confirming that the corresponding intermediate training data exists, the intermediate training data can be directly called as the input of the second sub-model for training, and when the intermediate training data set does not have relevant data, therefore, the data preprocessing module can be used to generate the corresponding intermediate training data for the training sample by calling the data processing function of the first sub-model, so as to return the generated intermediate training data as the reading result to the data loader, and the intermediate training data is cached in the data cache module, for example, the intermediate training data is used as the value, and the index of the training sample corresponding to the intermediate training data is used as the key.

[0103] According to yet another aspect of the present disclosure, a model training device is also provided. Figure 8 A schematic diagram of a model training device 2000 according to an embodiment of the present disclosure is shown.

[0104] like Figure 8 As shown, the model training device 2000 may include one or more processors 2010, and one or more memories 2020. The memory 2020 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 2010, the model training method described above may be executed.

[0105] The processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may be an X86 architecture or an ARM architecture.

[0106] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0107] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig. 9 The architecture of the computing device 3000 shown in FIG. Fig. 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the model training method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Fig. 9 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig. 9 One or more components of a computing device are shown.

[0108] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the model training method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0109] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the model training method according to the embodiments of the present disclosure.

[0110] Embodiments of the present disclosure provide a model training method, apparatus, computer program product, and computer-readable storage medium.

[0111] The method provided by the embodiment of the present disclosure is for a model to be trained including multiple sub-models, determines a first sub-model that does not participate in the training, and a second sub-model that participates in the training and is located after the first sub-model, and uses the first sub-model to complete data preloading for the training data set before model training, so that the result of the data preloading can be directly called in the model training to perform parameter updates for the second sub-model. The method of the embodiment of the present disclosure can avoid repeated calculations of sub-models that do not participate in the training between different training cycles through the above-mentioned data preloading, saving a lot of computing power and improving the training efficiency of the model.

[0112] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0114] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A model training method, comprising: Determine a first sub-model and a second sub-model in a model to be trained, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model; Before training the model to be trained, generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in a video memory of a computing device, and caching the generated intermediate training data set; In the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device without training the first sub-model, so as to update the parameters of the second sub-model.

2. The method of claim 1, wherein: The training of the model to be trained includes multiple rounds of training, wherein each round of training uses a different training data subset of the training data set; Wherein, in the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device, but the first sub-model is not trained, so as to update the parameters of the second sub-model, including: For each round of training of the model to be trained, for each training data in the training data subset used for this round of training: Determining whether there is intermediate training data corresponding to the training data in the intermediate training data set; When it is determined that intermediate training data corresponding to the training data exists in the intermediate training data set, using the intermediate training data as input of the second sub-model to train the second sub-model; When it is determined that there is no intermediate training data corresponding to the training data in the intermediate training data set, intermediate training data corresponding to the training data is generated based on the training data using the first sub-model, and the intermediate training data is added to the intermediate training data set.

3. The method of claim 1, wherein: Before training the model to be trained, generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in the video memory of the computing device, and caching the generated intermediate training data set includes: Constructing a data set category, wherein the data set category includes a data preprocessing module and a data caching module; and For each training data in the at least part of the training data: Using a data preprocessing module, preprocessing the training data through a data processing function to generate intermediate training data corresponding to the training data, wherein the data processing function includes a calling function for the first sub-model; The generated intermediate training data is cached in the data cache module in the form of a key-value pair, wherein the intermediate training data serves as a value and an index of training data corresponding to the intermediate training data in the training data set serves as a key.

4. The method of claim 3, wherein: In the training of the model to be trained, based on the cached intermediate training data set, only the second sub-model is trained in the video memory of the computing device without training the first sub-model, so as to update the parameters of the second sub-model, including: For each round of training of the model to be trained: Determine, using the data loader, the index of each training data in the training data subset for the training round; Using the data loader, obtaining intermediate training data corresponding to the training data from the data set category based on the index of the training data; and The second sub-model is trained by using the returned intermediate training data as input of the second sub-model.

5. The method of claim 4, wherein: Using the data loader, acquiring intermediate training data corresponding to the training data from the data set category based on the index of the training data includes: In case the intermediate training data is acquired from the data cache module, returning the intermediate training data; In the case that the intermediate training data is not obtained from the data cache module, the data preprocessing module is used to generate intermediate training data corresponding to the training data, the intermediate training data is returned, and the intermediate training data is cached in the data cache module in the form of key-value pairs.

6. The method of claim 1, before training the model to be trained, generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model in the video memory of the computing device, and caching the generated intermediate training data set comprises: After generating an intermediate training data set based on at least part of the training data in the training data set using the first sub-model, caching the generated intermediate training data set in the video memory of the computing device, and releasing the first sub-model from the video memory of the computing device; Wherein, the method further comprises: During the training of the model to be trained, the second sub-model is loaded into the video memory of the computing device, so as to train the second sub-model based on the intermediate training data set cached in the video memory of the computing device.

7. The method of claim 6, further comprising: When the available space of the video memory of the computing device meets a predetermined condition, flattening the storage space of the intermediate training data set to merge the storage space of the intermediate training data in the intermediate training data set into a continuous storage space, and releasing video memory fragments between the storage spaces of the intermediate training data in the intermediate training data set; When the available space in the video memory of the computing device does not meet a predetermined condition, the intermediate training data set cached in the video memory of the computing device is transferred to the central processing unit, and during the training of the model to be trained, the required intermediate training data in the intermediate training data set is loaded from the central processing unit into the video memory of the computing device.

8. The method of claim 1, wherein: The first sub-model includes one or more first-layer structures whose parameters will not be updated during the training of the model to be trained, and the second sub-model includes one or more second-layer structures whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the one or more first-layer structures are located before the one or more second-layer structures.

9. A model training device, comprising: A model determination module is configured to determine a first sub-model and a second sub-model in a model to be trained, wherein the first sub-model is a sub-model whose parameters will not be updated during the training of the model to be trained, the second sub-model is a sub-model whose parameters will be updated during the training of the model to be trained, and in the training of the model to be trained, the first sub-model runs before the second sub-model; a data preloading module configured to generate an intermediate training data set in a video memory of a computing device using the first sub-model based on at least part of the training data in the training data set before training the model to be trained, and cache the generated intermediate training data set; and The model training module is configured to train only the second sub-model in the video memory of the computing device, without training the first sub-model, during the training of the model to be trained, based on the cached intermediate training data set, so as to update the parameters of the second sub-model.

10. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images