Data processing method and device for model training and computing equipment

Abstract

The invention discloses a data processing method and device for model training, computing equipment and a storage medium, and belongs to the technical field of artificial intelligence. In a data processing process, a video memory prediction model is established based on model parameter information of a model, a target batch size interval is acquired according to the model parameter information and an available video memory space size of a video card used for model training, and a video memory prediction result is obtained based on the target batch size interval, the video memory prediction model and the model parameter information. According to the method, the target batch size and the corresponding data processing scheme are obtained, so that data in the model training process are processed according to the target batch size and the corresponding data processing scheme, the purpose of accelerating model training is achieved, and the problems that a video memory is limited and the utilization rate of a video card is low in the model training process are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, computing device, and storage medium for model training. Background Art

[0002] Deep neural network (DNN) models are often used to handle complex tasks such as image recognition, target detection, and natural language processing. As the complexity of tasks continues to increase, researchers have proposed deeper models to extract deep features of data, which requires saving video memory space to increase the speed of model training.

[0003] At present, one method of saving video memory space when training a model is to process data based on recalculation technology. Recalculation technology means deleting data that is not used temporarily during training from the video memory, and restoring the deleted data by recalculating it based on the saved input data before the deleted data is used next time. By using the above data processing method, video memory space can be saved, and the acceleration effect brought by saving video memory space may cover the time consumed by recalculation.

[0004] The above data processing method can save limited video memory space, which hinders the further acceleration of model training. Summary of the invention

[0005] The embodiment of the present application provides a data processing method, device, computing device and storage medium for model training, which can realize rapid training of the model. The technical solution is as follows.

[0006] In a first aspect, a data processing method for model training is provided, the method comprising: establishing a video memory prediction model based on model parameter information of a DNN model to be trained, the model parameter information being used to describe parameters of the DNN model, the video memory prediction model being used to predict the size of video memory space occupied by training the DNN model under different batch sizes; obtaining a target batch size interval based on the model parameter information and the size of available video memory space of a graphics card; obtaining a target batch size and a corresponding data processing scheme based on the target batch size interval, the video memory prediction model and the model parameter information, the data processing scheme being used to process at least one of input data and output data of a layer in the DNN model during model training, the training throughput corresponding to the target batch size and the corresponding data processing scheme being the largest within the target batch size interval.

[0007] An embodiment of the present application provides a data processing method for model training, which can achieve the purpose of accelerating model training. During the data processing process, a video memory prediction model is established based on the model parameter information of the model. According to the model parameter information and the size of the available video memory space of the graphics card used for model training, a target batch size interval is obtained. Then, based on the target batch size interval, the video memory prediction model and the model parameter information, the target batch size and the corresponding data processing scheme are obtained, so that the data in the model training process is processed according to the target batch size and the corresponding data processing scheme, thereby solving the problems of limited video memory and low graphics card utilization during model training.

[0008] In some embodiments, establishing a video memory prediction model based on model parameter information of a DNN model to be trained includes: based on the DNN model, obtaining model parameter information, the model parameter information indicating the data amount of the model parameters of the DNN model; based on the model parameter information, determining the video memory occupancy of each layer of the DNN model, the video memory occupancy of each layer of the DNN model including the data amount of the model parameters of each layer of the DNN model and the data amount of output data; based on the video memory occupancy of each layer of the DNN model, establishing a video memory prediction model.

[0009] In the above process, by collecting model parameter information and GPU memory information, it can be ensured that the use of video memory in model training can be more efficient.

[0010] In some embodiments, based on the model parameter information and the available video memory space size of the graphics card, obtaining the target batch size interval includes:

[0011] Based on the video memory usage of each layer of the DNN model, the maximum recomputation savings are obtained. The maximum recomputation savings represent the maximum video memory that can be saved by recomputation when the batch size is 1.

[0012] Based on the model parameter information of the DNN model, the available video memory space of the graphics card, and the maximum recalculation saving, the lower bound of the target batch size range is obtained;

[0013] Based on the video memory usage of each layer of the DNN model, the maximum offloading checkpoint savings are obtained. The maximum offloading checkpoint savings represent the maximum video memory that can be saved by the offloading checkpoint when the batch size is 1.

[0014] Based on the model parameter information of the DNN model, the available video memory space of the graphics card, the maximum recomputation savings, and the maximum offloading savings, the upper bound of the target batch size range is obtained.

[0015] In some embodiments, based on the target batch size interval, the video memory prediction model, and the model parameter information, obtaining the target batch size and the corresponding data processing solution includes:

[0016] Based on the video memory prediction model, obtain the theoretical video memory occupancy corresponding to each batch size within the target batch range;

[0017] For each batch size, if the theoretical video memory occupancy corresponding to the batch size is less than or equal to the available video memory space size, the preset data processing scheme is determined as the data processing scheme corresponding to the batch size;

[0018] If the theoretical video memory usage corresponding to the batch size is larger than the available video memory space, a data processing solution that meets the target video memory saving amount is generated based on the target video memory saving amount and the video memory usage of each layer of the DNN model;

[0019] Based on the training throughput of the data processing solutions corresponding to the multiple batch sizes, a target batch size and a corresponding data processing solution are determined.

[0020] In the above process, before generating the data processing plan, it will be judged whether the current graphics card can meet the training requirements, that is, the scenarios with unlimited and limited video memory will be identified. When it is determined that the training requirements can be met, the data processing plan will not be generated. This can improve the efficiency of obtaining data processing plans, avoid the acquisition of additional data processing plans for scenarios with unlimited video memory, and improve the utilization rate of video memory.

[0021] In some embodiments, based on the target video memory saving amount and the video memory occupancy of each layer of the DNN model, generating a data processing solution corresponding to the batch size that satisfies the target video memory saving amount includes:

[0022] Based on the theoretical video memory occupancy and the available video memory space size, a target video memory saving amount is obtained, where the target video memory saving amount indicates the amount of video memory that should be saved for training the DNN model;

[0023] Based on the memory usage of each layer of the DNN model, a model block sequence of the DNN model is obtained, where the model block sequence includes a plurality of model blocks sorted from low to high in terms of computational intensity, where the computational intensity is obtained based on model parameter information;

[0024] Based on the model block sequence and the video memory occupancy of each layer of the DNN model, a data processing solution is obtained in which the saving amount satisfies the target video memory saving amount;

[0025] If there is a data processing solution that satisfies the target video memory saving amount, the data processing solution is output as the data processing solution corresponding to the batch size.

[0026] In some embodiments, based on the model block sequence and the memory usage of each layer of the DNN model, a data processing scheme for obtaining a savings amount that satisfies a target memory savings amount includes:

[0027] Based on the model block sequence, obtain a recalculation plan of the batch size, where the recalculation plan indicates the model blocks to be recalculated during the model training process;

[0028] If the theoretical recalculation saving amount corresponding to the recalculation scheme is greater than or equal to the target video memory saving amount, the recalculation scheme is determined as the data processing scheme that meets the target video memory saving amount;

[0029] If the theoretical recomputation savings corresponding to the recomputation plan is less than the target video memory savings, an offload checkpoint plan is generated for the batch size, where the offload checkpoint plan indicates the checkpoint data to be offloaded during the model training process;

[0030] If the theoretical offloading saving amount of the offloading checkpoint scheme is greater than or equal to the minimum offloading data amount, the offloading checkpoint scheme and the recalculation scheme are determined as the data processing scheme for the batch size.

[0031] In the above process, the recalculation method implements an efficient and accurate recalculation operator selection strategy. The obtained recalculation operator set, that is, each model block in the recalculation scheme, can minimize the time overhead caused by recalculation, that is, minimize the time overhead of the recalculation data processing scheme. After obtaining the recalculation scheme, it is also determined whether it is necessary to generate an unloading checkpoint scheme. When it is necessary to generate an unloading checkpoint scheme, a set of unloaded checkpoint data that can meet the target video memory saving amount to the greatest extent will be calculated to minimize the time overhead caused by the unloading checkpoint, that is, minimize the time overhead of the data processing scheme of the unloading checkpoint.

[0032] In some embodiments, generating a target offload checkpointing scheme for the batch size includes:

[0033] Obtain a target unloading interval value sequence of the DNN model, where the target unloading interval value sequence consists of unloading interval values, where the unloading interval value refers to the number of model blocks between two unloadings, and the unloading interval value ensures that there is no overlapping time period between two unloadings;

[0034] Based on the minimum unloading data volume and the target unloading interval value sequence of the DNN model, a target unloading checkpoint solution is obtained. The minimum unloading data volume refers to the amount of checkpoint data that needs to be unloaded at least during the model training process.

[0035] In some embodiments, based on the training throughput of the data processing schemes corresponding to the multiple batch sizes, determining the target batch size and the corresponding data processing scheme includes:

[0036] Based on multiple batch sizes within the target batch size range and corresponding data processing methods, the DNN model is trained to obtain a training throughput rate corresponding to each batch size;

[0037] Based on the training throughput corresponding to each batch size, the target batch size and the corresponding data processing solution are obtained.

[0038] In the above process, when selecting the target batch size, through simple tests of multiple batch sizes within the target batch size range, the target batch size and the corresponding data processing solution in the actual production environment can be quickly obtained, which can effectively guide users to perform model training while maximizing training efficiency.

[0039] In a second aspect, a data processing device for model training is provided, which is used to execute the data processing method for model training. Specifically, the data processing device for model training includes a functional module for executing the data processing method for model training provided in the first aspect or any optional method of the first aspect.

[0040] In a third aspect, a computing device or a computing device cluster is provided, which includes a processor, and the processor is used to execute program code so that the computing device or the computing device cluster executes to implement the operations performed by the data processing method for model training as described above.

[0041] In a fourth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor to enable a computing device to perform operations performed by the above-mentioned data processing method for model training.

[0042] In a fifth aspect, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the method provided in various optional implementations of the above-mentioned first aspect.

[0043] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0046] Figure 3 is a flow chart of a data processing method for model training provided in an embodiment of the present application;

[0047] Figure 4It is a flow chart of a method for establishing a video memory prediction model provided by an embodiment of the present application;

[0048] Figure 5 is a flow chart of a method for obtaining a target batch size interval provided in an embodiment of the present application;

[0049] Figure 6 It is a flow chart of a method for obtaining a target batch size and a corresponding data processing solution provided in an embodiment of the present application;

[0050] Figure 7 It is a flow chart of a method for generating a data processing solution corresponding to a batch size provided by an embodiment of the present application;

[0051] Figure 8 is a schematic diagram of unloading checkpoint data provided by an embodiment of the present application;

[0052] Fig. 9 is a schematic diagram of another method for unloading checkpoint data provided in an embodiment of the present application;

[0053] Fig.10 This is a flow chart of a model training provided in an embodiment of the present application;

[0054] Fig.11 It is a structural schematic diagram of a data processing device for model training provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0056] Figure 1 1 is a schematic diagram of a computing device provided in an embodiment of the present application. It should be understood that the computing device described below can implement any function in any of the following methods. Generally, the electronic device 100 includes: a processor 101 and a memory 102.

[0057] The processor 101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0058] The memory 102 may include one or more computer-readable storage media, which may be non-transitory. The memory 102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 102 is used to store at least one program, which is used to be executed by the processor 101 to implement the data processing method for model training provided in the method embodiment of the present application.

[0059] In some embodiments, the electronic device 100 may further include: a peripheral device interface 103 and at least one peripheral device. The processor 101, the memory 102 and the peripheral device interface 103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 103 via a bus, a signal line or a circuit board.

[0060] In some embodiments, the computing device may be a separate physical server, or may be implemented as Figure 2The computing device cluster shown is a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking the computing device as a cloud server as an example, the computing device can also be called a cloud platform (i.e., the abbreviation of cloud computing platform), which refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data computing processing is processed and analyzed at the remote end and then returned to the user, which has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud platform can realize the rapid issuance and release of configurable computing resources with a small management cost or a low interaction complexity between users and service providers.

[0061] The data processing method for model training provided in the embodiment of the present application can achieve the purpose of accelerating DNN model training and solve the technical problems of limited video memory and low graphics card utilization during model training. Figure 3 The three parts are shown.

[0062] 301. Based on the model parameter information of the DNN model to be trained, establish a video memory prediction model, where the model parameter information is used to describe the parameters of the DNN model, and the video memory prediction model is used to predict the size of video memory space occupied by training the DNN model under different batch sizes.

[0063] 302. Based on the above model parameter information and the available video memory space size of the graphics card, obtain a target batch size range.

[0064] 303. Based on the above-mentioned video memory prediction model, the target batch size range and the above-mentioned model parameter information, obtain the target batch size and the corresponding data processing scheme, where the data processing scheme is used to process at least one of the input data and the output data in the model training, and the training throughput corresponding to the target batch size and the corresponding data processing scheme is the largest within the target batch size range.

[0065] Among them, the recalculation scheme indicates the model blocks to be recalculated during the model training process, and the unloading checkpoint scheme indicates the checkpoint data to be unloaded during the model training process.

[0066] During model training, for the layer indicated in the recalculation scheme to be recalculated, the output data of the layer is deleted, and before the deleted data is used next time, it is recalculated based on the saved input data to restore the data. For recalculation, the layer to be recalculated is called a checkpoint, and the input data of the layer to be recalculated is called the checkpoint data.

[0067] For the checkpoint data to be unloaded as indicated in the unloading checkpoint plan, the unloading method adopts asynchronous replication to unload the checkpoint data to the memory. Before the data is used next time, the data is retrieved back to the video memory space by asynchronous replication. Asynchronous replication means that the unloading of checkpoint data is parallel to the calculation of the model.

[0068] In an embodiment of the present application, a video memory prediction model is established based on the DNN model. Therefore, it is possible to predict the size of the video memory space to be occupied when training based on different batch sizes. Furthermore, after determining the target batch size range that meets the training requirements, it is possible to select from different batch sizes the batch size with the highest training throughput when the corresponding data processing solution is used for model training. This improves the model training speed in scenarios where video memory is limited.

[0069] Below, based on Figure 4 The process shown in the figure introduces the process of the first part above. Figure 4 is a flowchart of establishing a video memory prediction model provided by an embodiment of the present application. Figure 4 , the process includes the following steps.

[0070] 401. Obtain model parameter information of the DNN model to be trained, and determine the video memory occupancy of each layer of the DNN model based on the model parameter information.

[0071] In some embodiments, the model parameter information of the above-mentioned DNN model is obtained by scanning the model structure of the DNN model through the torchstat tool.

[0072] The model parameter information indicates the data volume of the model parameters of the DNN model, and the model parameters include parameters such as error gradient, weight, and weight gradient.

[0073] Among them, the video memory occupancy of each layer of the DNN model includes the data volume of the model parameters of each layer of the DNN model and the data volume of the output data. The data volume of the output data is the size of the feature map output after the calculation of the layer.

[0074] In some embodiments, the video memory occupancy of each layer of the model refers to the sum of the feature map size, error gradient size, weight size, and weight gradient size of each layer of the model. When training a deep learning model under the pytorch framework, the video memory occupancy of each layer of the model refers to the sum of the feature map size, weight size, and weight gradient size of each layer of the model.

[0075] 402. Based on the model parameter information and the memory occupancy of each layer, a memory prediction model is established, wherein the memory prediction model is used to predict the theoretical memory occupancy for training the DNN model under different batch sizes.

[0076] The theoretical memory usage in the model training process is represented by M(b). The memory prediction model in the above steps is shown in formula (1):

[0077]

[0078] In formula (1), b is used to represent the batch size of model training, L is used to represent the number of model layers, and O l It is used to represent the feature map size of the lth layer, E l Used to indicate the magnitude of the error gradient, P l Used to indicate the weight size, G l Used to represent the size of the weight gradient, b is a positive integer greater than zero.

[0079] In some embodiments, when training a deep learning model under the pytorch framework, the maximum memory usage of the video memory does not include the size of the error gradient. Therefore, the video memory prediction model is shown in formula (2):

[0080]

[0081] In formula (2), b is used to represent the batch size of model training, L is used to represent the number of model layers, and O l It is used to represent the feature map size of the lth layer, P l Used to indicate the weight size, G l Used to indicate the size of the weight gradient. b is a positive integer greater than zero.

[0082] Below, based on Figure 5 The process shown in the figure introduces the process of the second part mentioned above. Figure 5 is a flowchart of obtaining a target batch size interval provided in an embodiment of the present application. Figure 5 , the process includes the following steps.

[0083] 501. Based on the video memory usage of each layer of the DNN model, obtain the maximum recomputation saving, where the maximum recomputation saving represents the maximum video memory that can be saved by recomputation when the batch size is 1.

[0084] The maximum recomputation savings refers to the sum of the video memory that can be saved by recomputing each model block in the model when the batch size is 1. A model block refers to a collection of layers in a model, and a model block includes at least one layer in the model.

[0085] 502. Based on the model parameter information of the DNN model, the available video memory space size of the graphics card, and the maximum recalculation saving, obtain the lower bound of the target batch size range.

[0086] The lower bound of the target batch size interval is obtained by the following formula (3):

[0087]

[0088] In formula (3), b min It is used to represent the lower bound of the target batch size interval, M gpu It is used to indicate the size of the available video memory space of the graphics card, L is used to indicate the number of model layers, and P l Used to indicate the weight size, G l Used to indicate the size of the weight gradient, O l Used to represent the feature map size of the lth layer, max(M recp (1)) is used to indicate the maximum recalculation savings, and square brackets are used to indicate rounding down. min An integer greater than zero.

[0089] Among them, the available video memory space size M gpu Calculated based on the video memory information and video memory usage information of the graphics card used for model training.

[0090] It should be noted that the lower bound of the target batch size interval and the lower bound of the target batch size interval plus one should satisfy the relationship shown in formula (4) and formula (5):

[0091] M offload (b min )≤0 (4)

[0092] M offload (b min +1)>0 (5)

[0093] Among them, b min It is used to represent the lower bound of the target batch size interval, M offload (b min ) is used to indicate the min Minimum amount of offloaded data when training a model for batch size.

[0094] For current graphics card products, the calculation is high-speed, and the time consumed by recalculation can be completely covered by the model training time saved by increasing the batch size. In most models, the recalculation method is used to increase the maximum batch size of the model to the batch size obtained by the maximum recalculation savings. This batch size will have the highest training throughput in the interval [1, the batch size], so this batch size is used as the lower bound of the target batch size interval.

[0095] 503. Based on the video memory occupancy of each layer of the DNN model, a maximum offloading checkpoint saving is obtained, where the maximum offloading checkpoint saving represents the maximum video memory that can be saved by the offloading checkpoint when the batch size is 1.

[0096] The maximum unloading checkpoint saving refers to the sum of the data of all checkpoints to be unloaded determined according to the unloading interval value sequence when the batch size is 1.

[0097] The unloading interval value sequence is obtained by using the above model parameter information using a dynamic selection method. The unloading interval value in the unloading interval value sequence can ensure that during the model training process, the unloading time of a checkpoint data can be exactly hidden by using the unloading interval value plus the calculation time of a model block.

[0098] 504. Based on the model parameter information of the DNN model, the available video memory space size of the graphics card, the maximum recalculation saving amount and the maximum offloading saving amount, obtain the upper bound of the target batch size range.

[0099] The upper bound of the target batch size interval is obtained by the following formula (6):

[0100]

[0101] In formula (6), b max Used to represent the upper bound of the target batch size interval, M gpu It is used to indicate the size of the available video memory space of the graphics card, L is used to indicate the number of model layers, and P l Used to indicate the weight size, G l Used to indicate the size of the weight gradient, O l Used to represent the feature map size of the lth layer, max(M recp (1)) is used to represent the maximum recomputation savings, max(M off (1)) is used to represent the maximum offload checkpoint savings, and square brackets are used to indicate rounding down. max is a positive integer greater than zero.

[0102] Among them, the available video memory space size M gpu Calculated based on the video memory information and video memory usage information of the graphics card used for model training.

[0103] If the batch size is further increased, a large amount of checkpoint data that cannot be overlapped by the computation time in model training needs to be offloaded, and the computation speed of the GPU is limited by the speed of storage and communication. Therefore, data transfer operations that cannot be overlapped by the computation in model training should be avoided. Therefore, the batch size obtained by the maximum offload memory savings is used as the upper bound of the target batch size range to ensure operation efficiency.

[0104] Below, based on Figure 6 The process shown in the figure introduces the process of the third part mentioned above. Figure 6 is a flowchart of obtaining the target batch size and the corresponding data processing solution provided by the embodiment of the present application. Figure 6 , the process includes the following steps.

[0105] 601. Based on the video memory prediction model, obtain the theoretical video memory occupancy corresponding to each batch size in the target batch size range.

[0106] Among them, each batch size within the target batch range is substituted into the above-mentioned video memory prediction model to output the theoretical video memory occupancy corresponding to each batch size.

[0107] 602. For each batch size, based on the above-mentioned available video memory space size and theoretical video memory occupancy, determine whether to generate a data processing solution for the DNN model under this batch size. If so, execute step 603; if not, determine the preset data processing solution as the data processing solution corresponding to the batch size.

[0108] In the embodiment of the present application, according to the theoretical video memory occupancy and the available video memory space size corresponding to each batch size, determining whether to generate a data processing scheme for the DNN model under the batch size includes: comparing the theoretical video memory occupancy M(b) under the batch size with the available video memory space size M of the graphics card used. gpu Compare, if M(b) is less than or equal to M gpu , it means that the graphics card can currently meet the training requirements, and there is no need to generate a data processing solution for the DNN model under this batch size. The subsequent training can be carried out with the preset data processing solution; if M(b) is greater than M gpu , it means that the graphics card cannot meet the training requirements, and step 603 is executed.

[0109] 603. Generate a data processing plan corresponding to the batch size based on the target video memory saving amount and the video memory occupancy of each layer of the model.

[0110] Among them, for any batch size, the batch size may correspond to a data processing solution that can meet the target video memory saving amount. If the batch size cannot find a data processing solution that can meet the target video memory saving amount, the batch size is discarded from the target batch size range.

[0111] Among them, the data processing scheme corresponding to the generated batch size can be found in Figure 7 As shown, the process includes the following steps.

[0112] 6031. Based on the theoretical video memory occupancy and the available video memory space size, obtain a target video memory saving amount, where the target video memory saving amount indicates the amount of video memory that should be saved for training the DNN model.

[0113] The target video memory saving is the difference between the theoretical video memory usage and the available video memory space size, which is achieved by formula (7):

[0114] M target (b) = M(b) - M gpu (7)

[0115] Among them, b is used to represent the batch size of model training, M target (b) is used to represent the target memory savings when the model is trained with a batch size of b. M(b) is used to represent the theoretical memory usage when the model is trained with a batch size of b. gpu Used to indicate the size of the available video memory space of the graphics card. b is an integer greater than zero.

[0116] 6032. Based on the memory usage of each layer of the above model, obtain a model block sequence of the DNN model, where the model block sequence includes multiple model blocks sorted from low to high computing intensity.

[0117] Among them, the computing intensity of each model block in the DNN model is calculated by the memory occupancy of each layer of the above model, and the model blocks are sorted from low to high according to the computing intensity to obtain the model block sequence.

[0118] The computational intensity of each model block in the DNN model is obtained by formula (8):

[0119] k = the computational effort of the model block / the memory usage of the model block (8)

[0120] Among them, k is used to represent the computational intensity of each model block. The model block is a collection of layers in the model, and the computational amount of the model block is obtained through the above-mentioned model parameter information.

[0121] 6033. Based on the model block sequence, obtain a recalculation plan for the batch size. If the theoretical recalculation savings corresponding to the recalculation plan are greater than or equal to the target video memory savings, determine the recalculation plan as a data processing plan that meets the target video memory savings.

[0122] The theoretical recomputation savings refers to the amount of video memory that can be theoretically saved by recomputing the model blocks in the recomputation scheme.

[0123] In step 6033, the model blocks in the model block sequence are added to the recalculation scheme in order from front to back, and the current theoretical recalculation savings of the recalculation scheme are obtained. If the current theoretical recalculation savings of the recalculation scheme is greater than the target video memory savings, it means that the current recalculation scheme can meet the model training requirements, and the corresponding batch size is output, and the current recalculation scheme is output as the recalculation scheme of the batch size, and subsequent model training is performed based on the recalculation scheme; if the theoretical recalculation savings after all model blocks in the model block sequence are added to the recalculation scheme is still less than the target video memory savings, the recalculation scheme containing all model blocks is output as the target recalculation scheme of the corresponding batch size, and step 6034 is executed to further determine the unloading checkpoint scheme for the DNN model under the batch size.

[0124] The recalculation scheme is used to indicate the model blocks to be recalculated in the model, which can be represented as a set of model blocks. In the embodiment of the present application, the theoretical recalculation saving is obtained by formula (9):

[0125]

[0126] In formula (9), b is used to represent the batch size of model training, M recp (b) is used to represent the theoretical recomputation savings when training the model with a batch size of b, Rec(b) is used to represent the number of model blocks in the recomputation scheme, and m l (b) is used to represent the amount of video memory that can be saved by recomputing the lth model block in the recomputation scheme when the model is trained with a batch size of b, and b is an integer greater than zero.

[0127] In the above step 6033, the process of obtaining the recalculation scheme refers to, first, adding the model block located at the previous preset position in the model block sequence to the recalculation scheme according to the order in the model block sequence, obtaining the current theoretical recalculation saving, and increasing or decreasing the model blocks in the current recalculation scheme based on the size relationship between the current theoretical recalculation saving and the target video memory saving. If the current theoretical recalculation saving is greater than the target video memory saving, then the model block added later is taken out from the recalculation scheme. If the current theoretical recalculation saving is less than the target video memory saving, then the model blocks are continued to be added to the recalculation scheme according to the order in the model block sequence until the theoretical recalculation saving is just greater than or equal to the target video memory saving. If the theoretical recalculation saving after adding all the model blocks in the model block sequence to the recalculation scheme is still less than the target video memory saving, then the recalculation scheme containing all the model blocks is output as a recalculation scheme of the corresponding batch size, and step 6034 is executed to further determine the unloading checkpoint scheme for the DNN model.

[0128] It should be noted that the theoretical recomputation savings is just greater than or equal to the target video memory savings when the recomputation video memory optimization method can meet the model training requirements, that is, when the number of model blocks in the recomputation scheme is less than or exactly equal to the number of model blocks in the DNN model, the number of model blocks in the recomputation scheme satisfies the relationship shown in formula (10) and formula (11):

[0129]

[0130]

[0131] Where b is used to represent the batch size of model training, Rec(b) is used to represent the number of model blocks in the recalculation scheme, and m l (b) is used to represent the amount of video memory that can be saved by recalculating the lth model block in the recalculation scheme when the model is trained with a batch size of b. M target (b) is used to represent the target video memory savings when training the model with a batch size of b, where b is an integer greater than zero.

[0132] Among them, the number of model blocks in the recalculation scheme is exactly equal to the number of model blocks in the DNN model, which means that the theoretical recalculation savings after adding all model blocks in the model block sequence to the recalculation scheme is just greater than the target video memory savings, and reducing one model block will make the theoretical recalculation savings less than the target video memory savings.

[0133] 6034. If the theoretical recomputation savings corresponding to the recomputation plan is less than the target video memory savings, generate an offload checkpoint plan for the batch size.

[0134] Step 6034 includes the following steps 6034A and 6034B:

[0135] 6034A. Obtain a target unloading interval value sequence of the DNN model, where the target unloading interval value sequence consists of unloading interval values, wherein the unloading interval value refers to the number of model blocks between two unloadings, and the unloading interval value ensures that there is no overlapping time period between two unloadings.

[0136] In some embodiments, a dynamic selection method is used to obtain a target unloading interval value sequence of the DNN model according to the above model parameter information. Obtaining an unloading interval value sequence by dynamic selection indicates that the unloading interval values ​​in an unloading interval value sequence can be different values.

[0137] Among them, the unloading interval value can ensure that when the model is trained, the unloading time of a checkpoint data can be exactly hidden by the calculation time of the unloading interval value plus a model block, that is, the calculation time consumed for calculating the unloading interval value model blocks after the current checkpoint to be unloaded is less than the unloading time of the checkpoint data, and the calculation time consumed for calculating the unloading interval value + 1 model blocks after the checkpoint to be unloaded is greater than the unloading time of the checkpoint data.

[0138] In some embodiments, Figure 8 As shown, there is a situation where the unloading interval value needs to be set to 0. The calculation time of a model block can just hide the unloading time of one checkpoint data. The "Memcpy HtoD" in a box represents the unloading of a checkpoint data, and a "model block" represents the calculation time of the model block.

[0139] In some embodiments, Fig. 9 As shown, there is a situation where the unloading interval value needs to be set to 1, so that the calculation time of the two model blocks can just hide the unloading time of one checkpoint data. The "Memcpy HtoD" in a box represents the unloading of a checkpoint data, and a "model block" represents the calculation time of the model block.

[0140] 6034B. Obtain an unloading checkpoint scheme through the minimum unloading data volume and target unloading interval value sequence of the DNN model. If the theoretical unloading savings of the unloading checkpoint scheme is greater than or equal to the minimum unloading data volume, determine the unloading checkpoint scheme and the above-mentioned recalculation scheme as the data processing scheme for the batch size.

[0141] The minimum amount of unloaded data is obtained by formula (12):

[0142] M offload (b) = M target (b)-M recp (b) (12)

[0143] In formula (12), b is used to represent the batch size of model training, M offload (b) is used to represent the minimum amount of unloaded data when the model is trained with a batch size of b, M target (b) is used to represent the target memory savings when training the model with a batch size of b, M recp (b) is used to represent the theoretical recomputation savings when training the model with a batch size of b.

[0144] Among them, the process of obtaining the target unloading checkpoint plan is as follows: taking N as the unloading interval value in the target unloading interval value sequence, the checkpoint data of the first model block of the model is added to the unloading checkpoint plan from front to back according to the order of the original model blocks in the model, and then the checkpoint data of the N+1th model block after the model block just added to the unloading checkpoint plan are added to the unloading checkpoint plan one by one. Each time a model block is added, the amount of video memory that can be saved by unloading the current model block in the unloading checkpoint plan is summed to obtain the theoretical unloading savings. When the theoretical unloading savings is just greater than or equal to the minimum unloading data amount, the current unloading checkpoint plan is output as the target unloading checkpoint plan of the batch size.

[0145] The theoretical unloading savings are achieved through formula (13):

[0146]

[0147] In formula (13), b is used to represent the batch size of model training, M off (b) is used to represent the theoretical offloading memory savings when the model is trained with a batch size of b. Off(b) is used to represent the number of elements in the offloading checkpoint scheme when the model is trained with a batch size of b. m l (b) is used to represent the amount of video memory that can be saved by offloading the lth checkpoint in the offloading checkpoint scheme when the model is trained with a batch size of b.

[0148] It should be noted that the target offloading checkpoint scheme must satisfy the relationship shown in formula (14) and formula (15):

[0149]

[0150]

[0151] Where b is used to represent the batch size of model training, Off(b) is used to represent the number of elements in the target unloading checkpoint scheme when the model is trained with a batch size of b, and m l (b) is used to represent the amount of video memory that can be saved by unloading the lth checkpoint in the target offloading checkpoint scheme when the model is trained with a batch size of b, M offload (b) is used to represent the minimum amount of unloaded data when the model is trained with a batch size of b. b is an integer greater than zero.

[0152] In some embodiments, if the offload checkpoint solution determined based on step 6034 includes all offloadable checkpoints and the theoretical offload savings are still less than the minimum offload data volume, the batch size is discarded. If the theoretical offload savings of the offload checkpoint solution that includes all offloadable checkpoint data is still less than the minimum offload data volume, it means that the currently available solution still cannot meet the training requirements, and it can be recommended to use a smaller batch size for model training.

[0153] In this process, a recalculation scheme is determined to determine at which checkpoints the DNN model can be recalculated. If the recalculation still cannot meet the training requirements, a checkpoint offloading scheme is further determined to save video memory and achieve further optimization.

[0154] In the process of determining the offloading checkpoint scheme, the problem of how to minimize the communication overhead is fully considered. On the one hand, in some embodiments, the data transmission between the GPU and the CPU involved in the target offloading checkpoint scheme is performed by asynchronous replication to perform the offloading of each checkpoint data, without letting the computing flow wait for a long time for communication. The asynchronous replication means that the offloading checkpoint data and the model calculation are parallel; on the other hand, the target offloading interval value of appropriate size is dynamically selected to ensure that the offloading interval value plus the calculation time of a model block can be used during model training to better hide the communication time of one checkpoint data; finally, the number of offloading checkpoints is controlled so that the theoretical offloading savings just meet the minimum offloading data amount.

[0155] 604. Based on multiple batch sizes within the above target batch size range and corresponding data processing methods, the model is trained to obtain a training throughput corresponding to each batch size.

[0156] The data processing scheme includes at least a recalculation scheme and may also include an unloading checkpoint scheme.

[0157] Among them, training the model means that for each batch size, multiple iterative training is performed according to the batch size, the training throughput corresponding to each iteration is obtained, and the average of the training throughputs of multiple iterations is used as the training throughput corresponding to the batch size.

[0158] The multiple batch sizes in step 604 refer to batch sizes within the target batch size interval after discarding batch sizes that do not meet the target video memory saving amount.

[0159] 605. Based on the training throughputs corresponding to the respective batch sizes, a target batch size and a corresponding data processing solution are obtained, wherein the target batch size refers to a batch size that maximizes the training throughput.

[0160] In step 605, the training throughputs corresponding to each batch size are compared to obtain the maximum training throughput. According to the maximum training throughput, the corresponding batch size is obtained as the target batch size, and the target batch size and the corresponding data processing scheme are output. It should be noted that the determined target batch size may be a batch size corresponding to a preset data processing scheme, or may be a batch size corresponding to a data processing scheme determined based on steps 6031 to 6036.

[0161] 606. Based on the target batch size and the corresponding data processing scheme, the DNN model is trained.

[0162] Among them, for the recalculation scheme in the data processing scheme, during the training process, if the recalculation scheme indicates that the lth layer is the layer to be recalculated, then after the lth layer is calculated, the output data of the lth layer is deleted, and when the output data is needed, the input data is recalculated through the lth layer; if the data processing scheme includes an unloading checkpoint scheme, if the unloading checkpoint scheme indicates that the lth layer is the checkpoint to be unloaded, then after the lth layer is calculated, the input data of the lth layer in the GPU is unloaded to the CPU, and when the input data is needed, the input data of the lth layer in the CPU is retrieved to the GPU.

[0163] It should be noted that Fig.10 The process of automatic training based on the memory optimization strategy and the computational flow graph of the model is demonstrated. The computational flow graph is determined based on the DNN model and is used to represent the computational process in the DNN model. The solid arrows represent data dependencies, the dotted lines represent the recalculated paths, the circles represent intermediate variables, and the circles with crosses represent that the intermediate variables are discarded or unloaded. The implementation method is as follows: When the model is forward propagated, the intermediate variables of the recalculated model blocks are automatically deleted, and the checkpoint data to be unloaded is automatically exchanged to the CPU memory in an asynchronous replication manner. When the model is reverse propagated, the checkpoint data saved in the CPU memory is automatically exchanged back to the GPU in advance in an asynchronous replication manner, and the deleted intermediate variables are automatically restored based on the checkpoint data on the GPU.

[0164] The embodiment of the present application provides an application example. In this application example, in an experimental environment where the CPU is Intel(R) Xeon(R) Gold6248CPU@2.50GHz, the memory is 192GB DDR4, the GPU is a single 32GB Tesla V100, and the operating system is Ubuntu20.04, the 1.12.1 version of pytorch and the 10.2 version of cuda are used to optimize the ResNet2402 model. The experimental results are shown in Table 1.

[0165] Table 1

[0166]

[0167] The results in Table 1 show that under a given experimental environment, when no optimization is performed, the maximum batch size of the baseline model training is 3, and the average throughput is 1.58 images per second; after implementing the technical solution of the present application, the target batch size interval is [12, 27], the target batch size is 16, and the average throughput is 2.02 images per second, which is 31% higher than the baseline. The embodiment of the present application can simultaneously improve the batch size and training speed of the ResNet model on the GPU.

[0168] In an embodiment of the present application, a video memory prediction model is established based on the DNN model. Therefore, it is possible to predict the size of the video memory space to be occupied when training based on different batch sizes. Furthermore, after determining the target batch size range that meets the training requirements, it is possible to select from different batch sizes the batch size with the highest training throughput when the corresponding data processing solution is used for model training. In this way, in a scenario where video memory is limited, a batch size that can increase the training speed is selected, thereby optimizing the model training.

[0169] The above describes the method of the embodiment of the present application, and the following describes the device of the embodiment of the present application. It should be understood that the device described below has any function of the computing device in the above method. Figures 1 to 10 The data processing method for model training according to the embodiment of the present application is described in detail. Based on the same application concept, the following will be combined with Fig.11 The data processing device for model training according to the embodiment of the present application is described. It should be understood that the technical features described in the method embodiment are also applicable to the following device embodiment.

[0170] See also Fig.11 , an embodiment of the present application provides a data processing device for model training, the device comprising:

[0171] A video memory prediction model building module 1101 is used to build a video memory prediction model based on model parameter information of a DNN model to be trained, wherein the model parameter information is used to describe parameters of the DNN model, and the video memory prediction model is used to predict the size of video memory space occupied by training the DNN model under different batch sizes;

[0172] A batch size interval acquisition module 1102 is used to acquire a target batch size interval based on the model parameter information and the available video memory space size of the graphics card;

[0173] The data processing scheme acquisition module 1103 is used to obtain the target batch size and the corresponding data processing scheme based on the target batch size interval, the video memory prediction model and the model parameter information. The data processing scheme is used to process at least one of the input data and output data of the middle layer of the DNN model during model training. The training throughput corresponding to the target batch size and the corresponding data processing scheme is the largest within the target batch size interval.

[0174] In some embodiments, the video memory prediction model building module is used to:

[0175] Based on the DNN model, acquiring model parameter information, wherein the model parameter information indicates the data volume of the model parameters of the DNN model;

[0176] Based on the model parameter information, determine the memory usage of each layer of the DNN model, where the memory usage of each layer of the DNN model includes the data volume of the model parameters of each layer of the DNN model and the data volume of the output data;

[0177] A video memory prediction model is established based on the video memory occupancy of each layer of the DNN model.

[0178] In some embodiments, the batch size interval acquisition module is used to:

[0179] Based on the video memory occupancy of each layer of the DNN model, a maximum recomputation saving is obtained, where the maximum recomputation saving represents a maximum video memory saving that can be saved by recomputation when the batch size is 1;

[0180] Based on the model parameter information of the DNN model, the available video memory space of the graphics card, and the maximum recalculation saving, obtaining a lower bound of the target batch size interval;

[0181] Based on the video memory occupancy of each layer of the DNN model, a maximum offloading checkpoint saving is obtained, where the maximum offloading checkpoint saving represents the maximum video memory that can be saved by the offloading checkpoint when the batch size is 1;

[0182] Based on the model parameter information of the DNN model, the available video memory space size of the graphics card, the maximum recalculation savings and the maximum offloading savings, an upper bound of the target batch size interval is obtained.

[0183] In some embodiments, the data processing solution acquisition module includes:

[0184] A theoretical video memory occupancy acquisition unit, configured to acquire a theoretical video memory occupancy corresponding to each batch size within the target batch interval based on the video memory prediction model;

[0185] A scheme acquisition unit is used for, for each batch size, if the theoretical video memory occupancy corresponding to the batch size is less than or equal to the available video memory space size, determining a preset data processing scheme as the data processing scheme corresponding to the batch size; if the theoretical video memory occupancy corresponding to the batch size is greater than the available video memory space size, generating a data processing scheme corresponding to the batch size that meets the target video memory saving amount based on the target video memory saving amount and the video memory occupancy amount of each layer of the DNN model;

[0186] The batch size determination unit is used to determine a target batch size and a corresponding data processing scheme based on the training throughput of the data processing schemes corresponding to the multiple batch sizes.

[0187] In some embodiments, the solution acquisition unit includes:

[0188] A savings acquisition subunit, configured to acquire a target video memory savings based on the theoretical video memory occupancy and the available video memory space size, wherein the target video memory savings indicates the amount of video memory that should be saved for training the DNN model;

[0189] A model block sequence acquisition subunit, used to acquire a model block sequence of the DNN model based on the memory occupancy of each layer of the DNN model, wherein the model block sequence includes a plurality of model blocks sorted from low to high in terms of computational intensity, and the computational intensity is obtained based on the model parameter information;

[0190] A scheme acquisition subunit is used to acquire a data processing scheme whose savings meet the target memory savings based on the model block sequence and the memory occupancy of each layer of the DNN model; if a data processing scheme that meets the target memory savings exists, the data processing scheme is output as a data processing scheme corresponding to the batch size.

[0191] In some embodiments, the solution acquisition subunit includes:

[0192] A recalculation scheme acquisition subunit, used to acquire a recalculation scheme of the batch size based on the model block sequence, wherein the recalculation scheme indicates the model blocks to be recalculated during the model training process;

[0193] The offloading checkpoint scheme acquisition subunit determines the recalculation scheme as the data processing scheme that meets the target video memory saving amount if the theoretical recalculation saving amount corresponding to the recalculation scheme is greater than or equal to the target video memory saving amount; generates an offloading checkpoint scheme for the batch size if the theoretical recalculation saving amount corresponding to the recalculation scheme is less than the target video memory saving amount, and indicates the checkpoint data to be offloaded during the model training process; determines the offloading checkpoint scheme and the recalculation scheme as the data processing scheme for the batch size if the theoretical offloading saving amount of the offloading checkpoint scheme is greater than or equal to the minimum offloading data amount.

[0194] In some embodiments, the offload checkpoint scheme acquisition subunit is used to:

[0195] Obtaining a target unloading interval value sequence of the DNN model, wherein the target unloading interval value sequence is composed of unloading interval values, wherein the unloading interval value refers to the number of model blocks between two unloadings, and the unloading interval value ensures that there is no overlapping time period between two unloadings;

[0196] Based on the minimum unloading data volume of the DNN model and the target unloading interval value sequence, a target unloading checkpoint solution is obtained, wherein the minimum unloading data volume refers to the data volume of the checkpoint data that needs to be unloaded at least during the model training process.

[0197] In some embodiments, the batch size determination unit is configured to:

[0198] Based on multiple batch sizes within the target batch size interval and corresponding data processing methods, the DNN model is trained to obtain a training throughput rate corresponding to each batch size;

[0199] Based on the training throughput corresponding to each batch size, the target batch size and the corresponding data processing solution are obtained.

[0200] It should be understood that the data processing device used for model training corresponds to the computing device in the above-mentioned method embodiment, and the various modules in the device and the above-mentioned other operations and / or functions are respectively various steps and methods implemented by the computing device in the method embodiment. The specific details can be found in the above-mentioned method embodiment. For the sake of brevity, they will not be repeated here.

[0201] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a terminal to complete the data processing method for model training in the above embodiment. For example, the computer-readable storage medium is a non-temporary computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0202] An embodiment of the present application also provides a computer program product or a computer program, which includes a program code. The computer instructions are stored in a computer-readable storage medium. The processor in the terminal reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the terminal executes the above-mentioned data processing method for model training.

[0203] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the data processing method for model training in the above-mentioned method embodiments.

[0204] Among them, the device, equipment, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0205] Through the description of the above implementation methods, those skilled in the art can understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the video memory allocation or model training method embodiment for model training provided in the above embodiment belongs to the same concept, and its specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0206] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0209] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0210] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0211] In this application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the related concepts in a concrete way.

[0212] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the samples involved in this application were obtained with full authorization.

[0213] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0214] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method for model training, characterized in that: The method comprises: Based on the model parameter information of the DNN model to be trained, a video memory prediction model is established, wherein the model parameter information is used to describe the parameters of the DNN model, and the video memory prediction model is used to predict the size of the video memory space occupied by training the DNN model under different batch sizes; Based on the model parameter information and the available video memory space of the graphics card, a target batch size interval is obtained; Based on the target batch size range, the video memory prediction model and the model parameter information, a target batch size and a corresponding data processing scheme are obtained. The data processing scheme is used to process at least one of the input data and output data of the middle layer of the DNN model during model training. The training throughput corresponding to the target batch size and the corresponding data processing scheme is the largest within the target batch size range.

2. The method according to claim 1, characterized in that The step of establishing a video memory prediction model based on the model parameter information of the DNN model to be trained includes: Based on the DNN model, acquiring model parameter information, wherein the model parameter information indicates the data volume of the model parameters of the DNN model; Based on the model parameter information, determine the memory usage of each layer of the DNN model, where the memory usage of each layer of the DNN model includes the data volume of the model parameters of each layer of the DNN model and the data volume of the output data; A video memory prediction model is established based on the video memory occupancy of each layer of the DNN model.

3. The method according to claim 1 or 2, characterized in that: The obtaining of the target batch size interval based on the model parameter information and the available video memory space size of the graphics card includes: Based on the video memory occupancy of each layer of the DNN model, a maximum recomputation saving is obtained, where the maximum recomputation saving represents a maximum video memory saving that can be saved by recomputation when the batch size is 1; Based on the model parameter information of the DNN model, the available video memory space of the graphics card, and the maximum recalculation saving, obtaining a lower bound of the target batch size interval; Based on the video memory occupancy of each layer of the DNN model, a maximum offloading checkpoint saving is obtained, where the maximum offloading checkpoint saving represents the maximum video memory that can be saved by the offloading checkpoint when the batch size is 1; Based on the model parameter information of the DNN model, the available video memory space size of the graphics card, the maximum recalculation savings and the maximum offloading savings, an upper bound of the target batch size interval is obtained.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the target batch size and the corresponding data processing scheme based on the target batch size interval, the video memory prediction model and the model parameter information includes: Based on the video memory prediction model, obtaining a theoretical video memory occupancy corresponding to each batch size within the target batch interval; For each batch size, if the theoretical video memory occupancy corresponding to the batch size is less than or equal to the available video memory space size, determining the preset data processing scheme as the data processing scheme corresponding to the batch size; If the theoretical video memory occupancy corresponding to the batch size is greater than the available video memory space size, based on the target video memory saving amount and the video memory occupancy amount of each layer of the DNN model, generate a data processing solution corresponding to the batch size that meets the target video memory saving amount; Based on the training throughput of the data processing solutions corresponding to the multiple batch sizes, a target batch size and a corresponding data processing solution are determined.

5. The method according to claim 4, characterized in that The generating, based on the target video memory saving amount and the video memory occupancy amount of each layer of the DNN model, a data processing scheme corresponding to the batch size and satisfying the target video memory saving amount comprises: Based on the theoretical video memory occupancy and the available video memory space size, obtaining a target video memory saving amount, wherein the target video memory saving amount indicates an amount of video memory that should be saved for training the DNN model; Based on the memory usage of each layer of the DNN model, a model block sequence of the DNN model is obtained, wherein the model block sequence includes a plurality of model blocks sorted from low to high in terms of computing intensity, and the computing intensity is obtained based on the model parameter information; Based on the model block sequence and the video memory occupancy of each layer of the DNN model, obtaining a data processing solution whose saving amount satisfies the target video memory saving amount; If there is a data processing solution that satisfies the target video memory saving amount, the data processing solution is output as a data processing solution corresponding to the batch size.

6. The method according to claim 5, characterized in that The data processing scheme for obtaining a saving amount that satisfies the target video memory saving amount based on the model block sequence and the video memory occupancy of each layer of the DNN model includes: Based on the model block sequence, obtaining a recalculation scheme of the batch size, wherein the recalculation scheme indicates model blocks to be recalculated during the model training process; If the theoretical recalculation saving amount corresponding to the recalculation scheme is greater than or equal to the target video memory saving amount, determining the recalculation scheme as a data processing scheme that satisfies the target video memory saving amount; If the theoretical recomputation savings corresponding to the recomputation plan is less than the target video memory savings, generating an offload checkpoint plan for the batch size, the offload checkpoint plan indicating checkpoint data to be offloaded during model training; If the theoretical offloading savings of the offloading checkpoint solution is greater than or equal to the minimum offloading data amount, the offloading checkpoint solution and the recalculation solution are determined as the data processing solution for the batch size.

7. The method according to claim 6, characterized in that The scheme for generating a target offloading checkpoint for the batch size comprises: Obtaining a target unloading interval value sequence of the DNN model, wherein the target unloading interval value sequence is composed of unloading interval values, wherein the unloading interval value refers to the number of model blocks between two unloadings, and the unloading interval value ensures that there is no overlapping time period between two unloadings; Based on the minimum unloading data volume of the DNN model and the target unloading interval value sequence, a target unloading checkpoint solution is obtained, wherein the minimum unloading data volume refers to the data volume of the checkpoint data that needs to be unloaded at least during the model training process.

8. The method according to any one of claims 4 to 7, characterized in that: The determining of the target batch size and the corresponding data processing scheme based on the training throughput of the data processing schemes corresponding to the plurality of batch sizes comprises: Based on multiple batch sizes within the target batch size interval and corresponding data processing methods, the DNN model is trained to obtain a training throughput rate corresponding to each batch size; Based on the training throughput corresponding to each batch size, the target batch size and the corresponding data processing solution are obtained.

9. A data processing device for model training, characterized in that: The device comprises: A video memory prediction model establishment module, used to establish a video memory prediction model based on model parameter information of a DNN model to be trained, wherein the model parameter information is used to describe parameters of the DNN model, and the video memory prediction model is used to predict the size of video memory space occupied by training the DNN model under different batch sizes; A batch size interval acquisition module, used to acquire a target batch size interval based on the model parameter information and the available video memory space size of the graphics card; A data processing scheme acquisition module is used to acquire a target batch size and a corresponding data processing scheme based on the target batch size interval, the video memory prediction model and the model parameter information. The data processing scheme is used to process at least one of the input data and output data of the middle layer of the DNN model during model training. The training throughput corresponding to the target batch size and the corresponding data processing scheme is the largest within the target batch size interval.

10. A computing device, characterized in that: The computing device comprises a processor, and the processor is configured to execute program code so that the computing device performs the method according to any one of claims 1 to 8.

11. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, including a processor, wherein the processor is configured to execute program code, so that the computing device cluster executes the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by a processor to enable a computing device to execute the method according to any one of claims 1 to 8.

Images