A method, device and storage medium for controlling video memory for model training

Abstract

The present disclosure provides a graphics memory control method and device for model training, equipment and storage medium, relating to the technical field of artificial intelligence, especially to the technical field of neural network, large model, training optimization and the like. The specific implementation scheme is: according to the grouping information of the parameter gradient information required by one or more times of reverse calculation in the model training process, reconstructing the graphics memory space for the one or more times of reverse calculation; performing the one or more times of reverse calculation to obtain one or more times of reverse calculation results; storing the one or more times of reverse calculation results into the graphics memory space reconstructed for the one or more times of reverse calculation; releasing the graphics memory space reconstructed for the one or more times of reverse calculation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as neural networks, large models, and training optimization. Background Art

[0002] Video memory is crucial for training large models. Large models typically contain a vast number of parameters that need to be continuously read, updated, and stored during training. Video memory, as the portion of the graphics processing unit (GPU) dedicated to storing and processing data, provides fast storage and access support for model parameters, ensuring smooth model training. If video memory is insufficient, strategies such as recomputation may be necessary to conserve memory. However, recomputation can lead to performance degradation, so other methods are needed to conserve video memory and reduce or eliminate recomputation. Summary of the Invention

[0003] The present disclosure provides a video memory control method, apparatus, device, and storage medium for model training.

[0004] According to one aspect of the present disclosure, a method for controlling video memory for model training is provided, comprising:

[0005] reconstructing a video memory space for the one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations;

[0006] Executing the one or more reverse calculations to obtain one or more reverse calculation results;

[0007] Storing the one or more reverse calculation results in a video memory space reconstructed for the one or more reverse calculations;

[0008] Release the video memory space that was reconstructed for one or more reverse calculations.

[0009] According to another aspect of the present disclosure, a video memory control device for model training is provided, comprising:

[0010] A reconstruction module, configured to reconstruct a video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations;

[0011] A calculation module, configured to perform the one or more reverse calculations and obtain one or more reverse calculation results;

[0012] A storage module, configured to store the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0013] The release module is used to release the video memory space reconstructed for one or more reverse calculations.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0020] According to the embodiment of the present disclosure, the video memory space can be reconstructed for the grouping information of the parameter gradient information required for the reverse calculation before the reverse calculation, and the reconstructed video memory space can be released after the reverse calculation is completed and the model update is completed, thereby reducing the occupancy of the video memory space during subsequent model training, improving the recalculation speed, and thus improving the model training speed.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a method for controlling video memory for model training according to an embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of a single training step of the video memory zeroing method disclosed herein;

[0025] Figure 3 is a flow chart of a single training step of the video memory control method for model training according to the present disclosure;

[0026] Figure 4 A schematic diagram of a single training step of the video memory control method for model training according to the present disclosure;

[0027] Figure 5 is a flowchart of a method for controlling video memory for model training according to another embodiment of the present disclosure;

[0028] Figure 6 is a flowchart of a method for controlling video memory for model training according to another embodiment of the present disclosure;

[0029] Figure 7a This is a schematic diagram of not releasing video memory in a single training step of model training;

[0030] Figure 7b This is a schematic diagram of releasing video memory in a single training step of model training;

[0031] Figure 8 is a structural diagram of a video memory control device for model training according to an embodiment of the present disclosure;

[0032] Figure 9 This is a structural diagram of a video memory control device for model training according to another embodiment of the present disclosure.

[0033] Figure 10 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] Figure 1 FIG. 1 is a flow chart of a method 100 for controlling video memory for model training according to an embodiment of the present disclosure. In one embodiment, the method includes:

[0036] S101, reconstructing video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations during the model training process;

[0037] S102, performing the one or more reverse calculations to obtain one or more reverse calculation results;

[0038] S103, storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0039] S104: Release the video memory space reconstructed for one or more reverse calculations.

[0040] In the embodiments of the present disclosure, the training process of a model, such as a large model, may include multiple training steps. For example, each time a gradient is calculated based on a batch of data (batch) and the process of updating the model parameters using the gradient is completed, it can be regarded as a training step. A training step may include operations such as forward calculation, reverse calculation, and parameter gradient communication. During the parameter gradient communication process, the value of the parameter gradient obtained by reverse calculation needs to be sent from the video memory to the target device such as internal memory, other video memory, other computers, etc. In order to improve communication efficiency, the parameter gradient information can be grouped and continuous video memory space can be allocated to each group. There are many ways to group, for example, according to the layer of the model, the parameter gradient information involved in one or more layers is grouped. For example, the parameter gradient information is grouped according to various factors such as the computing tasks, computing resources, and data distribution of the model.

[0041] During the communication process, one or more data groups in a continuous memory space can be sent together to improve communication efficiency. Figure 2 As shown, before training begins, the parameter gradient information G0, G1, G2, and G3 are pre-grouped and stored in contiguous memory space in the video memory. The contiguously stored G0, G1, G2, and G3 and their values ​​can then be sent to the main memory or other storage space. If this parameter gradient information is used as resident video memory during model training, the parameter gradient information in this video memory will not be released. Instead, the parameter gradient values ​​will be set to zero before the end of each training step. In this case, regardless of whether each parameter gradient information is used or not, it will occupy a portion of video memory, and this portion of video memory cannot be saved to improve model performance.

[0042] In order to reduce the occupation of video memory, the grouping information of parameter gradient information required for one or more reverse calculations in the training step can be pre-stored (or called the parameter gradient grouping corresponding to the reverse calculation). Before a certain reverse calculation needs to be performed, the pre-stored grouping information of the parameter gradient information required for the reverse calculation can be obtained first, and a continuous video memory space can be reconstructed in the video memory for the parameter gradient information required for the reverse calculation (or simply called the reverse calculation reconstructed video memory space). Figure 3 As shown, a continuous video memory space can be reconstructed for the parameter gradient information G0, G1, G2, and G3 required for the reverse calculation, and then the reverse calculation is performed, and the reverse calculation results are written into the continuous video memory space respectively. Among them, the reverse calculation result of G0 is written into the video memory space corresponding to G0 in the video memory, the reverse calculation result of G1 is written into the video memory space corresponding to G1 in the video memory, the reverse calculation result of G2 is written into the video memory space corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space corresponding to G3 in the video memory.

[0043] In an embodiment of the present disclosure, if a training step includes only one reverse calculation, the corresponding video memory space can be reconstructed for all elements in the parameter gradient information required for the reverse calculation (such as identification information of the parameter gradient) before the reverse calculation. If a training step includes multiple reverse calculations, different reverse calculations have their own required parameter gradient information, and the parameter gradient information required for each reverse calculation can be grouped together. In one example, before executing a certain reverse calculation, the corresponding video memory space is reconstructed only for all elements in the grouping information required for the reverse calculation. Before executing the next reverse calculation, the corresponding video memory space is reconstructed only for all elements in the parameter gradient information required for the next reverse calculation. Figure 4 As shown, a continuous video memory space can be reconstructed for the parameter gradient information G0 and G1 required for reverse calculation 1, and then reverse calculation 1 can be executed. The results of reverse calculation 1 are written into the continuous video memory space respectively. Among them, the reverse calculation result of G0 is written into the video memory space corresponding to G0 in the video memory, and the reverse calculation result of G1 is written into the video memory space corresponding to G1 in the video memory. Then, a continuous video memory space is reconstructed for the parameter gradient information G2 and G3 required for reverse calculation 2, and then reverse calculation 2 is executed, and the results of reverse calculation 2 are written into the continuous video memory space respectively. Among them, the reverse calculation result of G2 is written into the video memory space corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space corresponding to G3 in the video memory.

[0044] In the embodiment of the present disclosure, after all reverse calculations for a time step are completed, all reverse calculation results in the reconstructed video memory space can be sent at once during the communication process. After the communication is completed, the reconstructed video memory space can be released.

[0045] According to the disclosed embodiments, video memory space can be reconstructed for the parameter gradient information required for reverse calculation before reverse calculation. The reconstructed video memory space can then be released after the reverse calculation is complete and the model update is complete. The reconstruction and release operations take very little time, thus reducing video memory usage during subsequent model training and thereby increasing model training speed. Furthermore, since video memory usage is reduced during model training, excessive recalculations are unnecessary, reducing the number of recalculations and accelerating the model training process.

[0046] Figure 5 is a flow chart of a video memory control method 500 for model training according to another embodiment of the present disclosure. The method 500 can be used to implement step S101 of the video memory control method 100 for model training. In one embodiment, the method 500 includes: reconstructing video memory space for one or more reverse calculations during the model training process based on grouping information of parameter gradient information required for the reverse calculation, and further includes:

[0047] S501, applying for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory;

[0048] S502: In-situ multiplexing of the requested video memory space and the video memory space required by the parameter gradient information.

[0049] In the embodiment of the present disclosure, the grouping information of the parameter gradient information required for each reverse calculation in a training step of the model training can be stored in the memory in advance. Figure 3 As shown in , the parameter gradient information required for reverse calculation in a training step includes G0, G1, G2 and G3, and these parameter gradient information belong to the same group. Figure 4 As shown in , in a training step, the parameter gradient information required for reverse calculation 1 includes G0 and G1, which belong to group 1; the parameter gradient information required for reverse calculation 2 includes G2 and G3, which belong to group 2. Before executing the reverse calculation, you can apply to reconstruct the video memory space based on the group information of the parameter gradient information required for the reverse calculation in the memory. For example, Figure 3 As shown, for the reverse calculation, the video memory space requested for G0 in the group information is the address range [A0, A1), the video memory space requested for G1 is the address range [A1, A2), the video memory space requested for G2 is the address range [A2, A3), and the video memory space requested for G3 is the address range [A3, A4). For another example, Figure 4 As shown, for reverse calculation 1, the memory space requested for G0 in group 1 is in the address range [B0, B1), and the memory space requested for G1 in group 1 is in the address range [B1, B2). After reconstructing the memory space, reverse calculation 1 is executed. Then, for reverse calculation 2, the memory space requested for G2 in group 2 is in the address range [B3, B4), and the memory space requested for G3 in group 2 is in the address range [B4, B5). After reconstructing the memory space, reverse calculation 2 is executed.

[0050] In the embodiment of the present disclosure, the video memory space applied for reverse calculation and the video memory space of the parameter gradient information required for reverse calculation can be in-place multiplexed. By in-place multiplexing, each parameter gradient information in the grouped information can have a corresponding video memory space in the video memory. For example, Figure 3 and Figure 4 As shown in the figure, the address ranges of the video memory space requested for G0 in the video memory pointed to by G0 are [A0, A1) and [B0, B1) respectively, and the address ranges of the video memory space requested for G1 in the video memory pointed to by G1 are [A1, A2) and [B1, B2) respectively. Figure 3 and Figure 4As shown, the video memory space requested for G2 in the video memory pointed to by G2 has an address range of [A2, A3) and [B3, B4) respectively, and the video memory space requested for G3 in the video memory pointed to by G3 has an address range of [A3, A4) and [B4, B5) respectively.

[0051] In the embodiment of the present disclosure, the reconstructed video memory space for the same group is usually continuous. The reconstructed video memory space for different groups can be continuous or discontinuous. For example, Figure 4 In one case, the memory spaces corresponding to G0, G1, G2, and G3 can be continuous. In another case, the memory spaces corresponding to G0 and G1 are continuous, the memory spaces corresponding to G2 and G3 are continuous, but the memory spaces corresponding to G1 and G2 are not continuous.

[0052] After applying for video memory space and performing in-situ multiplexing, the video memory space can be reconstructed for the parameter gradient information required for reverse calculation. Then, after performing reverse calculation, the reverse calculation results can be stored in the corresponding reconstructed video memory space. For example, Figure 3 and Figure 4 As shown, the reverse calculation result of G0 is written into the video memory space [A0, A1) or [B0, B1) corresponding to G0 in the video memory, the reverse calculation result of G1 is written into the video memory space [A1, A2) or [B1, B2) corresponding to G1 in the video memory, the reverse calculation result of G2 is written into the video memory space [A2, A3) or [B3, B4) corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space [A3, A4) or [B4, B5) corresponding to G3 in the video memory. In the subsequent communication process, the data in the video memory space [A0, A4) or [B0, B2) and [B3, B5) can be sent to the target device together.

[0053] According to the embodiments of the present disclosure, the video memory space required for reconstructing the corresponding parameter gradient information for each reverse calculation can be used. These parameter gradient information do not need to occupy the video memory space all the time, thereby reducing the occupancy of the video memory, saving video memory resources, and accelerating the model training process.

[0054] In the embodiment of the present disclosure, according to the identification information of multiple parameter gradients in the group information of parameter gradient information required for a certain reverse calculation, a video memory space with a continuous address can be applied in the video memory as the video memory space of the entire group information. Figure 3 As shown, the grouping information includes {parameter gradient G0, parameter gradient G1, parameter gradient G3, parameter gradient G4}; G0, G1, G3, and G4 can be understood as examples of identification information of parameter gradients. Before performing the reverse calculation, the address range requested for the parameter gradient information in the video memory is [A0, A4). Figure 4As shown, group 1 includes {parameter gradient G0, parameter gradient G1}; group 2 includes {parameter gradient G2, parameter gradient G3}. Before executing reverse calculation 1, the address range requested in the video memory for the parameter gradient information in group 1 is [B0, B2), and the address range requested for the parameter gradient information in group 2 is [B3, B5).

[0055] According to the embodiment of the present disclosure, the corresponding video memory space can be reconstructed for different reverse calculations in the video memory according to the grouping information of the identification information of the parameter gradient required for the reverse calculation, and storage space can be provided for the reverse calculation results. After the training step is completed, the reconstructed video memory space can also be released, thereby reducing the occupancy of the video memory space and accelerating the overall training process.

[0056] In one embodiment, S502 further includes: in-situ multiplexing the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the video memory space of the application to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory in the video memory.

[0057] In the embodiment of the present disclosure, after applying for a continuous video memory space for a certain group information, the storage unit required for the identification information of each parameter gradient in the group information can be reused in situ with the continuous video memory space. This allows the identification information of each parameter gradient to have a corresponding storage unit in the video memory. For example, Figure 3 and Figure 4 As shown, in-situ multiplexing is performed according to the identification information G0 of the parameter gradient, and G0 can be pointed to the address range [A0, A1) or [B0, B1) of the requested video memory space. In-situ multiplexing is performed according to the identification information G1 of the parameter gradient, and G1 can be pointed to the address range [A1, A2) or [B1, B2) of the requested video memory space. In-situ multiplexing is performed according to the identification information G2 of the parameter gradient, and G2 can be pointed to the address range [A2, A3) or [B3, B4) of the requested video memory space. In-situ multiplexing is performed according to the identification information G3 of the parameter gradient, and G3 can be pointed to the address range [A3, A4) or [B4, B5) of the requested video memory space.

[0058] According to an embodiment of the present disclosure, a corresponding storage unit can be reconstructed in the video memory for the identification information of each parameter gradient in the parameter gradient information, which is used to store the calculation results of the reverse calculation that requires the parameter gradient information. The reconstructed storage unit in the video memory can be released subsequently to reduce the occupation of the video memory space, provide more video memory space for subsequent model training, and accelerate the overall model training process.

[0059] In one embodiment, Figure 5As shown, the method 500 can also be used to implement step S102 in the video memory control method 100 for model training. In one embodiment, the method 500 may include: S503, performing the one or more reverse calculations to obtain one or more reverse calculation results.

[0060] In one embodiment, the method 500 can also be used to implement step S103 of the video memory control method 100 for model training. The method 500 may include: storing the one or more reverse calculation results into the video memory space reconstructed for the one or more reverse calculations, and further including:

[0061] S504: Store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

[0062] In an embodiment of the present disclosure, during the reverse calculation process, the value of each parameter gradient in the parameter gradient information corresponding to the reverse calculation can be generated. These values ​​can be generated sequentially, for example, the value of G0 is generated first, and then the value of G1 is generated. By storing the grouping information of the parameter gradient information into the memory in advance, the video memory space corresponding to each parameter gradient information in the grouping information can be reconstructed according to the needs of the reverse calculation. Then, in the process of executing the reverse calculation again, the calculated values ​​of each parameter gradient are written sequentially to the video memory space for reconstructing each parameter gradient information. For example, after the reverse calculation obtains the value V0 of G0, V0 is written to the video memory space corresponding to G0. Then, the reverse calculation is continued, and after the value V1 of G1 is calculated, V1 is written to the video memory space corresponding to G1.

[0063] According to the disclosed embodiments, the values ​​of the parameter gradients obtained by reverse calculation can be stored in the reconstructed video memory space. This continuous video memory space can improve the efficiency of communication with other devices or components. Furthermore, the reconstructed storage units in the video memory can be released later, reducing the memory space occupied and improving the model training speed.

[0064] Figure 6 6 is a flow chart of a method 600 for controlling video memory for model training according to another embodiment of the present disclosure. The method 600 can be used to implement step S102 of the method 100 for controlling video memory for model training. In one embodiment, the method 600 may include: performing one or more reverse calculations to obtain one or more reverse calculation results, and further including:

[0065] S601. When a training step includes multiple reverse calculations, the values ​​of part or all of the parameter gradients in the video memory space reconstructed by the first reverse calculation are used as input information for the second reverse calculation, and the second reverse calculation is performed to obtain the values ​​of the parameter gradients of the second reverse calculation.

[0066] In the embodiment of the present disclosure, multiple reverse calculations included in a training step may have an associated relationship. Assume that a training step includes a first reverse calculation and a second reverse calculation, and some or all parameter gradient values ​​in the calculation result of the first reverse calculation can be used in the second reverse calculation. In this case, the video memory space can be reconstructed for the first reverse calculation first; the first reverse calculation is executed to obtain the first reverse calculation result (the parameter gradient value). Then, the part or all parameter gradient values ​​obtained by the first reverse calculation are used as the input of the second reverse calculation, and the second reverse calculation is continued.

[0067] In the embodiment of the present disclosure, if there is no correlation between multiple reverse calculations in a training step, they can be independently directed to S101 to S103, and then S104 is executed to delete the values ​​of the parameter gradients obtained by these reverse calculations in the video memory, thereby releasing the video memory space reconstructed for these reverse calculations.

[0068] According to the embodiment of the present disclosure, the value of the parameter gradient obtained by the previous reverse calculation can be used as the input of the next reverse calculation, which reduces the number of recalculations and improves the efficiency of training.

[0069] In one embodiment, Figure 6 As shown, the method 600 can also be used to implement step S103 in the video memory control method 100 for model training. In one embodiment, the method 600 further includes: storing the one or more reverse calculation results into the video memory space reconstructed for the one or more reverse calculations, further including:

[0070] S602: Store the value of the parameter gradient of the second reverse calculation into the video memory space reconstructed for the second reverse calculation.

[0071] In the embodiment of the present disclosure, it is assumed that there is an association relationship between the first reverse calculation and the second reverse calculation included in a training step, and that some or all parameter gradient values ​​in the calculation result of the first reverse calculation can be used in the second reverse calculation. In this case, S101 to S103 can be first executed for the first reverse calculation: reconstruct the video memory space for the first reverse calculation according to the first grouping information of the parameter gradient information required for the first reverse calculation; execute the first reverse calculation to obtain the first reverse calculation result; and store the first reverse calculation result in the video memory space reconstructed for the first reverse calculation. Then, use some or all parameter gradient values ​​obtained by the first reverse calculation as the input of the second reverse calculation, and execute S101 to S103 for the second reverse calculation: reconstruct the video memory space for the second reverse calculation according to the second grouping information of the parameter gradient information required for the second reverse calculation; execute the second reverse calculation to obtain the second reverse calculation result; and store the second reverse calculation result in the video memory space reconstructed for the second reverse calculation.

[0072] In one example, see Figure 4 , the first reverse calculation is reverse calculation 1, and the second reverse calculation is reverse calculation 2. Based on the first grouping information of the parameter gradient information G0 and G1 required for the first reverse calculation during the model training process, the video memory space [B0, B1) and [B1, B2) are reconstructed for the reverse calculation 1; the reverse calculation 1 is executed to obtain the results V0 and V1 of the reverse calculation 1; the result of the reverse calculation 1 is stored in the video memory space reconstructed for the first reverse calculation, that is, V0 is stored in [B0, B1) and V1 is stored in [B1, B2). If reverse calculation 2 requires the value V1 of G1, the value V1 of G1 can be used as the input of reverse calculation 2, and the video memory space [B3, B4) and [B4, B5) for reverse calculation 2 is reconstructed according to the parameter gradient information G1 and the second grouping information of G2 required for reverse calculation 2; the reverse calculation 2 is executed to obtain the results V2 and V3 of reverse calculation 2; the result of reverse calculation 2 is stored in the video memory space reconstructed for reverse calculation 2, that is, V2 is stored in [B3, B4) and V3 is stored in [B4, B5).

[0073] According to an embodiment of the present disclosure, before each reverse calculation in the model training process, the video memory space is reconstructed for the parameter gradient information required for the second reverse calculation, and the reconstructed video memory space is released after all reverse calculations in the model training process are completed and the model update is completed. This can reduce the occupancy of the video memory space during the model training process and improve the model training speed.

[0074] In one embodiment, Figure 6As shown, the method 600 can also be used to implement step S104 in the video memory control method 100 for model training. In one embodiment, the method 600 further includes: releasing the video memory space reconstructed for one or more reverse calculations, further including:

[0075] S603: After all reverse calculations in a training step are completed, the video memory space reconstructed for all reverse calculations is released.

[0076] In the embodiment of the present disclosure, if a training step includes multiple reverse calculations and the video memory space has been reconstructed for each reverse calculation, the video memory space of all reverse calculations designed for the training step can be released after the training step is completed. Figure 4 As shown, the video memory spaces [B0, B2) and [B3, B5] reconstructed for reverse calculation 1 and reverse calculation 2 can be released. During release, the reconstructed video memory space can be unloaded from the video memory, and the data in the reconstructed video memory space can also be deleted. When a new batch of samples is used for the next training step, the video memory space can be reconstructed and reverse calculations can be performed based on the grouping information of the parameter gradient information required for the reverse calculation of the next training step. Furthermore, the reconstructed video memory space can be released at the end of each training step.

[0077] According to the embodiment of the present disclosure, since the process of reconstructing and releasing the video memory space takes a very short time, it can greatly reduce the unnecessary occupation of the video memory space during the training process, thereby improving the overall training speed.

[0078] Recomputation is a common strategy used in large model training. It can reduce GPU memory usage during model training and ensure that the model can be trained within limited GPU memory. During large model training, when certain conditions are triggered, recomputation can be used to re-perform specific calculations, making the model trainable. However, recomputation itself introduces redundant computations, resulting in additional performance loss. To mitigate this performance loss, it is necessary to explore GPU memory optimization methods to reduce peak GPU memory usage during model training, and to partially or completely disable recomputation to improve model performance.

[0079] Optimizing parameter gradient memory is an important optimization direction. In related technical solutions, parameter gradients are often fused into one or more tensors in continuous memory space to improve the communication performance of parameter gradients during data parallel training. However, this makes it difficult to release parameter gradient memory and thus cannot be optimized.

[0080] Before model training begins, all parameter gradients can be grouped and all parameter gradients in each group will be fused into a large tensor to improve parameter gradient communication performance. If the parameter gradients are used as resident memory during model training, the parameter gradients in this memory will not be released, but the parameter gradient values ​​will be set to 0, such as Figure 7a In this case, each parameter gradient, whether used or not, will occupy a portion of the video memory, and this portion of the video memory cannot be saved to improve model performance.

[0081] In one example, Figure 2 As shown in the figure, the parameter gradients are divided into two groups. In the actual model training process, after the reverse calculation of the current training step is completed, the parameter gradient group 1 and the parameter gradient group 2 need to be reset to zero before the next training step can be carried out. The peak memory usage during the training process occurs after the forward calculation is completed and before the reverse calculation begins. Figure 2 If parameter gradient group 1 includes G0 and G1, and parameter gradient group 2 includes G2 and G3, after the reverse calculation of a training step is completed, the reverse calculation results corresponding to G0, G1, G2, and G3 are reset to zero, but the video memory space corresponding to G0, G1, G2, and G3 is retained in the video memory. In the next training step, the video memory space corresponding to G0, G1, G2, and G3 is still used to store the respective reverse calculation results.

[0082] The disclosed embodiments can utilize dynamic gradient release to release model parameter gradients from video memory after each batch of training. During reverse computation, dynamic reconstruction is performed based on the grouping information to meet the video memory requirements of the reverse computation. This embodiment can reduce the overall video memory usage of the model, thereby reducing the video memory usage of recalculations and improving the training performance of large models.

[0083] like Figure 7b As shown in the figure, after each training step, the video memory of all parameter gradient groups can be released. When a certain parameter gradient needs to be reversely calculated, the parameter gradient group where this parameter gradient is located can be reconstructed, and the video memory space of this group can be applied for. The information required for the reconstruction process (such as the grouping information of the parameter gradient information) is recorded in the memory before the training starts. The reconstruction process mainly includes two operations: one is to apply for the video memory space of the group to be reconstructed based on the grouping information of the parameter gradient information in the memory; the second is to in-place multiplex the video memory space of the parameter gradient information in the group information generated by the reverse operator of the reverse calculation with the video memory space applied for the group information. For example, the name of the parameter gradient in the group information is pointed to the video memory space corresponding to the name of the reconstructed parameter gradient in the video memory through a pointer. The time consumed by these two operations in the reconstruction process is almost negligible. As shown in the figure, Figure 7bAs shown, after reconstructing memory space 1 for parameter gradient group 1 required for reverse calculation 1, reverse calculation 1 can be performed, and the calculation results of reverse calculation 1 are respectively stored in corresponding positions in reconstructed memory space 1. After reconstructing memory space 2 for parameter gradient group 2 required for reverse calculation 2, reverse calculation 2 can be performed, and the calculation results of reverse calculation 2 are respectively stored in corresponding positions in reconstructed memory space 2.

[0084] In one example, see Figure 4 The calculation results of G0 and G1 obtained by reverse calculation 1 are stored in the storage unit of G0 and the storage unit of G1 in the reconstructed video memory space respectively. The calculation results of G2 and G3 obtained by reverse calculation 2 are stored in the storage unit of G2 and the storage unit of G3 in the reconstructed video memory space respectively.

[0085] According to an embodiment of the present disclosure, after the forward calculation is completed and before the reverse calculation begins, the video memory usage can be made to exclude the video memory of the parameter gradient, thereby reducing the peak video memory usage. When the peak video memory is reduced, there is no need to save video memory space through too much recalculation. Therefore, the video memory usage of recalculation can be further reduced, thereby improving the model training performance. The present disclosure can be applied to the performance optimization of large model training processes that require recalculation strategies. By dynamically releasing and reconstructing the parameter gradient video memory, the peak video memory usage of the model is reduced, and some recalculations can be turned off to improve training performance.

[0086] Figure 8 FIG. 8 is a schematic diagram of a memory control device 800 for model training according to an embodiment of the present disclosure. In one embodiment, the device includes:

[0087] A reconstruction module 801 is configured to reconstruct a video memory space for one or more reverse calculations in a model training process based on grouping information of parameter gradient information required for the one or more reverse calculations.

[0088] A calculation module 802 is configured to perform the one or more reverse calculations to obtain one or more reverse calculation results;

[0089] A storage module 803 is configured to store the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0090] The release module 804 is used to release the video memory space reconstructed for one or more reverse calculations.

[0091] Figure 9: This is a schematic diagram of the structure of a video memory control device 900 for model training according to another embodiment of the present disclosure. The device 900 includes: a reconstruction module 901, a calculation module 902, a storage module 903, and a release module 904. The functions of the above modules can refer to the functions of the modules of the video memory control device 800 for model training in the above embodiment. In one embodiment, the reconstruction module 901 includes:

[0092] The space application submodule 9011 is used to apply for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory;

[0093] The in-situ multiplexing submodule 9012 is configured to perform in-situ multiplexing on the requested video memory space and the grouping information of the parameter gradient information in the memory.

[0094] In one embodiment, the in-situ multiplexing submodule 9012 is also used to in-situ multiplex the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory in the video memory.

[0095] In one embodiment, the storage module 903 is further configured to store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

[0096] In one embodiment, the calculation module 902 is also used to use the values ​​of part or all of the parameter gradients in the video memory space reconstructed for the first reverse calculation as input information for the second reverse calculation, and perform the second reverse calculation to obtain the values ​​of the parameter gradients of the second reverse calculation when multiple reverse calculations are included in a training step.

[0097] In one embodiment, the storage module 903 is further configured to store the value of the parameter gradient of the second reverse calculation into a video memory space reconstructed for the second reverse calculation.

[0098] In one embodiment, the release module 904 is further configured to release the video memory space reconstructed for all reverse calculations after all reverse calculations in a training step are completed.

[0099] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0100] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0102] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0103] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0104] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0105] The computing unit 1001 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the memory control method for model training. For example, in some embodiments, the memory control method for model training can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the memory control method for model training described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the memory control method for model training by any other suitable means, such as by means of firmware.

[0106] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0107] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0108] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0110] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0112] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0113] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A video memory control method for model training, comprising: Reconstructing a video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations during the model training process; Performing the one or more reverse calculations to obtain one or more reverse calculation results; Storing the one or more reverse calculation results in a video memory space reconstructed for the one or more reverse calculations; Release the video memory space reconstructed for one or more reverse calculations; Reconstructing the video memory space for the reverse calculation according to the grouping information of the parameter gradient information required for one or more reverse calculations during the model training process includes: Applying for reconstruction of video memory space for the one or more reverse calculations according to grouping information of the parameter gradient information required for the one or more reverse calculations in the memory; In-situ multiplexing of the requested video memory space and the video memory space required for the parameter gradient information; The in-situ multiplexing of the applied video memory space and the video memory space required for the parameter gradient information includes: in-situ multiplexing of the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the video memory.

2. The method according to claim 1, wherein Storing the one or more reverse calculation results in a video memory space reconstructed for the one or more reverse calculations includes: The value of each parameter gradient in the one or more reverse calculation results is stored in a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

3. The method according to claim 1 or 2, wherein: Performing the one or more reverse calculations to obtain one or more reverse calculation results includes: When a training step includes multiple reverse calculations, the values ​​of part or all of the parameter gradients in the video memory space reconstructed by the first reverse calculation are used as input information for the second reverse calculation, and the second reverse calculation is performed to obtain the values ​​of the parameter gradients of the second reverse calculation.

4. The method according to claim 3, storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations, comprising: The parameter gradient value of the second reverse calculation is stored in the video memory space reconstructed for the second reverse calculation.

5. The method according to claim 1 or 2, wherein: Release the video memory space reconstructed for one or more reverse calculations, including: After all the backward calculations in a training step are completed, the video memory space reconstructed for all the backward calculations is released.

6. A video memory control device for model training, comprising: A reconstruction module, configured to reconstruct a video memory space for one or more reverse calculations in a model training process based on grouping information of parameter gradient information required for the one or more reverse calculations; A calculation module, configured to perform the one or more reverse calculations to obtain one or more reverse calculation results; A storage module, configured to store the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations; A release module is used to release the video memory space reconstructed for one or more reverse calculations; Wherein, the reconstruction module includes: A space application submodule, configured to apply for reconstruction of video memory space for the one or more reverse calculations according to grouping information of the parameter gradient information required for the one or more reverse calculations in the memory; An in-situ multiplexing submodule, configured to perform in-situ multiplexing of the requested video memory space and the grouping information of the parameter gradient information in the memory; Among them, the in-situ multiplexing submodule is also used to in-situ multiplex the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory in the video memory.

7. The device according to claim 6, wherein The storage module is further configured to store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

8. The device according to claim 6 or 7, wherein: The calculation module is also used to use the values ​​of part or all of the parameter gradients in the video memory space reconstructed for the first reverse calculation as input information for the second reverse calculation, and perform the second reverse calculation to obtain the values ​​of the parameter gradients of the second reverse calculation when multiple reverse calculations are included in a training step.

9. The apparatus according to claim 8, wherein the storage module is further configured to store the value of the parameter gradient of the second reverse calculation into a video memory space reconstructed for the second reverse calculation.

10. The device according to claim 6 or 7, wherein: The release module is further configured to release the video memory space reconstructed for all reverse calculations after all reverse calculations in a training step are completed.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images