Video memory control method and device for model training, equipment and storage medium

Abstract

The invention provides a video memory control method, device and equipment for model training and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of neural networks, large models, training optimization and the like. The specific implementation scheme is as follows: according to grouping information of parameter gradient information required by one or more reverse calculations in a model training process, reconstructing a video memory space for the one or more reverse calculations; executing the one-time or multi-time reverse calculation to obtain one-time or multi-time reverse calculation results; storing the one-time or multiple-time reverse calculation result into a video memory space reconstructed for the one-time or multiple-time reverse calculation; and a video memory space reconstructed by one or more times of reverse calculation is released.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as neural networks, large models, and training optimization. Background Art

[0002] Video memory is very important in large model training. Large models usually contain a large number of parameters, which need to be constantly read, updated, and stored during the training process. Video memory, as the part of the graphics processing unit (GPU) dedicated to storing and processing data, can provide fast storage and access support for model parameters to ensure smooth model training. If the video memory is not enough, you may need to use strategies such as recalculation to save video memory. However, recalculation will lead to performance degradation, so other means should be used to save video memory to reduce or eliminate recalculation. Summary of the invention

[0003] The present invention provides a video memory control method, apparatus, device and storage medium for model training.

[0004] According to one aspect of the present disclosure, a video memory control method for model training is provided, comprising:

[0005] Reconstructing a video memory space for the one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations;

[0006] Execute the one or more reverse calculations to obtain one or more reverse calculation results;

[0007] storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0008] Release the video memory space reconstructed for one or more reverse calculations.

[0009] According to another aspect of the present disclosure, a video memory control device for model training is provided, comprising:

[0010] A reconstruction module, used for reconstructing a video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for the one or more reverse calculations;

[0011] A calculation module, used to perform the one or more reverse calculations to obtain one or more reverse calculation results;

[0012] A storage module, used for storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0013] The release module is used to release the video memory space reconstructed for one or more reverse calculations.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] According to the embodiments of the present disclosure, the video memory space can be reconstructed for the grouping information of the parameter gradient information required for the reverse calculation before the reverse calculation, and the reconstructed video memory space can be released after the reverse calculation is completed and the model update is completed, thereby reducing the occupancy of the video memory space during subsequent model training, and the recalculation speed can be increased, thereby increasing the model training speed.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a video memory control method for model training according to an embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of a single training step of the video memory zeroing method disclosed in the present invention;

[0025] Figure 3 is a flowchart of a single training step of the video memory control method for model training according to the present disclosure;

[0026] Figure 4 A schematic flow chart of a single training step of the video memory control method for model training according to the present disclosure;

[0027] Figure 5 is a flowchart of a method for controlling video memory for model training according to another embodiment of the present disclosure;

[0028] Figure 6 is a flowchart of a method for controlling video memory for model training according to another embodiment of the present disclosure;

[0029] Figure 7a This is a schematic diagram of not releasing video memory in a single training step of model training;

[0030] Figure 7b This is a schematic diagram of releasing video memory in a single training step of model training;

[0031] Figure 8 is a structural schematic diagram of a video memory control device for model training according to an embodiment of the present disclosure;

[0032] Fig. 9 is a structural schematic diagram of a video memory control device for model training according to another embodiment of the present disclosure

[0033] Fig.10 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0035] Figure 1 FIG. 1 is a flow chart of a method 100 for controlling a video memory for model training according to an embodiment of the present disclosure. In one embodiment, the method includes:

[0036] S101, reconstructing video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for one or more reverse calculations in a model training process;

[0037] S102, performing the one or more reverse calculations to obtain one or more reverse calculation results;

[0038] S103, storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0039] S104: Release the video memory space reconstructed for one or more reverse calculations.

[0040] In the disclosed embodiments, the training process of a model such as a large model may include multiple training steps. For example, each time a gradient is calculated based on a batch of data (batch) and the process of updating the model parameters using the gradient is completed, it can be regarded as a training step. A training step may include operations such as forward calculation, reverse calculation, and parameter gradient communication. During the parameter gradient communication process, the value of the parameter gradient obtained by reverse calculation needs to be sent from the video memory to the target device such as internal memory, other video memory, other computers, etc. In order to improve communication efficiency, the parameter gradient information can be grouped and continuous video memory space can be allocated to each group. There are many ways to group, for example, according to the layers of the model, the parameter gradient information involved in one or more layers is grouped. For example, the parameter gradient information is grouped according to various factors such as the computing tasks, computing resources, and data distribution of the model.

[0041] During the communication process, one or more data groups in continuous video memory space can be sent together to improve communication efficiency. Figure 2 As shown, before the training starts, the parameter gradient information G0, G1, G2 and G3 are grouped together in advance, and G0, G1, G2 and G3 and their values ​​are stored in a continuous video memory space in the video memory. The G0, G1, G2 and G3 and their values ​​stored continuously in the video memory can be sent to the internal memory or other storage space. If these parameter gradient information are used as the resident video memory during the model training process, the parameter gradient information in this video memory will not be released, but the value of the parameter gradient will be set to zero before the end of each training step. In this case, each parameter gradient information will occupy a part of the video memory regardless of whether it is used or not, and this part of the video memory cannot be saved to improve the model performance.

[0042] In order to reduce the occupation of video memory, the grouping information of parameter gradient information required for one or more reverse calculations in the training step can be stored in advance (or called the parameter gradient grouping corresponding to the reverse calculation). Before a certain reverse calculation needs to be performed, the grouping information of the parameter gradient information required for the reverse calculation can be obtained in advance, and a continuous video memory space can be reconstructed in the video memory for the parameter gradient information required for the reverse calculation (or simply called the reverse calculation reconstructed video memory space). Figure 3 As shown, a continuous video memory space can be reconstructed for the parameter gradient information G0, G1, G2 and G3 required for the reverse calculation, and then the reverse calculation is performed, and the continuous video memory space is written respectively according to the reverse calculation results. Among them, the reverse calculation result of G0 is written into the video memory space corresponding to G0 in the video memory, the reverse calculation result of G1 is written into the video memory space corresponding to G1 in the video memory, the reverse calculation result of G2 is written into the video memory space corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space corresponding to G3 in the video memory.

[0043] In the disclosed embodiment, if a training step includes only one reverse calculation, the corresponding video memory space can be reconstructed for all elements in the parameter gradient information required for the reverse calculation (e.g., identification information of the parameter gradient) before the reverse calculation. If a training step includes multiple reverse calculations, different reverse calculations have their own required parameter gradient information, and the parameter gradient information required for each reverse calculation can be grouped together. In one example, before executing a certain reverse calculation, the corresponding video memory space is reconstructed only for all elements in the grouping information required for the reverse calculation. Before executing the next reverse calculation, the corresponding video memory space is reconstructed only for all elements in the parameter gradient information required for the next reverse calculation. Figure 4 As shown, a continuous video memory space can be reconstructed for the parameter gradient information G0 and G1 required for reverse calculation 1, and then the reverse calculation 1 is executed. The continuous video memory spaces are written respectively according to the results of reverse calculation 1. Among them, the reverse calculation result of G0 is written into the video memory space corresponding to G0 in the video memory, and the reverse calculation result of G1 is written into the video memory space corresponding to G1 in the video memory. Then, a continuous video memory space is reconstructed for the parameter gradient information G2 and G3 required for reverse calculation 2, and then the reverse calculation 2 is executed, and the continuous video memory spaces are written respectively according to the results of reverse calculation 2. Among them, the reverse calculation result of G2 is written into the video memory space corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space corresponding to G3 in the video memory.

[0044] In the disclosed embodiment, after all reverse calculations of a time step are completed, all reverse calculation results in the reconstructed video memory space can be sent at once during the communication process. After the communication is completed, the reconstructed video memory space can be released.

[0045] According to the disclosed embodiment, the video memory space can be reconstructed for the parameter gradient information required for the reverse calculation before the reverse calculation, and the reconstructed video memory space can be released after the reverse calculation is completed and the model update is completed. The reconstruction and release operations take very little time, so the occupation of the video memory in the subsequent model training process can be reduced, thereby improving the model training speed. Furthermore, since the occupation of the video memory in the model training process is reduced, there is no need to perform too much recalculation, so the number of recalculations can be reduced, speeding up the model training process.

[0046] Figure 5 1 is a flow chart of a video memory control method 500 for model training according to another embodiment of the present disclosure. The method 500 can be used to implement step S101 in the video memory control method 100 for model training. In one embodiment, the method 500 includes: reconstructing the video memory space for the reverse calculation according to the grouping information of the parameter gradient information required for one or more reverse calculations during the model training process, and further includes:

[0047] S501, applying for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory;

[0048] S502: In-situ multiplexing of the requested video memory space and the video memory space required by the parameter gradient information.

[0049] In the disclosed embodiment, the grouping information of the parameter gradient information required for each reverse calculation in a training step of the model training can be stored in the memory in advance. Figure 3 As shown in , the parameter gradient information required for reverse calculation in a training step includes G0, G1, G2 and G3, and these parameter gradient information belong to the same group. Figure 4 As shown in FIG. 1 , in a training step, the parameter gradient information required for reverse calculation 1 includes G0 and G1, which belong to group 1; the parameter gradient information required for reverse calculation 2 includes G2 and G3, which belong to group 2. Before executing the reverse calculation, the video memory space can be reconstructed according to the group information of the parameter gradient information required for the reverse calculation in the memory. For example, Figure 3 As shown, for reverse calculation, the video memory space applied for G0 in the group information is the address range [A0, A1), the video memory space applied for G1 is the address range [A1, A2), the video memory space applied for G2 is the address range [A2, A3), and the video memory space applied for G3 is the address range [A3, A4). For another example, Figure 4 As shown, for reverse calculation 1, the video memory space applied for G0 in group 1 is the address range [B0, B1), and the video memory space applied for G1 in group 1 is the address range [B1, B2). After reconstructing the video memory space, reverse calculation 1 is performed. Then for reverse calculation 2, the video memory space applied for G2 in group 2 is the address range [B3, B4), and the video memory space applied for G3 in group 2 is the address range [B4, B5). After reconstructing the video memory space, reverse calculation 2 is performed.

[0050] In the embodiment of the present disclosure, the video memory space applied for reverse calculation and the video memory space of the parameter gradient information required for reverse calculation can be reused in situ. By in situ reuse, each parameter gradient information in the grouping information can have a corresponding video memory space in the video memory. For example, Figure 3 and Figure 4 As shown in the figure, the address range of the video memory space requested for G0 in the video memory pointed to by G0 is [A0, A1) and [B0, B1) respectively, and the address range of the video memory space requested for G1 in the video memory pointed to by G1 is [A1, A2) and [B1, B2) respectively. Figure 3 and Figure 4As shown, the video memory space applied for G2 in the video memory pointed to by G2 has an address range of [A2, A3) and [B3, B4) respectively, and the video memory space applied for G3 in the video memory pointed to by G3 has an address range of [A3, A4) and [B4, B5) respectively.

[0051] In the disclosed embodiment, the video memory space reconstructed for the same group is usually continuous. The video memory space reconstructed for different groups may be continuous or discontinuous. For example, Figure 4 In one case, the video memory spaces corresponding to G0, G1, G2, and G3 can be continuous. In another case, the video memory spaces corresponding to G0 and G1 are continuous, the video memory spaces corresponding to G2 and G3 are continuous, but the video memory spaces corresponding to G1 and G2 are not continuous.

[0052] After applying for video memory space and performing in-situ multiplexing, the video memory space can be reconstructed for the parameter gradient information required for reverse calculation. Then, after performing reverse calculation, the reverse calculation results can be stored in the corresponding reconstructed video memory space. For example, Figure 3 and Figure 4 As shown, the reverse calculation result of G0 is written into the video memory space [A0, A1) or [B0, B1) corresponding to G0 in the video memory, the reverse calculation result of G1 is written into the video memory space [A1, A2) or [B1, B2) corresponding to G1 in the video memory, the reverse calculation result of G2 is written into the video memory space [A2, A3) or [B3, B4) corresponding to G2 in the video memory, and the reverse calculation result of G3 is written into the video memory space [A3, A4) or [B4, B5) corresponding to G3 in the video memory. In the subsequent communication process, the data in the video memory space [A0, A4) or [B0, B2) and [B3, B5) can be sent to the target device together.

[0053] According to the embodiments of the present disclosure, the video memory space required for reconstructing the corresponding parameter gradient information for each reverse calculation can be used. These parameter gradient information do not need to occupy the video memory space all the time, thereby reducing the occupancy of the video memory, saving video memory resources, and accelerating the model training process.

[0054] In the embodiment of the present disclosure, according to the identification information of multiple parameter gradients in the group information of parameter gradient information required for a certain reverse calculation, a video memory space with a continuous address can be applied in the video memory as the video memory space of the entire group information. Figure 3 As shown, the grouping information includes {parameter gradient G0, parameter gradient G1, parameter gradient G3, parameter gradient G4}; wherein G0, G1, G3 and G4 can be understood as examples of identification information of parameter gradients. Before performing reverse calculation, in the video memory, the address range applied for parameter gradient information is [A0, A4). Figure 4As shown, group 1 includes {parameter gradient G0, parameter gradient G1}; group 2 includes {parameter gradient G2, parameter gradient G3}. Before performing reverse calculation 1, in the video memory, the address range applied for the parameter gradient information in group 1 is [B0, B2), and the address range applied for the parameter gradient information in group 2 is [B3, B5).

[0055] According to the embodiments of the present disclosure, corresponding video memory space can be reconstructed for different reverse calculations in the video memory according to the grouping information of the identification information of the parameter gradients required for the reverse calculations, storage space can be provided for the reverse calculation results, and the reconstructed video memory space can be released after the training step is completed, thereby reducing the occupancy of the video memory space and accelerating the overall training process.

[0056] In one embodiment, S502 further includes: in-situ multiplexing the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the video memory space of the application to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory.

[0057] In the embodiment of the present disclosure, after applying for a continuous video memory space for a certain group information, the storage unit required for the identification information of each parameter gradient in the group information can be reused in situ with the continuous video memory space. This allows the identification information of each parameter gradient to have a corresponding storage unit in the video memory. For example, Figure 3 and Figure 4 As shown, in-situ multiplexing is performed according to the identification information G0 of the parameter gradient, and G0 can be pointed to the address range [A0, A1) or [B0, B1) of the requested video memory space. In-situ multiplexing is performed according to the identification information G1 of the parameter gradient, and G1 can be pointed to the address range [A1, A2) or [B1, B2) of the requested video memory space. In-situ multiplexing is performed according to the identification information G2 of the parameter gradient, and G2 can be pointed to the address range [A2, A3) or [B3, B4) of the requested video memory space. In-situ multiplexing is performed according to the identification information G3 of the parameter gradient, and G3 can be pointed to the address range [A3, A4) or [B4, B5) of the requested video memory space.

[0058] According to the embodiments of the present disclosure, a corresponding storage unit can be reconstructed in the video memory for the identification information of each parameter gradient in the parameter gradient information, which is used to store the calculation results of the reverse calculation that requires the parameter gradient information. The reconstructed storage unit in the video memory can be released subsequently to reduce the occupation of the video memory space, provide more video memory space for subsequent model training, and accelerate the overall model training process.

[0059] In one embodiment, if Figure 5As shown, the method 500 can also be used to implement step S102 in the video memory control method 100 for model training. In one embodiment, the method 500 may include: S503, performing one or more reverse calculations to obtain one or more reverse calculation results.

[0060] In one embodiment, the method 500 can also be used to implement step S103 in the video memory control method 100 for model training. The method 500 may include: storing the one or more reverse calculation results in the video memory space reconstructed for the one or more reverse calculations, and further includes:

[0061] S504: Store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

[0062] In the embodiment of the present disclosure, during the reverse calculation process, the value of each parameter gradient in the parameter gradient information corresponding to the reverse calculation can be generated. These values ​​can be generated sequentially, for example, the value of G0 is generated first, and then the value of G1 is generated. By storing the grouping information of the parameter gradient information into the memory in advance, the video memory space corresponding to each parameter gradient information in the grouping information can be reconstructed according to the needs of the reverse calculation. Then, in the process of executing the reverse calculation again, the calculated values ​​of each parameter gradient are written sequentially to the video memory space for reconstructing each parameter gradient information. For example, after the value V0 of G0 is obtained by reverse calculation, V0 is written to the video memory space corresponding to G0. Then, the reverse calculation is continued, and after the value V1 of G1 is calculated, V1 is written to the video memory space corresponding to G1.

[0063] According to the embodiment of the present disclosure, the values ​​of each parameter gradient obtained by reverse calculation can be stored in the reconstructed video memory space, and the communication efficiency with other devices or components can be improved through the continuous video memory space. In addition, the reconstructed storage unit in the video memory can be released later, reducing the occupation of the video memory space and improving the model training speed.

[0064] Figure 6 6 is a flow chart of a video memory control method 600 for model training according to another embodiment of the present disclosure. The method 600 can be used to implement step S102 in the video memory control method 100 for model training. In one embodiment, the method 600 may include: performing one or more reverse calculations to obtain one or more reverse calculation results, and further including:

[0065] S601. When a training step includes multiple reverse calculations, part or all of the parameter gradient values ​​in the video memory space reconstructed by the first reverse calculation are used as input information for the second reverse calculation, and the second reverse calculation is performed to obtain the parameter gradient values ​​of the second reverse calculation.

[0066] In the disclosed embodiment, multiple reverse calculations included in a training step may have an associated relationship. Assume that a training step includes a first reverse calculation and a second reverse calculation, and part or all of the parameter gradient values ​​in the calculation result of the first reverse calculation can be used in the second reverse calculation. In this case, the video memory space can be reconstructed for the first reverse calculation first; the first reverse calculation is executed to obtain the first reverse calculation result (the parameter gradient value). Then, part or all of the parameter gradient values ​​obtained by the first reverse calculation are used as the input of the second reverse calculation, and the second reverse calculation is continued.

[0067] In the disclosed embodiment, if there is no correlation between multiple reverse calculations in a training step, they can be independently directed to S101 to S103, and then S104 is executed to delete the values ​​of the parameter gradients obtained by these reverse calculations in the video memory, thereby releasing the video memory space reconstructed for these reverse calculations.

[0068] According to the embodiments of the present disclosure, the value of the parameter gradient obtained by the previous reverse calculation can be used as the input of the next reverse calculation, which reduces the number of recalculations and improves the efficiency of training.

[0069] In one embodiment, if Figure 6 As shown, the method 600 can also be used to implement step S103 in the video memory control method 100 for model training. In one embodiment, the method 600 further includes: storing the one or more reverse calculation results in the video memory space reconstructed for the one or more reverse calculations, and further includes:

[0070] S602: Store the value of the parameter gradient of the second reverse calculation into a video memory space reconstructed for the second reverse calculation.

[0071] In the disclosed embodiment, it is assumed that there is an association relationship between the first reverse calculation and the second reverse calculation included in a training step, and part or all of the parameter gradient values ​​in the calculation result of the first reverse calculation can be used in the second reverse calculation. In this case, S101 to S103 can be first performed for the first reverse calculation: reconstruct the video memory space for the first reverse calculation according to the first grouping information of the parameter gradient information required for the first reverse calculation; perform the first reverse calculation to obtain the first reverse calculation result; store the first reverse calculation result in the video memory space reconstructed for the first reverse calculation. Then, part or all of the parameter gradient values ​​obtained by the first reverse calculation are used as the input of the second reverse calculation, and S101 to S103 are performed for the second reverse calculation: reconstruct the video memory space for the second reverse calculation according to the second grouping information of the parameter gradient information required for the second reverse calculation; perform the second reverse calculation to obtain the second reverse calculation result; store the second reverse calculation result in the video memory space reconstructed for the second reverse calculation.

[0072] In one example, see Figure 4 , the first reverse calculation is reverse calculation 1, and the second reverse calculation is reverse calculation 2. According to the first grouping information of the parameter gradient information G0 and G1 required for the first reverse calculation in the model training process, the video memory space [B0, B1) and [B1, B2) are reconstructed for the reverse calculation 1; the reverse calculation 1 is executed to obtain the results V0 and V1 of the reverse calculation 1; the result of the reverse calculation 1 is stored in the video memory space reconstructed for the first reverse calculation, that is, V0 is stored in [B0, B1) and V1 is stored in [B1, B2). If reverse calculation 2 requires the value V1 of G1, the value V1 of G1 can be used as the input of reverse calculation 2, and the video memory space [B3, B4) and [B4, B5) for reverse calculation 2 is reconstructed according to the parameter gradient information G1 and the second grouping information of G2 required for reverse calculation 2; the reverse calculation 2 is executed to obtain the results V2 and V3 of reverse calculation 2; the result of reverse calculation 2 is stored in the video memory space reconstructed for the reverse calculation 2, that is, V2 is stored in [B3, B4) and V3 is stored in [B4, B5).

[0073] According to an embodiment of the present disclosure, before each reverse calculation in the model training process, the video memory space is reconstructed for the parameter gradient information required for the second reverse calculation, and the reconstructed video memory space is released after all reverse calculations in the model training process are completed and the model update is completed. This can reduce the occupancy of the video memory space in the model training process and improve the model training speed.

[0074] In one embodiment, if Figure 6As shown, the method 600 can also be used to implement step S104 in the video memory control method 100 for model training. In one embodiment, the method 600 further includes: releasing the video memory space reconstructed for one or more reverse calculations, further including:

[0075] S603: After all the reverse calculations in a training step are completed, the video memory space reconstructed for all the reverse calculations is released.

[0076] In the disclosed embodiment, if a training step includes multiple reverse calculations and the video memory space has been reconstructed for each reverse calculation, the video memory space of all reverse calculations designed for the training step can be released after the training step is completed. Figure 4 As shown, the video memory spaces [B0, B2) and [B3, B5] reconstructed for reverse calculation 1 and reverse calculation 2 can be released. When releasing, the reconstructed video memory space can be unloaded from the video memory, and the data in the reconstructed video memory space can also be deleted. When a new batch of samples is used for the next training step, the video memory space can be reconstructed and the reverse calculation can be performed based on the grouping information of the parameter gradient information required for the reverse calculation of the next training step. And at the end of each training step, the reconstructed video memory space can be released.

[0077] According to the embodiments of the present disclosure, since the time required for the process of reconstructing and releasing the video memory space is very short, the unnecessary occupation of the video memory space during the training process can be greatly reduced, and therefore the overall training speed can be improved.

[0078] The recalculation strategy is a common strategy in large model training. It can reduce the memory usage during model training and ensure that the model can be trained under the constraints of limited memory space. During the training of a large model, when certain conditions are triggered, the recalculation strategy can be used to recalculate specific calculations to make the model trainable. However, recalculation itself introduces redundant calculations and brings additional performance loss. In order to reduce the performance loss of recalculation, it is necessary to explore memory optimization methods to reduce the peak memory usage of model training, so as to partially or completely turn off recalculation and improve model performance.

[0079] Optimizing parameter gradient memory is an important optimization direction. In related technical solutions, parameter gradients are often fused into tensors in one or more continuous memory spaces to improve the communication performance of parameter gradients during data parallel training. However, this makes it difficult to release parameter gradient memory and thus impossible to optimize parameter gradient memory.

[0080] Before model training begins, all parameter gradients can be grouped, and all parameter gradients in each group will be fused into a large tensor to improve parameter gradient communication performance. If the parameter gradient is used as the resident memory during the model training process, the parameter gradient in this memory will not be released, but the value of the parameter gradient will be set to 0, such as Figure 7a In this case, each parameter gradient will occupy a portion of the video memory regardless of whether it is used or not, and this part of the video memory cannot be saved to improve the model performance.

[0081] In one example, Figure 2 As shown in the figure, the parameter gradients are divided into two groups. In the actual model training process, after the reverse calculation of the current training step is completed, the parameter gradient group 1 and the parameter gradient group 2 need to be reset to zero before the next training step can be carried out. The peak value of the video memory in the training process occurs after the forward calculation is completed and before the reverse calculation begins. Figure 2 , if parameter gradient group 1 includes G0 and G1, and parameter gradient group 2 includes G2 and G3. After the reverse calculation of a training step is completed, the reverse calculation results corresponding to G0, G1, G2 and G3 will be set to zero, but the video memory space corresponding to G0, G1, G2 and G3 will be reserved in the video memory. In the next training step, the video memory space corresponding to G0, G1, G2 and G3 will still be used to store their respective reverse calculation results.

[0082] The disclosed embodiment can use a dynamic gradient release method to release the model parameter gradient from the video memory after each batch of training. During reverse calculation, dynamic reconstruction is performed according to the grouping information to meet the video memory requirements of reverse calculation. This embodiment can reduce the overall video memory usage of the model, thereby reducing the video memory usage of heavy calculations and improving the training performance of large models.

[0083] like Figure 7b As shown, after each training step, the video memory of all parameter gradient groups can be released. When it is necessary to perform reverse calculation on a certain parameter gradient, the parameter gradient group where this parameter gradient is located can be reconstructed, and the video memory space of this group can be applied for. The information required for the reconstruction process (such as the grouping information of the parameter gradient information) is recorded in the memory before the training starts. The reconstruction process mainly includes two operations: one is to apply for the video memory space of the group to be reconstructed according to the grouping information of the parameter gradient information in the memory; the second is to in-place multiplex the video memory space of the parameter gradient information in the group information generated by the reverse operator of the reverse calculation with the video memory space applied for the group information. For example, the name of the parameter gradient in the group information is pointed to the video memory space corresponding to the name of the reconstructed parameter gradient in the video memory through a pointer. The time consumed by these two operations in the reconstruction process is almost negligible. As shown Figure 7bAs shown, after reconstructing the video memory space 1 for the parameter gradient group 1 required for reverse calculation 1, reverse calculation 1 can be performed, and the calculation results of reverse calculation 1 are respectively stored in the corresponding positions in the reconstructed video memory space 1. After reconstructing the video memory space 2 for the parameter gradient group 2 required for reverse calculation 2, reverse calculation 2 can be performed, and the calculation results of reverse calculation 2 are respectively stored in the corresponding positions in the reconstructed video memory space 2.

[0084] In one example, see Figure 4 , the calculation results of G0 and G1 obtained by reverse calculation 1 are stored in the storage unit of G0 and the storage unit of G1 in the reconstructed video memory space respectively. The calculation results of G2 and G3 obtained by reverse calculation 2 are stored in the storage unit of G2 and the storage unit of G3 in the reconstructed video memory space respectively.

[0085] According to the embodiments of the present disclosure, after the forward calculation is completed and before the reverse calculation begins, the video memory occupancy can be made to exclude the video memory of the parameter gradient, thereby reducing the peak video memory occupancy. If the peak video memory is reduced, there is no need to save video memory space through too much recalculation. Therefore, the video memory occupancy of the recalculation can be further reduced, and the model training performance can be improved. The present disclosure can be applied to the performance optimization of large model training processes that require recalculation strategies. The model peak video memory occupancy is reduced by the dynamic release and reconstruction of the parameter gradient video memory, and some recalculations can be turned off to improve training performance.

[0086] Figure 8 8 is a schematic diagram of a memory control device 800 for model training according to an embodiment of the present disclosure. In one embodiment, the device includes:

[0087] A reconstruction module 801 is used to reconstruct a video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for one or more reverse calculations in a model training process;

[0088] A calculation module 802 is used to perform the one or more reverse calculations to obtain one or more reverse calculation results;

[0089] A storage module 803 is used to store the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations;

[0090] The release module 804 is used to release the video memory space reconstructed for one or more reverse calculations.

[0091] Fig. 9900 is a schematic diagram of the structure of a video memory control device for model training according to another embodiment of the present disclosure. The device 900 includes: a reconstruction module 901, a calculation module 902, a storage module 903 and a release module 904. The functions of the above modules can refer to the functions of the modules of the video memory control device for model training 800 in the above embodiment. In one embodiment, the reconstruction module 901 includes:

[0092] The space application submodule 9011 is used to apply for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory;

[0093] The in-situ multiplexing submodule 9012 is used to perform in-situ multiplexing on the requested video memory space and the grouping information of the parameter gradient information in the memory.

[0094] In one embodiment, the in-situ multiplexing submodule 9012 is also used to in-situ multiplex the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory in the video memory.

[0095] In one embodiment, the storage module 903 is further used to store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

[0096] In one embodiment, the calculation module 902 is also used to use part or all of the parameter gradient values ​​in the video memory space reconstructed by the first reverse calculation as input information for the second reverse calculation, and perform the second reverse calculation to obtain the parameter gradient values ​​of the second reverse calculation when multiple reverse calculations are included in a training step.

[0097] In one implementation, the storage module 903 is further used to store the value of the parameter gradient of the second reverse calculation into the video memory space reconstructed for the second reverse calculation.

[0098] In one implementation, the release module 904 is further configured to release the video memory space reconstructed for all reverse calculations after all reverse calculations in a training step are completed.

[0099] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0100] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0101] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0102] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0103] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0104] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0105] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a video memory control method for model training. For example, in some embodiments, the video memory control method for model training may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the video memory control method for model training described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a graphics memory control method for model training in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video memory control method for model training, comprising: Reconstructing video memory space for one or more reverse calculations according to grouping information of parameter gradient information required for one or more reverse calculations during model training; Execute the one or more reverse calculations to obtain one or more reverse calculation results; Storing the one or more reverse calculation results in a video memory space reconstructed for the one or more reverse calculations; Release the video memory space reconstructed for one or more reverse calculations.

2. The method according to claim 1, wherein: Reconstructing the video memory space for the reverse calculation according to the grouping information of the parameter gradient information required for one or more reverse calculations in the model training process, including: Applying for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory; The applied video memory space and the video memory space required by the parameter gradient information are multiplexed in situ.

3. The method according to claim 2, wherein: In-situ multiplexing of the applied video memory space and the video memory space required for the parameter gradient information includes: The storage unit required by the identification information of each parameter gradient in the parameter gradient information is multiplexed in-situ with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the video memory.

4. The method according to claim 3, wherein: Storing the one or more reverse calculation results in a video memory space reconstructed for the one or more reverse calculations includes: The value of each parameter gradient in the one or more reverse calculation results is stored in a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

5. The method according to any one of claims 1 to 4, wherein: Executing the one or more reverse calculations to obtain one or more reverse calculation results includes: When a training step includes multiple reverse calculations, part or all of the parameter gradient values ​​in the video memory space reconstructed by the first reverse calculation are used as input information for the second reverse calculation, and the second reverse calculation is performed to obtain the parameter gradient values ​​of the second reverse calculation.

6. The method according to claim 5, storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations, comprising: The value of the parameter gradient of the second reverse calculation is stored in the video memory space reconstructed for the second reverse calculation.

7. The method according to any one of claims 1 to 6, wherein: Release the video memory space reconstructed for one or more reverse calculations, including: After all the backward calculations in a training step are completed, the video memory space reconstructed for all the backward calculations is released.

8. A video memory control device for model training, comprising: A reconstruction module, used to reconstruct the video memory space for one or more reverse calculations in the model training process according to the grouping information of the parameter gradient information required for the one or more reverse calculations; A calculation module, used to perform the one or more reverse calculations to obtain one or more reverse calculation results; A storage module, used for storing the one or more reverse calculation results into a video memory space reconstructed for the one or more reverse calculations; The release module is used to release the video memory space reconstructed for one or more reverse calculations.

9. The device according to claim 8, wherein: The reconstruction module comprises: A space application submodule, used for applying for reconstruction of video memory space for the one or more reverse calculations according to the grouping information of the parameter gradient information required for the one or more reverse calculations in the memory; The in-situ multiplexing submodule is used to perform in-situ multiplexing on the applied video memory space and the grouping information of the parameter gradient information in the memory.

10. The device according to claim 9, wherein: The in-situ multiplexing submodule is also used to in-situ multiplex the storage unit required for the identification information of each parameter gradient in the parameter gradient information with the applied video memory space to obtain the storage unit corresponding to the identification information of each parameter gradient in the memory in the video memory.

11. The device according to claim 10, wherein: The storage module is further used to store the value of each parameter gradient in the one or more reverse calculation results into a storage unit corresponding to the identification information of each parameter gradient reconstructed for the one or more reverse calculations.

12. The device according to any one of claims 8 to 11, wherein: The calculation module is also used to use the values ​​of part or all of the parameter gradients in the video memory space reconstructed by the first reverse calculation as input information for the second reverse calculation, and perform the second reverse calculation to obtain the values ​​of the parameter gradients of the second reverse calculation when multiple reverse calculations are included in one training step.

13. The device according to claim 12, wherein the storage module is further used to store the value of the parameter gradient of the second reverse calculation into a video memory space reconstructed for the second reverse calculation.

14. The device according to any one of claims 8 to 13, wherein: The release module is also used to release the video memory space reconstructed for all reverse calculations after all reverse calculations in a training step are completed.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images