Gradient information storage method and device, electronic equipment and storage medium
When training a neural network model, the gradient information is adaptively stored and deleted according to the video memory size of the video memory space and gradient information, the problem of insufficient video memory capacity of the graphics processor is solved and the training efficiency of the model is improved.
Patent Information
- Application Number
- CN202411593347.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-02-28
AI Technical Summary
When training neural network models, graphics processors (GPUs) are prone to insufficient memory capacity, especially when using gradient descent algorithm optimization, the sample size processed during forward propagation and back propagation is large, resulting in high demand for gradient information storage.
By adaptively storing and deleting gradient information according to the storage capacity of the target memory space and the video memory size occupied by each set of gradient information, it is ensured that the video memory size occupied by the gradient information to be retained is less than or equal to the storage capacity of the target memory space.
It effectively avoids the problem of insufficient memory space, reduces the time for recalculating gradient information at the backpropagation node, and improves the training efficiency of neural network models.
Smart Images
Figure CN119536642A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to a method, device, electronic device, storage medium and computer program product for storing gradient information. Background Art
[0002] When training a neural network model, it is usually not trained in a cloud computing environment for security reasons. When the size of the neural network model is large, the storage capacity of the video memory space is high. In particular, when the gradient descent algorithm is used to optimize the neural network model, the amount of samples processed in the forward propagation process and the back propagation process will be relatively large, and the gradient information of each computing node of the neural network model will be stored during the forward propagation process. The graphics processing unit (GPU) is prone to insufficient video memory capacity. Summary of the invention
[0003] The present invention is proposed in view of the above problems. The embodiments of the present invention provide a method, device, electronic device, storage medium and computer program product for storing gradient information. This solution can adaptively retain the gradient information generated during the training of the neural network model according to the storage capacity of the video memory space, effectively reducing the time for recalculating the gradient information at the back propagation node, and the training efficiency of the model is high.
[0004] According to one aspect of the present invention, a method for storing gradient information is provided, which is used to store gradient information calculated during the training of a neural network model. The method includes: obtaining first capacity information corresponding to a target video memory space used to store gradient information of the neural network model, the first capacity information being used to represent the storage capacity of the target video memory space; obtaining video memory size information corresponding to each group of gradient information of the neural network model, the video memory size information being used to represent the size of video memory occupied by the gradient information, each group of gradient information corresponding to a unique computing node of the neural network model; based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information other than the gradient information to be retained is deleted, wherein the size of video memory occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
[0005] Optionally, the neural network model includes multiple computing nodes; based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: based on a preset arrangement order of the computing nodes, deleting the gradient information corresponding to at least some of the computing nodes in the neural network model until the total video memory size occupied by the undeleted gradient information is less than or equal to the storage capacity of the target video memory space, and storing the undeleted gradient information as the gradient information to be retained in the neural network model in the target video memory space.
[0006] Optionally, the video memory size occupied by each group of gradient information is equal, the neural network model includes at least two node sets, each node set includes multiple computing nodes; according to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: according to the first capacity information and the video memory size information corresponding to each group of gradient information, a first target number of groups is determined, the first target number of groups is the maximum number of groups of gradient information that can be accommodated in the target video memory space; according to the first target number of groups, at least A second target number of groups corresponding to each node set in the two node sets, the second target number of groups being the number of groups of gradient information to be retained corresponding to the node set, and the sum of the second target numbers of groups corresponding to at least two node sets being equal to the first target number of groups; for each node set in the at least two node sets, based on a preset arrangement order of the computing nodes, deleting the gradient information corresponding to at least part of the computing nodes in the node set until the number of groups of the gradient information not deleted is equal to the second target number of groups corresponding to the node set, and storing the gradient information not deleted in the target video memory space as the gradient information to be retained corresponding to the node set.
[0007] Optionally, the size of video memory occupied by each group of gradient information is equal, the neural network model includes at least two node sets, each node set includes multiple computing nodes; based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: based on the first capacity information and the video memory size information corresponding to each group of gradient information, a first target number of groups is determined, the first target number of groups being the maximum number of groups of gradient information that can be accommodated in the target video memory space; based on a preset arrangement order of the computing nodes, the gradient information corresponding to at least part of the computing nodes in the neural network model is deleted until the number of groups of the undeleted gradient information is equal to the first target number of groups, and the undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
[0008] Optionally, the target video memory space is at least part of the storage space of the preset video memory, and obtaining first capacity information corresponding to the target video memory space used to store gradient information of the neural network model includes: obtaining second capacity information and third capacity information of the preset video memory, the second capacity information being used to represent the total storage capacity of the preset video memory, and the third capacity information being used to represent the inherent occupied capacity in the preset video memory; based on the second capacity information and the third capacity information, determining the first capacity information, wherein the storage capacity represented by the first capacity information is equal to the difference between the total storage capacity and the inherent occupied capacity.
[0009] Optionally, based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information other than the gradient information to be retained is deleted, including: determining the initial number of groups or the initial video memory size corresponding to the gradient information to be retained in the neural network model based on the first capacity information and the video memory size information corresponding to each group of gradient information; outputting gradient indication information for indicating the initial number of groups or the initial video memory size; determining the target number of groups or the target video memory size corresponding to the gradient information to be retained in the neural network model in response to the user's feedback operation on the initial number of groups or the initial video memory size based on the gradient indication information; storing the gradient information that meets the target number of groups or the target video memory size as the gradient information to be retained in the target video memory space and deleting the remaining gradient information.
[0010] Optionally, in response to a user's feedback operation on the initial number of groups or the initial memory size based on the gradient indication information, a target number of groups or a target memory size corresponding to the gradient information to be retained in the neural network model is determined, including: when the feedback operation is a modification operation, if the modified number of groups or the modified memory size input by the user is greater than the initial number of groups or the initial memory size, an error prompt message is output.
[0011] According to another aspect of the present invention, there is also provided a storage device for gradient information, comprising: a first acquisition module, used to acquire first capacity information corresponding to a target video memory space for storing gradient information of the neural network model, the first capacity information being used to represent the storage capacity of the target video memory space; a second acquisition module, used to acquire video memory size information corresponding to each group of gradient information of the neural network model, the video memory size information being used to represent the size of video memory occupied by the gradient information, each group of gradient information corresponding to a unique computing node of the neural network model; a storage module, used to store the gradient information to be retained in the neural network model in the target video memory space and delete the gradient information except the gradient information to be retained according to the first capacity information and the video memory size information corresponding to each group of gradient information, wherein the size of video memory occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
[0012] According to yet another aspect of the present invention, there is provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned method for storing gradient information when the processor is running.
[0013] According to yet another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned method for storing gradient information when running.
[0014] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned method for storing gradient information when running.
[0015] The above technical solution can effectively avoid insufficient storage space of the video memory when training the model by adaptively storing and deleting part of the gradient information according to the storage capacity of the target video memory space used to store the gradient information and the size of the video memory occupied by each group of gradient information. In addition, by storing and deleting part of the gradient information based on the storage capacity of the target video memory space, the video memory space can be fully utilized, and the excessive amount of gradient information deleted due to indiscriminate deletion of gradient information can be effectively avoided, which in turn leads to too long calculation time for recalculating the deleted gradient information in the back propagation stage, which is beneficial to saving the training time of the model and improving the training efficiency of the model.
[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0018] Figure 1 A schematic block diagram showing a method for storing gradient information according to an embodiment of the present invention is shown;
[0019] Figure 2 A schematic diagram of the architecture of a neural network model according to an embodiment of the present invention is shown;
[0020] Figure 3 A schematic block diagram of a storage device for gradient information according to an embodiment of the present invention is shown;
[0021] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.
[0023] As described above, in the related art, the gradient information of each computing node of the neural network model will be stored during the forward propagation process, and the graphics processing unit (GPU) is prone to insufficient video memory capacity. If excessive gradient information is deleted, the model training time will increase significantly. In order to at least solve the above technical problems, the embodiments of the present invention provide a method, device, electronic device, storage medium and computer program product for storing gradient information. This solution can adaptively retain the gradient information generated during the training of the neural network model according to the storage capacity of the video memory space, which is conducive to avoiding the problem of insufficient video memory of the graphics processing unit, and can effectively reduce the time for recalculating the gradient information at the back propagation node, and the training efficiency of the model is relatively high.
[0024] See also Figure 1As shown, it is a schematic block diagram of a method for storing gradient information according to an embodiment of the present invention. According to one aspect of the present invention, a method for storing gradient information is provided, which is used to store gradient information calculated during the training of a neural network model, and the method includes: step S110, step S120 and step S130.
[0025] Exemplarily, when training a neural network model, the training process may include two stages, namely, a forward propagation stage and a backward propagation stage. In the forward propagation stage, each neuron of each layer of the subnetwork of the neural network model (i.e., a "computational node" in an embodiment of the present invention) may receive the input of the previous layer of the subnetwork to which it belongs for weighted summation, and substitute the result of the weighted summation into the activation function to obtain the intermediate activation value of the computational node (i.e., the "gradient information" in an embodiment of the present invention). The intermediate activation value may be used to calculate the gradient for optimizing the neural network model in the backward propagation stage. It can be understood that there are intermediate activation values of some computational nodes in the neural network model, and after deletion, the intermediate activation values of these computational nodes may be recalculated in the backward propagation stage.
[0026] In step S110, first capacity information corresponding to a target video memory space for storing gradient information of a neural network model is obtained, where the first capacity information is used to represent the storage capacity of the target video memory space.
[0027] Exemplarily, when training a neural network model, the video memory space of the graphics processor can temporarily store the data of the neural network model, and the data of the neural network model can include at least part of the gradient information obtained in the forward propagation stage. In the video memory space, the space used to store the gradient information of the neural network model can be recorded as the target video memory space, and the target video memory space can have first capacity information, and the first capacity information can indicate the maximum amount of data of the gradient information that can be stored in the target video memory space, that is, the storage capacity of the target video memory space. It can be understood that the storage capacity of the target video memory space is less than the storage capacity of the entire video memory space. In the video memory space, other video memory spaces except the target video memory space can be used to store other information of the neural network model other than the gradient information when training the neural network model, such as weights, biases, losses, input data, etc.
[0028] In step S120, the video memory size information corresponding to each group of gradient information of the neural network model is obtained, where the video memory size information is used to indicate the size of the video memory occupied by the gradient information, and each group of gradient information corresponds to a unique computing node of the neural network model.
[0029] Exemplarily, the neural network model may have multiple computing nodes, and in the forward propagation stage, a set of gradient information corresponding to each computing node may be obtained. Specifically, each computing node may obtain a weighted summation result based on the weight and input data, and the sum of the cumulative result and the bias may obtain an intermediate activation value through an activation function, and the intermediate activation value is a set of gradient information corresponding to the computing node. Each set of gradient information may occupy a certain size of video memory space, and in the video memory space, the video memory size occupied by the set of gradient information may be represented by the video memory size information. Accordingly, each set of gradient information corresponds to a set of video memory size information. It can be understood that the video memory size occupied by each set of gradient information may be the same or different, which is related to the tensor size and data type of the sample data input to the neural network model. When the tensor size and data type of each set of sample data input are consistent, the video memory size occupied by each set of gradient information is the same.
[0030] In step S130, based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information except the gradient information to be retained is deleted, wherein the video memory size occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
[0031] Exemplarily, the total video memory size occupied by each group of gradient information can be determined based on the video memory size information corresponding to each group of gradient information, and the storage capacity of the target video memory space used to store each group of gradient information can be determined based on the first capacity information. When the total video memory size occupied by each group of gradient information meets the preset condition, the gradient information to be retained can be determined in each group of gradient information, and the gradient information to be retained can be stored in the target storage space, and the gradient information in each group of gradient information other than the gradient information to be retained can be deleted. Specifically, the preset condition can be used to determine whether to delete part of the gradient information in each group of gradient information, which can be defined by the user according to actual needs. The preset condition can be, for example, that the total video memory size occupied by each group of gradient information is greater than the storage capacity of the target video memory space, and can also be, for example, that the ratio of the total video memory size occupied by each group of gradient information to the storage capacity of the target video memory space is greater than a preset ratio (for example, 0.9). In some embodiments, the gradient information to be retained can be determined according to a preset retention rule, and the preset retention rule can be defined based on the algorithm rule of the trained neural network model, which can be used to determine the gradient information that needs to be retained in the neural network model. The preset retention rule may, for example, be to retain the gradient information of a specific computing node in the neural network model, and the specific computing node may include: a computing node of the input layer, a computing node of the output layer, a computing node of a preset key layer (for example, in a residual network, a computing node connecting each residual block), etc. In other embodiments, the gradient information to be deleted may be determined according to the preset deletion rule, and the gradient information to be retained may be the gradient information in each group of gradient information except the gradient information to be deleted. Similarly, the preset deletion rule may also be defined based on the algorithmic rule of the trained neural network model, which may be used to determine the gradient information that can be deleted in the neural network model. The preset deletion rule may, for example, be to delete the gradient information of a specific computing node in the neural network model, and the specific computing node may include: a computing node corresponding to the gradient information that does not participate in the calculation process of the back propagation stage, a computing node corresponding to the gradient information that can be calculated by forward propagation communication (for example, in a residual network, the computing nodes of the residual block whose gradient information can be restored by the retained forward propagation path), a computing node of the linear layer, etc. It can be understood that when determining the gradient information to be retained according to the preset retention rules, the retention priority of the gradient information corresponding to each type of computing node can be set. More specifically, for any type of computing node, the retention priority of the gradient information corresponding to each computing node of the computing node can also be set. After determining the storage capacity of the target video memory space, the gradient information to be retained can be determined based on the retention priority of the gradient information corresponding to each computing node. The deletion logic of the gradient information to be deleted according to the preset deletion rules can be similar to the retention logic of the gradient information to be retained according to the preset retention rules, which will not be repeated here.It can be understood that the size of the video memory occupied by the gradient information to be retained determined according to the preset retention rule / deletion rule is less than or equal to the storage capacity of the target video memory space.
[0032] The above technical solution can effectively avoid insufficient storage space of the video memory when training the model by adaptively storing and deleting part of the gradient information according to the storage capacity of the target video memory space used to store the gradient information and the size of the video memory occupied by each group of gradient information. In addition, by storing and deleting part of the gradient information based on the storage capacity of the target video memory space, the video memory space can be fully utilized, and the excessive amount of gradient information deleted due to indiscriminate deletion of gradient information can be effectively avoided, which in turn leads to too long calculation time for recalculating the deleted gradient information in the back propagation stage, which is beneficial to saving the training time of the model and improving the training efficiency of the model.
[0033] Optionally, the neural network model includes multiple computing nodes; based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: based on a preset arrangement order of the computing nodes, deleting the gradient information corresponding to at least some of the computing nodes in the neural network model until the total video memory size occupied by the undeleted gradient information is less than or equal to the storage capacity of the target video memory space, and storing the undeleted gradient information as the gradient information to be retained in the neural network model in the target video memory space.
[0034] Exemplarily, the preset arrangement order of each computing node in the neural network model can be set according to the architecture of the trained neural network model. Taking the neural network model in which the network layer includes an input layer, a feature extraction layer, an intermediate processing layer, and an output layer as an example, each network layer of the neural network model can have one or more computing nodes, and the first arrangement order of the computing nodes of each network layer can be, for example: computing nodes of the input layer - computing nodes of the feature extraction layer - computing nodes of the intermediate processing layer - computing nodes of the output layer. When a certain network layer has multiple computing nodes, the second arrangement order of each computing node of the network layer can be determined according to the calculation order of each computing node of the network layer and / or the dependency relationship between each computing node. For each computing node of the neural network model, the preset arrangement order of the computing node can be determined based on the first arrangement order and / or the second arrangement order of the computing node. After determining the gradient information that can be deleted in the neural network model, the gradient information corresponding to some computing nodes can be deleted based on the preset arrangement order of the computing nodes corresponding to the determined gradient information that can be deleted, and the deleted gradient information satisfies: the occupied video memory size is greater than or equal to the difference between the total video memory size occupied by all the gradient information that has not been deleted and the storage capacity of the target video memory space. In some embodiments, the gradient information may be deleted group by group in a preset arrangement order until the total video memory size occupied by the remaining undeleted gradient information is less than or equal to the storage capacity of the target video memory space, or until the video memory size occupied by the deleted gradient information is greater than or equal to the difference between the total video memory size occupied by all undeleted gradient information and the storage capacity of the target video memory space, and the remaining undeleted gradient information may be stored in the target video memory space as the gradient information to be retained in the neural network model. In other embodiments, all the gradient information to be deleted may be determined based on the storage capacity of the target video memory space and the preset arrangement order of each computing node. Specifically, after deleting the portion of the gradient information, the remaining undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
[0035] The above technical solution can delete gradient information based on the preset arrangement order of computing nodes in the neural network model, and can select part of the gradient information for deletion according to the preset arrangement order of computing nodes in multiple groups of deletable gradient information. This deletion method can dynamically adapt to the architecture of the neural network model, so that the deleted gradient information can be as simple and fast as possible in the calculation process, which is conducive to further reducing the time for recalculating gradient information in the back propagation stage.
[0036] Optionally, the video memory size occupied by each group of gradient information is equal, the neural network model includes at least two node sets, each node set includes multiple computing nodes; according to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: according to the first capacity information and the video memory size information corresponding to each group of gradient information, a first target number of groups is determined, the first target number of groups is the maximum number of groups of gradient information that can be accommodated in the target video memory space; according to the first target number of groups, at least A second target number of groups corresponding to each node set in the two node sets, the second target number of groups being the number of groups of gradient information to be retained corresponding to the node set, and the sum of the second target numbers of groups corresponding to at least two node sets being equal to the first target number of groups; for each node set in the at least two node sets, based on a preset arrangement order of the computing nodes, deleting the gradient information corresponding to at least part of the computing nodes in the node set until the number of groups of the gradient information not deleted is equal to the second target number of groups corresponding to the node set, and storing the gradient information not deleted in the target video memory space as the gradient information to be retained corresponding to the node set.
[0037] Exemplarily, the tensor size and data type of each group of sample data input into the neural network model can be consistent. In this case, the video memory size occupied by each group of gradient information obtained in the forward propagation stage is the same. Multiple computing nodes in the neural network model can be divided into at least two node sets, and accordingly, each node set can include multiple computing nodes. The embodiment of the present invention does not specifically limit the division rule of the node set. In some embodiments, it can be divided according to the network layer to which the computing node belongs. In other embodiments, the multiple network layers of the neural network model can be divided into at least two network blocks according to the purpose (for example, one or more network layers for feature extraction, and one or more network layers for feature processing belong to network layers of different purposes respectively), and the node set can be divided according to the network block to which the computing node belongs. In other embodiments, the computing nodes can also be divided into at least two node sets according to the dependency relationship between the computing nodes (for example, serial dependency, parallel dependency, jump connection, etc.). When the video memory size occupied by each group of gradient information is the same, the first target number of groups can be determined according to the first capacity information and the video memory size information corresponding to each group of gradient information. Specifically, the first capacity information may indicate that the storage capacity of the target video memory space is U, and the video memory size information corresponding to each group of gradient information may indicate that the video memory size occupied by each group of gradient information is a, then the maximum number of groups of gradient information that can be accommodated in the target video memory space (i.e., the first target number of groups) is equal to z, wherein U / a-1<z≤U / a and z is an integer. After determining the first target number of groups, the number of groups of gradient information to be retained corresponding to each node set may be allocated according to a preset allocation rule, and the number of groups of gradient information to be retained corresponding to each node set may be recorded as the second target number of groups. It can be understood that the sum of the second target number of groups corresponding to each node set may be less than or equal to the first target number of groups. It should be noted that the sum of the second target number of groups corresponding to each node set in the embodiment of the present invention is preferably equal to the first target number of groups. The embodiment of the present invention does not limit the preset allocation rule, which can be defined by the user. For example, the preset allocation rule may be an average allocation (the second target number of groups corresponding to each node set is equal), or an allocation according to a preset allocation ratio (the ratio of the second target number of groups corresponding to each node set is equal to the preset allocation ratio), or an allocation in response to the user's allocation operation information, where the allocation operation information is the specific value of the second target number of groups corresponding to each node set set by the user based on the first target number of groups. After determining the second target number of groups corresponding to each computing node set, for each node set, each computing node in the node set may have a preset arrangement order. For the description of the preset arrangement order, please refer to the aforementioned embodiment, which will not be repeated here.In some embodiments, the gradient information may be deleted group by group according to the preset arrangement order of each computing node in the node set, until the number of groups of remaining undeleted gradient information corresponding to the node set is equal to the second target number of groups, and the remaining undeleted gradient information corresponding to the node set may be stored in the target video memory space as the gradient information to be retained in the neural network model. In other embodiments, all the gradient information to be deleted may be determined based on the second target number of groups corresponding to the node set and the preset arrangement order of each computing node in the node set, and after deleting the part of the gradient information, the remaining undeleted gradient information may be stored in the target video memory space as the gradient information to be retained in the neural network model.
[0038] The above technical solution can retain and delete gradient information based on the characteristics of each node set by allocating the number of groups of gradient information to be retained for each node set, thereby realizing neural network models with different architectures. The retention and deletion strategies of gradient information for neural network models of various architectures can be flexibly set, so that the stored gradient information can be quickly and efficiently applied to the calculation of the back propagation stage of the current neural network model.
[0039] Optionally, the size of video memory occupied by each group of gradient information is equal, the neural network model includes at least two node sets, each node set includes multiple computing nodes; based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: based on the first capacity information and the video memory size information corresponding to each group of gradient information, a first target number of groups is determined, the first target number of groups being the maximum number of groups of gradient information that can be accommodated in the target video memory space; based on a preset arrangement order of the computing nodes, the gradient information corresponding to at least part of the computing nodes in the neural network model is deleted until the number of groups of the undeleted gradient information is equal to the first target number of groups, and the undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
[0040] Exemplarily, the specific process of determining the first target number of groups and the description of the preset arrangement order can all be referred to the above embodiments, which will not be repeated here. In some embodiments, the gradient information can be deleted group by group according to the preset arrangement order of each computing node until the number of groups of the remaining undeleted gradient information is equal to the first target number of groups, and the remaining undeleted gradient information can be stored in the target video memory space as the gradient information to be retained in the neural network model. It should be noted that the number of groups of gradient information to be retained can be less than or equal to the first target number of groups, and the number of groups of gradient information to be retained in the embodiment of the present invention is preferably equal to the first target number of groups. In other embodiments, the number of groups of gradient information to be deleted can be calculated. Specifically, the number of groups of all gradient information obtained in the forward propagation stage is recorded as w. When the first target number of groups is equal to z, the number of groups of gradient information to be deleted k≥wz (the specific value of the gradient information to be deleted can be defined according to actual needs). The gradient information can be deleted group by group according to the preset arrangement order of each computing node until the number of groups of deleted gradient information is equal to k, and the remaining undeleted gradient information can be stored in the target video memory space as the gradient information to be retained in the neural network model. Alternatively, all the gradient information to be deleted can be obtained according to the preset arrangement order of each computing node and the number k of groups of gradient information to be deleted (the number of all the groups of gradient information to be deleted obtained is equal to k), and after deleting the part of the gradient information, the remaining undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
[0041] The above technical solution can calculate the number of groups of gradient information to be retained when the video memory occupied by each gradient information is equal. The calculation method for the number of groups of gradient information to be retained is simple and the corresponding calculation amount is small, so that the gradient information to be retained can be determined quickly and efficiently.
[0042] Optionally, the target video memory space is at least part of the storage space of the preset video memory, and obtaining first capacity information corresponding to the target video memory space used to store gradient information of the neural network model includes: obtaining second capacity information and third capacity information of the preset video memory, the second capacity information being used to represent the total storage capacity of the preset video memory, and the third capacity information being used to represent the inherent occupied capacity in the preset video memory; based on the second capacity information and the third capacity information, determining the first capacity information, wherein the storage capacity represented by the first capacity information is equal to the difference between the total storage capacity and the inherent occupied capacity.
[0043] Exemplarily, the preset video memory may be the video memory of a graphics processor used when training a neural network model. Those skilled in the art may understand that the video memory has a storage space for storing data, and the target video memory space may be at least part of the storage space of the preset video memory. The preset video memory may have second capacity information and third capacity information. The second capacity information may represent the total storage capacity of the preset video memory (that is, the maximum amount of data that can be stored), and the third capacity information may represent the inherent occupied capacity in the preset video memory. The inherent occupied capacity refers to the amount of data that needs to occupy a fixed video memory size when training a neural network model. The data occupying a fixed video memory size may include, for example, model parameters, optimizer state information, sample data input to the model, data output by the model, model architecture information (such as the number of network layers, network layer type), etc. The storage capacity of the target storage space may be equal to the difference between the total storage capacity of the preset video memory and the inherent occupied capacity.
[0044] The above technical solution can fully consider the data that needs to occupy a fixed video memory size in the video memory currently used to train the neural network model. The size of the target storage space for storing gradient information can be determined based on this part of data. For different neural network models, the size of the target storage space can be flexibly determined, which is conducive to the reasonable allocation of the entire storage space of the video memory, thereby making the data amount of the gradient information to be retained more reasonable.
[0045] Optionally, based on the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information other than the gradient information to be retained is deleted, including: determining the initial number of groups or the initial video memory size corresponding to the gradient information to be retained in the neural network model based on the first capacity information and the video memory size information corresponding to each group of gradient information; outputting gradient indication information for indicating the initial number of groups or the initial video memory size; determining the target number of groups or the target video memory size corresponding to the gradient information to be retained in the neural network model in response to the user's feedback operation on the initial number of groups or the initial video memory size based on the gradient indication information; storing the gradient information that meets the target number of groups or the target video memory size as the gradient information to be retained in the target video memory space and deleting the remaining gradient information.
[0046] Exemplarily, when training a neural network model on a graphics processor, the graphics processor may determine the initial number of groups (less than or equal to the maximum number of groups of gradient information that the target video memory space can accommodate) or the initial video memory size (less than or equal to the storage capacity of the target storage space) corresponding to the gradient information to be retained in the neural network model according to a preset storage rule based on the first capacity information and the video memory size information corresponding to each group of gradient information. The preset storage rule may be, for example, that the initial number of groups / initial video memory size corresponding to the gradient information to be retained is equal to the maximum number of groups / storage capacity of the gradient information that the target video memory space can accommodate. The preset storage rule may also be, for example, that the initial number of groups / initial video memory size corresponding to the gradient information to be retained is equal to the preset number of groups / preset capacity, wherein the preset number of groups is less than or equal to the maximum number of groups of gradient information that the target video memory space can accommodate, and the preset capacity is less than or equal to the storage capacity of the target storage space. The graphics processor according to an embodiment of the present invention may be connected to an interactive device, and the interactive device may include an input component (such as a keyboard, a mouse, etc.) and a display component (such as a display screen). The graphics processor may perform corresponding operations in response to user operations on the input component, and may also display specific information to the user through the display component. The graphics processor can output gradient indication information indicating the initial video memory size after the initial group number to the user through the display component, and the user can perform a feedback operation on the graphics processor through the input component based on the gradient indication information. The feedback operation can be, for example, a confirmation operation for confirming that the initial group number is the target group number corresponding to the gradient information to be retained, or a confirmation operation for confirming that the initial video memory size is the target video memory size of the gradient information to be retained. The feedback operation can also be a modification operation for modifying the initial group number, or a modification operation for modifying the target video memory size of the initial video memory size. The graphics processor can determine the target group number or target video memory size corresponding to the gradient information to be retained in the neural network model in response to the user's feedback operation. After determining the target group number or target video memory size, the gradient information to be retained can be determined based on the target group number or target video memory size, and the determined gradient information to be retained can be stored in the target video memory space and the remaining gradient information can be deleted. It can be understood that the determined number of groups / video memory size of the gradient information to be retained is equal to the target number of groups / target video memory size.
[0047] The above technical solution can adjust the data amount of the gradient information to be retained based on the user's own needs by interacting with the user, making the selection of the gradient information to be retained more flexible.
[0048] Optionally, in response to a user's feedback operation on the initial number of groups or the initial memory size based on the gradient indication information, a target number of groups or a target memory size corresponding to the gradient information to be retained in the neural network model is determined, including: when the feedback operation is a modification operation, if the modified number of groups or the modified memory size input by the user is greater than the initial number of groups or the initial memory size, an error prompt message is output.
[0049] Exemplarily, the user can perform a modification operation on the initial number of groups or the initial video memory size through the input component based on the gradient indication information. When the number of groups / video memory size modified by the user is greater than the initial number of groups / initial video memory size, the image processor can output an error prompt information to the user through the display component in response to the modification operation. For example, a pop-up box can be displayed on the display screen to prompt the user that the modification has failed.
[0050] The above technical solution can effectively prevent the storage capacity of the target storage space from being unable to meet the storage requirements after the user modifies the initial number of groups or the initial video memory size, thereby effectively avoiding the possibility of model training failure due to insufficient video memory.
[0051] See also Figure 2 As shown, it is a schematic diagram of the architecture of a neural network model according to an embodiment of the present invention. Each node in the figure can represent a computing node, and the number next to the node can represent the intermediate activation value corresponding to the node. Figure 2 The neural network model shown in has three network layers: the bottom input layer containing 4 computing nodes, the middle layer containing 2 computing nodes, and the top input layer containing only 1 computing node. Figure 2It can be seen that the gradient information corresponding to the two computing nodes on the left of the input layer can be used as the input of the left computing node of the middle layer, and the gradient information corresponding to the two computing nodes on the right of the input layer can be used as the input of the right computing node of the middle layer. The gradient information corresponding to the two computing nodes in the middle layer can be used as the input of the output layer. The video memory size occupied by the gradient information corresponding to each computing node of the neural network model is 1, and the video memory size required to store all the gradient information of the neural network model is 7. When the storage capacity of the target video memory space is less than 7, for example, in this embodiment, the storage capacity of the target video memory space can be 3, and the gradient information corresponding to some computing nodes can be deleted according to the preset arrangement order of each computing node. In the embodiment of the present invention, the preset arrangement order of the computing nodes can be the computing nodes of the input layer-the computing nodes of the middle layer-the computing nodes of the output layer, and the preset arrangement order of the computing nodes of each layer can be randomly arranged by the graphics processor, or arranged from small to large according to the calculation amount of the computing nodes of this layer. It can be understood that the number of groups of gradient information to be deleted in the embodiment of the present invention is 4. Assume that in the back propagation phase, for any one of the computing nodes in the middle layer and the computing nodes in the output layer, the computing node performs a summation operation on the gradient information input to the computing node. For example, the four computing nodes in the input layer have corresponding gradient information of 2, 1, 4, and 5 from left to right. Then the left computing node in the middle layer performs a summation operation on the input "2" and "1", and the obtained gradient information is "3". The right computing node in the middle layer performs a summation operation on the input "4" and "5", and the obtained gradient information is "9". The output layer performs a summation operation on the gradient information corresponding to the two computing nodes in the middle layer, and the obtained gradient information is 12. Each computing node can store the corresponding gradient information respectively. When the number of groups of gradient information to be deleted is 4, according to the aforementioned preset arrangement order, the 4 groups of gradient information stored on the 4 computing nodes in the input layer can be deleted, and the video memory size occupied by the remaining gradient information is 3, which is equal to the storage capacity of the target video memory space.
[0052] See also Figure 3 As shown, it is a schematic block diagram of a storage device for gradient information according to an embodiment of the present invention. According to another aspect of the present invention, a storage device 300 for gradient information is provided, which is used to store gradient information calculated during the training process of a neural network model. The device 300 includes:
[0053] A first acquisition module 310 is used to acquire first capacity information corresponding to a target video memory space used to store gradient information of the neural network model, wherein the first capacity information is used to represent the storage capacity of the target video memory space;
[0054] A second acquisition module 320 is used to obtain video memory size information corresponding to each set of gradient information of the neural network model, wherein the video memory size information is used to indicate the size of video memory occupied by the gradient information, and each set of gradient information corresponds to a unique computing node of the neural network model;
[0055] The storage module 330 is used to store the gradient information to be retained in the neural network model in the target video memory space and delete the gradient information other than the gradient information to be retained according to the first capacity information and the video memory size information corresponding to each group of gradient information, wherein the video memory size occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
[0056] See also Figure 4 As shown, it is a schematic block diagram of an electronic device according to an embodiment of the present invention. According to another aspect of the present invention, an electronic device 400 is also provided, including: a processor 410 and a memory 420, wherein the memory 410 stores computer program instructions, and the computer program instructions are used by the processor 410 to execute the above-mentioned method for storing gradient information when the processor 410 is running.
[0057] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned method for storing gradient information of the embodiment of the present invention, and is used to implement the corresponding module in the above-mentioned storage device for gradient information or the corresponding module in the above-mentioned storage device for gradient information according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above-mentioned storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.
[0058] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned method for storing gradient information when running.
[0059] A person skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned storage device for gradient information by reading the above-mentioned specific description of the storage method for gradient information, and for the sake of brevity, it will not be described in detail here.
[0060] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.
[0061] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0062] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.
[0063] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0064] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.
[0065] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.
[0066] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.
[0067] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the storage device for gradient information according to an embodiment of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0068] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.
[0069] The above is only a specific embodiment or description of a specific embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for storing gradient information, used to store gradient information calculated during the training of a neural network model, characterized in that: include: Acquire first capacity information corresponding to a target video memory space used to store gradient information of the neural network model, wherein the first capacity information is used to represent a storage capacity of the target video memory space; Obtaining video memory size information corresponding to each set of gradient information of the neural network model, wherein the video memory size information is used to indicate the size of video memory occupied by the gradient information, and each set of gradient information corresponds to a unique computing node of the neural network model; According to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information except the gradient information to be retained is deleted, wherein the video memory size occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
2. The method according to claim 1, characterized in that: The neural network model includes a plurality of computing nodes; According to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: Based on a preset arrangement order of the computing nodes, the gradient information corresponding to at least part of the computing nodes in the neural network model is deleted until the total video memory size occupied by the undeleted gradient information is less than or equal to the storage capacity of the target video memory space, and the undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
3. The method according to claim 1, characterized in that The video memory occupied by each group of gradient information is equal in size, the neural network model includes at least two node sets, and each node set includes multiple computing nodes; According to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: Determine a first target number of groups according to the first capacity information and the video memory size information corresponding to each group of gradient information, wherein the first target number of groups is a maximum number of groups of gradient information that can be accommodated by the target video memory space; Determining a second target number of groups corresponding to each node set in the at least two node sets according to the first target number of groups, wherein the second target number of groups is the number of groups of gradient information to be retained corresponding to the node sets, and the sum of the second target numbers of groups corresponding to the at least two node sets is equal to the first target number of groups; For each node set of the at least two node sets, based on a preset arrangement order of the computing nodes, gradient information corresponding to at least part of the computing nodes in the node set is deleted until the number of groups of the undeleted gradient information is equal to a second target number of groups corresponding to the node set, and the undeleted gradient information is stored in the target video memory space as the gradient information to be retained corresponding to the node set.
4. The method according to claim 1, characterized in that: The video memory occupied by each group of gradient information is equal in size, the neural network model includes at least two node sets, and each node set includes multiple computing nodes; According to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space, and the gradient information other than the gradient information to be retained is deleted, including: Determine a first target number of groups according to the first capacity information and the video memory size information corresponding to each group of gradient information, wherein the first target number of groups is a maximum number of groups of gradient information that can be accommodated by the target video memory space; Based on a preset arrangement order of the computing nodes, the gradient information corresponding to at least part of the computing nodes in the neural network model is deleted until the number of groups of the undeleted gradient information is equal to the first target number of groups, and the undeleted gradient information is stored in the target video memory space as the gradient information to be retained in the neural network model.
5. The method according to claim 1, characterized in that The target video memory space is at least a portion of the storage space of the preset video memory, and the obtaining of the first capacity information corresponding to the target video memory space for storing the gradient information of the neural network model includes: Acquire second capacity information and third capacity information of the preset video memory, wherein the second capacity information is used to indicate the total storage capacity of the preset video memory, and the third capacity information is used to indicate the inherent occupied capacity of the preset video memory; The first capacity information is determined based on the second capacity information and the third capacity information, wherein the storage capacity represented by the first capacity information is equal to the difference between the total storage capacity and the inherent occupied capacity.
6. The method according to any one of claims 1 to 5, characterized in that: According to the first capacity information and the video memory size information corresponding to each group of gradient information, the gradient information to be retained in the neural network model is stored in the target video memory space and the gradient information other than the gradient information to be retained is deleted, including: Determining the initial number of groups or the initial memory size corresponding to the gradient information to be retained in the neural network model according to the first capacity information and the memory size information corresponding to each group of gradient information; Outputting gradient indication information for indicating the initial number of groups or the initial video memory size; In response to a user's feedback operation on the initial number of groups or the initial video memory size based on the gradient indication information, determining a target number of groups or a target video memory size corresponding to the gradient information to be retained in the neural network model; The gradient information that meets the target number of groups or the target video memory size is stored in the target video memory space as the gradient information to be retained, and the remaining gradient information is deleted.
7. The method according to claim 6, characterized in that The step of determining a target number of groups or a target memory size corresponding to the gradient information to be retained in the neural network model in response to a user's feedback operation on the initial number of groups or the initial memory size based on the gradient indication information comprises: When the feedback operation is a modification operation, if the modified number of groups or the modified video memory size input by the user is greater than the initial number of groups or the initial video memory size, an error prompt message is output.
8. A gradient information storage device for storing gradient information calculated during neural network model training, characterized in that: include: A first acquisition module, used to acquire first capacity information corresponding to a target video memory space used to store gradient information of the neural network model, wherein the first capacity information is used to represent the storage capacity of the target video memory space; A second acquisition module is used to obtain video memory size information corresponding to each group of gradient information of the neural network model, wherein the video memory size information is used to indicate the size of video memory occupied by the gradient information, and each group of gradient information corresponds to a unique computing node of the neural network model; A storage module is used to store the gradient information to be retained in the neural network model in the target video memory space and delete the gradient information other than the gradient information to be retained according to the first capacity information and the video memory size information corresponding to each group of gradient information, wherein the video memory size occupied by the gradient information to be retained is less than or equal to the storage capacity of the target video memory space.
9. An electronic device, comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the method for storing gradient information according to any one of claims 1 to 7 when the processor is running the computer program instructions.
10. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the method for storing gradient information as described in any one of claims 1 to 7 when running.
11. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the method for storing gradient information according to any one of claims 1 to 7 when running.