Training method, device, electronic device and storage medium for neural network model

Abstract

The present disclosure provides a training method, device, electronic device and storage medium for a neural network model, and relates to the field of machine learning, and in particular to the field of deep learning technology. The specific implementation scheme is: based on multiple sub-tensors with consistent behavior, a fused tensor is constructed and the fused tensor is stored in a preset video memory; a mapping relationship between the fused tensor and the sub-tensors is constructed; when a target sub-tensor needs to be used, the target sub-tensor is extracted from the fused tensor based on the mapping relationship to train the neural network model; when a fused tensor needs to be used, the fused tensor is extracted from the preset video memory to train the neural network model. The present disclosure fuses sub-tensors with the same behavior and processes them in units of fused tensors, which can reduce the amount of calculation and save computing resources. Moreover, compared with the communication fusion method, it can reduce the occupied video memory. Moreover, when necessary, the sub-tensors can also be processed separately, making model training more flexible.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to the field of deep learning technology. Background Art

[0002] In deep learning, model training is an iterative convergence process that begins by initializing model parameters and then adjusts them based on the training dataset to achieve the weights and biases that minimize loss. Each iteration consists of three phases: forward propagation, backpropagation, and parameter update. In distributed training, communication operations must be inserted at appropriate locations to ensure training efficiency.

[0003] The training of neural network models requires processing large amounts of data and multiple computing methods. How to effectively utilize computing resources to improve training efficiency is a concern in the industry. Summary of the Invention

[0004] The present disclosure provides a training method, device, electronic device and storage medium for a neural network model.

[0005] According to one aspect of the present disclosure, a method for training a neural network model is provided, comprising:

[0006] Based on multiple sub-tensors with consistent behavior, a fused tensor is constructed and stored in the preset video memory;

[0007] Construct the mapping relationship between the fused tensor and the sub-tensors;

[0008] When the target sub-tensor is needed, the target sub-tensor is extracted from the fusion tensor based on the mapping relationship, and the neural network model is trained based on the target sub-tensor;

[0009] When the fused tensor is needed, the fused tensor is extracted from the preset video memory and the neural network model is trained based on the fused tensor.

[0010] According to another aspect of the present disclosure, there is provided a training apparatus for a neural network model, comprising:

[0011] The fusion module is used to construct a fused tensor based on multiple sub-tensors with consistent behavior and store the fused tensor in the preset video memory;

[0012] Construction module, used to build the mapping relationship between fusion tensor and sub-tensor;

[0013] A first training module is used to extract the target sub-tensor from the fusion tensor based on the mapping relationship when the target sub-tensor is needed, and train the neural network model based on the target sub-tensor;

[0014] The second training module is used to extract the fused tensor from the preset video memory when the fused tensor is needed, and train the neural network model based on the fused tensor.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and,

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.

[0021] In the disclosed embodiments, sub-tensors with the same behavior are fused and processed as fused tensors, which can reduce the amount of computation and conserve computing resources. Furthermore, compared to communication fusion, this approach can reduce the amount of video memory used. Furthermore, if necessary, sub-tensors can be processed individually, making model training more flexible.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0024] Figure 1 is a flowchart of a training method for a neural network model according to an embodiment of the present disclosure;

[0025] FIG2( a ) is a schematic diagram of the arrangement order of multiple sub-tensors according to an embodiment of the present disclosure;

[0026] FIG2( b ) is a schematic diagram of the structure of a tensor group according to an embodiment of the present disclosure;

[0027] Figure 3(a)-Figure 3(f)is a schematic diagram of constructing a fused tensor according to an embodiment of the present disclosure;

[0028] Figure 4 is a scenario diagram of a training method for a neural network model according to an embodiment of the present disclosure;

[0029] FIG5( a ) is a schematic diagram of a scenario of a training method for a neural network model according to another embodiment of the present disclosure;

[0030] FIG5( b ) is a specific timing diagram according to another embodiment of the present disclosure;

[0031] FIG5( c ) is a schematic diagram of a scenario in the parameter updating stage according to another embodiment of the present disclosure;

[0032] Figure 6 is a structural diagram of a training device for a neural network model according to an embodiment of the present disclosure;

[0033] Figure 7 is a structural diagram of a training device for a neural network model according to another embodiment of the present disclosure;

[0034] Figure 8 It is a block diagram of an electronic device used to implement a training method of a neural network model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Traditional tensor fusion methods are mainly divided into two types: layer tensor fusion (Layer & Tensor fusion) and communication fusion (Fused AllReduce).

[0037] 1) Layer tensor fusion is based on horizontal or vertical merging between layers to significantly reduce the number of layers, achieving efficient and portable inference capabilities. Horizontal merging refers to combining convolution, bias, and activation layers into a CBR (convolution bias ReLU) structure, while vertical merging refers to combining layers with the same structure but different weights into a single CBR layer. The combined CBR layer only requires a single kernel launch to complete the computation.

[0038] After training, neural network models require inference deployment. This involves deploying a pre-trained neural network model to real-world scenarios, such as image classification, object detection, and online translation. Because layer tensor fusion is only applicable during the inference deployment phase and requires customized fusion solutions for different network structures, it is not universally applicable.

[0039] 2) Communication fusion involves inserting communication operations into the distributed training process, synchronizing gradients across multiple machines to ensure training efficiency. To fully utilize network bandwidth and shorten communication time, the communication fusion solution fuses multiple sub-tensors into a single tensor before communication, and then restores the tensor into multiple independent sub-tensors after communication.

[0040] While communication fusion can improve the communication performance of distributed training, it introduces redundant concatenation and splitting operations in each iteration, affecting overall training efficiency. Furthermore, the temporary tensor fusion operation allocates additional video memory on top of the original tensor. Even if this memory is released promptly after communication ends, it will still increase the peak video memory usage during training.

[0041] In view of this, in order to improve the training efficiency, a training method for a neural network model is provided in the embodiment of the present disclosure. This method proposes a special structure and process of tensor fusion, which improves the efficiency of model training while being universal. Figure 1 A training method for a neural network model provided in an embodiment of the present disclosure is described.

[0042] like Figure 1 FIG. 1 is a flow chart of a training method for a neural network model provided by an embodiment of the present disclosure, including the following contents:

[0043] S101: Based on multiple sub-tensors with consistent behaviors, a fused tensor is constructed and the fused tensor is stored in a preset video memory.

[0044] The sub-tensor type can be model parameters (param), gradients (grad), momentum (momentum), model states (modelstates), etc. For example, the model parameters of a convolutional layer can be a sub-tensor. The gradient corresponding to each model parameter can also be a sub-tensor.

[0045] S102, constructing a mapping relationship between the fused tensor and the sub-tensors.

[0046] That is, the fused tensor can be further encapsulated to construct a tensor group, which needs to contain a mapping relationship between the sub-tensors and the fused tensor. The mapping relationship can be defined as the position encoding of the sub-tensors in the fused tensor.

[0047] S103: When the target sub-tensor needs to be used, the target sub-tensor is extracted from the fusion tensor based on the mapping relationship, and the neural network model is trained based on the target sub-tensor.

[0048] S104: When the fused tensor needs to be used, the fused tensor is extracted from the preset video memory, and the neural network model is trained based on the fused tensor.

[0049] Therefore, in the embodiment of the present disclosure, sub-tensors with the same behavior are fused, and when the fused tensor needs to be used, it can be processed in units of the fused tensor, thereby reducing the number of calls to the kernel, reducing the amount of calculation, saving computing resources, and improving training efficiency compared to processing each sub-tensor separately. In addition, in the embodiment of the present disclosure, the mapping relationship between the sub-tensors in the fused tensor is maintained in the tensor group, and each sub-tensor can be processed separately in the previous manner without affecting the model training process. Compared with the hierarchical fusion solution, there is no need to worry about the specific networking structure, so it has better scalability and versatility. Compared with the communication fusion solution, in the embodiment of the present disclosure, there is no need to temporarily generate a fused tensor every time the fused tensor needs to be used, thereby further reducing the amount of calculation and saving computing resources. Moreover, there is no need to occupy additional video memory to temporarily cache the fused tensor, which can improve the utilization of video memory.

[0050] To facilitate understanding, some aspects involved in the embodiments of the present disclosure are described below.

[0051] Constructing a fused tensor based on multiple sub-tensors with consistent behavior in S101. In some embodiments, the multiple sub-tensors with consistent behavior may be determined based on the following methods, including:

[0052] If the computational behaviors of the multiple sub-tensors are consistent, determining that the behaviors of the multiple sub-tensors are consistent; and / or,

[0053] In the case where the communication behaviors of the multiple sub-tensors are consistent, it is determined that the behaviors of the multiple sub-tensors are consistent.

[0054] In the disclosed embodiments, sub-tensors with consistent computational behavior are constructed into a single fused tensor, while sub-tensors with consistent communication behavior are constructed into another fused tensor. When executing these computational or communication behaviors, only one kernel call is required for each sub-tensor to complete the operation. This eliminates the need for separate kernel calls for each sub-tensor, thereby improving the training efficiency of the neural network.

[0055] For example, there is a parameter update operation in the training process of a neural network model, and this update operation is not related to the specific parameters, that is, the update methods of various model parameters are basically the same. Therefore, the {parameter A, parameter B, ..., parameter N} used for the parameter update operation can be determined as having consistent operation behavior, and {parameter A, parameter B, ..., parameter N} can be fused to obtain a fused tensor for parameter update.

[0056] In addition to the aforementioned parameter update behavior, there are other consistent computational behaviors in the optimizer. For example, these include: gradient synchronization behavior (such as the need to calculate the mean gradient in distributed training), gradient explosion detection behavior, gradient vanishing detection behavior, etc.

[0057] Among them, the aforementioned exploding gradients and vanishing gradients are two phenomena that lead to optimizer instability. During neural network model training, the optimizer uses gradients to determine the direction and magnitude of neural network parameter updates, thereby updating the model parameters in the correct direction and with the appropriate magnitude. In deep networks or recurrent neural networks, the update process may accumulate a very large gradient. Excessively large gradients can significantly update the model parameters, making it difficult for the model parameters to converge and the neural network model unstable. From a deep perspective, the learning speeds of different layers vary greatly. Gradient explosion manifests itself as layers near the output learning very well, while layers near the input learn very slowly. Training can take a long time, and the weights of the first few layers remain essentially the same as the randomly initialized values. In extreme cases, the values ​​of the model parameter weights become so large that the results display NaN values, where NaN represents infinity and non-numeric values.

[0058] The manifestation of vanishing gradients is that the closer the parameters are to the output layer, the larger the gradient, making them the parameters that are primarily learned; while the parameters farther away from the output layer can only be learned at a very slow rate with a gradient close to 0. This situation is equivalent to a vicious cycle. Because the values ​​of the nodes near the output layer are obtained by the forward operation performed by the layer with a slow learning rate, and because the forward layer has a slow learning rate, the parameters may not have learned the characteristics, the input of the subsequent layers will be more random. This is equivalent to learning on random data. Even if the learning rate is fast, it may not actually learn useful features. This process makes it even more difficult for the parameters of the previous layers to learn valid values.

[0059] Therefore, both gradient explosion and gradient disappearance need to be checked. The inspection method can be implemented by calling the kernel corresponding to the gradient explosion and disappearance check operator (check_inf_and_scaleop). Based on the tensor fusion method provided in the embodiment of the present disclosure, it is possible to avoid calling the parallel processing function corresponding to the gradient explosion and disappearance check for each gradient separately (which corresponds to at least one kernel). Specifically, using the fused tensor in the embodiment of the present disclosure, it is possible to aggregate the gradients, obtain the gradient group, and then call the corresponding kernel based on the gradient group to obtain the inspection result.

[0060] In the disclosed embodiments, exploding and vanishing gradient checks are independent of specific model parameters. Therefore, all parameters of the entire neural network model can be fused into a single fused tensor, effectively improving check efficiency. Specifically, when checking for exploding or vanishing gradients, the kernel only needs to be called once, rather than separately for each model parameter. Other gradient synchronization and parameter updates are similar. Therefore, fusing sub-tensors with these computational behaviors can reduce the number of kernel calls, improve model training efficiency, and reduce resource consumption.

[0061] In some embodiments, consistent behavior can also be understood as copying central processing units (CPUs) between different devices. For example, in the forward pass of a context autoencoder (CAE), the exponential moving average (EMA) component can be fused into a single fused tensor to avoid performing separate EMA operations on each sub-tensor, thus saving resources and improving model training efficiency.

[0062] In other embodiments, extensive research has revealed that sub-tensors with consistent computational or communication behaviors have consistent types and devices. This allows obtaining the type and device corresponding to each sub-tensor, and determining that multiple sub-tensors with the same type and device are consistent in behavior.

[0063] The types corresponding to each sub-tensor can be divided into two categories: floating-point and integer. Floating-point types are further divided into 16-bit, 32-bit, and 64-bit types based on precision. Integer types are further divided into 8-bit unsigned integers, 8-bit signed integers, 16-bit signed integers, 32-bit signed integers, and 64-bit signed integers based on the presence or absence of a sign bit and precision. Each type can be further divided into CPUs and graphics processing units (GPUs) based on the device storing the tensor.

[0064] To improve the accuracy of operations during neural network model training, the tensor types required for each step should be the same. Therefore, sub-tensors with the same type and device can be determined to behave in a consistent manner. The determination of consistent behavior based on the tensor type and device can be determined autonomously by the computer or specified by the user based on needs, and this is not limited in the present embodiment.

[0065] In the disclosed embodiments, sub-tensors of the same type and device are determined to have consistent behavior. This consistent behavior is independent of the specific network structure of the model. For any network structure, the disclosed embodiments primarily consider whether the sub-tensors behave consistently, and can fuse sub-tensors with the same behavior, thereby saving computing resources and effectively improving the training efficiency of the neural network model. Because the network structure does not need to be considered, the disclosed embodiments are more scalable and versatile.

[0066] For S102, the mapping relationship can be the arrangement order of multiple sub-tensors in the fused tensor, as shown in Figure 2(a). The sub-tensor can be in the form of a single tensor, and the single tensor is shown as sub-tensor 1 in Figure 2(a). The sub-tensor can also be in the form of an array, and the form of the array is shown as sub-tensor 2 in Figure 2(a). The constructed tensor group can not only include the arrangement position of the sub-tensors in the fused tensor (that is, the mapping relationship), but also include the attributes of each sub-tensor. For example, the attributes of the sub-tensor may include whether the sub-tensor is trainable, whether it can be split, etc. With these attributes, it is convenient to train based on these attributes during the model training process.

[0067] The structure of its tensor group is shown in Figure 2(b). The tensor list contains multiple sub-tensors, corresponding to the fused tensor in the GPU. The bidirectional arrows indicate the mapping between the sub-tensors and the fused tensor. The tensor group includes the mapping relationship between the sub-tensors and the fused tensor, as well as information such as the sub-tensor's device, type, and attributes. If the fused tensor of the neural network needs to be reconstructed, reconstruction can be performed based on this record.

[0068] In order to shorten the time spent in the model training phase, in the embodiment of the present disclosure, the preset video memory is the GPU video memory, and the construction of the fusion tensor is completed in the initialization phase of the model training, which may include the following operations: based on the amount of data of each sub-tensor in the multiple sub-tensors that need to be fused, determining the video memory capacity required for the fusion tensor; releasing the video memory capacity occupied by each sub-tensor in the multiple sub-tensors, and storing the fusion tensor in the preset video memory.

[0069] Taking the model parameters as an example, in the initialization phase before model training, a parameter list is first constructed as shown in FIG3(a). The parameter list can be specified by the user according to the needs, or it can be automatically generated by the computer according to certain rules. The certain rules can be to automatically identify sub-tensors of the same type and device as sub-tensors with consistent behavior, and to construct a parameter list based on these sub-tensors with consistent behavior. It should be noted that the parameter list used by the same neural network model can be one or more. The number of parameter lists can be determined according to actual needs, and the embodiment of the present disclosure does not limit this. For each parameter list, the sub-tensors in the parameter list are calculated, the size of the sub-tensors is calculated, and the arrangement order of the sub-tensors is recorded. Based on the recorded arrangement order and the size of the sub-tensors, a video memory that can accommodate the fusion parameters (i.e., the fusion tensor) is applied for inside the GPU, and the initialization of the fusion parameters is completed in the CPU at the same time, that is, the unique identifier of the fusion parameter of the parameter list is defined in the CPU, and the storage space required for the fusion parameter is applied. After initialization, as indicated by the arrows in Figure 3(b), the sub-tensors in the parameter list are copied to the CPU in the order they appear in the parameter list, completing the construction of the fusion parameters. To conserve GPU memory, the GPU memory is then released. The process of releasing GPU memory is illustrated in Figure 3(c). After the fusion parameters are constructed on the CPU, the GPU memory occupied by the sub-tensors is released. The dashed line indicates the clearing of the GPU memory occupied by the sub-tensors. To facilitate model training, the fusion parameters are stored on the GPU, as shown in Figure 3(d). After the copying is complete, the CPU memory is released, as shown in Figure 3(e). After completing these operations, the constructed parameter group (i.e., tensor group) facilitates the mapping relationship. As shown in Figure 3(f), the bidirectional arrows indicate the mapping relationship between the sub-tensors and the fusion parameters on the GPU. The number of GPUs can be multiple or one. In the case of multiple GPUs, the fusion parameters need to be synchronized across multiple GPUs.

[0070] In the disclosed embodiment, the step of fusing tensors is placed in the initialization stage of model training, and the fused tensors are retained during subsequent training. There is no need to fuse and release the fused tensors multiple times, ensuring that the entire model training can improve training efficiency based on the fused tensors, and can also reasonably train the model based on the mapping relationship.

[0071] In addition, the above-mentioned fusion tensor construction operation can be applied not only in the model initialization stage, but also at the beginning of model training.

[0072] Regarding the target sub-tensor in S103, when the target sub-tensor needs to be used, the target sub-tensor is extracted from the fusion tensor based on the mapping relationship, and the neural network model is trained based on the target sub-tensor. It can be implemented as follows: when some sub-tensors in the fusion tensor perform the target behavior, some sub-tensors are extracted from the fusion tensor based on the mapping relationship; the kernel corresponding to the target behavior is called to process the target sub-tensor to obtain the processing result; and the neural network model is trained based on the processing result.

[0073] Among them, the target sub-tensor can be one sub-tensor or multiple sub-tensors, and during implementation, it can be determined according to training requirements.

[0074] Examples of extracting target subtensors, such as neural network structures such as Figure 4 As shown in the figure, there are multiple layers in the neural network structure, including input layer, fully connected layer, activation layer, bias layer, output layer, etc. The convolution layer is also divided into multiple convolution kernels. Figure 4 Each layer is fused as a sub-tensor into a single fused tensor. When a target sub-tensor is needed, it can be extracted for model training. For example, when performing a convolution operation on a 3×3 convolutional layer, the sub-tensor corresponding to the 3×3 convolutional layer can be called based on the mapping relationship in the tensor group to complete the neural network model training process.

[0075] In the embodiment of the present disclosure, for the tensor group constructed based on the mapping relationship, it is possible to call the sub-tensors separately, making the present disclosure more universal.

[0076] For the fused tensor in S104, when each sub-tensor in the fused tensor performs the same behavior, determine the kernel corresponding to the same behavior; call the kernel to execute the same behavior to obtain the execution result; and train the neural network model based on the execution result.

[0077] Continue to use Figure 4 The neural network model structure is illustrated using distributed training as an example. The training sample data is evenly divided into multiple batches, as shown in Figure 5(a). Each batch is then divided into multiple sub-batches, with each GPU responsible for training one sub-batch. Since each GPU is responsible for a sub-batch of data, the sub-gradients it obtains may differ, necessitating gradient synchronization.

[0078] The gradient synchronization phase mainly involves two steps: gradient mean and AllReduce communication. Based on the gradient synchronization method, the model parameters are updated in reverse. Since the gradient mean is independent of the specific model parameters, the gradient fusion method can be used to Figure 4All sub-gradients of the neural network layers in the training set are fused into a fused tensor (i.e., the gradient group in Figure 5(a)). A kernel can be launched to calculate the gradient mean, avoiding multiple kernel calls for multiple gradients, reducing the number of kernel launches, and improving training efficiency.

[0079] The comparison chart of reducing the number of kernel calls is shown in Figure 5(b). When the fused tensor is not used, each sub-gradient will call the kernel once, while when the fused tensor is used, the kernel only needs to be called once, thereby accelerating the model training speed and saving training time.

[0080] In some possible implementations, a parameter group, a gradient group, a momentum group, and a master parameter group can be constructed in the initialization phase of the neural network model based on the method provided in the embodiment of the present disclosure. As shown in Figure 5(c), the group (tensor group) of the parameter update calculation phase under the tensor fusion strategy proposed in the embodiment of the present disclosure includes a training parameter group, a gradient group, a momentum group, and a master parameter group. Under the selected optimizer, different parameters have the same update formula, that is, the behavior is consistent. Therefore, by directly using fused_tensor (fused tensor) for calculation, the number of calls to the optimizer calculation op (fusion operator) can be reduced from O(N) to O(1) while the amount of calculation remains unchanged.

[0081] In the optimizer compute node, in addition to the aforementioned exploding / vanishing gradient checks and parameter updates, other operations with similar behaviors also apply, such as gradient clipping.

[0082] For example, gradient nan / inf (explosion / vanishing) checks require obtaining the nan / inf information of all parameter gradients. If check_inf_and_scaleop is called for each parameter separately and the check results are aggregated at the end, the same effect can be achieved by directly calling the relevant op once with fused_param (fused tensor).

[0083] Similarly, gradient clipping constrains all parameter gradients to a certain range based on a certain norm. For operations like pow, elementwise_add, and scale involved in norm calculation and gradient compression, using fusedparam directly reduces performance loss. The specific timing diagram is similar to Figure 5(b) and is not repeated here.

[0084] In summary, the disclosed embodiments enable a single kernel call to complete distributed training of neural network models, even when different sub-tensors perform the same action. This reduces the number of kernel launches from N to 1, effectively saving unnecessary tensor read and write time and kernel launch time.

[0085] It should be noted that the disclosed solution is applicable to the model pre-training stage of various fields, for example, it is applicable to the model pre-training stage of natural language processing (NLP), computer vision (CV), protein structure prediction, etc.

[0086] In the field of natural language processing, the present disclosure can be applied to models such as the Generative Pre-Training (GPT) model, the Enhanced Language Representation with Informative Entities (ERNIE) model, and the Mixture of Experts networks (MOE) model in NLP.

[0087] Regardless of the language model used, in the disclosed embodiments, when training a language model in parallel using multiple GPUs, the sub-gradients generated by each GPU for each model parameter are used as sub-tensors to generate a gradient group, i.e., a fused tensor of gradients. This allows the kernel to be called once to calculate the mean gradient during backpropagation.

[0088] Similarly, when checking for exploding / vanishing gradients, the relevant kernel can be called once to check the gradient value.

[0089] For the model parameters, when updating the parameters, since the same gradient mean is used, the kernel can be called once to complete the parameter update, and then synchronized to each GPU card, thereby reducing the amount of calculation and lowering performance loss.

[0090] Similarly, in the field of computer vision, the present disclosure is applicable to models such as Computer Aided Engineering (CAE) and PaddlePaddle Large Scale Classification (PLSC) in CV.

[0091] Regardless of the visual model used, in the disclosed embodiments, when training a language model in parallel using multiple GPUs, the sub-gradients generated by each GPU for each model parameter are used as sub-tensors to generate a gradient group, i.e., a fused tensor of gradients. This allows the kernel to be called once to calculate the mean gradient during backpropagation.

[0092] Similarly, when checking for exploding / vanishing gradients, the relevant kernel can be called once to check the gradient value.

[0093] For the model parameters, when updating the parameters, since the same gradient mean is used, the kernel can be called once to complete the parameter update, and then synchronized to each GPU card, thereby reducing the amount of calculation and lowering performance loss.

[0094] In the field of protein prediction, the present disclosure can be applied to models such as AlphaFold2, which predicts protein structure based on sequence.

[0095] Regardless of the protein prediction model used, in the disclosed embodiments, when training a language model in parallel using multiple GPUs, the sub-gradients generated by each GPU for each model parameter are used as sub-tensors to generate a gradient group, i.e., a fused tensor of gradients. This allows for backpropagation and gradient mean calculations to be performed by invoking a kernel once using the gradient group.

[0096] Similarly, when checking for exploding / vanishing gradients, the relevant kernel can be called once to check the gradient value.

[0097] For the model parameters, when updating the parameters, since the same gradient mean is used, the kernel can be called once to complete the parameter update, and then synchronized to each GPU card, thereby reducing the amount of calculation and lowering performance loss.

[0098] Based on the same technical concept, the embodiment of the present disclosure also provides a training device for a neural network model, such as Figure 6 Shown, including:

[0099] A fusion module 601 is configured to construct a fused tensor based on multiple sub-tensors with consistent behaviors and store the fused tensor in a preset video memory;

[0100] A construction module 602 is used to construct a mapping relationship between the fusion tensor and the sub-tensors;

[0101] A first training module 603 is configured to extract a target sub-tensor from the fused tensor based on a mapping relationship when a target sub-tensor is needed, and train a neural network model based on the target sub-tensor;

[0102] The second training module 604 is used to extract the fused tensor from the preset video memory when the fused tensor is needed, and train the neural network model based on the fused tensor.

[0103] In some embodiments, Figure 6 On the basis of Figure 7 As shown, it also includes:

[0104] The first tensor set determination module 701 is configured to determine a plurality of sub-tensors with consistent behaviors based on the following method:

[0105] If the computational behaviors of the multiple sub-tensors are consistent, determining that the behaviors of the multiple sub-tensors are consistent; and / or,

[0106] In the case where the communication behaviors of the multiple sub-tensors are consistent, it is determined that the behaviors of the multiple sub-tensors are consistent.

[0107] In some embodiments, Figure 6 On the basis of Figure 7 As shown, it also includes:

[0108] The second tensor set determination module 702 is configured to determine a plurality of sub-tensors with consistent behaviors based on the following method:

[0109] Get the type and device corresponding to each sub-tensor;

[0110] Multiple sub-tensors of the same type and device are determined to have consistent behavior.

[0111] In some embodiments, the second training module 604 is configured to:

[0112] When each sub-tensor in the fused tensor performs the same behavior, determine the kernel corresponding to the same behavior;

[0113] Call the kernel to perform the same action and get the execution result;

[0114] Train the neural network model based on the execution results.

[0115] In some embodiments, the first training module 603 is configured to:

[0116] When the target sub-tensor in the fused tensor performs the target behavior, the target sub-tensor is extracted from the fused tensor based on the mapping relationship;

[0117] Call the kernel corresponding to the target behavior to process the target sub-tensor and obtain the processing result;

[0118] The neural network model is trained based on the processing results.

[0119] In some embodiments, the preset video memory is GPU video memory, and the fusion module 601 performs the following operations during the initialization phase of model training, including:

[0120] Determine the video memory capacity required for the fused tensor based on the data volume of each sub-tensor in the multiple sub-tensors;

[0121] Releases the video memory capacity occupied by each of the multiple sub-tensors and stores the fused tensor in the default video memory.

[0122] In some embodiments, the operation behaviors of the multiple sub-tensors include: gradient synchronization behavior, gradient explosion check behavior, gradient disappearance check behavior, and parameter update behavior.

[0123] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0125] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0128] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the training method of the neural network model. For example, in some embodiments, the training method of the neural network model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the training method of the neural network model described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the training method of the neural network model by any other appropriate means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain. The disclosed embodiments may employ a server to execute a neural network model training method.

[0136] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0137] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training a neural network model, comprising: Constructing a fused tensor based on multiple sub-tensors with consistent behaviors and storing the fused tensor in a preset video memory; Constructing a mapping relationship between the fused tensor and the sub-tensors; When a target sub-tensor is needed, extracting the target sub-tensor from the fused tensor based on the mapping relationship, and training a neural network model based on the target sub-tensor; When the fused tensor needs to be used, extracting the fused tensor from the preset video memory, and training the neural network model based on the fused tensor, including: When each sub-tensor in the fused tensor performs the same behavior, determining a kernel corresponding to the same behavior; Calling the kernel to execute the same behavior and obtaining an execution result; A neural network model is trained based on the execution result.

2. The method according to claim 1, further comprising: Multiple sub-tensors with consistent behavior are determined based on the following: If the operation behaviors of the multiple sub-tensors are consistent, determining that the behaviors of the multiple sub-tensors are consistent; and / or, In a case where the communication behaviors of the plurality of sub-tensors are consistent, it is determined that the behaviors of the plurality of sub-tensors are consistent.

3. The method according to claim 1, further comprising: Multiple sub-tensors with consistent behavior are determined based on the following: Get the type and device corresponding to each sub-tensor; Multiple sub-tensors of the same type and device are determined to be multiple sub-tensors with consistent behaviors.

4. The method according to claim 1, wherein In a case where a target sub-tensor is needed, extracting the target sub-tensor from the fused tensor based on the mapping relationship, and training a neural network model based on the target sub-tensor, including: When the target sub-tensor in the fused tensor performs a target behavior, extracting the target sub-tensor from the fused tensor based on the mapping relationship; Calling the kernel corresponding to the target behavior to process the target sub-tensor and obtain a processing result; The neural network model is trained based on the processing results.

5. The method according to any one of claims 1 to 4, wherein the preset video memory is GPU video memory, and the following operations are performed during the initialization phase of model training, including: Determining a video memory capacity required for the fused tensor based on the data amount of each sub-tensor in the multiple sub-tensors; Release the video memory capacity occupied by each sub-tensor in the multiple sub-tensors, and store the fused tensor in the preset video memory.

6. The method according to claim 5, wherein the operations on the plurality of sub-tensors comprise: Gradient synchronization behavior, gradient explosion check behavior, gradient disappearance check behavior, and parameter update behavior.

7. A training device for a neural network model, comprising: A fusion module, configured to construct a fused tensor based on multiple sub-tensors with consistent behaviors and store the fused tensor in a preset video memory; A construction module, configured to construct a mapping relationship between the fused tensor and the sub-tensors; a first training module, configured to extract the target sub-tensor from the fused tensor based on the mapping relationship when a target sub-tensor is needed, and train a neural network model based on the target sub-tensor; A second training module is configured to extract the fused tensor from the preset video memory when the fused tensor is needed, and train the neural network model based on the fused tensor; the second training module is configured to: When each sub-tensor in the fused tensor performs the same behavior, determining a kernel corresponding to the same behavior; Calling the kernel to execute the same behavior and obtaining an execution result; A neural network model is trained based on the execution result.

8. The apparatus according to claim 7, further comprising: The first tensor set determination module is configured to determine the plurality of sub-tensors with consistent behaviors based on the following method: In a case where the operation behaviors of the multiple sub-tensors are consistent, determining that the behaviors of the multiple sub-tensors are consistent; and / or, In a case where the communication behaviors of the plurality of sub-tensors are consistent, it is determined that the behaviors of the plurality of sub-tensors are consistent.

9. The apparatus according to claim 7, further comprising: The second tensor set determination module is configured to determine the plurality of sub-tensors having consistent behaviors based on the following method: Get the type and device corresponding to each sub-tensor; Multiple sub-tensors of the same type and device are determined to be multiple sub-tensors with consistent behaviors.

10. The device according to claim 7, wherein The first training module is used to: When the target sub-tensor in the fused tensor performs a target behavior, extracting the target sub-tensor from the fused tensor based on the mapping relationship; Calling the kernel corresponding to the target behavior to process the target sub-tensor and obtain a processing result; The neural network model is trained based on the processing results.

11. The apparatus according to any one of claims 7 to 10, wherein the preset video memory is GPU video memory, and the fusion module performs the following operations during the initialization phase of model training, including: Determining a video memory capacity required for the fused tensor based on the data amount of each sub-tensor in the multiple sub-tensors; Release the video memory capacity occupied by each sub-tensor in the multiple sub-tensors, and store the fused tensor in the preset video memory.

12. The apparatus according to claim 11, wherein the operations on the plurality of sub-tensors comprise: Gradient synchronization behavior, gradient explosion check behavior, gradient disappearance check behavior, and parameter update behavior.

13. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images