Method and device calculating weights

Abstract

Embodiments of the present application provide a method for calculating weights and a related device. The method includes: obtaining an input gradient of a current layer; calculating, based on a result of multiplication of the input gradient, a scaling factor, and input information of the current layer, a first weight gradient of the current layer; and updating, based on the first weight gradient of the current layer, weights of the current layer. The proposed technique makes it possible to prevent gradient overflow in the calculated weight gradient, thereby improving the stability of calculation.

Description

METHOD AND DEVICE FOR CALCULATING WEIGHTSTECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of training neural networks and more particularly, to a method for calculating weights, and a related device.BACKGROUND

[0002] Many technical and high-performance operations require usage of high precision numerical formats for computation, but in many cases lower precision arithmetic operations may suffice. Another reason for using lower precision operations is lower resource consumption and faster computation.

[0003] A model consists of a series of layers, each of them possibly having a set of weights, that can be modified during a training process. As the model training process is a time and resource consuming process, it can greatly benefit from reduced precision. In FP16 and BF16 computations and memory consumption can be up to two times lower than those in FP32. However, reduced precision raises problems of gradient overflow, gradient underflow and computational instability.

[0004] Therefore, how to avoid gradient underflow and computational instability has become the current technical problem to be solved.SUMMARY

[0005] Embodiments of the present application provide a method for calculating weights and a related device.

[0006] According to a first aspect, an embodiment of this application provides a method for calculating weights, where the method includes:

[0007] obtaining an input gradient of a current layer; i

[0008] calculating, based on a result of multiplication of the input gradient, a scaling factor, and input information of the current layer, a first weight gradient of the current layer; and

[0009] updating, based on the first weight gradient of the current layer, weights of the current layer.

[0010] According to the method above, by multiplying the input gradient by the scaling factor, it is possible to prevent gradient overflow in the calculated weight gradient, thereby improving the stability of calculation.

[0011] In a possible design, the method further includes: updating, based on the first weight gradient of the current layer, the scaling factor.

[0012] In a possible design, the method further includes: normalizing, if a norm of the first weight gradient is higher than a threshold, the first weight gradient to a second weight gradient.

[0013] According to the method above, by normalizing the calculated weight gradient, it is possible to realize spike prevention.

[0014] In a possible design, the updating, based on the first weight gradient of the current layer, the weights of the cunent layer includes: updating, based on the second weight gradient of the current layer, the weights of the current layer.

[0015] In a possible design, the method further includes: updating, based on a result of division of the first weight gradient and the scaling factor, the first weight gradient to a third weight gradient of the current layer.

[0016] According to the method above, after calculating the weight gradient, the weight gradient can be divided by the same scaling factor, and this is done locally to ensure that the next layer calculation is not affected by the current layer scaling.

[0017] In a possible design, the updating, based on the first weight gradient of the current layer, the weights of the current layer includes: updating, based on the third weight gradient of the current layer, the weights of the current layer.

[0018] In a possible design, the method further includes: normalizing, if a norm of the first weight gradient is higher than a threshold, the third weight gradient to a fourth weight gradient.

[0019] In a possible design, the updating, based on the first weight gradient of the current layer, the weights of the current layer includes: updating, based on the fourth weight gradient of the current layer, the weights of the current layer.

[0020] According to a second aspect, an embodiment of this application provides a method for calculating an output gradient, where the method includes:

[0021] Obtaining an input gradient of a current layer;

[0022] calculating, based on a result of multiplication of the input gradient, a scaling factor, and weights of the current layer, a first output gradient of the current layer; and

[0023] passing, to a next layer, the first output gradient of the current layer.

[0024] In a possible design, the method further includes: updating, based on a result of division of the first output gradient and the scaling factor, the first output gradient to a second output gradient of the current layer.

[0025] According to a third aspect, an embodiment of this application provides a computing device, and the computing device has function of implementing the method in the first aspect or any possible design of the first aspect, or has the function of implementing the method in the second aspect or any possible design of the second aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or the software includes one or more modules corresponding to the function.

[0026] According to a fourth aspect, an embodiment of this application provides a computer readable storage medium including instructions. When the instructions are run on a computer, the computer is enabled to perform the method in the first aspect or any possible design of the first aspect, or enabled to perform the method in the second aspect or any possible design of the second aspect.

[0027] According to a fifth aspect, an embodiment of this application provides a computing device, including a processor, a memory, and a communications interface. The processor is connected to the memory and the communications interface. The memory is configured to store instructions, the processor is configured to execute the instructions, and the communications interface is configured to communicate with another network element under control of the processor. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible design of the first aspect, or enabled to perform the method in the second aspect or any possible design of the second aspect.

[0028] According to a sixth aspect, an embodiment of this application provides a computing device cluster, including at least one computing device, where the computing device includes aprocessor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method in the first aspect or any possible design of the first aspect, or executes the method in the second aspect or any possible design of the second aspect.

[0029] According to a seventh aspect, a chip system is provided, where the chip system includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that a server on which a chip is disposed performs the method in the first aspect or any possible design of the first aspect, or performs the method in the second aspect or any possible design of the second aspect.

[0030] According to an eighth aspect, a computer program product is provided, where when the computer program product is run on a server, the server is enabled to perform the method in the first aspect or any possible design of the first aspect, or perform the method in the second aspect or any possible design of the second aspect.DESCRIPTION OF DRAWINGS

[0031] FIG. 1 shows an example of an application to a model training process.

[0032] FIG. 2 shows a method for updating weights provided by an embodiment of the present application.

[0033] FIG. 3 shows a method for calculating an output weight provided by an embodiment of the present application.

[0034] FIG. 4 shows a flowchart for calculating a weight gradient and an output gradient.

[0035] FIG. 5 shows a detailed flowchart for calculating a weight gradient and an output gradient.

[0036] FIG. 6 is a schematic block diagram of a computing device 600 according to an embodiment of this present application.

[0037] FIG. 7 is a schematic block diagram of a computing device 700 according to an embodiment of this application.

[0038] FIG. 8 is a schematic block diagram of a computing device 800 according to an embodiment of this application.DESCRIPTION OF EMBODIMENTS

[0039] The following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the accompanying drawings. Apparently, the described embodiments are merely some but not all of the embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0040] In the specification, claims, and accompanying drawings of the present application, the term “include” and any other variants of the term mean to cover the non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes an unlisted step or unit, or optionally further includes another inherent step or unit of the process, the method, the system, the product, or the device.

[0041] This application will present various aspects, embodiments or features in the context of a system including a plurality of devices, components, modules, and the like. It is to be understood and appreciated that various systems may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the figures. In addition, combinations of these schemes can also be used.

[0042] In addition, in the embodiments of the present application, words such as “exemplary” and “for example” are used to represent examples or illustrations. Any embodiment or design described in this application as “exemplary” should not be construed as a preferred or advantageous embodiment over other embodiments or designs. Rather, the use of the word example is intended to present a concept in a concrete way.

[0043] In this application, “relevant” and “corresponding” can sometimes be used interchangeably, and it should be noted that, when the difference is not emphasized, the meanings to be expressed are the same.

[0044] The service scenarios described in the embodiments of the present application are for the purpose of illustrating the technical solutions of the embodiments of the present application more clearly, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. In spite of the evolution of an architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to solving similar technical problems.

[0045] For the convenience of description, the following will explain several concepts involved in the embodiment of the application.

[0046] A reduced precision training is a process of training a model in data formats using lower bits compared to FP32. For example, in this application embodiment, two training modes of FP16 and BF16 are considered.

[0047] Gradient scaling is an act of modifying gradients in a model training process.

[0048] Loss scaling is a gradient scaling.

[0049] Tensor scaling is a gradient scaling.

[0050] Gradient normalization is an act of modifying the magnitude of a gradient.

[0051] Loss spike indicates a sudden sharp increase in loss magnitude.

[0052] Gradient explosion indicates a sudden sharp increase in gradient magnitude.

[0053] Model stabilization refers to a process of removing loss spikes and gradient explosions.

[0054] Training stabilization refers to a model stabilization.

[0055] Automatic loss scaler is a system for automatic adjustment of scaling value.

[0056] Automatic gradient scaler is used for an automatic loss scaler.

[0057] Many technical and high-performance operations require usage of high precision numerical formats for computation, but in many cases lower precision arithmetic operations may suffice. Another reason for using lower precision operations is lower resource consumption and faster computation.

[0058] A model consists of a series of layers, each of them possibly having a set of weights, that can be modified during a training process. As the model training process is a time and resource consuming process, it can greatly benefit from reduced precision. In FP16 and BF16 computations and memory consumption can be up to two times lower than those in FP32.However, reduced precision raises problems of gradient overflow, gradient underflow and computational instability.

[0059] Gradient overflow and gradient underflow occur when values in the computation exceed the format defined boundaries. During gradient overflow the value becomes greater than the numerical limits of the format and can no longer be distinguished from infinite. This is especially prominent in FP 16 format, as numerical limits are lower than those in FP32. Gradient underflow occurs when the value can no longer be distinguished from zero. This can lead to large discrepancies from FP32 computation.

[0060] Computational instability happens due to different dynamic ranges having different numerical errors associated with them. This imprecision in computation can lead to discrepancies and model training divergence.

[0061] In the prior art, in Loss-scaling for deep neural network training with reduced precision, a loss scaling approach is used. At first, they scale loss by value S that is computed to shift loss into more precise dynamic range. Then the gradients are calculated and unsealed. The obtained values are converted to FP32 to make loss update more precise.

[0062] In Loss-scaling for deep neural network training with reduced precision, only global loss scaling approach is used. This can lead to negative impact of single layer training process on the others. Another disadvantage is usage of mixed precision, which does not allow to get the full benefits of reduced precision in model training process.

[0063] In the prior art, in scaled learning for training DNNs, the main idea is to assign a separate learning rate to each layer and update the learning rate based on the noise-to-signal ratio. Weights are updated using separate learning rates.

[0064] In Scaled learning for training DNNs, usage of separate learning rates does not provide any protection from loss spikes. It also fails to protect the model from overflows and underflows.

[0065] In the prior art, in a method and apparatus for training neural networks, each layer is assigned a layer-wise loss scaling factor that changes during training.

[0066] In the method and apparatus for training the neural networks, the separate loss scaling factor is beneficial for overflow and underflow problems, but does not protect from loss spikes.

[0067] This approach aims to solve the following problems: gradient overflow, gradient underflow, computational instability and loss spiking.

[0068] Gradient overflow manifests in a low precision model training process when the compute values exceed the numerical range of the model. It can lead to incorrect result of a series of computations, if it occurs during the training process.

[0069] Gradient underflow manifests in a low precision model training process when the compute values exceed the threshold that separates it from zero. It can lead to incorrect result of a series of computations, if it occurs during the training process.

[0070] Computational instability occurs in the model when the values goes beyond the range of numerical stability during computation. This problem usually occurs in BF16 format.

[0071] Loss spiking is an event of increase in loss magnitude in a single training step. This can lead to gradient overflow, weight corruption and model divergence.

[0072] In the methods provided in this application, separate adaptive gradient scaling based on local gradient statistics allows to independently shift each gradient of each layer into the range of high calculation accuracy. It collects local gradient statistics from each weight gradient tensor and updates input gradient before weight gradient computation. Each statistic is collected separately and independently.

[0073] In the methods provided in this application, local weight gradient normalization allows independent normalization of the gradient amplitude of each layer separately. It computes LI norm separately for each layer. If the norm is higher than a threshold, then normalize the gradient. This allows to decrease the impact of gradients from different layers on each other.

[0074] This solution is applied to each DNN layer containing weights during model initialization. FIG. 1 shows an example of such application to the model training process. Model forward pass is computed without any changes. During backward pass, before the gradient is computed, a separate scaling mechanism is applied to a gradient tensor. The scaling mechanism consists of a local scaling factor, an update mechanism and a normalization mechanism.

[0075] FIG. 2 shows a method for updating weights provided by an embodiment of the present application. The method specifically includes the following steps.

[0076] Step 210: Obtain an input weight.

[0077] For example, an output gradient of the previous DNN layer can be used as an input gradient of the current DNN layer.

[0078] Step 220: Calculate a weight gradient of a current DNN layer based on the input gradient, a scaling factor, and input information of the current DNN layer.

[0079] For example, the weight gradient of the current DNN layer can be obtained by multiplying the input gradient of the current DNN layer by a scaling factor, and then multiplying it by the input information of the current DNN layer.

[0080] It should be understood that the scaling factor can also be referred to as a local scaling factor. The local scaling factor is separate and independent for each layer of the model, to which the mechanism is applied. To update the scaling factor based on qa> / 3, where a, ft are user-defined parameters, qais a quantile of the distribution of gradient modules. The default value of a is set to 0,9999999. The default value of / ? is set to 8192. The scale is decreased if the quantile exceeds the threshold, otherwise it is increased.

[0081] Optionally, in some embodiments, after calculating the weight gradient, the weight gradient can be divided by the same scaling factor. This is done locally to ensure that the next layer calculation is not affected by the current layer scaling.

[0082] Step 230: Update weights of the current DNN layer based on the weight gradient.

[0083] For example, after obtaining the weight gradient, it is possible to update the weights of the current DNN layer in this step based on the weight gradient.

[0084] In one example, if the current step is the last step of the current DNN layer, then the training ends.

[0085] In another example, if the current step is not the last step of the current DNN layer, the gradient for the next step is calculated in the current DNN layer using the method shown in FIG. 2, and the weights for the next step are updated based on the weight gradient calculated during the next step.

[0086] Optionally, in some embodiments, the calculated weight gradient can also be normalized and the weights are updated based on the normalized weight gradient.

[0087] By analyzing the norms of the gradients during model training, localized gradient normalization can be detected by observing a constant norm in different layers.

[0088] Normalization mechanism is computing the LI norm of the weight gradient andcomparing it to a user-defined threshold y. If the norm of the weight gradient exceeds the threshold it is normalized to be equal to that threshold.

[0089] For example, the default value of y is 1.

[0090] Local weight gradient normalization allows independent normalization of the gradient amplitude of each layer separately. It computes LI norm separately for each layer. If the norm of the weight gradient is higher than the threshold then normalize the weight gradient. This allows to decrease the impact of gradients from different layers on each other.

[0091] FIG. 3 shows a method for calculating an output weight provided by an embodiment of the present application. The method specifically includes the following steps.

[0092] Step 310: Obtain an input weight.

[0093] For example, an output gradient of the previous DNN layer can be used as an input gradient of the current DNN layer.

[0094] Step 320: Calculate an output gradient of a current DNN layer based on an input gradient, a scaling factor, and weights of the current DNN layer.

[0095] For example, the output gradient of the current DNN layer can be obtained by multiplying the input gradient of the current DNN layer by a scaling factor, and then multiplying the result by the weights of the current DNN layer.

[0096] Optionally, in some embodiments, after calculating the output gradient, the output gradient can be divided by the same scaling factor. This is done locally to ensure that the next layer calculation is not affected by the current layer scaling.

[0097] Step 330: Pass the output gradient calculated by the current DNN layer to a next DNN layer.

[0098] For example, pass the output gradient calculated by the current DNN layer to the next DNN layer.

[0099] For example, the output gradient of the cunent DNN layer can be used as the input gradient of the next DNN layer.

[0100] In one example, if the current DNN layer is the last DNN layer, then the training ends.

[0101] In another example, if the current DNN layer is not the last DNN layer, then the training continues.

[0102] The solution can be used in training DNNs during a training process and can be implemented into existing pipelines during a model creation step and later used during the whole training process. The local gradient scaling allows to solve the overflow and underflow problem, while local normalization can prevent loss spikes during the model training process.

[0103] FIG. 4 shows a flowchart for calculating a weight gradient and an output gradient.

[0104] FIG. 5 shows a detailed flowchart of the mechanisms.

[0105] The present embodiment provides a separate adaptive gradient scaling module, and the separate adaptive gradient scaling module can be used in any model without any limitations on layer type and architecture.

[0106] The input data of the separate adaptive gradient scaling module consists of input gradient.

[0107] The separate adaptive gradient scaling module contains the following parts:

[0108] Scaling submodule, where the scaling submodule is used to update the input gradient.

[0109] Statistic calculation submodule, where the statistic calculation submodule is used to calculate the statistic based on the computed weight tensor.

[0110] Unsealing submodule, where the unsealing submodule is used to update the weight and output gradient.

[0111] The parameters of the separate adaptive gradient scaling module are as following:

[0112] a - the quantile to compute. The parameter must be in range [0, 1]. The default value is 0.9999999.

[0113] p - the quantile threshold. The default value is 8192.

[0114] finer ease ~ scale increase factor. When increased the scale is multiplied by this factor.

[0115] fdecrease- scale decrease factor. When decreased the scale is multiplied by this factor.

[0116] The statistic computation is based on quantile analysis of the weight gradient distribution. This statistic can be estimated by several ways, including counting number of tensor elements exceeding the threshold.

[0117] The present embodiment provides a local weight gradient normalization module, and the local weight gradient normalization module is used to normalize the weight gradient i iduring the training process.

[0118] The local weight gradient normalization module contains: normalization statistic collection submodule and normalization submodule.

[0119] The parameters of the local weight gradient normalization module are as following:

[0120] y - the normalization threshold. Default value is 1.

[0121] The normalization statistic is analyzed by collecting LI norm of weight gradient. Other method includes collecting other norms of the tensor and comparing them to the target threshold.

[0122] The normalization submodule changes the norm of the gradient by dividing it by the current norm value and multiplying it by a threshold.

[0123] The main point of the present embodiment is preventing gradient overflow, underflow and spike. This approach is not limited to DNNs and can be used in other methods that have adjustable parameters.

[0124] The same goal can possibly be reached by detecting computational instability and dynamically switching to higher precision.

[0125] FIG. 6 is a schematic block diagram of a computing device 600 according to an embodiment of this application. As shown in FIG. 6, the computing device 600 includes: an obtaining unit 601, a calculating unit 602 and an updating unit 603.

[0126] The obtaining unit 601 is configured to obtain an input gradient of a current layer.

[0127] The calculating unit 602 is configured to calculate, based on a result of multiplication of the input gradient, a scaling factor, and input information of the current layer, a first weight gradient of the current layer.

[0128] The updating unit 603 is configured to update, based on the first weight gradient of the current layer, weights of the current layer.

[0129] In some embodiments, the updating unit 603 is further configured to update, based on the first weight gradient of the current layer, the scaling factor.

[0130] In some embodiments, the computing device 600 further includes: a normalizing unit, configured to normalize, if a norm of the first weight gradient is higher than a threshold, the first weight gradient to a second weight gradient.

[0131] In some embodiments, the updating unit 603 is specifically configured to update,based on the second weight gradient of the current layer, the weights of the current layer.

[0132] In some embodiments, the updating unit 603 is specifically configured to update, based on a result of division of the first weight gradient and the scaling factor, the first weight gradient to a third weight gradient of the current layer.

[0133] In some embodiments, the updating unit 603 is specifically configured to update, based on the third weight gradient of the current layer, the weights of the current layer.

[0134] In some embodiments, the normalizing unit is further configured to normalize, if a norm of the first weight gradient is higher than a threshold, the third weight gradient to a fourth weight gradient.

[0135] In some embodiments, the updating unit 603 is specifically configured to update, based on the fourth weight gradient of the current layer, the weights of the current layer.

[0136] FIG. 7 is a schematic block diagram of a computing device 700 according to an embodiment of this application. As shown in FIG. 7, the computing device 700 includes: an obtaining unit 701, a calculating unit 702 and a passing unit 703.

[0137] The obtaining unit 701 is configured to obtain an input gradient of a current layer.

[0138] The calculating unit 702 is configured to calculate, based on a result of multiplication of the input gradient, a scaling factor, and weights of the current layer, a first output gradient of the current layer.

[0139] The passing unit 703 is configured to pass the first output gradient of the current layer.

[0140] In some embodiments, the computing device 700 further includes: an updating unit, configured to update, based on a result of division of the first output gradient and the scaling factor, the first output gradient to a second output gradient of the current layer.

[0141] As shown in FIG.8, a computing device 800 may include a transceiver 801, a processor 802, and a memory 803. The memory 803 may be configured to store code, instructions, and the like executed by the processor 802.

[0142] It should be understood that the processor 802 may be an integrated circuit chip and has a signal processing capability. In an implementation process, steps of the foregoing method embodiments may be completed by using a hardware integrated logic circuit in the processor, or by using instructions in a form of software. The processor may be a general purpose processor,a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a system on chip (SoC) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor may implement or perform the methods, the steps, and the logical block diagrams that are disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the methods disclosed with reference to the embodiments of the present application may be directly performed and completed by a hardware decoding processor, or may be performed and completed by using a combination of hardware in the decoding processor and a software module. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor reads information in the memory and completes the steps of the foregoing methods in combination with hardware in the processor.

[0143] It may be understood that the memory 803 in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), or a flash memory. The volatile memory may be a random access memory (Random Access Memory, RAM) and is used as an external cache. By way of example rather than limitation, many forms of RAMs may be used, and are, for example, a static random access memory (Static RAM, SRAM), a dynamic random access memory (Dynamic RAM, DRAM), a synchronous dynamic random access memory (Synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), an enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), a synchronous link dynamic random access memory (Synchronous link DRAM, SLDRAM), and a direct rambus random access memory (Direct Rambus RAM, DR RAM).

[0144] It should be noted that the memory in the systems and the methods described in thisspecification includes but is not limited to these memories and a memory of any other appropriate type.

[0145] An embodiment of this application further provides a system chip, where the system chip includes an input / output interface, at least one processor, at least one memory, and a bus. The at least one memory is configured to store instructions, and the at least one processor is configured to invoke the instructions of the at least one memory to perform operations in the methods in the foregoing embodiments.

[0146] An embodiment of this application further provides a computer storage medium, where the computer storage medium may store a program instruction for performing any of the foregoing methods.

[0147] Optionally, the storage medium may be specifically the memory 803.

[0148] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0149] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may beimplemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0151] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0152] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0153] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disc.

[0154] The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A method for calculating weights, comprising: obtaining an input gradient of a current layer; calculating, based on a result of multiplication of the input gradient, a scaling factor, and input information of the current layer, a first weight gradient of the current layer; and updating, based on the first weight gradient of the current layer, weights of the current layer.

2. The method according to claim 1, wherein the method further comprises: updating, based on the first weight gradient of the current layer, the scaling factor.

3. The method according to claim 1 or 2, wherein the method further comprises: normalizing, if a norm of the first weight gradient is higher than a threshold, the first weight gradient to a second weight gradient.

4. The method according to claim 3, wherein the updating, based on the first weight gradient of the current layer, the weights of the current layer comprises: updating, based on the second weight gradient of the current layer, the weights of the current layer.

5. The method according to any one of claims 1-4, wherein the method further comprises: updating, based on a result of division of the first weight gradient and the scaling factor, the first weight gradient to a third weight gradient of the current layer.

6. The method according to claim 5, wherein the updating, based on the first weight gradient of the current layer, the weights of the current layer comprises: updating, based on the third weight gradient of the current layer, the weights of the current layer.

7. The method according to claim 5, wherein the method further comprises: normalizing, if a norm of the first weight gradient is higher than a threshold, the third weight gradient to a fourth weight gradient.

8. The method according to claim 7, wherein the updating, based on the first weightgradient of the current layer, the weights of the current layer comprises: updating, based on the fourth weight gradient of the current layer, the weights of the current layer.

9. A method for calculating an output gradient, comprising: obtaining an input gradient of a current layer; calculating, based on a result of multiplication of the input gradient, a scaling factor, and weights of the current layer, a first output gradient of the current layer; and passing, to a next layer, the first output gradient of the current layer.

10. The method according to claim 9, wherein the method further comprises: updating, based on a result of division of the first output gradient and the scaling factor, the first output gradient to a second output gradient of the current layer.

11. A computing device, comprising: an obtaining unit, configured to obtain an input gradient of a current layer; a calculating unit, configured to calculate, based on a result of multiplication of the input gradient, a scaling factor, and input information of the current layer, a first weight gradient of the current layer; and an updating unit, configured to update, based on the first weight gradient of the current layer, weights of the current layer.

12. The computing device according to claim 11, wherein the updating unit is further configured to update, based on the first weight gradient of the current layer, the scaling factor.

13. The computing device according to claim 11 or 12, wherein the computing device further comprises: a normalizing unit configured to normalize, if a norm of the first weight gradient is higher than a threshold, the first weight gradient to a second weight gradient.

14. The computing device according to claim 13, wherein the updating unit is specifically configured to update, based on the second weight gradient of the current layer, the weights of the current layer.

15. The computing device according to any one of claims 11-14, wherein the updating unit is further configured to update, based on a result of division of the firstweight gradient and the scaling factor, the first weight gradient to a third weight gradient of the current layer.

16. The computing device according to claim 15, wherein the updating unit is specifically configured to update, based on the third weight gradient of the current layer, the weights of the current layer.

17. The computing device according to claim 15, wherein the normalizing unit is further configured to normalize, if a norm of the first weight gradient is higher than a threshold, the third weight gradient to a fourth weight gradient.

18. The computing device according to claim 17, wherein the updating unit is specifically configured to update, based on the fourth weight gradient of the current layer, the weights of the current layer.

19. A computing device, comprising: an obtaining unit, configured to obtain an input gradient of a current layer; a calculating unit, configured to calculate, based on a result of multiplication of the input gradient, a scaling factor, and weights of the current layer, a first output gradient of the current layer; and a passing unit, configured to pass the first output gradient of the current layer.

20. The computing device according to claim 19, wherein the computing device further comprises: an updating unit configured to update, based on a result of division of the first output gradient and the scaling factor, the first output gradient to a second output gradient of the current layer.

21. A computer readable storage medium storing instructions that, when run on a server, enable the server to perform the method according to any one of claims 1 to 10.

22. A computing device cluster, comprising at least one computing device, wherein the computing device comprises a processor and a memory coupled with the processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device executes the method according to any one of claims 1 to 10.

23. A computer program product, wherein when the computer program product is run ona server, the server is enabled to perform the method according to any one of claims 1 to 10.

Images