Gradient descent for near-zero learning rates

Abstract

A system (100) and method (200) for iteratively updating parameters (132) according to a gradient descent algorithm (130). In a given nth iteration of the method, one or more processors can determine a gradient value of a gradient vector of the parameters in a first dimension (210), determine a product value based at least in part on (i) a sum of (ii) a product of the determined gradient value and a learning rate of the gradient descent algorithm (220), determine an updated parameter value according to a function that includes the product value (230), and update the parameters to equal the updated parameter value (240).

Description

[0001] Cross-references to related applications

[0002] This application is a continuation of U.S. Patent Application No. 17 / 069,227, filed October 13, 2020, which claims the benefit of U.S. Provisional Patent Application No. 63 / 077,868, filed September 14, 2020, the disclosure of which is incorporated herein by reference. Background Technology

[0003] Gradient descent is used to update the parameters of a machine learning algorithm, thereby improving the parameters over several iterations until they converge to a value that improves the accuracy of the machine learning algorithm. In each iteration, the parameters can be adjusted by a certain amount depending on a cost function that indicates the degree of inaccuracy of the parameters. The adjustment can be further scaled according to the learning rate, where a small learning rate leads to a small adjustment, and a high learning rate leads to a larger adjustment.

[0004] Typically, the learning rate starts with a small or even zero value to avoid large initial adjustments to the parameters before a path to the optimal solution can be found. However, some gradient descent algorithms cannot support small or zero learning rate values ​​because inputting such values ​​into the algorithm can introduce errors, such as division-by-zero errors. One reason why very small learning rate values ​​may become zero and / or then cause division-by-zero errors when used as a denominator or in the calculation of denominators is that the machine representation of rational numbers in floating-point form and computer arithmetic, as well as floating-point arithmetic, have some inaccuracies that cause very small values ​​to drop to zero. Similarly, when rational numbers close to zero are represented as floating-point values ​​and used for division or multiplication, the result may be a floating-point representation of zero. Summary of the Invention

[0005] This technique broadly relates to a method and system for iteratively updating parameters in a machine learning algorithm based on a gradient descent algorithm that supports zero and near-zero learning rate values. The gradient descent algorithm iteratively updates the parameters using a product value that is at least partially a product of the accumulated linear gradient value and the learning rate. Using the product value, instead of the accumulated linear gradient value, avoids the learning rate appearing in the denominator of the function used to determine the updated parameters, thus avoiding division-by-zero errors and similar errors associated with zero and near-zero learning rate values.

[0006] One aspect of this disclosure relates to a method for iteratively updating parameters according to a gradient descent algorithm. The method may include: in the nth iteration of the gradient descent algorithm, one or more processors determining a gradient value of the gradient vector of the parameters in a first dimension; the one or more processors determining a product value based at least in part on (i) the sum of the product of the product determined in the (n-1)th iteration and (ii) the product of the determined gradient value and the learning rate of the gradient descent algorithm; the one or more processors determining an updated parameter value according to a function including the product value; and the one or more processors updating the parameters to be equal to the updated parameter value.

[0007] In some examples, the product value may be based at least in part on (i) the product of the accumulated linear gradient value and the learning rate and (ii) the parameter value determined in the (n-1)th iteration.

[0008] In some examples, the product value can be equal to the difference between (i) the product of the accumulated linear gradient value and the learning rate and (ii) the parameter value determined in the (n-1)th iteration scaled according to the scaling factor.

[0009] In some examples, the method may include: determining a cumulative squared gradient value by the one or more processors based on the square of the gradient value and the cumulative squared gradient value of the (n-1)th iteration. The scaling value may be based on each of the cumulative squared gradient value and the cumulative squared gradient value of the (n-1)th iteration.

[0010] In some examples, the scaling value can be equal to:

[0011]

[0012] Where n i It is the cumulative squared gradient value, n i-1 It is the cumulative squared gradient value of the (n-1)th iteration, and lr_power is a predetermined parameter.

[0013] In some examples, the function includes the product value: the parameter value can be set to zero in response to the ratio between the absolute value of the product value and the learning rate exceeding a predetermined fixed value; and the parameter value can be set to a non-zero value in response to the ratio between the absolute value of the product value and the learning rate being less than or equal to a predetermined fixed value.

[0014] In some examples, the method may include determining non-zero values ​​based on a parametric function. The denominator of the parametric function may include each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value, such that the parametric function returns a division-by-zero error only if each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value is equal to zero.

[0015] In some examples, the parameter function can be:

[0016]

[0017] Where λ2 and β are the first and second predetermined fixed values, λ1 is the third predetermined fixed value, η is the learning rate, and p i It is the product value, and n i It is the cumulative squared gradient value.

[0018] In some examples, the learning rate can be predefined to start from zero and increase over time.

[0019] In some examples, the learning rate can be predefined to start near zero and increase over time.

[0020] In some examples, the gradient vector of the parameters can include multiple dimensions. The method can be executed independently for each corresponding dimension of the gradient vector.

[0021] In some examples, the parameters can be those of a neural network.

[0022] In some examples, the gradient descent algorithm can be any of the following: stochastic gradient descent, Follow Regularized Leader (FTRL) learning algorithm, Adagrad, Adam, or RMSProp.

[0023] In some examples, the learning rate can be predefined as changing over time, and the number of cores included in the one or more processors assigned to execute the method can remain constant as the learning rate changes over time.

[0024] Another aspect of this disclosure relates to a system comprising: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the stored instructions to iteratively update parameters according to a gradient descent algorithm and to perform any of the methods disclosed herein. In some examples, the one or more processors may include a constant number of cores assigned to iteratively update parameters according to a gradient descent algorithm. Attached Figure Description

[0025] Figure 1 This is a block diagram illustrating an example system according to aspects of this disclosure.

[0026] Figure 2 It is a flowchart illustrating an example routine according to an aspect of this disclosure.

[0027] Figure 3 and 4 It's a diagram. Figure 2 The flowchart of the example subroutine of the routine.

[0028] Figure 5 This is a flowchart illustrating the relationship between the gradient descent algorithm processing and the learning rate according to aspects of this disclosure. Detailed Implementation

[0029] Figure 1 This is a block diagram illustrating an example system 100 including a computing environment for processing queries. System 100 may include one or more processors 110 and memory 120 communicating with said one or more processors. Processor 110 may include well-known processors or other lesser-known processor types. Alternatively, processor 110 can be a dedicated controller, such as an ASIC, graphics processing unit (GPU), or tensor processing unit (TPU). Memory 120 may include a non-transitory computer-readable medium capable of storing information accessible to processor 110, such as a hard disk drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. System 100 may implement any of many architectures and technologies, including but not limited to direct-attached storage (DAS), network-attached storage (NAS), storage area network (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), hybrid architecture networks, etc. In addition to storage devices, data centers may also include many other devices, such as cables, routers, etc.

[0030] Memory 120 is capable of storing information accessible to the one or more processors 110, including data received at or generated by the processor 110 and instructions executable by the one or more processors 110. Data can be retrieved, stored, or modified by the processor 110 according to the instructions. For example, while systems and methods are not limited to a specific data structure, data can be stored as a structure with multiple different fields and records or documents or caches in computer registers, data repositories, etc. Data can also be formatted in computer-readable formats—such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memories, including other network locations, or information used by functions to compute relevant data. Instructions can include various algorithms for instructing the processor 110 to perform operations. Instructions can include directly executable instruction sets, such as machine code stored in object code format for direct processing by the processor 110, or indirectly executable instruction sets, such as scripts that are interpreted or compiled in advance as needed, scripts or sets of independent source code modules. In this regard, the terms “instruction”, “step”, “routine” and “procedure” are used interchangeably in this document.

[0031] The instructions may include a gradient descent algorithm 130, programmed to instruct the one or more processors 110 to receive training data 140, process the training data 140 according to one or more specified objectives of the gradient descent algorithm 130, and incrementally adjust the gradient descent algorithm 130 based on the success or failure of the processing results to better achieve the specified objectives when processing new data in the future. The gradient descent algorithm 130 can be used in supervised learning algorithms, where the success or failure of the processing results can be determined based on known characteristics of the training data 140. Alternatively, the gradient descent algorithm 130 can be used in fully or partially unsupervised learning algorithms, where the success or failure of the processing results can be derived at least partially from the processing results themselves, such as through clustering or association algorithms. In the case of fully or partially unsupervised learning algorithms, at least some of the training data 140 may have unknown characteristics, such as new data to be processed by the gradient descent algorithm in the future.

[0032] The gradient descent algorithm 130 can operate on one or more parameters 132 of the system's machine learning structure. The machine learning structure can be a machine learning algorithm or other algorithms, and for this machine learning structure, the output of the structure is a function of the input and one or more variables or parameters. The machine learning structure can be modified by adjusting the parameters 132 to improve the accuracy of the output. By way of example, in a neural network, parameters can include weights or embeddings between the nodes and layers of the network, dense layers, or any combination thereof. The machine learning structure can also operate on one or more hyperparameters, such as the number of layers or the learning rate 134. The learning rate 134 can specify the magnitude of the adjustment of the machine learning structure for a given iteration of the gradient descent algorithm. It should be recognized that different machine learning structures may include different parameters and hyperparameters.

[0033] In some examples, gradient descent algorithm 130 can operate on one or more hyperparameters of a machine learning architecture. For instance, the learning rate 134 can be adjusted based on the output of gradient descent algorithm 130. For example, if gradient descent algorithm 130 causes a large change in parameter 132, it can be determined that the learning rate 134 should be increased to slow the rate of change of parameter 132 in each iteration of gradient descent algorithm 130. Conversely, if gradient descent algorithm 130 causes a small change in parameter 132, it can be determined that the learning rate 134 should be maintained or decreased. Alternatively, hyperparameters such as the learning rate can be predetermined. For example, the learning rate 134 can change over time according to a predetermined function such as a step function, or it can remain constant. In the case where the learning rate is a step function, the learning rate can start from zero or near zero and increase over time. Values ​​“near zero” include values ​​where the function used by gradient descent algorithm 130 to determine the updated parameters, divided by the learning rate, returns an error or inaccurate result.

[0034] although Figure 1 While each of processor 110 and memory 120 is illustrated as a single block functionally, processor 110 and memory 120 may actually include multiple processors and memories that may or may not be stored in the same physical housing. For example, some of the data and instructions stored in memory may be stored on a removable CD-ROM, while other data and instructions may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from processor 110 but still accessible to processor 110.

[0035] Figure 2 This is a flowchart illustrating example routine 200 used to update the parameter "w" of the gradient descent algorithm. Example routine 200 illustrates the steps of a single iteration "i" of the gradient descent algorithm, whereby the previous iteration of the gradient descent algorithm is defined as "i-1". The parameter determined in the (i-1)th iteration is referred to herein as w. i-1 The parameter determined in the i-th iteration is called w. i This routine can be executed by one or more processors, such as Figure 1 The processor 110 shown.

[0036] In block 210, the one or more processors determine the gradient value g of parameter w. i Gradient value g i The cost function, which characterizes the success or failure of the processing outcome, can be used with respect to the most recently determined parameter value w. i-1 This can be calculated, for example, by comparing the actual parameters output by the gradient descent algorithm with the predicted output of the parameters.

[0037] The gradient value g of the i-th iteration of the gradient descent algorithm i It can be used to calculate the cumulative linear gradient value m i and cumulative squared gradient value n i One or both. The cumulative linear gradient value m i This can be derived by summing the gradients calculated in each iteration of the gradient descent algorithm. Similarly, the cumulative squared gradient value n i It can be derived by summing the squares of the gradients calculated in each iteration of the gradient descent algorithm.

[0038] In block 220, the one or more processors determine the product value p. i Product value p i It can be based on algorithm p i-1 The product value in the previous iteration and g i The sum of the products of the gradient descent algorithm's learning rate η. For example, the product value p iIt can be equal to the product of the accumulated linear gradient value and the learning rate. For a further example, p i This can be achieved by further subtracting the scaling factor p. i-1 To export. Used for scaling p i-1 The scaling value can be based on the cumulative squared gradient value n i The scaling value can be based on the previous iteration n. i-1 The cumulative squared gradient value.

[0039] p i Example calculations in Figure 3 The subroutine 300 is shown. At box 310, the accumulated linear gradient value m is... i Determined. Value m i It can be m i-1 and g i The sum of . At box 320, the cumulative squared gradient value n i Determined. Value n i It can be n i-1 and g i 2 The sum. At box 330, the scaling value s i Calculated using the following scaling function:

[0040] (1)s i =n i -lr_power -n i-1 -1r_power

[0041] Here, "lr_power" is a predefined parameter. At box 340, p i It is determined to be equal to the following product-valued function:

[0042] (2)p i =(η*m i )-(s i *w i-1 ).

[0043] return Figure 2 At box 230, the updated parameter value w i According to including the product value p i The function is used to determine this.

[0044] w i Example calculations in Figure 4 The subroutine 400 is shown. At box 410, the absolute value of the product, |p, is determined. i Whether the ratio between | and the learning rate η exceeds a predetermined fixed value λ1. This determines the evaluation of the cumulative linear gradient value m. i Whether it is large or small, the cumulative squared gradient value n since the previous iteration is also considered. iThe more recent change in the ratio is used to predetermine a fixed value λ1 as a threshold. If the ratio is greater than the threshold, the operation can continue at box 420, where the updated parameter value is determined to be equal to zero. Otherwise, if the ratio is less than or equal to the threshold, the operation can continue at box 430, where the updated parameter value is determined according to the following parameter value function:

[0045] (3)

[0046] Both λ² and β are predetermined fixed values, and “sgn” is the sign function. The denominator of the parameter-valued function (3) includes the cumulative squared gradient value n. i And each of two separate predetermined fixed values ​​λ2 and β. Furthermore, only when the cumulative squared gradient value n... i The parameter function will only return a division-by-zero error when both the predetermined fixed values ​​λ2 and β are equal to zero. i It is usually set to an initial non-zero value, and in addition, for n i Since all of λ, 2, and β are equal to zero, it is not an efficient configuration. Therefore, the parameter-valued function (3) avoids errors caused by division by zero.

[0047] Return again Figure 2 At box 240, the parameter w of the machine learning structure is updated to be equal to the determined parameter value w of the i-th iteration. i Updating the parameters w can cause the machine learning structure to output different results for a given input, thus the cost function of the machine learning structure is iteratively reduced or optimized over several iterations of the gradient descent algorithm. In this way, the machine learning structure can converge to an optimal or relatively optimal set of parameters that minimizes or relatively reduces the cost function, resulting in a more accurate determination of future data inputs by the machine learning structure.

[0048] The example routine above demonstrates a single iteration of the machine learning process, whereby a single parameter w of the machine learning structure is incrementally moved closer to the optimal solution. It should be understood that this routine can be executed iteratively, thus, with each iteration, based on the training data fed into the system between iterations, the parameter continues to move closer to the optimal solution. The amount of time between iterations is specified by the learning rate, which can be fixed or, according to a predetermined schedule, vary over time as a function of the machine learning process. Additionally, it should be understood that the system may include several parameters, and the routine can be executed independently for each parameter, whereby each parameter represents a different dimension of the gradient vector. With each iteration, based on the training data provided to the system between iterations, the parameter set can be moved closer to the optimal solution.

[0049] Figure 2The advantage of example routine 200 is that it avoids the error of dividing zero or near-zero learning rate values ​​by zero. This is because the denominator of the parameter value function is not multiplied by the learning rate.

[0050] When an alternative parameter value function inversely proportional to the learning rate is considered, the benefit of avoiding the division-by-zero error of zero or near-zero learning rate values ​​becomes apparent. For learning rates at or near zero, the alternative function will return a division-by-zero error. To avoid the effects of near-zero learning rates in such parameter value functions, the learning rate must start with a value greater than near zero. However, starting the machine learning process with a relatively large learning rate can cause the machine learning algorithm to fail to converge to a good solution, i.e., to the set of parameter values ​​that minimizes the cost function. Therefore, an alternative way to reduce the tuning of the machine learning architecture parameters is needed to ensure that the gradient descent algorithm converges to a good solution. One such way is to slow down the processing of the training data, so that less data is considered between each iteration of the gradient descent algorithm. This can be achieved by limiting the number of processor cores assigned to process the training data using the machine learning architecture. Reducing processing will ensure smaller tuning of the parameters in each iteration. Eventually, the processing speed can be increased after the gradient descent algorithm begins to move towards a good solution, but this will only happen after several iterations of the gradient descent algorithm.

[0051] Figure 5 The illustrations show the difference in training data processing speed between the gradient descent algorithm of this application and a hypothetical alternative gradient descent algorithm (hereinafter referred to as "Alt_GD"), which is affected by a division-by-zero error with a zero or near-zero learning rate. First curve 501 illustrates the processing power of Alt_GD plotted against the learning rate, such as the number of processor cores assigned. Second curve 502 illustrates the processing power of the gradient descent algorithm of this disclosure plotted against the learning rate, such as the number of processor cores assigned.

[0052] The first difference between curves 501 and 502 is that curve 501 starts with a non-zero learning rate, while curve 502 is able to start with a lower learning rate. This means that Alt_GD makes larger changes to the parameters at the start of the operation than the gradient descent algorithm disclosed herein. This introduces the risk of moving too quickly toward a less optimal solution.

[0053] The second difference between curves 501 and 502 is that curve 501 starts with a low processing rate and causes the processing to slope upwards over time, while curve 502 starts and remains at full processing capacity. This means that the early iterations of Alt_GD operate on machine learning structures that have received relatively little training data, while the early iterations of the gradient descent algorithm of this disclosure operate on machine learning structures that have received relatively more training data. In this way, the gradient descent algorithm of this disclosure converges to the solution faster than Alt_GD.

[0054] To illustrate the improved processing of the gradient descent algorithm disclosed herein, Figure 5 The shaded areas represent a comparison of the amount of processing completed after four iterations of the corresponding gradient descent algorithm. For Alt_GD, the total processing performed at time T1 (at the fourth iteration of Alt_GD, indicated by the fourth point along curve 501) is shown as the shaded area below curve 501. For the gradient descent algorithm of this disclosure, the total processing performed at time T1' (at the fourth iteration of Alt_GD, indicated by the fourth point along curve 502) is shown as the shaded area below curve 502. It can be seen that the area below 501 is significantly smaller than the area below curve 502. This demonstrates that curve 502 has a faster overall processing speed. It also shows that the preliminary results for the gradient descent algorithm of this disclosure, such as the results available at times T1 and T1', are more accurate than those for Alt_GD. In other words, the gradient descent algorithm of this disclosure allows accurate preliminary results to be available more quickly.

[0055] Figure 2 The above example routine 200 demonstrates how to update parameters using a Follow Regularized Leader (FTRL) learning algorithm. Therefore, example routine 200 can outperform alternative FTRL algorithms that involve dividing by the learning rate. It should be understood that the same fundamental principles can be applied to other algorithms, such as stochastic gradient descent, Adagrad, Adam, RMSProp, etc. In other words, for other gradient descent algorithms that divide by the learning rate, the principles of this disclosure—such as determining the product value and determining the updated parameter values ​​based on the product value—can be applied to avoid having to divide by the learning rate.

[0056] Although the techniques described herein have been illustrated with reference to specific embodiments, it is to be understood that these embodiments merely illustrate the principles and applications of the technology. Therefore, it is to be understood that numerous modifications can be made to the illustrative embodiments, and other arrangements can be designed without departing from the spirit and scope of the technology as defined by the appended claims.

[0057] Most of the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be used without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be presented illustratively rather than by limiting the subject matter defined by the claims. As an example, the preceding operations need not be performed in the exact order described above. Instead, the steps can be performed in different orders, such as in reverse or simultaneously. Steps can also be omitted unless otherwise stated. Furthermore, the provision of examples described herein and sentences expressed with phrases such as "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, the examples are not intended to illustrate only one embodiment among many possible embodiments. Further, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method for iteratively updating parameters according to a gradient descent algorithm, the method comprising: In the nth iteration: The gradient value of the gradient vector of the parameter in the first dimension is determined by one or more processors; The product value is determined by the one or more processors based at least in part on the sum of: (i) the product value determined in the (n-1)th iteration and (ii) the product of the determined gradient value and the learning rate of the gradient descent algorithm; The updated parameter values ​​are determined by the one or more processors based on a function that includes the product value; as well as The parameters are updated by the one or more processors to be equal to the updated parameter values, wherein the product value is equal to the difference between: (i) the product of the accumulated linear gradient value and the learning rate and (ii) the parameter value determined in the (n-1)th iteration scaled according to the scaling value. The method further includes the one or more processors determining the cumulative squared gradient value based on the square of the gradient value and the cumulative squared gradient value of the (n-1)th iteration, wherein the scaling value is based on each of the cumulative squared gradient value and the cumulative squared gradient value of the (n-1)th iteration.

2. The method according to claim 1, wherein, The scaling value is equal to: Where n i It is the cumulative squared gradient value, n i-1 It is the cumulative squared gradient value of the (n-1)th iteration, and lr_power is a predetermined parameter.

3. The method according to claim 1, wherein, The function including the product value: In response to the ratio between the absolute value of the product and the learning rate exceeding a predetermined fixed value, the parameter value is determined to be equal to zero; as well as In response to the ratio between the absolute value of the product and the learning rate being less than or equal to the predetermined fixed value, the parameter value is determined to be equal to a non-zero value.

4. The method of claim 3, further comprising: The non-zero value is determined according to a parametric function, wherein the denominator of the parametric function includes each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value, and wherein the parametric function returns a division-by-zero error only when each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value is equal to zero.

5. The method according to claim 3, wherein, The parameter function is: Where λ2 and β are the first and second predetermined fixed values, λ1 is the third predetermined fixed value, η is the learning rate, and p i It is the product value, and n i It is the cumulative squared gradient value.

6. The method according to claim 1, wherein, The learning rate is predefined to start from zero and increase over time.

7. The method according to claim 1, wherein, The learning rate is predefined to start near zero and increase over time.

8. The method according to claim 1, wherein, The gradient vector of the parameters includes multiple dimensions, and the method is performed independently for each corresponding dimension of the gradient vector.

9. The method according to claim 1, wherein, The parameters are those of the neural network.

10. The method according to claim 1, wherein, The gradient descent algorithm is any one of the following: stochastic gradient descent, Follow Regularized Leader (FTRL) learning algorithm, Adagrad, Adam, or RMSProp.

11. The method according to any one of claims 1 to 10, wherein, The learning rate is predefined to change over time, and the number of cores included in the one or more processors assigned to perform the method remains constant as the learning rate changes over time.

12. A system comprising: The memory is used to store instructions; as well as One or more processors are coupled to the memory and configured to execute stored instructions to iteratively update parameters according to a gradient descent algorithm, wherein, for the nth iteration of the gradient descent algorithm, the one or more processors are configured to: Determine the gradient value of the gradient vector of the parameter in the first dimension; The product value is determined at least in part based on the sum of: (i) the product value determined in the (n-1)th iteration and (ii) the product of the determined gradient value and the learning rate of the gradient descent algorithm; The updated parameter value is determined based on a function that includes the product value; and The parameter is updated to be equal to the updated parameter value, wherein the product value is equal to the difference between: (i) the product of the accumulated linear gradient value and the learning rate and (ii) the parameter value determined in the (n-1)th iteration scaled according to the scaling value. The one or more processors are configured to determine the cumulative squared gradient value based on the square of the gradient value and the cumulative squared gradient value of the (n-1)th iteration, wherein the scaling value is based on each of the cumulative squared gradient value and the cumulative squared gradient value of the (n-1)th iteration.

13. The system according to claim 12, wherein, The scaling value is equal to: Where n i It is the cumulative squared gradient value, n i-1 It is the cumulative squared gradient value of the (n-1)th iteration, and lr_power is a predetermined parameter.

14. The system according to claim 12, wherein, The function including the product value: In response to the ratio between the absolute value of the product and the learning rate exceeding a predetermined fixed value, the parameter value is determined to be equal to zero; as well as In response to the ratio between the absolute value of the product and the learning rate being less than or equal to the predetermined fixed value, the parameter value is determined to be equal to a non-zero value.

15. The system according to claim 14, wherein, The one or more processors are configured to determine the non-zero value according to a parameter function, wherein the denominator of the parameter function includes each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value, and wherein the parameter function returns a division-by-zero error only when each of the cumulative squared gradient value, the first predetermined fixed value, and the second predetermined fixed value is equal to zero.

16. The system according to claim 14, wherein, The parameter function is: Where λ2 and β are the first and second predetermined fixed values, λ1 is the third predetermined fixed value, η is the learning rate, and p i It is the product value, and n i It is the cumulative squared gradient value.

17. The system according to claim 12, wherein, The learning rate is predefined to start from zero and increase over time.

18. The system according to claim 12, wherein, The learning rate is predefined to start near zero and increase over time.

19. The system according to claim 12, wherein, The gradient vector of the parameters includes multiple dimensions, and the one or more processors are configured to update the parameters of each corresponding dimension of the gradient vector independently of each other.

20. The system according to claim 12, wherein, The parameters are those of the neural network.

21. The system according to claim 12, wherein, The gradient descent algorithm is any one of the following: stochastic gradient descent, Follow Regularized Leader (FTRL) learning algorithm, Adagrad, Adam, or RMSProp.

22. The system according to any one of claims 12 to 21, wherein, The one or more processors include a constant number of cores assigned to iteratively update the parameters according to the gradient descent algorithm.

Images