Hardware-oriented Adam algorithm second-order moment estimation optimization method and system

Abstract

The present invention discloses a hardware-oriented Adam algorithm second-order moment estimation optimization method and system, the method comprising: initialization; repeatedly executing the following steps until reaching a set number of iteration steps or satisfying a convergence condition: taking derivative of the objective function for the parameter to be optimized in this step to obtain the gradient of this step; obtaining the larger and smaller values ​​of the second-order distance square root formula according to the second-order exponential decay factor, the absolute value of the gradient of this step and the square root value of the second-order moment estimation in the previous step; inputting it into the second-order distance square root formula to obtain the second-order moment estimation square root value of this step to calculate the update step of this step, and the steps for determining the second-order distance square root formula are as follows: obtaining an approximate formula for the square root of the sum of squares; obtaining the larger and smaller values ​​for representing the second-order distance square root formula; determining the second-order distance square root formula. The present invention can realize hardware-oriented Adam algorithm second-order moment estimation optimization with low resource occupancy and high throughput, and can avoid the loss of accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and circuit design, and in particular to a hardware-oriented Adam algorithm second-order moment estimation optimization method and system. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] The essence of training a neural network is to repeat the forward and backward propagation of the network and continuously search for suitable parameters. The direction of updating the parameters is the direction that minimizes the loss function, which is the difference between the true value and the objective function value. The neural network calculates the partial derivative of the update parameter based on the loss function. The partial derivative does not directly act on the parameter to be updated, but first obtains the update factor through an optimizer, and then updates the parameter to be updated together with the learning rate.

[0004] Common optimization algorithms include Stochastic Gradient Descent (SGD), Momentum-SGD, Adagrad, Adadelta, and Adaptive Moment Estimation (Adam). The Adam algorithm combines the advantages of the Momentum and Adadelta algorithms, considers both the first-order moment estimation and the second-order moment estimation during the iteration process, and dynamically adjusts the learning rate of each parameter.

[0005] At present, the Adam algorithm has been widely used due to its good performance and fast convergence speed. The first-order moment in the Adam algorithm is equivalent to the expectation of the current gradient. Its function is to maintain inertia, so that the difference between the previous and next updates will not be too large during each update, that is, to ensure that the gradient is smoothly and stably transitioned. The second-order moment in the Adam algorithm is equivalent to the expectation of the square of the current gradient. Its function is environmental perception. For example, when the first-order moment is small and the second-order moment is large, it means that the current gradient is positive and negative, so the first-order moment offsets each other and the second-order moment gradually accumulates. At this time, the update may be in an oscillating state.

[0006] Adam considers both inertial maintenance and environmental perception issues, and is currently one of the optimization algorithms with the fastest convergence speed, best performance, and most widespread use. This is accompanied by an increase in computational complexity, which will result in exponential speed resource consumption when implemented in hardware.

[0007] At present, there are roughly three ways to implement Adam. The first is to use a general-purpose processor, but it has disadvantages such as low computing efficiency and inability to parallelize operations. The second GPU implementation will bring huge power consumption overhead. The last one is to implement it using a field-programmable gate array (FPGA), which has high parallelism and low power consumption and can solve the problems of the above two processors. However, limited resource allocation and loss of precision also become obstacles to hardware acceleration. According to the second-order moment calculation formula v of the Adam algorithm t =β2·v t-1 +(1-β2)·g t 2 And the parameter update formula When calculating the second-order moment estimation of the Adam algorithm, it is necessary to go through square operations, division operations, square root operations, and division operations in sequence. These operations deployed in hardware (FPGA) will consume a lot of hardware resources, especially LUT resources. Just one square root operation will use about 5,000 LUT resources and 675 trigger resources. At the same time, since the core of the Adam algorithm is to find the minimum value of the objective function along the direction of gradient descent, there are always values ​​with a very small order of magnitude passing through the Adam algorithm. If this value is squared again, the order of magnitude will double. When a fixed-point number is used for square operations in hardware, the resulting value is very likely to exceed the representation range of the minimum precise value, and thus be recorded as a value of 0, resulting in a large loss of precision and causing the algorithm to fail to converge.

[0008] Therefore, hardware implementation of the Adam algorithm second-order distance estimation must consider resource usage on the one hand, and avoid loss of accuracy on the other hand. At the same time, data throughput must also be considered. Summary of the invention

[0009] The embodiment of the present invention provides a hardware-oriented Adam algorithm second-order moment estimation optimization method, which is used to solve the problems of excessive resource occupation and easy loss of precision when the Adam algorithm is implemented in FPGA, and realizes hardware-oriented Adam algorithm second-order moment estimation optimization with low resource occupation and high throughput, and can avoid the loss of precision. The method includes:

[0010] Initialize the second-order moment estimate square root value, the second-order exponential decay factor, the initial iteration number and the preset iteration step;

[0011] Repeat the following steps until the set number of iterations is reached or the convergence condition is met:

[0012] Derivative the objective function for the parameters to be optimized in this step to obtain the gradient of this step;

[0013] According to the second-order exponential decay factor, the absolute value of the gradient in this step and the estimated square root value of the second-order moment in the previous step, the larger and smaller values ​​of the second-order distance square root formula are obtained;

[0014] The larger value and the smaller value are input into the second-order distance square root formula to obtain the second-order moment estimated square root value of this step, and the second-order moment estimated square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step;

[0015] The steps for determining the formula for the sum of squares of the second-order distances are as follows:

[0016] The second-order moment estimate of this step in the parameter update formula of the Adam algorithm to be optimized is replaced by the square root of the second-order moment estimate of this step;

[0017] The second-order moment estimation value calculation formula in the Adam algorithm is organized into the second-order moment estimation value calculation formula in the form of the square root of the sum of the squares of the first temporary value and the second temporary value;

[0018] Get the approximate formula for the square root of the sum of squares;

[0019] Obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value;

[0020] Based on the approximate formula for the larger value, smaller value, and square root of the sum of squares, determine the formula for the sum of squares of the second-order distances.

[0021] The embodiment of the present invention also provides a hardware-oriented Adam algorithm second-order moment estimation optimization system, which is used to solve the problems of excessive resource occupation and easy loss of precision when the Adam algorithm is implemented in FPGA, and realizes hardware-oriented Adam algorithm second-order moment estimation optimization with low resource occupation and high throughput, and can avoid the loss of precision. The system includes a control module, a comparison module and a second-order moment estimation module; wherein,

[0022] The control module is used to initialize the second-order moment estimated square root value, the second-order exponential decay factor, the initial iteration step number and the preset iteration step; and repeat the following steps until the set iteration step number is reached or the convergence condition is met:

[0023] Derivative the objective function for the parameters to be optimized in this step to obtain the gradient of this step;

[0024] The second-order exponential decay factor, the gradient of this step and the estimated square root value of the second-order moment of the previous step are input into the comparison module to obtain the larger and smaller values ​​of the second-order moment square root sum formula;

[0025] The larger value and the smaller value are input into the second-order moment estimation module to obtain the second-order moment estimation square root value of this step, the second-order moment estimation module is used to obtain the second-order moment estimation square root value of this step based on the second-order distance square root sum formula, the second-order moment estimation square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step;

[0026] The steps for determining the formula for the sum of squares of the second-order distances are as follows:

[0027] The second-order moment estimate of this step in the parameter update formula of the Adam algorithm to be optimized is replaced by the square root of the second-order moment estimate of this step;

[0028] The second-order moment estimation value calculation formula in the Adam algorithm is organized into the second-order moment estimation value calculation formula in the form of the square root of the sum of the squares of the first temporary value and the second temporary value;

[0029] Get the approximate formula for the square root of the sum of squares;

[0030] Obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value;

[0031] Based on the approximate formula for the larger value, smaller value, and square root of the sum of squares, determine the formula for the sum of squares of the second-order distances.

[0032] In the embodiment of the present invention, the second-order moment estimated square root value, the second-order exponential decay factor, the initial iteration step number and the preset iteration step are initialized; the following steps are repeatedly performed until the set iteration step number is reached or the convergence condition is met: the objective function is differentiated for the parameter to be optimized in this step to obtain the gradient of this step; according to the second-order exponential decay factor, the absolute value of the gradient of this step and the square root value of the second-order moment estimated in the previous step, the larger value and the smaller value of the second-order distance square root formula are obtained; the larger value and the smaller value are input into the second-order distance square root formula to obtain the second-order moment estimated square root value of this step, and the second-order moment estimated square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step. , the update step is used to train the neural network in hardware to obtain the update step of this step; wherein, the steps for determining the square root of the second-order distance formula are as follows: replace the second-order moment estimate of this step in the parameter update formula to be optimized of the Adam algorithm with the square root of the second-order moment estimate of this step; organize the calculation formula of the second-order moment estimate of this step in the Adam algorithm into the square root of the square of the first temporary value and the second temporary value; obtain the square root approximate formula; obtain the larger value and smaller value used to represent the square root of the second-order distance formula according to the first temporary value and the second temporary value; determine the square root of the second-order distance formula according to the larger value, the smaller value, and the square root approximate formula. Compared with the existing technical solutions, the embodiment of the present invention determines the formula for the sum of square roots of second-order distances, and in this process, determines the approximate formula for the square root of the sum of squares, and obtains the larger and smaller values ​​of the formula for the sum of square roots of second-order distances based on the second-order exponential decay factor, the absolute value of the gradient of this step, and the estimated square root value of the second-order moment of the previous step. Further, the larger and smaller values ​​are input into the formula for the sum of square roots of second-order distances to obtain the estimated square root value of the second-order moment of this step, which can solve the large consumption of hardware resources caused by square operations, square root operations, and division operations in hardware implementation of the Adam algorithm, avoid the loss of precision caused by continuous squaring and square rooting of data when using fixed-point numbers, and greatly improve the data throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0034] Figure 1 The process of the hardware-oriented Adam algorithm second-order moment estimation optimization method in the embodiment of the present invention is as follows Figure 1 ;

[0035] Figure 2Flow chart of the steps of determining the formula for the sum of squares of second-order distances in an embodiment of the present invention;

[0036] Figure 3 The process of the hardware-oriented Adam algorithm second-order moment estimation optimization method in the embodiment of the present invention is as follows Figure 2 ;

[0037] Figure 4 This is a flow chart of the application of the second-order moment estimated square root value of this step in the overall training process of the neural network by Adam in an embodiment of the present invention;

[0038] Figure 5 Schematic diagram of the hardware-oriented Adam algorithm second-order moment estimation optimization system in an embodiment of the present invention Figure 1 ;

[0039] Figure 6 is a schematic diagram of a comparison module in an embodiment of the present invention;

[0040] Figure 7 is a schematic diagram of a second-order moment estimation module in an embodiment of the present invention;

[0041] Figure 8 A diagram showing the relationship between the modules for calculating the square root of the second-order moment estimate in this step in an embodiment of the present invention;

[0042] Fig. 9 Schematic diagram of the hardware-oriented Adam algorithm second-order moment estimation optimization system in an embodiment of the present invention Figure 2 . DETAILED DESCRIPTION

[0043] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0044] Figure 1 The process of the hardware-oriented Adam algorithm second-order moment estimation optimization method in the embodiment of the present invention is as follows Figure 1 ,include:

[0045] Step 101, initializing the second-order moment estimated square root value, the second-order exponential decay factor, the initial iteration step number and the preset iteration step;

[0046] Step 102, repeatedly perform the following steps until the set number of iterations is reached or the convergence condition is met:

[0047] Step 1021, deriving the objective function for the parameters to be optimized in this step to obtain the gradient of this step;

[0048] Step 1022, obtaining a larger value and a smaller value of the second-order moment square root formula according to the second-order exponential decay factor, the absolute value of the gradient in this step, and the estimated square root value of the second-order moment in the previous step;

[0049] Step 1023, input the larger value and the smaller value into the second-order distance square root formula to obtain the second-order moment estimated square root value of this step, the second-order moment estimated square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step;

[0050] Figure 2 Flow chart of the steps of determining the formula for the sum of squares of second-order distances in an embodiment of the present invention, wherein the steps of determining the formula for the sum of squares of second-order distances are as follows:

[0051] Step 201, replacing the second-order moment estimate of this step in the parameter update formula to be optimized of the Adam algorithm with the square root of the second-order moment estimate of this step;

[0052] Step 202, the second-order moment estimation value calculation formula of this step in the Adam algorithm is organized into the second-order moment estimation value calculation formula of this step in the form of the square root of the sum of the squares of the first temporary value and the second temporary value;

[0053] Step 203, obtaining an approximate formula for the square root of the sum of squares;

[0054] Step 204, obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value;

[0055] Step 205, determining the square root formula of the second order distance according to the larger value, the smaller value, and the square root approximation formula.

[0056] In step 201, the second-order moment estimate of this step in the parameter update formula to be optimized of the Adam algorithm is replaced by the square root of the second-order moment estimate of this step. In each subsequent round, there is no need to calculate the second-order moment estimate, and each update step can be completed using only the square root of the second-order moment estimate.

[0057] In one embodiment, the gradient calculation formula is: Among them, g t is the gradient of this step; θ is the parameter to be optimized; θ t-1 is the parameter to be optimized in this step.

[0058] f t (θ t-1 ) can be expressed as the objective function, i.e., a random scalar function that is differentiable with respect to the parameter θ.

[0059] Among them, the second-order moment estimate square root value is vr t The initial iteration number is assigned a value of 0.

[0060] In addition, after obtaining the gradient of this step, the absolute value of the gradient of this step is taken, and the absolute value of the gradient of this step is used for subsequent calculations. This is because a and b in the formula for the sum of squares of the second-order moments must be non-negative numbers. It should be noted that it is not necessary to take the absolute value of the gradient of this step when calculating the first-order moment.

[0061] In one embodiment, the larger and smaller values ​​of the second-order moment square root formula are obtained according to the second-order exponential decay factor, the absolute value of the gradient of this step, and the estimated square root value of the second-order moment of the previous step, including:

[0062] The first temporary value is calculated using the following formula based on the second-order exponential decay factor and the estimated square root of the second-order moment of the previous step: Among them, β2 is the second-order exponential decay factor, vr t-1 It is the estimated square root of the second moment of the previous step;

[0063] The second temporary value is obtained using the following formula based on the second-order exponential decay factor and the gradient of this step: g t is the gradient of this step;

[0064] Compare the first temporary value and the second temporary value, take the larger value as the larger value of the second-order distance square root sum formula, and take the smaller value as the smaller value of the second-order distance square root sum formula.

[0065] In one embodiment, obtaining the square root approximation formula includes:

[0066] Presets Where a = vr t cosθ,b=vr t sinθ,

[0067] Assume a≥b, where sinθ≥cosθ,

[0068] Presets in,

[0069] Arrange the first formula The range of the first formula is

[0070] when or When , the value of the first formula is 0, and the approximate formula for determining the square root is

[0071] The above process is also the proof process of the approximate formula for the square root of the sum of squares.

[0072] The error of the first formula above is The maximum error is only about.

[0073] In the square root approximation formula, one square operation, one square root operation, and two division operations of the second-order moment estimate in each iteration of the original Adam algorithm are simplified to operations such as addition and shift that are easy to implement in hardware and do not lose precision. After testing on the FPGA development board (Ultra96_V2), a single square root operation occupies 5157 LUT (look up table) resources and 675 trigger resources; a single division operation occupies 227 LUT resources and 9 DSP resources. In the optimized algorithm, a single addition operation only occupies 39 LUT resources, does not require DSP and trigger resources, and a single shift operation consumes almost no resources. After testing, for one Adam unit, the method of the present invention reduces LUT resources by about 83% and DSP resources by 50% compared with the original Adam.

[0074] In one embodiment, the step of determining the formula for the sum of squares of second-order distances includes:

[0075] The second-order moment estimate v in the calculation formula of the second-order moment estimate in the Adam algorithm is t Expressed as the estimated square root value vr of the second-order moment of this step t ;

[0076] The second-order moment estimate calculation formula in this step of the Adam algorithm is The calculation formula for the second-order moment estimate of this step, which is organized into the form of square root:

[0077]

[0078] in, is the first temporary value, is the second temporary value;

[0079] According to the first temporary value and the second temporary value, a larger value for expressing the formula of the sum of squares of the second order distances is obtained. and smaller value

[0080] Approximate formula based on larger value, smaller value, square root and root root The formula for determining the sum of squares of the second-order distances is:

[0081] The above is the derivation process of the formula for the sum of squares of the second-order moment estimate. By using the formula for the sum of squares of the second-order distance, there is no need to calculate the second-order moment estimate v at each step. t , we only need to use the second-order distance square root formula to calculate the estimated square root value vr of each stept , we can calculate and update the parameters to be optimized and complete the entire update process. t It can be finally expressed as the addition of two numbers, and the entire updating process does not require square open operations.

[0082] In order to further reduce the consumption of hardware resources and improve data throughput without affecting the accuracy, the calculation of dividing by 3 in the formula for the sum of squares of the second-order distances is optimized. In one embodiment, the formula for the sum of squares of the second-order distances is as follows:

[0083] vr t ≈a+(b>>2)+(b>>4)+(b>>6)

[0084] Among them, VR t is the square root of the second-order moment estimate for this step, where a is a larger value and b is a smaller value.

[0085] In the embodiment of the present invention, 1 / 3 is expressed as an initial value of 1 / 4 (1 / 2 2 )Sum of geometric sequence with common ratio 1 / 4, taking the first three digits for simplicity.

[0086] Figure 3 The process of the hardware-oriented Adam algorithm second-order moment estimation optimization method in the embodiment of the present invention is as follows Figure 2 In one embodiment, the method further comprises:

[0087] Step 301, initializing the first-order moment estimate, initial learning rate, first-order exponential decay factor and constant;

[0088] Step 302, calculating the first-order moment estimate of this step according to the first-order exponential decay factor, the first-order moment estimate of the previous step and the gradient of this step;

[0089] Step 303, input the initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate value of this step, and the square root value of the second-order moment estimate of this step into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step.

[0090] In one embodiment, the following formula is used to calculate the first-order moment estimate of this step according to the first-order exponential decay factor, the first-order moment estimate of the previous step, and the gradient of this step:

[0091] m t =β1·m t-1 +(1-β1)·g t

[0092] Among them, m t is the first-order moment estimate of this step, β1 is the first-order exponential decay factor, m t-1 is the first-order moment estimate of the previous step and the gradient of this step, gt is the gradient of this step.

[0093] In one embodiment, the parameter update formula of the optimized Adam algorithm is as follows:

[0094]

[0095] Among them, θ t is the parameter to be optimized in this step, θ t-1 is the parameter to be optimized in the previous step, Ir is the initial learning rate, m t is the first-order moment estimate of this step, vr t is the square root of the second-order moment estimate of this step, ε is a constant, and t is the number of iterations.

[0096] The derivation process of the parameter update formula to be optimized of the above optimized Adam algorithm is as follows:

[0097] Update the parameters to be optimized of the original Adam algorithm v t Replace with v t 2 , the updated formula is rewritten as follows:

[0098]

[0099] Then the optimized Adam algorithm's parameter update formula can be obtained approximately.

[0100] The code corresponding to the hardware-oriented Adam algorithm second-order moment estimation optimization method in the embodiment of the present invention is deployed on the hardware board to complete the neural network parameter training, saving about 83% of LUT resources and 50% of DSP resources, while increasing the speed by about 10 times, thus better achieving the purpose of hardware acceleration.

[0101] Through the above embodiment, the square root value of the second-order moment estimate of this step is obtained, and the square root value of the second-order moment estimate of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step. Figure 4 This is a flow chart of applying the second-order moment estimated square root value of this step to Adam in the overall training process of the neural network in an embodiment of the present invention, including:

[0102] Step 401, use the parameters to be optimized in the previous step to perform network forward propagation to obtain the output value of this step;

[0103] Step 402, the output value of this step is brought into the objective function, and the network back propagation phase is entered. The chain rule is used to derive the parameters to be optimized according to the objective function to obtain the derivative value;

[0104] Step 403, input the obtained derivative value into the Adam algorithm to obtain the parameters to be optimized in this step;

[0105] Step 404, determine whether the iteration termination condition is met, if so, end the iteration, if not, go to step 401.

[0106] The embodiment of the present invention further proposes a hardware-oriented Adam algorithm second-order moment estimation optimization system, which is a hardware circuit implementation of the hardware-oriented Adam algorithm second-order moment estimation optimization method. Figure 5 Schematic diagram of the hardware-oriented Adam algorithm second-order moment estimation optimization system in an embodiment of the present invention Figure 1 , including a control module 501, a comparison module 502 and a second-order moment estimation module 503; wherein,

[0107] The control module 501 is used to initialize the second-order moment estimated square root value, the second-order exponential decay factor, the initial iteration step number and the preset iteration step; and repeatedly perform the following steps until the set iteration step number is reached or the convergence condition is met:

[0108] Derivative the objective function for the parameters to be optimized in this step to obtain the gradient of this step;

[0109] Input the second-order exponential decay factor, the gradient of this step and the second-order moment estimated square root value of the previous step into the comparison module 402 to obtain the larger value and the smaller value of the second-order moment square root sum formula;

[0110] The larger value and the smaller value are input into the second-order moment estimation module 503 to obtain the second-order moment estimation square root value of this step, the second-order moment estimation module is used to obtain the second-order moment estimation square root value of this step based on the second-order distance square root sum formula, the second-order moment estimation square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, the update step is used to train the neural network in hardware to obtain the update step of this step;

[0111] The steps for determining the formula for the sum of squares of the second-order distances are as follows:

[0112] The second-order moment estimate of this step in the parameter update formula of the Adam algorithm to be optimized is replaced by the square root of the second-order moment estimate of this step;

[0113] The second-order moment estimation value calculation formula in the Adam algorithm is organized into the second-order moment estimation value calculation formula in the form of the square root of the sum of the squares of the first temporary value and the second temporary value;

[0114] Get the approximate formula for the square root of the sum of squares;

[0115] Obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value;

[0116] Based on the approximate formula for the larger value, smaller value, and square root of the sum of squares, determine the formula for the sum of squares of the second-order distances.

[0117] Figure 6 Schematic diagram of a comparison module in an embodiment of the present invention. In one embodiment, the comparison module includes:

[0118] A first multiplier 601 is used to multiply the square root of the second-order exponential decay factor and the square root of the previous second-order moment estimate to obtain a first temporary value;

[0119] A first shift register 602 is used to right-shift the gradient updated in this step according to the bit value determined by the second-order exponential decay factor to obtain a second temporary value;

[0120] A comparator 603 is used to compare the first temporary value and the second temporary value to obtain a comparison result;

[0121] A first two-way selector 604, configured to obtain a larger value of the first temporary value and the second temporary value according to the comparison result;

[0122] The second two-way selector 605 is used to obtain the smaller value between the first temporary value and the second temporary value according to the comparison result.

[0123] Figure 7 : is a schematic diagram of a second-order moment estimation module in an embodiment of the present invention. In one embodiment, the second-order moment estimation module includes:

[0124] The second shift register 701 is used to right shift the smaller value by two bits to obtain a third temporary value;

[0125] The third shift register 702 is used to right shift the smaller value by four bits to obtain a fourth temporary value;

[0126] The fourth shift register 703 is used to right-shift the smaller value by six bits to obtain a fifth temporary value;

[0127] A first adder 704, configured to perform addition calculation on the larger value and the third temporary value to obtain a sixth temporary value;

[0128] A second adder 705 is used to add the fourth temporary value and the fifth temporary value to obtain a seventh temporary value;

[0129] The third adder 706 is used to perform addition calculation on the seventh temporary value and the sixth temporary value to obtain the square root value of the second-order moment estimate of this step.

[0130] based on Figure 6 and Figure 7 , Figure 8The figure is a relationship diagram of related modules for calculating the square root value of the second-order moment estimate in this step in an embodiment of the present invention. According to the implementation of the relationship module, the square and square root operations are avoided, and the update process is optimized to the simplest addition and shift operations, which effectively reduces a large amount of resource waste, increases data throughput, and avoids the situation where the square and square root operations mistakenly calculate smaller values ​​as zero.

[0131] Fig. 9 Schematic diagram of the hardware-oriented Adam algorithm second-order moment estimation optimization system in an embodiment of the present invention Figure 2 In one embodiment, it also includes a parameter update module 901 to be optimized, which is used to input the initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate value of this step, and the square root value of the second-order moment estimate of this step into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step;

[0132] The control module is also used to:

[0133] Initialize the first-order moment estimate, initial learning rate, first-order exponential decay factor and constant;

[0134] Calculate the estimated value of the first-order moment of this step according to the first-order exponential decay factor, the estimated value of the first-order moment of the previous step and the gradient of this step;

[0135] The initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate value of this step, and the square root value of the second-order moment estimate of this step are input into the parameter update module to be optimized to obtain the update step of this step.

[0136] In summary, the method and system proposed in the embodiments of the present invention have the following beneficial effects:

[0137] First, the formula for the square root of the second-order moment and the approximate formula for the square root of the sum of squares are proposed, and the one square operation, one square root operation, and two division operations of the second-order moment estimate in each iteration of the original Adam algorithm are simplified into operations such as addition and shift that are convenient for hardware implementation and do not lose precision. After testing on the FPGA development board (Ultra96_V2), a single square root operation occupies 5157 LUT (look up table) resources and 675 trigger resources; a single division operation occupies 227 LUT resources and 9 DSP resources. In the optimized algorithm, a single addition operation only occupies 39 LUT resources, does not require DSP and trigger resources, and a single shift operation consumes almost no resources. After testing, for one Adam unit, the present invention reduces LUT resources by about 83% and DSP resources by 50% compared with the original Adam.

[0138] Second, the process of calculating the parameters to be optimized by estimating the second-order moment replaces the continuous square and square root operations, avoiding the situation of erroneous assignment of zero values ​​due to the limited bit width of the hardware storage space. Because the Adam algorithm searches for the minimum value of the objective function along the direction of gradient descent, the gradient value to be processed by the algorithm will always reach a relatively small value. After a square root operation, the order of magnitude will be multiplied exponentially, and the required bit width will become twice the bit width of the original value. Therefore, it is necessary to consider the problem that the data exceeds the hardware storage bit width, thereby affecting the accuracy. The method of the present invention does not undergo a square root operation. In theory, as long as the input data can pass normally, the above problem will not occur and cause any loss of accuracy.

[0139] Third, since the delay of addition and shift operations is smaller than that of operations such as squaring, square root and division, the method of the present invention can increase the data throughput by about 10 times in circuit implementation compared to the original Adam algorithm, better meeting the needs of neural network hardware acceleration.

[0140] Fourth, the hardware implementation of the hardware-oriented Adam algorithm second-order moment estimation optimization system uses FPGA as the hardware acceleration platform and HLS high-level synthesis as the tool to implement the hardware of the Adam algorithm after the second-order moment estimation optimization. Compared with the original Adam algorithm, it saves hardware resources by orders of magnitude, improves data throughput by more than 70%, and avoids accuracy issues. It should be emphasized that the Adam algorithm, as an algorithm for neural network parameter optimization, must generally be deployed on FPGA together with the neural network. The original Adam consumes 2623 LUT resources for a single iteration. Even if it can be implemented in hardware alone, it is difficult to find a board with sufficient resources to meet the needs of the neural network using the original Adam algorithm. Therefore, it is urgent to improve the Adam algorithm to meet the needs of hardware deployment.

[0141] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hardware-oriented Adam algorithm second-order moment estimation optimization system, characterized in that: It includes a control module, a comparison module and a second-order moment estimation module; wherein, The control module is used to initialize the second-order moment estimated square root value, the second-order exponential decay factor, the initial iteration step number and the preset iteration step; and repeat the following steps until the set iteration step number is reached or the convergence condition is met: Derivative the objective function for the parameters to be optimized in this step to obtain the gradient of this step; The second-order exponential decay factor, the gradient of this step and the estimated square root value of the second-order moment of the previous step are input into the comparison module to obtain the larger and smaller values ​​of the second-order moment square root sum formula; The larger value and the smaller value are input into the second-order moment estimation module to obtain the second-order moment estimation square root value of this step, the second-order moment estimation module is used to obtain the second-order moment estimation square root value of this step based on the second-order distance square root sum formula, the second-order moment estimation square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step; The steps for determining the formula for the sum of squares of the second-order distances are as follows: The second-order moment estimate of this step in the parameter update formula of the Adam algorithm to be optimized is replaced by the square root of the second-order moment estimate of this step; The second-order moment estimation value calculation formula in the Adam algorithm is organized into the second-order moment estimation value calculation formula in the form of the square root of the sum of the squares of the first temporary value and the second temporary value; Get the approximate formula for the square root of the sum of squares; Obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value; According to the approximate formula of the larger value, the smaller value, and the square root of the sum of squares, the formula for the sum of squares of the second-order distances is determined; The comparison module includes: A first multiplier is used to multiply the square root of the second-order exponential decay factor and the square root value of the previous second-order moment estimate to obtain a first temporary value; A first shift register is used to right-shift the gradient updated in this step according to the bit value determined by the second-order exponential decay factor to obtain a second temporary value; A comparator, used for comparing the first temporary value and the second temporary value to obtain a comparison result; A first two-way selector, used for obtaining a larger value of the first temporary value and the second temporary value according to the comparison result; A second two-way selector, used for obtaining a smaller value of the first temporary value and the second temporary value according to the comparison result; The second-order moment estimation module includes: The second shift register is used to right shift the smaller value by two bits to obtain a third temporary value; A third shift register is used for right shifting the smaller value by four bits to obtain a fourth temporary value; A fourth shift register, used for right shifting the smaller value by six bits to obtain a fifth temporary value; A first adder, configured to perform addition calculation on the larger value and the third temporary value to obtain a sixth temporary value; a second adder, configured to perform addition calculation on the fourth temporary value and the fifth temporary value to obtain a seventh temporary value; The third adder is used for performing addition calculation on the seventh temporary value and the sixth temporary value to obtain the square root value of the second-order moment estimate in this step.

2. The system according to claim 1, characterized in that It also includes a parameter update module to be optimized, which is used to input the initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate value of this step, and the square root value of the second-order moment estimate of this step into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step; The control module is also used to: Initialize the first-order moment estimate, initial learning rate, first-order exponential decay factor and constant; Calculate the estimated value of the first-order moment of this step according to the first-order exponential decay factor, the estimated value of the first-order moment of the previous step and the gradient of this step; The initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate value of this step, and the square root value of the second-order moment estimate of this step are input into the parameter update module to be optimized to obtain the update step of this step.

3. A hardware-oriented Adam algorithm second-order moment estimation optimization method, characterized in that: Applied to the system of claim 1 or 2, the method comprises: Initialize the second-order moment estimate square root value, the second-order exponential decay factor, the initial iteration number and the preset iteration step; Repeat the following steps until the set number of iterations is reached or the convergence condition is met: Derivative the objective function for the parameters to be optimized in this step to obtain the gradient of this step; According to the second-order exponential decay factor, the absolute value of the gradient in this step and the estimated square root value of the second-order moment in the previous step, the larger and smaller values ​​of the second-order distance square root formula are obtained; The larger value and the smaller value are input into the second-order distance square root formula to obtain the second-order moment estimated square root value of this step, and the second-order moment estimated square root value of this step is used to input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step, and the update step is used to train the neural network in hardware to obtain the update step of this step; The steps for determining the formula for the sum of squares of the second-order distances are as follows: The second-order moment estimate of this step in the parameter update formula of the Adam algorithm to be optimized is replaced by the square root of the second-order moment estimate of this step; The second-order moment estimation value calculation formula in the Adam algorithm is organized into the second-order moment estimation value calculation formula in the form of the square root of the sum of the squares of the first temporary value and the second temporary value; Get the approximate formula for the square root of the sum of squares; Obtaining a larger value and a smaller value for representing a formula for the sum of squares of second-order distances according to the first temporary value and the second temporary value; Based on the approximate formula for the larger value, smaller value, and square root of the sum of squares, determine the formula for the sum of squares of the second-order distances.

4. The method according to claim 3, characterized in that The gradient calculation formula for this step is: Among them, g t is the gradient of this step; θ is the parameter to be optimized; θ t-1 is the parameter to be optimized in this step.

5. The method according to claim 3, characterized in that After obtaining the gradient of this step, it also includes: Take the absolute value of the gradient of this step.

6. The method according to claim 3, characterized in that According to the second-order exponential decay factor, the absolute value of the gradient of this step, and the estimated square root value of the second-order moment of the previous step, the larger and smaller values ​​of the second-order distance square root formula are obtained, including: The first temporary value is calculated using the following formula based on the second-order exponential decay factor and the estimated square root of the second-order moment of the previous step: Among them, β2 is the second-order exponential decay factor, vr t-1 It is the estimated square root of the second moment of the previous step; The second temporary value is obtained using the following formula based on the second-order exponential decay factor and the gradient of this step: g t is the gradient of this step; Compare the first temporary value and the second temporary value, take the larger value as the larger value of the second-order distance square root sum formula, and take the smaller value as the smaller value of the second-order distance square root sum formula.

7. The method according to claim 3, characterized in that Get the approximate formula for the square root, including: Presets Where a = vr t cosθ,b=vr t sinθ, Assume a≥b, where sinθ≥cosθ, Presets in, Arrange the first formula The range of the first formula is when or When , the value of the first formula is 0, and the approximate formula for determining the square root is 8. The method according to claim 7, characterized in that The step of determining the formula for the sum of squares of second-order distances comprises: The second-order moment estimate vt in the calculation formula of the second-order moment estimate in the Adam algorithm is expressed as the square root of the second-order moment estimate vr t ; The second-order moment estimate calculation formula in this step of the Adam algorithm is The calculation formula for the second-order moment estimate of this step, which is organized into the form of square root: in, is the first temporary value, is the second temporary value; According to the first temporary value and the second temporary value, a larger value for expressing the formula of the sum of squares of the second order distances is obtained. and smaller value Approximate formula based on larger value, smaller value, square root and root root The formula for determining the sum of squares of the second-order distances is:

9. The method according to claim 8, characterized in that Also includes: Rewrite the division operation in the formula for the sum of squares of second-order distances into the form of summing the first three even powers of 1 / 2 of the dividend in the division operation; The formula for the sum of squares of the second-order distances is as follows: vr t ≈a+(b>>2)+(b>>4)+(b>>6) Among them, VR t is the square root of the second-order moment estimate for this step, where a is a larger value and b is a smaller value.

10. The method according to claim 3, characterized in that Also includes: Initialize the first-order moment estimate, initial learning rate, first-order exponential decay factor and constant; Calculate the estimated value of the first-order moment of this step according to the first-order exponential decay factor, the estimated value of the first-order moment of the previous step and the gradient of this step; The initial learning rate, the first-order exponential decay factor, the second-order exponential decay factor and constant, the first-order moment estimate of this step, and the square root of the second-order moment estimate of this step are input into the parameter update formula to be optimized of the optimized Adam algorithm to obtain the update step of this step.

11. The method according to claim 3, characterized in that The parameter update formula of the optimized Adam algorithm is as follows: Among them, θ t is the parameter to be optimized in this step, θ t-1 is the parameter to be optimized in the previous step, Ir is the initial learning rate, m t is the first-order moment estimate of this step, vr t is the square root of the second-order moment estimate of this step, ε is a constant, and t is the number of iterations.

Images