Method for training a neural network and optimizer for updating parameters of a neural network

Abstract

Disclosed are a method for training a neural network, an optimizer for updating parameters of a neural network, an electronic device, and a computer-readable storage medium. Various embodiments of the present disclosure combine the advantages of first-order momentum information and second-order momentum information, fully utilize the derivative information and curvature information of a loss function at each time step in the process of training a neural network, and release the memory for storing the derivative information after obtaining the first-order momentum information, thereby being capable of improving the training speed of the neural network while reducing the memory overhead.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence services, and more specifically to a method for training a neural network, an optimizer for updating neural network parameters, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Deep learning is an important branch of artificial intelligence that uses multi-layered neural network models to model and predict complex data. With the continuous expansion of deep learning applications, how to efficiently train deep neural network models has become a hot research topic in the field.

[0003] Currently, industry and academia have proposed various optimizers to update the parameters in neural networks for training. During training, these optimizers adjust the model parameters in the neural network by calculating the gradient of the loss function, so that the loss function can converge to the extreme value more quickly, thereby shortening the training time required for the neural network.

[0004] These optimizers can be applied to multiple fields. For example, in computer vision, they can be used for tasks such as image classification, object detection, and image generation. In natural language processing, they can be used for tasks such as language modeling, text translation, text generation, and speech recognition. In recommender systems, they can be used for tasks such as product recommendation and ad recommendation. Furthermore, these optimizers can also be applied to artificial intelligence scientific computing, such as solving and simulating partial differential equations (PDEs) describing physical phenomena. In reinforcement learning, they can be used for tasks such as playing chess and creating chat models that follow human instructions and values.

[0005] However, for very large neural networks, using optimizers during training requires significant memory and computational resources. Therefore, further improvements are needed to the techniques for training neural networks using optimizers. Summary of the Invention

[0006] This disclosure provides a method for training a neural network, an optimizer for updating neural network parameters, an electronic device, and a computer-readable storage medium.

[0007] This disclosure provides a method for training a neural network, comprising: iteratively training the neural network using a loss function at multiple time steps, wherein, for each of the multiple time steps, based on the values ​​of the parameters of the neural network at that time step, determining first-order momentum information corresponding to that time step, the first-order momentum information incorporating the derivative information of the loss function at that time step; determining second-order momentum information corresponding to that time step based on the first-order momentum information, the second-order momentum information incorporating the curvature information of the loss function at that time step; and updating the parameters of the neural network based on the first-order momentum information and the second-order momentum information to obtain parameters of the neural network for the next time step.

[0008] This disclosure provides an optimizer for updating neural network parameters. The optimizer updates the neural network using a loss function at each of multiple time steps. The optimizer includes: a first-order momentum information calculation circuit configured to determine first-order momentum information corresponding to the time step based on the values ​​of the neural network parameters at that time step, wherein the first-order momentum information incorporates the derivative information of the loss function at that time step; a second-order momentum information calculation circuit configured to determine second-order momentum information corresponding to the time step based on the first-order momentum information, wherein the second-order momentum information incorporates the curvature information of the loss function at that time step; and a parameter update circuit configured to update the parameters of the neural network based on the first-order momentum information and the second-order momentum information to obtain parameters for the neural network at the next time step.

[0009] This disclosure provides an electronic device, including: a processor; and a memory, wherein the memory stores a computer-executable program, and when the processor executes the computer-executable program, the above-described method is performed.

[0010] This disclosure provides an apparatus, including: a processor; and a memory storing computer instructions that, when executed by the processor, implement the method described above.

[0011] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method.

[0012] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0013] The various embodiments of this disclosure combine the advantages of first-order momentum information and second-order momentum information, making full use of the derivative and curvature information of the loss function at each time step during the training of the neural network, and releasing the memory used to store the derivative information after obtaining the first-order momentum information, which can improve the training speed of the neural network while reducing memory overhead. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this disclosure.

[0015] Figure 1 A schematic diagram of an application scenario according to an embodiment of this disclosure is shown.

[0016] Figure 2 This is a flowchart illustrating a method for training a neural network according to an embodiment of the present disclosure.

[0017] Figure 3 This is a schematic diagram illustrating an example loss function according to an embodiment of the present disclosure.

[0018] Figure 4 This is a schematic diagram illustrating an optimizer for updating neural network parameters according to an embodiment of the present disclosure.

[0019] Figure 5 This is a diagram illustrating the effect of implementing a method according to an embodiment of the present disclosure.

[0020] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0021] Figure 7 An architectural diagram of a computing device according to an embodiment of the present disclosure is shown.

[0022] Figure 8 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0024] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0025] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0026] Optionally, the various models involved in the embodiments of this disclosure can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Network nodes (i.e., neurons) are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at the nodes in the input layer of the neural network is propagated to the nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0027] Currently, various optimizers have been proposed by industry and academia for training neural networks. During training, these optimizers adjust the model parameters in the neural network by calculating the gradient of the loss function, so that the loss function can converge to the extreme value more quickly, thereby shortening the training time required for the neural network.

[0028] For example, industry and academia have proposed the Stochastic Gradient Descent (SGD) optimizer, a gradient-based optimizer that updates parameters by calculating the gradient of the loss function with respect to the model parameters. The SGD optimizer uses the same learning rate when updating each parameter, which typically needs to be manually adjusted for the model to converge. The SGD optimizer only utilizes first-order momentum information, resulting in slow convergence, and it may get stuck near local minima when dealing with non-convex optimization problems with high curvature.

[0029] For example, industry and academia have proposed the Adaptive Moment Estimation (Adam) optimizer, a gradient-based optimizer that combines momentum gradient descent and adaptive learning rate. The Adam optimizer utilizes first-order and second-order momentum information to improve the convergence speed of the loss function; however, the second-order momentum information it uses does not include information about the curvature of the loss function. In some tasks, this results in slow convergence and may not even converge to the optimal point.

[0030] Traditional second-order optimizers utilizing first- and second-order momentum information have been proposed, but they require approximately six times the memory of the neural network's parameters. Such traditional second-order optimizers may be unsuitable for neural network models with extremely large parameter counts, which can reach tens of billions, hundreds of billions, or even trillions. When training these neural network models, traditional second-order optimizers often need to perform complex calculations based on the massive number of parameters to update the parameters. Especially for traditional second-order optimizers, due to the significant computational resources required to obtain second-order momentum information (and / or loss function curvature information), and the inability to release memory resources promptly during the computation of second-order momentum information (and / or loss function curvature information), their application in the field of deep learning is difficult.

[0031] To this end, this disclosure provides an improved optimizer that can not only quickly acquire first-order momentum information and second-order momentum information containing loss function curvature information with less computation, but also release some memory in a timely manner after acquiring the first-order momentum information, thereby improving the training speed of the neural network while reducing memory overhead.

[0032] The various embodiments of this disclosure are described in detail with reference to the following accompanying drawings.

[0033] First refer to Figure 1 This document describes application scenarios of the methods and corresponding apparatuses according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically illustrated.

[0034] The apparatus for training a neural network according to embodiments of this disclosure can be integrated into various electronic devices, for example, Figure 1The device can be any electronic device in server 110 and multiple terminals 120. For example, a device for training a neural network can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet, laptop, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Similarly, the device for training a neural network can also be integrated into server 110. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0035] It is understood that the apparatus for inference on various possible complex neural network models using the method for training neural network models according to the embodiments of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. Of course, this disclosure is not limited thereto.

[0036] During the training of the neural network model, the terminal and / or server can use the training sample set stored in the terminal and / or server to train the neural network. It is worth noting that the neural network model in this paper can be a neural network model applicable to various scenarios, especially neural network models used in computer vision, natural language processing, and recommender systems.

[0037] This disclosure provides a method for training a neural network, comprising: iteratively training the neural network using a loss function at multiple time steps, wherein, for each of the multiple time steps, based on the values ​​of the parameters of the neural network at the current time step, determining first-order momentum information corresponding to the time step, the first-order momentum information incorporating the derivative information of the loss function at the time step; determining second-order momentum information corresponding to the time step based on the first-order momentum information, the second-order momentum information incorporating the curvature information of the loss function at the time step; and updating the parameters of the neural network at the next time step based on the first-order momentum information and the second-order momentum information.

[0038] The various embodiments of this disclosure combine the advantages of first-order momentum information and second-order momentum information. During the training of a neural network, second-order momentum information, incorporating the curvature of the loss function, is directly obtained from the first-order momentum information. This allows for convenient and full utilization of the derivative and curvature information of the loss function at each time step, significantly improving the training speed of the neural network. Some embodiments of this disclosure also release the memory used to store derivative information after acquiring the first-order momentum information, thereby reducing memory overhead while improving the training speed of the neural network.

[0039] The following combination Figures 2 to 8 A more detailed description will be given of all or part of the embodiments according to this disclosure.

[0040] Figure 2 This is a flowchart illustrating a method 20 for training a neural network according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram illustrating an example loss function according to an embodiment of the present disclosure.

[0041] The method 20 for training a neural network according to embodiments of this disclosure can be applied to any electronic device. It is understood that the electronic device can be various types of hardware devices, such as personal digital assistants (PDAs), audio / video devices, mobile phones, MP3 players, personal computers, laptops, servers, etc. For example, the electronic device can be... Figure 1 This includes servers and user terminals, etc. The following description uses a server as an example, but those skilled in the art should understand that this disclosure is not limited thereto.

[0042] Before training a neural network, its various parameters need to be initialized to prepare it for training. The initialization process includes, but is not limited to, setting initial values ​​for the weights and biases of each neuron in the neural network. In some embodiments of this disclosure, random initialization can be used to set random initial values ​​for the various parameters in the neural network model, or all parameters in the neural network model can be set to preset values ​​(e.g., 0 or 1), etc. This disclosure is not limited thereto.

[0043] After initializing the neural network, it can optionally be trained using multiple time steps. For example, a time step refers to a discrete point in time during the training process, with each time step corresponding to one update of the neural network's parameters. To better update the neural network's parameters at each time step to a state that allows the loss function of the neural network to converge, an optimizer according to embodiments of this disclosure can be used to determine the amount by which each parameter of the neural network needs to change at that time step. The loss function is a function of the neural network's predicted value for the target label and the true value of the target label, and includes, but is not limited to, the mean squared error function, the cross-entropy function, etc. The optimizer according to embodiments of this disclosure can extract the direction and magnitude information of the changes required for each parameter of the neural network from the loss function and the information of the neural network's parameters at that time step. This information includes, but is not limited to, first-order momentum information, second-order momentum information, derivative information, and curvature information, as detailed below. Of course, this disclosure is not limited thereto.

[0044] For example, method 20 according to an embodiment of this disclosure includes: iteratively training the neural network using a loss function at multiple time steps. After the multiple time steps are completed, the value of the loss function converges. Optionally, at each of the multiple time steps, operations S201 to S204 are performed sequentially, wherein operation S202 is an optional operation.

[0045] First, in operation S201, based on the values ​​of the parameters of the neural network at the time step, the first-order momentum information corresponding to the time step is determined, and the first-order momentum information is fused with the derivative information of the loss function at the time step.

[0046] For example, the first-order momentum information according to embodiments of this disclosure can also be referred to as first-order moment information, which indicates the direction information of the gradient of the loss function estimated by the optimizer according to embodiments of this disclosure at the time step. Specifically, the first-order momentum information may incorporate the derivative information of the loss function at the time step. This derivative information includes, but is not limited to, the values ​​of the partial derivatives of the loss function at each trainable parameter in the neural network at that time step. For example, assuming the loss function is L and a trainable parameter in the neural network is w, then the derivative information corresponding to that parameter is...

[0047] Furthermore, first-order momentum information can also incorporate other information. For example, considering the dependencies between time steps during training, the first-order momentum information at a certain time step can also incorporate the values ​​of first-order momentum information or second-order momentum information from the previous time step, and so on. Optionally, the parameters, first-order momentum information, and second-order momentum information of the neural network at a certain time step all contain information from the previous time step and can be recursively applied to the initial time step. After each parameter update of the neural network at a time step, the values ​​of the parameters, first-order momentum information, and second-order momentum information of the neural network at the current step are stored as constants in memory for easy subsequent calls, and the computation graph used to calculate the parameters, first-order momentum information, and second-order momentum information of the neural network at the current step is cleared to save storage overhead.

[0048] For example, in operation S201, the first-order momentum information of the neural network at the previous time step and the values ​​of the parameters of the neural network at that time step can be further obtained; and the first-order momentum information corresponding to the time step can be determined based on the derivative information of the loss function at the current time step and the value of the first-order momentum information of the neural network at the previous time step. Of course, this disclosure is not limited thereto.

[0049] Optionally, determining the first-order momentum information corresponding to the current time step based on the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step includes: obtaining pre-configured first-order moment coefficients, and based on the first-order moment coefficients, calculating a weighted sum of the values ​​of the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step, and determining the weighted sum as the first-order momentum information corresponding to the time step.

[0050] by Figure 3 For example, suppose the loss function according to an embodiment of this disclosure is of the form y = 4.0x 2 y=x 2 , and y = 0.1x 2 Any term in the equation, where x represents the difference between the sample labels and the output value of the neural network, and y represents the value of the loss function. That is, x could be a function of the various parameters in the neural network, the sample data, and the labels corresponding to the sample data as independent variables. When x is not equal to 0, the first-order momentum information can indicate the direction of gradient descent. For example... Figure 3 As shown, y = 4.0x 2 The corresponding direction of descent is different from y = 0.1x 2 The corresponding gradient descent direction is more erratic; therefore, with y = 4.0x... 2 When used as a loss function, the update amount of the neural network parameters should be less than that of y = 0.1x. 2This can be considered as a loss function. Of course, this disclosure is not limited to this.

[0051] Next, in some embodiments according to this disclosure, optionally, in operation S202, the memory used to store the derivative information of the loss function at the time step is released.

[0052] At this time, the memory of the device used to train the neural network may store all state information of the neural network at that time step, including but not limited to the weights and biases of each parameter of the neural network model after being updated in the previous time step, the first-order momentum information corresponding to the previous time step, and the derivative information. Furthermore, the memory may also include the second-order momentum information corresponding to the previous time step. This information will consume a large amount of storage resources. However, after operation S201, the first-order momentum information has already incorporated the derivative information of the loss function at that time step. The optimizer according to embodiments of this disclosure will then use the first-order momentum information instead of the derivative information, thereby safely releasing the memory used to store the derivative information of the loss function at that time step. As mentioned above, since the derivative information may include the values ​​of the partial derivatives of all trainable parameters in the neural network at that time step, it will consume a large amount of memory resources. Embodiments of this disclosure reduce the overhead required by the optimizer according to embodiments of this disclosure by releasing the memory used to store the derivative information of the loss function at that time step.

[0053] Next, in operation S203, based on the first-order momentum information corresponding to the time step, the second-order momentum information corresponding to the time step is determined, and the second-order momentum information is fused with the curvature information of the loss function at the time step.

[0054] For example, the second-order momentum information according to embodiments of this disclosure can also be referred to as second-order moment information, which indicates the step size of the gradient of the loss function estimated by the optimizer according to embodiments of this disclosure at the time step. Specifically, the second-order momentum information may incorporate the curvature information of the loss function at the time step. This curvature information includes, but is not limited to, the values ​​of the second-order partial derivatives of the loss function at each trainable parameter in the neural network at that time step. For example, assuming the loss function is L, and the two trainable parameters in the neural network are w1 and w2, then the derivative information corresponding to parameter w1 is... and Furthermore, second-order momentum information can be integrated with other information. For example, considering the dependencies between different time steps during training, the second-order momentum information at a certain time step can also be integrated with the second-order momentum information from the previous time step, and so on.

[0055] For example, in operation S203, a random vector can be further generated, where each element has a 50% probability of being positive one and a 50% probability of being negative one, and the number of elements in the random vector is consistent with the number of parameters in the neural network. Based on the random vector, the values ​​of the neural network's parameters at the time step, and the first-order momentum information corresponding to the time step, the second-order momentum information corresponding to the time step is determined using a differentiation operation. The differentiation operation can extract the curvature information of the loss function at the time step from the first-order momentum information at a relatively fast computation speed, exceeding the computational speed of traditional second-order optimizers.

[0056] Optionally, determining the second-order momentum information corresponding to the time step using differentiation includes: multiplying the random vector by the first-order momentum information to obtain a vector composed of the elements on the diagonal of the Hessian matrix; performing a nonlinear transformation on the vector composed of the elements on the diagonal of the Hessian matrix corresponding to the time step to obtain a vector composed of the elements on the diagonal of the nonlinearly transformed Hessian matrix; obtaining pre-configured first-order moment coefficients and second-order moment coefficients, and calculating a weighted sum of the vector composed of the elements on the diagonal of the nonlinearly transformed Hessian matrix and the second-order momentum information of the neural network at the previous time step based on the first-order moment coefficients and second-order moment coefficients, and determining the weighted sum as the second-order momentum information corresponding to the time step. The diagonal elements of the Hessian matrix contain curvature information of the loss function with respect to the neural network parameters. Multiplying the random vector by the first-order momentum information allows for a convenient and quick solution to the vector composed of the elements on the diagonal of the Hessian matrix without explicitly solving for the value of each element in the Hessian matrix. Of course, this disclosure is not limited to this.

[0057] Continue with Figure 3 For example, suppose the loss function according to an embodiment of this disclosure is of the form y = 4.0x 2 y=x 2 , and y = 0.1x 2 Any term in the equation, where x represents the difference between the sample label and the output value of the neural network, and y represents the value of the loss function. When x equals 0, the first-order momentum information of the three loss functions is the same. However, since the second-order momentum information can indicate changes in the gradient (e.g., the step size of the gradient), it contains more information; for example, it can indicate y = 4.0x. 2 The corresponding degree of curvature is less than that of y = 0.1x. 2 The degree of curvature is higher. Of course, this disclosure is not limited to this.

[0058] In operation S204, the parameters of the neural network are updated based on the first-order momentum information and the second-order momentum information to obtain the parameters of the neural network for the next time step.

[0059] As described above, first-order momentum information provides the direction information of the gradient of the loss function at that time step, and second-order momentum information provides the step size and magnitude information of the gradient of the loss function at that time step. From this information, the update amount of each parameter of the neural network at that time step can be derived. In some embodiments of this disclosure, this update amount can be regarded as a nonlinear transformation of first-order momentum information and second-order momentum information. Finally, the parameters of the neural network will be updated according to this update amount in order to better adapt to the sample data in the next time step and improve the accuracy of the output data.

[0060] Specifically, the update direction of each parameter of the neural network at a given time step can be derived from the first-order momentum information, and the magnitude correction of the step size of each parameter of the neural network at that time step can be derived from the second-order momentum information. In some embodiments of this disclosure, the update amount can be regarded as a function of the first-order momentum information and the second-order momentum information. For example, in this function, the first-order momentum information serves as the numerator representing the update direction, while the second-order momentum information serves as the denominator representing the magnitude correction of the update step size.

[0061] For example, in some embodiments of this disclosure, operation S204 further includes a de-biasing operation on the first-order momentum information and the second-order momentum information. As mentioned above, both the first-order and second-order momentum information may be data estimated by the optimizer, and these estimates may be inaccurate, thus requiring correction. For example, a correction factor may be determined based on the step size, the first-order and / or second-order momentum information from the previous time step, the form of the loss function, etc., to correct the first-order and second-order momentum information at the current time step. Based on the corrected first-order and second-order momentum information, the update amount of each parameter of the neural network at this time step is determined, thereby determining the parameters of the neural network for the next time step. Of course, this disclosure is not limited thereto.

[0062] For example, in operation S204, pre-configured first-order moment coefficients and second-order moment coefficients can be further obtained. Based on the first-order moment coefficients and the second-order moment coefficients, a de-biasing operation is performed on the first-order momentum information and the second-order momentum information to obtain de-biased first-order momentum information and second-order momentum information. Based on the de-biased first-order momentum information and second-order momentum information, the parameters of the neural network are updated to obtain the parameters of the neural network for the next time step. Of course, this disclosure is not limited thereto.

[0063] Optionally, updating the parameters of the neural network based on the first-order momentum information and the second-order momentum information includes: performing a nonlinear transformation on the first-order momentum information and the second-order momentum information to obtain an update amount for updating the parameters of the neural network; and superimposing the update amount with the value of the parameters of the neural network at the time step to update the parameters of the neural network.

[0064] Therefore, the method 20 for training a neural network according to the present disclosure combines the advantages of first-order momentum information and second-order momentum information, fully utilizes the derivative information and curvature information of the loss function at each time step during the training of the neural network, and releases the memory used to store the derivative information after obtaining the first-order momentum information, which can improve the training speed of the neural network while reducing the memory overhead.

[0065] Figure 4 This is a schematic diagram illustrating an optimizer 400 that updates neural network parameters according to an embodiment of the present disclosure. Figure 5 This is a diagram illustrating the effects of implementing a method according to an embodiment of the present disclosure.

[0066] The Optimizer 400 can be applied to multiple fields. For example, in computer vision, it can be used for tasks such as image classification, object detection, and image generation. In natural language processing, it can be used for tasks such as language modeling, text translation, text generation, and speech recognition. In recommender systems, it can be used for tasks such as product recommendation and ad recommendation. Furthermore, it can be applied to artificial intelligence scientific computing, such as solving and simulating partial differential equations (PDEs) describing physical phenomena. In reinforcement learning, it can be used for tasks such as playing chess and creating chat models that follow human instructions and values.

[0067] refer to Figure 4 The described optimizer 400 is used to update the neural network using a loss function at each of multiple time steps, and includes: a first-order momentum information calculation circuit, a memory release circuit, a second-order momentum information calculation circuit, and a parameter update circuit. The optimizer 400 can be configured to perform the method 20 detailed above. The optimizer 400 may also be referred to herein as an apparatus for training a neural network. Furthermore, the optimizer 400 may include more or fewer circuits, and this disclosure is not limited thereto.

[0068] Suppose the neural network has N parameters, which are represented as w = (w1, w2, ..., w...). N ) TThe loss function for this neural network is denoted as L(w). Before training this neural network, it may be necessary to initialize it, that is, assign the value of w0 to w. w0 is also called the initial network parameters.

[0069] Further assuming the learning rate during the training process of the neural network is η t Furthermore, the learning rate can vary depending on the learning rate strategy. In practice, optimizer 400 also requires first-moment coefficients β1 and second-moment coefficients β2 for subsequent calculations.

[0070] In the first time step, the optimizer 400 sets both the first-order momentum information m0 and the second-order momentum information v0 corresponding to the previous time step to zero, that is, m0 = 0, v0 = 0. At any other time step t, the optimizer 400 can obtain the first-order momentum information m0 corresponding to the previous time step (t-1). t-1 and second-order momentum information v t-1 Where t is an integer greater than 1. Of course, this disclosure is not limited to this.

[0071] The following examples illustrate the various operations in method 20 executed by optimizer 400 using time step t as an example. Those skilled in the art should understand that this disclosure is not limited thereto.

[0072] First, at time step t, the first-order momentum information calculation circuit in optimizer 400 is configured to perform operation S201: determining the first-order momentum information corresponding to the time step based on the values ​​of the parameters of the neural network at that time step, wherein the first-order momentum information incorporates the derivative information of the loss function at that time step. Specifically, the first-order momentum information calculation circuit can perform the following operations: acquiring the first-order momentum information of the neural network at the previous time step and the values ​​of the parameters of the neural network at that time step; and determining the first-order momentum information corresponding to the time step based on the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step. Optionally, determining the first-order momentum information corresponding to the current time step based on the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step includes: obtaining pre-configured first-order moment coefficients, and calculating a weighted sum of the values ​​of the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step based on the first-order moment coefficients, and determining the weighted sum as the first-order momentum information corresponding to the time step. This disclosure is not limited thereto.

[0073] For example, assuming the derivative of the loss function at time step t is g, it can be calculated according to formula (1). In formula (1), N is the number of parameters of the neural network.

[0074]

[0075] Next, the first-order momentum information calculation circuit calculates the first-order momentum information m corresponding to the previous time step (t-1) according to formula (2). t-1 and the derivative information g and the first-order momentum information m t This enables the use of g to update first-order momentum information.

[0076] m t =β1m t-1 +(1-β1)g (2)

[0077] Then, optionally, the memory release circuit in the optimizer 400 is configured to perform operation S202: release the memory used to store the derivative information of the loss function at the time step. Memory consumption is reduced by removing the storage of the derivative information g.

[0078] Next, the second-order momentum information calculation circuit in the optimizer 400 can be configured to perform operation S203: based on the first-order momentum information corresponding to the time step, determine the second-order momentum information corresponding to the time step, wherein the second-order momentum information incorporates the curvature information of the loss function at the time step. The second-order momentum information calculation circuit can be configured to generate a random vector, wherein the elements in the random vector have a 50% probability of being positive one and a 50% probability of being negative one, and the number of elements in the random vector is consistent with the number of parameters of the neural network; and based on the random vector, the values ​​of the parameters of the neural network at the time step, and the first-order momentum information corresponding to the time step, determine the second-order momentum information corresponding to the time step using a differentiation operation, wherein the second-order momentum information incorporates the curvature information of the loss function at the time step. The step of determining the second-order momentum information corresponding to the time step using differentiation includes: multiplying the random vector by the first-order momentum information to obtain a vector composed of the elements on the diagonal of the Hessian matrix; performing a nonlinear transformation on the vector composed of the elements on the diagonal of the Hessian matrix corresponding to the time step to obtain a vector composed of the elements on the diagonal of the Hessian matrix after the nonlinear transformation; obtaining pre-configured first-order moment coefficients and second-order moment coefficients, and calculating a weighted sum of the vector composed of the elements on the diagonal of the Hessian matrix after the nonlinear transformation and the second-order momentum information of the neural network in the previous time step based on the first-order moment coefficients and second-order moment coefficients, and determining the weighted sum as the second-order momentum information corresponding to the time step.

[0079] Specifically, the second-order momentum information calculation circuit can automatically generate a random vector z, whose dimension is consistent with w, and whose element value is either +1 or -1, with a probability of 50% for both +1 and -1. Then, the second-order momentum information calculation circuit uses automatic differentiation based on formula (3) and the random vector z to calculate the first-order momentum information m. t The derivative is taken to obtain the curvature information of the loss function at the time step. As shown in Equation (4), for the random vector z and the first-order momentum information m t Differentiating the product of the products yields the result. According to The vector consisting of the elements on the diagonal of the Hessian matrix can be obtained by solving the problem.

[0080]

[0081]

[0082] Where i is the index of the sample data, and H is the Hessian matrix (of size N). 2 ), that is (1-β1)Hz i An approximate estimate of the size is only N. According to formula (3), the value of the Hessian matrix H can be obtained directly without explicitly solving for it. It reduces the amount of computation and memory required.

[0083] and in, for The expected value, where diag{H} represents the set of elements on the diagonal of H, can be further represented as That is, This is an approximation of (1-β1)diag{H}. Based on the following formula (5), diag{H} and... approximate.

[0084]

[0085] Here, diag{H} has a size of N, and its inverse matrix is ​​a simple element-wise division matrix. -1 = 1 / diag{H}. Compared to the traditional second-order optimizer which requires N... 2 Only a value of H can be used to calculate the second-order momentum information, and optimizer 400 significantly reduces the computational load. More specifically, according to formulas (3) and (4), the solution obtained at this time... The curvature information has already been included, and the entire solution process only involves the first-order momentum information m. tSince the derivative information g is not involved, releasing the memory used to store the derivative information g of the loss function at the time step in the aforementioned steps does not affect the subsequent process of obtaining second-order momentum information, including curvature information. Therefore, optimizer 400 can significantly reduce memory requirements.

[0086] In the known In this case, formula (6) can be used to calculate the second-order momentum information v corresponding to the previous time step (t-1). t-1 The second-order moment coefficient β2 determines the second-order momentum information v corresponding to time step t. t Among them, formula (6) is obtained through It achieves an approximation of the square of the vector composed of the diagonal elements of the Hessian matrix.

[0087]

[0088] From formula (6), we can see that the second-order momentum information v t The vector consisting of the diagonal elements of the Hessian matrix after nonlinear transformation. The value v of the second-order momentum information corresponding to the neural network at the previous time step. t-1 The weighted sum.

[0089] Then, the parameter update circuit in the optimizer 400 is configured to perform operation S204: update the parameters of the neural network based on the first-order momentum information and the second-order momentum information to obtain the parameters of the neural network for the next time step.

[0090] Optionally, the parameter update circuit is configured to acquire pre-configured first-order moment coefficients and second-order moment coefficients; perform a debiasing operation on the first-order momentum information and the second-order momentum information based on the first-order moment coefficients and the second-order moment coefficients to obtain debiased first-order momentum information and second-order momentum information; and update the parameters of the neural network based on the debiased first-order momentum information and second-order momentum information to obtain the parameters of the neural network for the next time step. Optionally, updating the parameters of the neural network based on the first-order momentum information and the second-order momentum information includes: performing a nonlinear transformation on the first-order momentum information and the second-order momentum information to obtain an update amount for updating the parameters of the neural network; and superimposing the update amount with the value of the parameters of the neural network at the time step to update the parameters of the neural network.

[0091] Optionally, the parameter update circuit can first use formulas (7) and (8) to update the first-order momentum information m. t and second-order momentum information v t Perform a debiasing operation to obtain the debiased first-order momentum information m.t and second-order momentum information v t .

[0092]

[0093]

[0094] Then the parameter update circuit will correspondingly use formula (9) to change the parameters of the neural network from w t-1 Updated to w t .

[0095]

[0096] Here, ε is a very small floating-point number used to ensure that m does not occur. t When divided by zero, and η t The learning rate. The update amount in formula (9) for updating the parameters of the neural network. Nonlinear transformation of first-order momentum information and second-order momentum information was achieved.

[0097] In each of the above operations, whether it's the first-order momentum information calculation circuit, the memory release circuit, the second-order momentum information calculation circuit, or the parameter update circuit, it requires at most five times the amount of memory as the number of parameters N in the neural network. For example, when calculating second-order momentum information, the second-order momentum information calculation circuit needs to store a maximum of five vectors w simultaneously. t-1 ,m t ,v t-1 ,z i ,h i It consumes 5N of memory. Compared to traditional implementations that require at least 6N of memory, the optimizer disclosed herein can reduce memory overhead by 1 / 6.

[0098] like Figure 5 As shown, in actual experiments, the optimizer 400 was used to train the transformer-based translation neural network model. Whether during training or validation, its loss function converged in about 8000 time steps. This achieved similar convergence results and training time as traditional implementation methods while reducing the maximum memory usage.

[0099] As can be seen, the various optimizers 400 disclosed herein combine the advantages of first-order momentum information and second-order momentum information, making full use of the derivative and curvature information of the loss function at each time step during the training of the neural network, and releasing the memory used to store the derivative information after acquiring the first-order momentum information, which can improve the training speed of the neural network while reducing memory overhead.

[0100] According to another aspect of this disclosure, an electronic device is also provided for implementing the method 20 according to an embodiment of this disclosure or carrying an optimizer 400 according to an embodiment of this disclosure. Figure 6 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0101] like Figure 6 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code that, when executed by the one or more processors 2010, can perform the methods described above.

[0102] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 or ARM architecture.

[0103] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0104] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 7 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 7As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 7 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 7 One or more components in the computing device shown.

[0105] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 8 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0106] like Figure 8 As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the method described with reference to the above figures according to embodiments of the present disclosure can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0107] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a method according to an embodiment of this disclosure.

[0108] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0110] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A method for training a neural network, comprising: The neural network is iteratively trained using a loss function at multiple time steps, wherein, for each of the multiple time steps, From memory, obtain the first-order momentum information of the neural network at the previous time step, the second-order momentum information at the previous time step, and the values ​​of the parameters of the neural network at the time step; Based on the values ​​of the parameters of the neural network at the time step, the derivative information of the loss function at the time step is determined, and the derivative information of the loss function at the time step is stored in memory; Based on the derivative information of the loss function at the time step and the value of the first-order momentum information of the neural network at the previous time step, the first-order momentum information corresponding to the time step is determined, and the first-order momentum information is fused with the derivative information of the loss function at the time step. The first-order momentum information corresponding to the time step is stored in memory; After storing the first-order momentum information corresponding to the time step in memory, release the memory used to store the derivative information of the loss function at the time step; Based on the first-order momentum information corresponding to the time step, the second-order momentum information corresponding to the time step is determined, and the second-order momentum information is fused with the curvature information of the loss function at the time step; The second-order momentum information corresponding to the time step is stored in memory; Based on the first-order momentum information and the second-order momentum information, the parameters of the neural network are updated to obtain the parameters of the neural network for the next time step; and The values ​​of the parameters of the neural network used for the next time step are stored in memory.

2. The method as described in claim 1, wherein, The loss function is a function of the neural network's predicted value of the target label and the actual value of the target label.

3. The method as described in claim 1, wherein, The step of determining the first-order momentum information corresponding to the current time step based on the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step includes: Obtain the pre-configured first-order moment coefficients, and based on the first-order moment coefficients, calculate the weighted sum of the derivative information of the loss function at the current time step and the first-order momentum information of the neural network at the previous time step, and determine the weighted sum as the first-order momentum information corresponding to the time step.

4. The method of claim 1, wherein, The determination of the second-order momentum information corresponding to the time step based on the first-order momentum information of the time step includes: Generate a random vector, wherein each element in the random vector has a 50% probability of being positive one and a 50% probability of being negative one, and the number of elements in the random vector is the same as the number of parameters in the neural network; and Based on the random vector, the values ​​of the parameters of the neural network at the time step, and the first-order momentum information corresponding to the time step, the second-order momentum information corresponding to the time step is determined by using the differentiation operation. The second-order momentum information is fused with the curvature information of the loss function at the time step.

5. The method of claim 4, wherein, The process of determining the second-order momentum information corresponding to the time step using the differentiation operation includes: The random vector is multiplied by the first-order momentum information to obtain a vector composed of the elements on the diagonal of the Hessian matrix. A nonlinear transformation is performed on the vector composed of the elements on the diagonal of the Hessian matrix corresponding to the time step to obtain the vector composed of the elements on the diagonal of the Hessian matrix after the nonlinear transformation. Obtain the pre-configured first-order moment coefficients and second-order moment coefficients, and based on the first-order moment coefficients and second-order moment coefficients, calculate the weighted sum of the vector composed of the elements on the diagonal of the Hessian matrix after nonlinear transformation and the value of the second-order momentum information of the neural network at the previous time step, and determine the weighted sum as the second-order momentum information corresponding to the time step.

6. The method of claim 1, wherein, The step of updating the parameters of the neural network based on the first-order momentum information and the second-order momentum information includes: Obtain the pre-configured first-order moment coefficients and second-order moment coefficients; Based on the first-order moment coefficients and the second-order moment coefficients, a de-biasing operation is performed on the first-order momentum information and the second-order momentum information to obtain the de-biased first-order momentum information and second-order momentum information. Based on the debiased first-order momentum information and second-order momentum information, the parameters of the neural network are updated to obtain the parameters of the neural network for the next time step.

7. The method of claim 1, wherein, The step of updating the parameters of the neural network based on the first-order momentum information and the second-order momentum information includes: A nonlinear transformation is performed on the first-order momentum information and the second-order momentum information to obtain the update amount for updating the parameters of the neural network; and The update amount is superimposed on the value of the parameters of the neural network at the time step to update the parameters of the neural network.

8. An optimizer for updating parameters of a neural network, wherein the optimizer is configured to update the neural network using a loss function at each of a plurality of time steps, the optimizer comprising: The first-order momentum information calculation circuit is configured as follows: From memory, obtain the first-order momentum information of the neural network at the previous time step, the second-order momentum information at the previous time step, and the values ​​of the parameters of the neural network at the time step; Based on the values ​​of the parameters of the neural network at the time step, the derivative information of the loss function at the time step is determined, and the derivative information of the loss function at the time step is stored in memory; Based on the derivative information of the loss function at the time step and the value of the first-order momentum information of the neural network at the previous time step, the first-order momentum information corresponding to the time step is determined, and the first-order momentum information is fused with the derivative information of the loss function at the time step. The first-order momentum information corresponding to the time step is stored in memory; The memory release circuit is configured to: after storing the first-order momentum information corresponding to the time step in memory, release the memory used to store the derivative information of the loss function at the time step; The second-order momentum information calculation circuit is configured as follows: Based on the first-order momentum information corresponding to the time step, the second-order momentum information corresponding to the time step is determined, and the second-order momentum information is fused with the curvature information of the loss function at the time step; The second-order momentum information corresponding to the time step is stored in memory; as well as The parameter update circuit is configured as follows: Based on the first-order momentum information and the second-order momentum information, the parameters of the neural network are updated to obtain the parameters of the neural network for the next time step; and The values ​​of the parameters of the neural network used for the next time step are stored in memory.

9. An electronic device, comprising: processor; as well as A memory, wherein the memory stores a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-7.

Images