Training method and device of neural network, computer device and storage medium

Abstract

The application discloses a neural network training method and device, computer equipment and a storage medium, and belongs to the field of artificial intelligence (AI) technology. The method comprises the following steps: obtaining a training data set and a neural network to be trained, wherein the training data set comprises a plurality of training samples; iteratively training the neural network to be trained by using the plurality of training samples in the training data set; obtaining a training sample input into the neural network to be trained in an Nth iteration process and an output result of the neural network to be trained for the training sample in the Nth iteration process, wherein N is a positive integer; determining a momentum coefficient used in the Nth iteration process based on the training sample and the output result in the Nth iteration process; and updating parameters of the neural network to be trained in the Nth iteration process based on the momentum coefficient. The application improves the training efficiency of the neural network.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for training a neural network, a computer device, and a storage medium. Background Technology

[0002] With the popularization of neural networks, they are being applied to more and more fields. Before using a neural network, it is usually necessary to design an initial neural network and train it using training samples to ensure its performance.

[0003] In related technologies, gradient descent is commonly used to train neural networks. The training process involves multiple iterations. Each iteration includes: inputting training samples into an initial neural network and obtaining the output of the initial neural network for those training samples; then, based on the error of the output relative to the training samples, calculating the gradient used to update the parameters of the initial neural network in this iteration, so that the weights and biases of the neurons in the initial neural network are updated according to this gradient during this iteration.

[0004] However, the training efficiency of training the initial neural network using this gradient descent method needs further improvement. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and storage medium for training a neural network, thereby improving the training efficiency of training an initial neural network. The technical solution provided by this application is as follows:

[0006] Firstly, this application provides a method for training a neural network. The method includes: acquiring a training dataset and a neural network to be trained, wherein the training dataset includes multiple training samples; iteratively training the neural network to be trained using the multiple training samples in the training dataset, wherein during the iterative training process: acquiring the training samples input to the neural network to be trained in the Nth iteration, and the output results of the neural network to be trained in response to the training samples in the Nth iteration, where N is a positive integer; determining the momentum coefficient used in the Nth iteration based on the training samples and output results in the Nth iteration; and updating the parameters of the neural network to be trained in the Nth iteration based on the momentum coefficient. The momentum coefficient is used to measure the contribution of the cumulative value of the gradients used in the iterations prior to the Nth iteration to the actual update amount.

[0007] By determining the momentum coefficient used in the Nth iteration based on the training samples input to the neural network in the Nth iteration and the output of the neural network to the training samples in the Nth iteration, and then updating the parameters of the neural network in the Nth iteration based on the momentum coefficient, it is possible to update the parameters of the neural network according to different momentum coefficients in different iterations. This is beneficial for finding a better optimization path for training the neural network and helps to improve the efficiency of training the neural network.

[0008] In one possible implementation, determining the momentum coefficients used in the Nth iteration based on the training samples and output results in the Nth iteration includes: determining the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration based on the training samples and output results in the Nth iteration.

[0009] In any iteration, by determining the momentum coefficients used by each hidden layer of the neural network to be trained, the momentum coefficients can be adjusted at the hidden layer level, which refines the granularity of momentum coefficient adjustment and can further improve the efficiency of training the neural network.

[0010] Optionally, based on the training samples and output results in the Nth iteration process, the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration process are determined respectively, including: obtaining the reference gradient of any hidden layer in the Nth iteration process based on the training samples and output results in the Nth iteration process; obtaining the reference gradient of any hidden layer in the iteration process before the Nth iteration process; and determining the momentum coefficients used by any hidden layer in the Nth iteration process based on the reference gradients of any hidden layer in the Nth and previous iteration processes.

[0011] Furthermore, based on the training samples and output results in the Nth iteration process, the momentum coefficient used in the Nth iteration process is determined, including: based on the training samples and output results in the Nth iteration process, the reference momentum coefficient used in the Nth iteration process is determined; and based on the user-input adjustment instructions and the reference momentum coefficient, the momentum coefficient is obtained.

[0012] By updating the reference momentum coefficient based on user-input-based adjustment instructions, human experience can be injected into the determination of the momentum coefficient, which can further improve the efficiency of training neural networks.

[0013] Optionally, obtaining the momentum coefficient based on the user-input adjustment instruction and the reference momentum coefficient further includes: when the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction, determining the reference momentum coefficient as the momentum coefficient; when the reference momentum coefficient is outside the momentum coefficient range, updating the reference momentum coefficient based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient.

[0014] In one possible implementation, when the reference momentum coefficient is outside the momentum coefficient range, the reference momentum coefficient is updated based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient, including: when the reference momentum coefficient is greater than the upper limit of the momentum coefficient range, the upper limit is determined as the momentum coefficient; when the reference momentum coefficient is less than the lower limit of the momentum coefficient range, the lower limit is determined as the momentum coefficient.

[0015] Optionally, based on the momentum coefficient, the parameters of the neural network to be trained are updated during the Nth iteration, including:

[0016] Based on the momentum coefficient, the reference gradient of the neural network to be trained is updated during the Nth iteration;

[0017] Based on the updated gradient, the parameters of the neural network to be trained are updated.

[0018] Secondly, this application provides a training apparatus for a neural network, comprising: an acquisition module for acquiring a training dataset and a neural network to be trained, wherein the training dataset includes multiple training samples; the acquisition module is further configured to iteratively train the neural network to be trained using the multiple training samples in the training dataset, wherein during the iterative training process: the acquisition of the training samples input to the neural network to be trained in the Nth iteration, and the output results of the neural network to be trained in response to the training samples in the Nth iteration, wherein N is a positive integer; a determination module for determining the momentum coefficient used in the Nth iteration based on the training samples and the output results in the Nth iteration; and an update module for updating the parameters of the neural network to be trained in the Nth iteration based on the momentum coefficient.

[0019] Optionally, a determination module is specifically used to: determine the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration process based on the training samples and output results in the Nth iteration process.

[0020] Optionally, the determination module is specifically used for: obtaining the reference gradient of any hidden layer in the Nth iteration based on the training samples and output results in the Nth iteration; obtaining the reference gradient of any hidden layer in the iteration process before the Nth iteration; and determining the momentum coefficient used by any hidden layer in the Nth iteration based on the reference gradient of any hidden layer in the Nth and previous iteration processes.

[0021] Optionally, the determination module is specifically used to: determine the reference momentum coefficient used in the Nth iteration based on the training samples and output results in the Nth iteration; and obtain the momentum coefficient based on the user-input adjustment instructions and the reference momentum coefficient.

[0022] Optionally, the determination module is specifically used to: determine the reference momentum coefficient as the momentum coefficient when the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction; and update the reference momentum coefficient based on the upper and lower limits of the momentum coefficient range when the reference momentum coefficient is outside the momentum coefficient range to obtain the momentum coefficient.

[0023] Optionally, the determination module is specifically used to: determine the upper limit value as the momentum coefficient when the reference momentum coefficient is greater than the upper limit value of the momentum coefficient range; and determine the lower limit value as the momentum coefficient when the reference momentum coefficient is less than the lower limit value of the momentum coefficient range.

[0024] Optionally, the update module is specifically used to: update the reference gradient of the neural network to be trained during the Nth iteration based on the momentum coefficient; and update the parameters of the neural network to be trained based on the updated gradient.

[0025] Thirdly, this application provides a computer device, which includes a processor and a memory, wherein a computer program is stored in the memory; when the processor executes the computer program, the computer device implements the method provided by the first aspect of this application and any optional implementation thereof.

[0026] Fourthly, this application provides a non-transient computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor, implements the method provided by the first aspect of this application and any optional implementation thereof.

[0027] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the method provided by the first aspect of this application and any optional implementation thereof. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a training device involved in a neural network training method provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of a training device involved in another neural network training method provided in this application embodiment;

[0030] Figure 3 This is a schematic diagram illustrating an application scenario involving a neural network training method provided in an embodiment of this application;

[0031] Figure 4 This is a flowchart of a neural network training method provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram illustrating an application scenario involving another neural network training method provided in this application embodiment;

[0033] Figure 6 This is a flowchart of a method for determining the momentum coefficient used in the Nth iteration based on training samples and output results, provided in an embodiment of this application.

[0034] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] With the development of deep learning technology, more and more users are using deep neural networks (DNNs) to solve problems. Deep learning is a type of machine learning technique that uses deep neural networks to solve problems. Its main characteristic is the use of multiple nonlinear transformation structures to process and analyze data. Deep learning is mainly applied in perception and decision-making scenarios within the field of artificial intelligence, such as image and speech recognition, natural language translation, and computer games. A deep neural network is a framework for deep learning; it is a neural network with at least one hidden layer. Similar to shallow neural networks, deep neural networks can model complex nonlinear systems at a higher level of abstraction, improving the capabilities of the model.

[0037] Similar to deep neural networks, shallow neural networks have also been widely used. The main difference between shallow and deep neural networks lies in the number of hidden layers (also known as depth). Both deep and shallow neural networks are collectively referred to as neural networks. Depending on the application requirements, neural networks of different depths can be used in practical scenarios. Before using a neural network, it needs to be trained to achieve better performance and improve the efficiency and accuracy of solving problems using the network. Typically, stochastic gradient descent (SGD) is used to train the neural network. Also known as incremental gradient descent, SGD is essentially an optimizer. It calculates the gradient of the loss function based on mini-batch training samples and iteratively optimizes the parameters such as the weights and biases of neurons in the neural network based on the gradient, thus training the neural network. The loss function is the objective function to be optimized during training; this loss function is a function differentiable with respect to the parameters of the neural network.

[0038] The stochastic gradient descent method accumulates the gradients used in iterations prior to the Nth iteration and combines this accumulated gradient with the gradient obtained from the loss function in the Nth iteration. This combined gradient is used as the actual update amount for updating the neural network parameters in the Nth iteration. Stochastic gradient descent enables the loss function of the neural network to converge to the optimal solution more quickly. The momentum coefficient measures the contribution of the accumulated gradients used in iterations prior to the Nth iteration to the actual update amount. The momentum coefficient ranges from [0,1]. In one implementation, the actual update amount in the Nth iteration = reference gradient obtained from the loss function in the Nth iteration + momentum coefficient × accumulated gradients used in iterations prior to the Nth iteration. This accumulated gradient can be equal to the actual update amount in the (N-1)th iteration.

[0039] Currently, in training neural networks using stochastic gradient descent, the same momentum coefficient is used throughout the entire training process. This means that the cumulative gradient contributes the same amount to the actual update in each iteration. However, for multi-layered neural networks, using the same momentum coefficient throughout training limits the possibility of finding a better optimization path. Therefore, the training efficiency of this stochastic gradient descent method needs further improvement.

[0040] This application provides a method for training a neural network. During the iterative training of the neural network to be trained, the momentum coefficient used in the Nth iteration is determined based on the training samples input to the neural network in the Nth iteration and the output results of the neural network in response to the training samples in the Nth iteration. Then, based on the momentum coefficient, the parameters of the neural network to be trained are updated in the Nth iteration. This method allows the momentum coefficient used in the Nth iteration to be calculated in each iteration, enabling the neural network parameters to be updated according to different momentum coefficients in different iterations. This facilitates finding a better optimization path for training the neural network and helps improve the efficiency of neural network training.

[0041] The neural network training method provided in this application can be applied to a neural network training device. This training device is used to execute the neural network training method provided in this application. Optionally, the training device can be implemented using a terminal, physical machine, bare-metal server, cloud server, virtual machine, or container, etc.

[0042] The training device can be logically divided into multiple parts, each with a different function. For example, such as Figure 1 As shown, the training device 10 may include the following functional modules: an acquisition module 101, a determination module 102, and a training module 103. The acquisition module 101 is used to acquire the training dataset and the neural network to be trained, and during the iterative training process of the neural network to be trained, to acquire the training samples input to the neural network to be trained in the Nth iteration, and the output results of the neural network to be trained in response to the training samples in the Nth iteration. The determination module 102 is used to determine the momentum coefficient used in the Nth iteration based on the training samples and output results in the Nth iteration. The update module 103 is used to update the parameters of the neural network to be trained in the Nth iteration based on the momentum coefficient. For example, as... Figure 2 As shown, the training device 10 may further include an initialization module 104, which is used to initialize the parameters of each network layer in the neural network (i.e., assign initial values ​​to the parameters of the neural network) when training the neural network is started.

[0043] Furthermore, multiple components of the training device can be deployed independently on terminals, physical machines, bare metal servers, cloud servers, virtual machines, or containers. Alternatively, multiple components of the training device can be deployed distributedly on one or more of terminals, multiple physical machines, multiple bare metal servers, multiple cloud servers, multiple virtual machines, and multiple containers, with these distributed components working together to achieve all the functions of the training device.

[0044] In one possible implementation, the training device can be deployed on one or more of the following: physical machines, bare-metal servers, cloud servers, virtual machines, and containers within a cloud platform. The cloud platform contains a large amount of infrastructure resources owned by the cloud service provider. For example, the cloud platform deploys computing resources, storage resources, and network resources, and these computing resources can be a large number of computer devices (e.g., servers). The training device can utilize the infrastructure resources deployed on the cloud platform to implement the neural network training method provided in this application embodiment, thereby training the neural network.

[0045] At this time, as Figure 3 As shown in the embodiments of this application, the neural network training method can be abstracted by a cloud service provider on a cloud platform into a cloud service for training neural networks (hereinafter referred to as the training cloud service) and provided to users. After a user purchases the training cloud service on the cloud platform, the cloud platform can use a training device to provide the user with the cloud service for training neural networks. Furthermore, according to application requirements, the cloud platform can provide users with training cloud services of different service granularities. The implementation of the training cloud service is explained below using three service granularities as examples:

[0046] In the first feasible approach, after purchasing the training cloud service, the user sends the neural network to be trained to the cloud platform. The cloud platform then uses the training cloud service to train the neural network provided by the user and provides the user with the trained neural network.

[0047] In the second possible implementation, after purchasing the training cloud service, the user sends the neural network to be trained and the relevant user parameters of the neural network to the cloud platform. The cloud platform uses its development capabilities to develop a training code package for training the neural network based on the relevant user parameters and the neural network training method provided in the embodiments of this application. Then, the user uses the training code package to train the neural network and provides the trained neural network to the user.

[0048] In the third possible implementation, after purchasing the training cloud service, the user downloads the relevant code for implementing the neural network training method provided in the embodiments of this application to the user's terminal. Then, the user uses their own development capabilities to develop a training code package for training the neural network to be trained. The user then sends the neural network to be trained and the training code package to the cloud platform. The cloud platform uses its resources to run the training code package to train the neural network to be trained, and then provides the user with the trained neural network.

[0049] Optionally, in this embodiment, the cloud platform can be a central cloud platform, an edge cloud platform, or a cloud platform including both a central cloud and an edge cloud; this embodiment does not specifically limit its type. Furthermore, when the cloud platform includes both a central cloud and an edge cloud, the training device can be partially deployed in the edge cloud platform and partially deployed in the central cloud platform.

[0050] It should be understood that the above-described division and deployment methods of functional modules in the training device are merely illustrative examples. This application does not limit the division and deployment methods of functional modules in the training device. In actual applications, the deployment can be adapted according to the computing power of the computing device used to deploy the training device or the specific application requirements.

[0051] The implementation process of a neural network training method provided in this application embodiment will be described below. For example... Figure 4 As shown, the implementation process of the training method for this neural network may include the following steps:

[0052] Step 401: Obtain the training dataset and the neural network to be trained.

[0053] The training dataset and the neural network to be trained can be provided by the user. For example, Figure 5 As shown, users can provide training samples to construct and provide a neural network to be trained, so that the training device can train the neural network based on the training samples. The training dataset includes multiple training samples.

[0054] Step 402: Use multiple training samples in the training dataset to iteratively train the neural network to be trained. During the iterative training process: obtain the training samples input to the neural network to be trained in the Nth iteration, and the output results of the neural network to be trained for the training samples, where N is a positive integer.

[0055] The training process of a neural network involves multiple iterations. In the Nth iteration, the training device inputs training samples into the neural network. The network processes these samples based on its current parameters and outputs a result tailored to the training samples. Then, the training device uses these training samples and the output result to determine if the loss function in the Nth iteration has reached its optimal solution. If the loss function has not reached its optimal solution, the gradient of the loss function is determined based on the training samples and the output result of the Nth iteration. The parameters of the neural network are then adjusted based on this gradient to facilitate the (N+1)th iteration. Therefore, the training device can obtain the training samples from the Nth iteration and the neural network's output result for those training samples. Here, N is a positive integer.

[0056] Step 403: Based on the training samples and output results in the Nth iteration, determine the momentum coefficient used in the Nth iteration.

[0057] Optionally, when the neural network includes multiple hidden layers, the momentum coefficients used by different hidden layers can be the same or different during the same iteration process when training the neural network. Correspondingly, when different hidden layers use different momentum coefficients during the same iteration process, the process of determining the momentum coefficients used in the Nth iteration process based on the training samples and output results in the Nth iteration process includes: determining the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration process based on the training samples and output results in the Nth iteration process.

[0058] The following example illustrates the process of determining the momentum coefficient of a target hidden layer among multiple hidden layers in a neural network, using the example of different momentum coefficients used by different hidden layers in the same iteration. When different hidden layers use the same momentum coefficient in the same iteration, please refer to this implementation process for determining the momentum coefficient. Here, the target hidden layer is any one of the multiple hidden layers. Figure 6 As shown, its implementation process includes:

[0059] Step 4031a: Based on the training samples and output results in the Nth iteration, obtain the reference gradient of the target hidden layer in the Nth iteration.

[0060] The training device can obtain the value of the loss function in the Nth iteration based on the training samples and output results. Since the loss function is a function differentiable with respect to the parameters of the neural network, the reference gradient of the target hidden layer in the Nth iteration can be determined by taking the derivative of the loss function, based on the value of the loss function in the Nth iteration and the parameter values ​​of the target hidden layer in the Nth iteration.

[0061] Step 4031b: Obtain the reference gradient of the target hidden layer during the iteration process before the Nth iteration.

[0062] In each iteration, training samples are input into the neural network to be trained, and the neural network to be trained will generate output results based on the training samples. Therefore, the reference gradient for the corresponding iteration can be determined based on the training samples and output results in each iteration.

[0063] Step 4031c: Based on the reference gradient in the Nth and previous iterations of the target hidden layer, determine the momentum coefficient used by the target hidden layer in the Nth iteration.

[0064] During the Nth iteration, the accumulation level of the target hidden layer's parameters in the current gradient direction can be determined based on the reference gradient from the Nth and previous iterations. If the accumulation level is insufficient, the specified momentum coefficient is adjusted accordingly to obtain the momentum coefficient used by the target hidden layer in the Nth iteration. This allows for adjustments to the target hidden layer's parameters based on the momentum coefficient to achieve a suitable accumulation level in the current gradient direction. The specified momentum coefficient can be set according to requirements. For example, it can be set to 0.9 or 0.95 based on experience.

[0065] The degree of accumulation can be determined based on the reference gradient in the Nth iteration and earlier iterations of the target hidden layer. In one implementation, during the Nth iteration, the cumulative absolute value of the reference gradient in the (N-1)th iteration and earlier iterations of the target hidden layer can be determined, and its L2 norm (hereinafter referred to as the first L2 norm) can be calculated. Then, the L2 norm of the reference gradient in the Nth iteration (hereinafter referred to as the second L2 norm) is determined, and the ratio of the first L2 norm to the second L2 norm is calculated. If this ratio is large, it indicates that the accumulation in the current gradient direction is already large, and the accumulation in this direction can be reduced by using a momentum coefficient. If the ratio is small, it indicates that the accumulation in the current gradient direction is still small, and the accumulation in the current gradient direction can be increased by using a momentum coefficient. Furthermore, the adjustment effect of the momentum coefficient in this adjustment process is as follows: when the learning curve oscillates significantly during training, a larger momentum coefficient is used for adjustment; when the learning curve oscillates less significantly, a smaller momentum coefficient is used for adjustment. By controlling the oscillations, the training process is adjusted and accelerated.

[0066] Optionally, the cumulative absolute value L1(i) of the reference gradient in the i-th and previous iterations of the target hidden layer, the cumulative absolute value L1(i-1) of the reference gradient in the (i-1)-th iteration, the reference gradient T(i) of the target hidden layer in the i-th iteration, and the momentum coefficient M(i) used by the target hidden layer in the i-th iteration satisfy: L1(i) = L1(i-1) × M(i) + |T(i)|. Here, i is a positive integer, and |x| represents the absolute value of x.

[0067] Therefore, during the Nth iteration, a weighting coefficient can be determined based on the ratio of the first L2 norm to the second L2 norm of the target hidden layer. The result of adjusting the specified momentum coefficient using this weighting coefficient is then determined as the momentum coefficient used by the target hidden layer during the Nth iteration. Adjusting the specified momentum coefficient using this weighting coefficient includes multiplying the weighting coefficient by the specified momentum coefficient to determine the adjusted momentum coefficient.

[0068] Optionally, the process of determining the weighting coefficients includes: determining an adjustment coefficient based on a specified momentum coefficient and the degree of accumulation; and then adjusting the ratio of the first L2 norm to the second L2 norm of the target hidden layer based on the adjustment coefficient to obtain the weighting coefficients. In one possible implementation, adjusting the ratio based on the adjustment coefficient includes: adjusting the specified momentum coefficient based on the adjustment coefficient; and then adjusting the ratio based on the adjusted specified momentum coefficient. Furthermore, to reduce the difference in magnitude between the specified momentum coefficient and the ratio, the specified momentum coefficient can first be spatially mapped, then the mapped specified momentum coefficient can be adjusted using the adjustment coefficient, and then the adjusted specified momentum coefficient can be spatially inversely mapped to obtain the specified momentum coefficient adjusted according to the adjustment coefficient.

[0069] In one possible implementation, spatial mapping of the specified momentum coefficient includes: obtaining the natural logarithm of the specified momentum coefficient. Inverse spatial mapping of the adjusted specified momentum coefficient includes: obtaining the base-e exponent of the adjusted specified momentum coefficient. Adjusting the ratio based on the adjusted specified momentum coefficient includes: determining the ratio of the ratio to the adjusted specified momentum coefficient as a weighting coefficient.

[0070] By determining the momentum coefficients for different hidden layers, the momentum coefficients used by the hidden layers can be dynamically adjusted layer by layer during gradient descent iterations. This makes it easier for the training device to find a better optimization path for training the neural network, thereby enabling the neural network to achieve higher training accuracy.

[0071] It should be noted that after determining the momentum coefficient used in the Nth iteration based on the training samples and output results, this momentum coefficient can be directly used to train the neural network. Alternatively, after determining the momentum coefficient used in the Nth iteration based on the training samples and output results, the momentum coefficient determined based on the training samples and output results can be updated according to user instructions. The implementation process of updating the momentum coefficient according to user instructions is explained below, and for ease of distinction, the momentum coefficient before adjustment according to user instructions will be referred to as the reference momentum coefficient in the following text.

[0072] Optionally, the process of updating the momentum coefficient according to the user's instructions includes: when the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction, the reference momentum coefficient is determined as the momentum coefficient; when the reference momentum coefficient is outside the momentum coefficient range, the reference momentum coefficient is updated based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient.

[0073] In one possible implementation, when the reference momentum coefficient is outside the momentum coefficient range, the reference momentum coefficient is updated based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient. This includes: when the reference momentum coefficient is greater than the upper limit of the momentum coefficient range, the upper limit is determined as the momentum coefficient; when the reference momentum coefficient is less than the lower limit of the momentum coefficient range, the lower limit is determined as the momentum coefficient. Alternatively, when the reference momentum coefficient is outside the momentum coefficient range, other implementation methods can be used to update the reference momentum coefficient. For example, when the reference momentum coefficient is greater than the upper limit of the momentum coefficient range, the weighted sum of the reference momentum coefficient and the upper limit is determined as the momentum coefficient. This application does not specifically limit the implementation method.

[0074] By updating the reference momentum coefficient based on user-input-based adjustment instructions, human experience can be injected into the determination of the momentum coefficient, which can further improve the efficiency of training neural networks.

[0075] For example, step 403 can be performed by the optimizer, such as... Figure 5 As shown, after the optimizer determination module obtains the training samples and output results, it can determine the reference gradients of each hidden layer in the neural network during the Nth iteration based on these training samples and output results (e.g., ...). Figure 5 (g1 to gn), and calculate the cumulative absolute value of the reference gradient of each hidden layer in the N-1th iteration and earlier (e.g., g ...). Figure 5 From v1 to vn), and then based on the cumulative value corresponding to each hidden layer and the reference gradient in the Nth iteration, determine the momentum coefficients used by each hidden layer in the Nth iteration (e.g., ...). Figure 5 (m1 to mn). Furthermore, the user can provide adjustment instructions to the optimizer, which will then adjust the momentum coefficients (i.e., reference momentum coefficients) determined based on the accumulated values ​​and parameter gradients, to obtain the momentum coefficients used to train the neural network.

[0076] Step 404: Based on the momentum coefficient, update the parameters of the neural network to be trained during the Nth iteration.

[0077] Optionally, the reference gradient of the neural network to be trained can be updated based on the momentum coefficient during the Nth iteration, and the parameters of the neural network to be trained can be updated based on the updated gradient. For example, stochastic gradient descent can be used to train the neural network. That is, during the Nth iteration, the gradients used in the iterations before the Nth iteration are accumulated, and the accumulated gradient value is combined with the reference gradient of the Nth iteration to obtain the actual update amount for iterating the parameters of the neural network during the Nth iteration, i.e., the updated gradient is obtained, and the parameters of the neural network to be trained are updated using the updated gradient. Combining the accumulated gradient value with the reference gradient of the Nth iteration includes: determining the weighted accumulated value and the sum of the gradients of the Nth iteration. The weighting coefficient of the accumulated value is the momentum coefficient determined in step 403. In one possible implementation, the actual update amount during the Nth iteration = the reference gradient obtained from the loss function during the Nth iteration + the momentum coefficient × the accumulated value of the gradients used in the iterations before the Nth iteration.

[0078] In summary, in the neural network training method provided in this application embodiment, by determining the momentum coefficient used in the Nth iteration based on the training samples input to the neural network in the Nth iteration and the output results of the neural network in response to the training samples in the Nth iteration during the iterative training process of the neural network to be trained, and then updating the parameters of the neural network to be trained in the Nth iteration based on the momentum coefficient, it is possible to calculate the momentum coefficient used in the Nth iteration in each iteration, so that the parameters of the neural network are updated according to different momentum coefficients in different iterations. This is beneficial for finding a better optimization path for training the neural network and helps to improve the efficiency of training the neural network.

[0079] It should be noted that the order of steps in the neural network training method provided in this application embodiment can be appropriately adjusted, and steps can also be added or removed as needed. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0080] This application also provides a neural network training device. Please refer to the schematic diagram of the neural network training device. Figure 1 ,like Figure 1 As shown, the training device 70 for the neural network includes:

[0081] The acquisition module 101 is used to acquire the training dataset and the neural network to be trained, wherein the training dataset includes multiple training samples.

[0082] The acquisition module 101 is also used to iteratively train the neural network to be trained using multiple training samples in the training dataset. During the iterative training process, the module acquires the training samples input to the neural network to be trained in the Nth iteration and the output results of the neural network to be trained in response to the training samples in the Nth iteration, where N is a positive integer.

[0083] The determination module 102 is used to determine the momentum coefficient used in the Nth iteration based on the training samples and output results in the Nth iteration process.

[0084] Update module 103 is used to update the parameters of the neural network to be trained during the Nth iteration based on the momentum coefficient.

[0085] Optionally, the determining module 102 is specifically used to: determine the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration based on the training samples and output results in the Nth iteration process.

[0086] Optionally, the determining module 102 is specifically used for: obtaining the reference gradient of any hidden layer in the Nth iteration based on the training samples and output results in the Nth iteration; obtaining the reference gradient of any hidden layer in the iteration process before the Nth iteration; and determining the momentum coefficient used by any hidden layer in the Nth iteration based on the reference gradient of any hidden layer in the Nth and previous iteration processes.

[0087] Optionally, the determining module 102 is specifically used to: determine the reference momentum coefficient used in the Nth iteration based on the training samples and output results in the Nth iteration; and obtain the momentum coefficient based on the adjustment instructions input by the user and the reference momentum coefficient.

[0088] Optionally, the determining module 102 is specifically used to: determine the reference momentum coefficient as the momentum coefficient when the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction; and update the reference momentum coefficient based on the upper and lower limits of the momentum coefficient range when the reference momentum coefficient is outside the momentum coefficient range to obtain the momentum coefficient.

[0089] Optionally, the determining module 102 is specifically used to: determine the upper limit value as the momentum coefficient when the reference momentum coefficient is greater than the upper limit value of the momentum coefficient range; and determine the lower limit value as the momentum coefficient when the reference momentum coefficient is less than the lower limit value of the momentum coefficient range.

[0090] Optionally, the update module 103 is specifically used to: update the reference gradient of the neural network to be trained during the Nth iteration based on the momentum coefficient; and update the parameters of the neural network to be trained based on the updated gradient.

[0091] In summary, in the neural network training apparatus provided in this application embodiment, by determining the momentum coefficient used in the Nth iteration based on the training samples input to the neural network in the Nth iteration and the output results of the neural network in response to the training samples in the Nth iteration during the iterative training process of the neural network to be trained, and then updating the parameters of the neural network to be trained based on the momentum coefficient in the Nth iteration, the momentum coefficient used in the Nth iteration can be calculated in each iteration, so that the parameters of the neural network can be updated according to different momentum coefficients in different iterations. This is beneficial for finding a better optimization path for training the neural network and helps to improve the efficiency of training the neural network.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding content in the foregoing method embodiments, and will not be repeated here.

[0093] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 1 or Figure 2 The training apparatus shown can be deployed within this computer device. For example... Figure 7 As shown, the computer device 70 includes a memory 701, a processor 702, a communication interface 703, and a bus 704. The memory 701, processor 702, and communication interface 703 are interconnected via the bus 704. Furthermore, the computer device 70 may include multiple processors 702 to allow different processors to implement the functions of the aforementioned different functional modules.

[0094] The memory 701 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 701 can store executable code. When the executable code stored in the memory 701 is executed by the processor 702, the processor 702 and the communication interface 703 are used to execute the neural network training method provided in this embodiment. The memory 701 may also include other software modules and data required for running processes, such as an operating system. The operating system can be LINUX, UNIX, WINDOWS™, etc.

[0095] The processor 702 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits.

[0096] The processor 702 can also be an integrated circuit chip with signal processing capabilities. In implementation, some or all of the functions of the neural network training method of this application can be completed by the integrated logic circuits in the hardware of the processor 702 or by instructions in software form. The aforementioned processor 702 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 702 reads the information in memory 701 and combines it with its hardware to complete the neural network training method of this application embodiment.

[0097] The communication interface 703 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the computer device 70 and other devices or communication networks. For example, the communication interface 703 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, or other devices with network access capabilities.

[0098] Bus 704 may include a pathway for transmitting information between various components of computer device 70 (e.g., memory 701, processor 702, communication interface 703).

[0099] Each of the aforementioned computer devices 70 establishes a communication path through a communication network. Each computer device 70 is used to implement part of the functions of the neural network training method provided in the embodiments of this application. Any computer device 70 can be a computer device in a cloud data center (e.g., a server), or a computer device in an edge data center, etc.

[0100] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0101] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product providing data synchronization cloud services includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer device, they fully or partially implement the flow or function of the neural network training method provided in the embodiments of this application.

[0102] Computer equipment can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide data synchronization cloud services.

[0103] This application also provides a storage medium, which is a non-volatile computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the training method of the neural network provided in this application is implemented.

[0104] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute the neural network training method provided in this application.

[0105] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0106] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "multiple" refers to two or more, unless otherwise expressly defined.

[0107] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0108] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a neural network, characterized in that, The method includes: Obtain a training dataset and a neural network to be trained, wherein the training dataset includes multiple training samples, and the trained neural network to be trained is used for image or speech recognition or natural language translation. The neural network to be trained is iteratively trained using the multiple training samples in the training dataset. During the iterative training process, the training samples input to the neural network to be trained in the Nth iteration are obtained, and the output results of the neural network to be trained in response to the training samples in the Nth iteration are obtained, where N is a positive integer. Based on the training samples and output results in the Nth iteration process, the momentum coefficient used in the Nth iteration process is determined. The momentum coefficient is used to measure the contribution of the cumulative value of the gradient used in the iterations before the Nth iteration process to the actual update amount. Based on the momentum coefficient, the parameters of the neural network to be trained are updated during the Nth iteration.

2. The method according to claim 1, characterized in that, The determination of the momentum coefficient used in the Nth iteration based on the training samples and the output results in the Nth iteration includes: Based on the training samples and output results in the Nth iteration, the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration are determined respectively.

3. The method according to claim 2, characterized in that, The determination of the momentum coefficients used by each hidden layer of the neural network to be trained during the Nth iteration, based on the training samples and the output results during the Nth iteration, includes: Based on the training samples and the output results in the Nth iteration, obtain the reference gradient of any hidden layer in the Nth iteration; Obtain the reference gradient of any hidden layer during the iteration process prior to the Nth iteration; Based on the reference gradient in the Nth and previous iterations of any hidden layer, determine the momentum coefficient used by any hidden layer in the Nth iteration.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of the momentum coefficient used in the Nth iteration based on the training samples and the output results in the Nth iteration includes: Based on the training samples and the output results in the Nth iteration, the reference momentum coefficient used in the Nth iteration is determined. The momentum coefficient is obtained based on the adjustment instructions input by the user and the reference momentum coefficient.

5. The method according to claim 4, characterized in that, The process of obtaining the momentum coefficient based on the user-input adjustment instruction and the reference momentum coefficient further includes: When the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction, the reference momentum coefficient is determined as the momentum coefficient; When the reference momentum coefficient is outside the momentum coefficient range, the reference momentum coefficient is updated based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient.

6. The method according to claim 5, characterized in that, When the reference momentum coefficient is outside the momentum coefficient range, updating the reference momentum coefficient based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient includes: When the reference momentum coefficient is greater than the upper limit of the momentum coefficient range, the upper limit is determined as the momentum coefficient; When the reference momentum coefficient is less than the lower limit of the momentum coefficient range, the lower limit is determined as the momentum coefficient.

7. The method according to any one of claims 1 to 3, 5, and 6, characterized in that, The step of updating the parameters of the neural network to be trained during the Nth iteration based on the momentum coefficient includes: Based on the momentum coefficient, the reference gradient of the neural network to be trained is updated during the Nth iteration. The parameters of the neural network to be trained are updated based on the updated gradient.

8. A training device for a neural network, characterized in that, The device includes: The acquisition module is used to acquire a training dataset and a neural network to be trained, wherein the training dataset includes multiple training samples, and the trained neural network to be trained is used for image or speech recognition or natural language translation. The acquisition module is further configured to iteratively train the neural network to be trained using the multiple training samples in the training dataset. During the iterative training process, the module acquires the training samples input to the neural network to be trained during the Nth iteration and the output results of the neural network to be trained for the training samples during the Nth iteration, wherein N is a positive integer. The determination module is used to determine the momentum coefficient used in the Nth iteration based on the training samples and the output results in the Nth iteration process. The momentum coefficient is used to measure the contribution of the cumulative value of the gradient used in the iterations before the Nth iteration process to the actual update amount. An update module is used to update the parameters of the neural network to be trained during the Nth iteration based on the momentum coefficient.

9. The apparatus according to claim 8, characterized in that, The determining module is specifically used for: Based on the training samples and output results in the Nth iteration, the momentum coefficients used by each hidden layer of the neural network to be trained in the Nth iteration are determined respectively.

10. The apparatus according to claim 9, characterized in that, The determining module is specifically used for: Based on the training samples and the output results in the Nth iteration, obtain the reference gradient of any hidden layer in the Nth iteration; Obtain the reference gradient of any hidden layer during the iteration process prior to the Nth iteration; Based on the reference gradient in the Nth and previous iterations of any hidden layer, determine the momentum coefficient used by any hidden layer in the Nth iteration.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The determining module is specifically used for: Based on the training samples and the output results in the Nth iteration, the reference momentum coefficient used in the Nth iteration is determined. The momentum coefficient is obtained based on the adjustment instructions input by the user and the reference momentum coefficient.

12. The apparatus according to claim 11, characterized in that, The determining module is specifically used for: When the reference momentum coefficient is within the momentum coefficient range specified by the adjustment instruction, the reference momentum coefficient is determined as the momentum coefficient; When the reference momentum coefficient is outside the momentum coefficient range, the reference momentum coefficient is updated based on the upper and lower limits of the momentum coefficient range to obtain the momentum coefficient.

13. The apparatus according to claim 12, characterized in that, The determining module is specifically used for: When the reference momentum coefficient is greater than the upper limit of the momentum coefficient range, the upper limit is determined as the momentum coefficient; When the reference momentum coefficient is less than the lower limit of the momentum coefficient range, the lower limit is determined as the momentum coefficient.

14. The apparatus according to any one of claims 8 to 10, 12, and 13, characterized in that, The update module is specifically used for: Based on the momentum coefficient, the reference gradient of the neural network to be trained is updated during the Nth iteration. The parameters of the neural network to be trained are updated based on the updated gradient.

15. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores a computer program; when the processor executes the computer program, the computer device implements the method according to any one of claims 1 to 7.

16. A non-transient computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by a processor, the processor performs the method according to any one of claims 1 to 7.

Images