Parameter optimization method and related apparatus

Abstract

This application discloses a parameter optimization method and related apparatus for optimizing the parameters of a neural network in a multi-task learning scenario. The method is applied to a neural network including a backbone network and multiple branch networks. The output of the backbone network is connected to the input of each branch network. Based on this neural network, the method includes: calculating the first-order gradient based on the loss function of the neural network; calculating the Fisher information matrix of the backbone network based on the loss function of the backbone network; optimizing the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the backbone network; calculating the Fisher information matrix of the target branch network based on the loss function of the target branch network; and optimizing the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the target branch network.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a parameter optimization method and related apparatus. Background Technology

[0002] With the development of deep learning, it has demonstrated superior performance in many application areas, including image recognition, object detection, and natural language processing. The optimizer is a crucial component of deep learning; it can be understood as an optimization algorithm used to optimize the parameters of a neural network. These optimization algorithms include first-order and second-order algorithms, with second-order algorithms converging faster than first-order algorithms.

[0003] Optimizing neural network parameters can be categorized into two scenarios: single-task learning and multi-task learning. Single-task learning refers to a neural network learning a single task, while multi-task learning refers to a neural network learning multiple tasks. Because second-order optimization algorithms converge relatively quickly, they are typically used for parameter optimization in both single-task and multi-task learning scenarios.

[0004] In multi-task learning scenarios, neural networks typically consist of a backbone network and multiple branch networks. The output of the backbone network is the input of each branch network, and each branch network is used to learn one task.

[0005] Currently, for neural networks in the aforementioned multi-task learning scenarios, the process of optimizing neural network parameters using a second-order optimization algorithm is as follows: The backbone network and a branch network are considered as a sub-network; correspondingly, the neural network in a multi-task learning scenario can be considered as multiple sub-networks. A loss function is constructed for each sub-network, and then the parameters of each sub-network are optimized using a second-order optimization algorithm based on the loss function. Since constructing the loss function for each sub-network requires backward calculation from the sub-network's output along its hierarchical structure, the current parameter optimization method has a relatively high overhead. Summary of the Invention

[0006] This application provides a parameter optimization method and related apparatus for optimizing the parameters of a neural network in a multi-task learning scenario.

[0007] The first aspect of this application provides a parameter optimization method applied to a neural network including a backbone network and multiple branch networks. The output of the backbone network is the input of each of the multiple branch networks. This neural network can be a neural network for multi-task learning, such as BERT (Bidirectional Encoder Representations from Transformers) in the field of natural language processing or Domain-Adversarial Training of Neural Networks in the field of transfer learning. The method based on this neural network (DANN) includes: calculating the first-order gradient based on the loss function of the neural network. Specifically, during backpropagation, the first-order gradient of the backbone network is obtained by differentiating the loss function of the neural network with respect to the parameters. Backpropagation refers to calculating the intermediate variables of each layer of the neural network and the derivative of the loss function with respect to each parameter in the order from the output layer to the input layer. The intermediate variables can be the output values ​​of each layer of the neural network. The Fisher information matrix of the backbone network is calculated based on the loss function of the backbone network. Similar to the calculation of the first-order gradient, it also requires backpropagation and calculation of the Fisher information matrix of the backbone network based on the loss function of the backbone network. The parameters of the backbone network are optimized based on the first-order gradient and the Fisher information matrix of the backbone network. The Fisher information matrix is ​​required for the second-order optimization algorithm, which calculates the second-order gradient based on the Fisher information matrix and the first-order gradient, and then optimizes the parameters based on the second-order gradient.

[0008] The Fisher information matrix of the target branch network is calculated based on the loss function of the target branch network, where the target branch network is one of multiple branch networks. Similar to the calculation process of the first gradient, the Fisher information matrix of the target branch network is also calculated through backpropagation and based on the loss function of the target branch. The parameters of the target branch network are optimized based on the loss function of the neural network and the loss function of the target branch network.

[0009] There is no specific order among the loss functions for constructing the neural network, the backbone network, and the branch networks.

[0010] The neural network is divided into a backbone network and multiple branch networks, and loss functions for the neural network, the backbone network, and the branch networks are constructed. This allows for optimization of the backbone network parameters based on the loss functions of both the backbone and the neural network, and vice versa. Since optimizing the backbone network parameters requires calculating the Fisher information matrix based on its loss function, and this calculation involves obtaining the derivative of the backbone network's loss function, only the derivative of one loss function (excluding the first gradient) is needed to optimize the backbone network parameters. Existing methods require obtaining the derivatives of the loss functions of multiple sub-networks to optimize the backbone network parameters. Therefore, this embodiment requires obtaining fewer derivatives of the loss functions, reducing overhead. Furthermore, the more branch networks there are or the more iterations are performed during optimization, the more significant the overhead reduction effect of this embodiment becomes.

[0011] Furthermore, this embodiment of the application segments the neural network, allowing the parameter optimization process of the backbone network and the parameter update process of the branch networks to be performed in parallel, thereby improving the efficiency of parameter optimization. Moreover, existing methods require backpropagation from the branch networks to the backbone network; therefore, if existing methods are used, the accuracy of the backbone network's parameter updates is affected by the parameter update process of the branch networks, resulting in lower accuracy. In this embodiment, however, backpropagation can start from the backbone network, and the accuracy of the backbone network's parameter updates is not affected by the parameter update process of the branch networks, resulting in higher accuracy.

[0012] As one implementation method, before calculating the first-order gradient of the loss function based on the neural network, this method further includes: constructing the loss function of the neural network. Specifically, samples can be input into the neural network, and then the loss function of the neural network is calculated based on the output value of the neural network and the true label of the sample. The true label of the sample can be understood as the label used to manually label the sample, or it can be understood as the theoretical value of the output of the neural network with the sample as input. For example, in the field of image recognition, if an image is used as a sample, and this image is a picture of a dog, and the image of this dog is labeled as 'a' through manual labeling, then 'a' is the true label of the sample; if a neural network is used to recognize this image, the recognition result of the neural network will theoretically also be 'a'.

[0013] As one implementation method, the loss function of a neural network can be constructed by: constructing the loss function of the neural network based on the output of the neural network, wherein the output of the neural network includes the outputs of all branch networks.

[0014] This implementation provides a specific scheme for constructing the loss function of a neural network.

[0015] As one implementation method, before calculating the Fisher information matrix of the backbone network based on the loss function of the backbone network, the method also includes: constructing the loss function of the backbone network.

[0016] As one implementation method, the loss function for constructing the backbone network includes: constructing the loss function of the backbone network based on the output of the backbone network.

[0017] Because the neural network is segmented, in this implementation, the loss function of the backbone network is constructed based on the output of the backbone network.

[0018] As one implementation, before optimizing the parameters of the target branch network based on the first-order gradient and the Fisher information matrix of the target branch network, the method also includes: constructing the loss function of the target branch network.

[0019] As one implementation method, constructing the loss function of the target branch network includes: constructing the loss function of the target branch network based on the output of the target branch network, wherein the output of the target branch network is a part of the output of the neural network.

[0020] Because the neural network is segmented, in this implementation, the loss function of the target branch network is constructed based on the output of the target branch network.

[0021] As one implementation method, the loss function of the backbone network is constructed by: constructing the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network.

[0022] Since the Fisher Information Matrix (FILM) can be obtained by sampling from the model's predicted probability distribution, this implementation method can predict a reference value corresponding to each output of the backbone network based on the probability distribution satisfied by the backbone network's output. Then, the loss function of the backbone network can be calculated based on this reference value and the backbone network's output. This loss function can be used to calculate the Fisher Information Matrix, which is then used to optimize the parameters. It's important to understand that the sampled Fisher Information Matrix obtained from the model's predicted probability distribution is actually a sampled Fisher Information Matrix. In multi-task learning scenarios, the deviation between the sampled Fisher Information Matrix and the true Fisher Information Matrix is ​​small, so the sampled Fisher Information Matrix can be directly used as the Fisher Information Matrix. Therefore, this implementation method can improve the accuracy of parameter optimization.

[0023] As one implementation method, constructing the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network includes: taking at least one sample as the input of the neural network and obtaining the output of the backbone network. For example, in the field of image recognition, the data of a certain image is used as a sample input to the backbone network, and the processing result of the backbone network on the image data is the output of the backbone network; obtaining a reference value of the output of the backbone network based on the probability distribution satisfied by the output of the backbone network. This reference value is obtained based on sampling of the probability distribution, so it can also be called a sample value; and constructing the loss function of the backbone network based on the output of the backbone network and the reference value of the output of the backbone network.

[0024] At least one sample is input into the neural network to obtain the output of the backbone network. Based on the probability distribution satisfied by the output of the backbone network, a reference value of the output of the backbone network is obtained. Based on the output of the backbone network and the reference value of the output of the backbone network, the loss function of the backbone network is constructed, thereby realizing the construction of the loss function required for sampling the Fisher matrix.

[0025] As one implementation approach, before constructing the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network, the method includes: determining the probability distribution satisfied by the output of the backbone network based on the topology of the backbone network, wherein the topology of the backbone network includes the number of network layers, the number of neurons in each layer, and the way the neurons are interconnected; for example, for the BERT network used for multi-task learning, the multivariate normal distribution can be determined as the probability distribution satisfied by the output value of the backbone network based on the topology of its backbone network, and this probability distribution can be a multivariate normal distribution or a multinomial distribution, etc.

[0026] Since the output of the backbone network is related to the topology of the backbone network, determining the probability distribution satisfied by the output of the backbone network based on the topology of the backbone network can more accurately determine the probability distribution satisfied by the output of the backbone network.

[0027] As one implementation method, constructing the loss function of the target branch network includes: constructing the loss function of the target branch network based on the probability distribution satisfied by the output of the target branch network.

[0028] Since the Fisher information matrix can be obtained by sampling from the model's predicted probability distribution, this implementation method can predict the reference value corresponding to each output of the target branch network based on the probability distribution satisfied by the output of the target branch network. Then, the loss function of the target branch network can be calculated based on this reference value and the output of the target branch network. This loss function can be used to calculate the Fisher information matrix, which is then used to optimize the parameters. It's important to understand that the sampled Fisher matrix obtained from the model's predicted probability distribution is actually a sampled Fisher matrix. In multi-task learning scenarios, the deviation between the sampled Fisher matrix and the true Fisher information matrix is ​​small, so the sampled Fisher matrix can be directly used as the Fisher information matrix. Therefore, this implementation method can improve the accuracy of parameter optimization.

[0029] One implementation approach involves constructing the loss function of the target branch network based on the probability distribution satisfied by the output of the target branch network. This includes: using at least one sample as the input to the neural network to obtain the output of the target branch network. For example, in the field of image recognition, the data of a certain image is used as a sample input to the target branch network, and the processing result of the target branch network on the image data is the output of the target branch network; obtaining a reference value for the output of the target branch network based on the probability distribution satisfied by the output of the target branch network. This reference value is obtained based on sampling from the probability distribution, so it can also be called a sampled value; and constructing the loss function of the target branch network based on the output of the target branch network and the reference value of the output of the target branch network.

[0030] At least one sample is input into the neural network to obtain the output of the target branch network. Based on the probability distribution satisfied by the output of the target branch network, a reference value of the output of the target branch network is obtained. Based on the output of the target branch network and the reference value of the output of the target branch network, the loss function of the target branch network is constructed, thereby realizing the construction of the loss function required for sampling the Fisher matrix.

[0031] As one implementation, multiple branch networks are used to learn multiple tasks, and each branch network in the multiple branch networks is used to learn one task among the multiple tasks; before constructing the loss function of the target branch based on the probability distribution satisfied by the output of the target branch network, the method includes: determining the probability distribution satisfied by the output of the target branch network based on the type of task learned by the target branch network, which can be a multivariate normal distribution or a multinomial distribution, etc.

[0032] Since the output of the target branch network is related to the topology of the target branch network, and the topology of the target branch network reflects the type of task that the target branch network has learned, determining the probability distribution satisfied by the output of the target branch network based on the type of task that the target branch network has learned can more accurately determine the probability distribution satisfied by the output of the target branch network.

[0033] As one implementation method, determining the probability distribution satisfied by the output of the target branch network based on the type of task corresponding to the target branch network includes: based on the type of task corresponding to the target branch network being a classification task, using a multinomial distribution as the probability distribution satisfied by the output value of the target branch network, where the classification task can be a multi-class classification task or a binary classification task.

[0034] When the type of task corresponding to the target branch network is a classification task, the output value of the target branch network is discrete and follows a multinomial distribution. Therefore, using the multinomial distribution as the probability distribution satisfied by the output of the target branch network can ensure that the loss function constructed based on the probability distribution is more accurate.

[0035] As one implementation method, determining the probability distribution satisfied by the output of the target branch network based on the type of task corresponding to the target branch network includes: based on the type of task corresponding to the target branch network being a regression task, using a multivariate normal distribution as the probability distribution satisfied by the output of the target branch network.

[0036] Regression tasks involve predicting multiple independent continuous variables. According to the law of large numbers, the output of regression tasks follows a multivariate normal distribution. Therefore, when the task type is regression, using the multivariate normal distribution as the probability distribution satisfied by the output value of the target branch network can ensure that the loss function constructed based on the probability distribution is relatively accurate.

[0037] A second aspect of this application provides a parameter optimization apparatus applied to a neural network including a backbone network and multiple branch networks, wherein the output of the backbone network is the input of each of the multiple branch networks. The apparatus includes: a computation unit for calculating a first-order gradient based on a loss function of the neural network; a computation unit further for calculating a Fisher information matrix of the backbone network based on a loss function of the backbone network; an optimization unit for optimizing the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the backbone network; a computation unit further for calculating a Fisher information matrix of a target branch network based on a loss function of a target branch network, where the target branch network is one of the multiple branch networks; and an optimization unit further for optimizing the parameters of the target branch network based on the first-order gradient and the Fisher information matrix of the target branch network.

[0038] As one implementation, the device also includes: a first building unit for constructing the loss function of the neural network.

[0039] In one implementation, the device also includes a second building unit for constructing the loss function of the backbone network.

[0040] As one implementation method, the second building unit is also used to construct the loss function of the target branch network.

[0041] As one implementation, the first building unit is specifically used to construct the loss function of the neural network based on the output of the neural network, where the output of the neural network includes the outputs of all branch networks.

[0042] This implementation provides a specific scheme for constructing the loss function of a neural network.

[0043] As one implementation, the second building unit is used to construct the loss function of the backbone network based on the output of the backbone network.

[0044] As one implementation, the second building unit is used to construct the loss function of the target branch network based on the output of the target branch network, where the output of the target branch network is part of the output of the neural network.

[0045] As one implementation, the second building unit is used to construct the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network.

[0046] As one implementation method, the second building unit is used to obtain the output of the backbone network by taking at least one sample as the input of the neural network;

[0047] The reference value of the backbone network output is obtained based on the probability distribution satisfied by the backbone network output.

[0048] The loss function of the backbone network is constructed based on the output of the backbone network and the reference value of the output of the backbone network.

[0049] As one implementation, the second building unit is also used to determine the probability distribution satisfied by the output of the backbone network based on the topology of the backbone network.

[0050] As one implementation, the second building unit is also used to construct the loss function of the target branch network based on the probability distribution satisfied by the output of the target branch network.

[0051] As one implementation, the second building unit is also used to obtain the output of the target branch network by taking at least one sample as the input of the neural network;

[0052] The reference value corresponding to the output of the target branch network is obtained based on the probability distribution satisfied by the output of the target branch network.

[0053] The loss function of the target branch network is constructed based on the output of the target branch network and the reference value corresponding to the output of the target branch network.

[0054] As one implementation, multiple branch networks are used to learn multiple tasks, and each branch network in the multiple branch networks is used to learn one task among the multiple tasks; the second building unit is also used to determine the probability distribution satisfied by the output of the target branch network based on the type of task learned by the target branch network.

[0055] As one implementation, the second building unit is also used to take the multinomial distribution as the probability distribution satisfied by the output of the target branch network, based on the fact that the type of task learned by the target branch network is a classification task.

[0056] As one implementation method, the second building unit is also used to determine the type of task learned by the target branch network as a regression task, and to use the multivariate normal distribution as the probability distribution satisfied by the output of the target branch network.

[0057] For details on the implementation, related explanations, and technical effects of each of the above units, please refer to the description of the first aspect of the embodiments of this application.

[0058] A third aspect of this application provides a computer device, including: one or more processors and a memory; wherein the memory stores computer-readable instructions; the one or more processors read the computer-readable instructions to cause the computer device to perform the method as described in either the first or second aspect.

[0059] A fourth aspect of this application provides a computer-readable storage medium including computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any implementation of the first or second aspect.

[0060] A fifth aspect of this application provides a chip including one or more processors. Part or all of the processors are configured to read and execute a computer program stored in a memory to perform the methods in any possible implementation of the first or second aspect described above.

[0061] Optionally, the chip includes a memory, which is connected to the processor via a circuit or wire. Further optionally, the chip also includes a communication interface, to which the processor is connected. The communication interface is used to receive data and / or information that needs to be processed. The processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing result through the communication interface. The communication interface can be an input / output interface.

[0062] In some implementations, some of the processors may implement some steps of the above method through dedicated hardware. For example, the processing of neural network models may be implemented by a dedicated neural network processor or graphics processor.

[0063] The method provided in this application embodiment can be implemented by a single chip or by multiple chips working together.

[0064] A sixth aspect of this application provides a computer program product, which includes computer software instructions that can be loaded by a processor to implement the method described in either the first or second aspect. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the architecture of the second-order optimization training system in the embodiments of this application;

[0066] Figure 2 This is a schematic diagram of one embodiment of the neural network in this application.

[0067] Figure 3 This is a schematic diagram of one embodiment of the parameter optimization method in this application;

[0068] Figure 4 This is a schematic diagram of the first embodiment of constructing the loss function in this application.

[0069] Figure 5 This is a schematic diagram of a second embodiment of constructing the loss function in this application.

[0070] Figure 6 This is a schematic diagram of a third embodiment of constructing the loss function in this application.

[0071] Figure 7 This is a schematic diagram of the fourth embodiment of constructing the loss function in this application;

[0072] Figure 8 This is a schematic diagram of another embodiment of the neural network in this application;

[0073] Figure 9 This is a schematic diagram of the iterative calculation process in the embodiments of this application;

[0074] Figure 10 This is a flowchart illustrating the process of constructing the loss function of the backbone network and the loss functions of each branch network in the embodiments of this application.

[0075] Figure 11 This is a schematic diagram of one embodiment of the parameter optimization device in this application;

[0076] Figure 12 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0078] It should be understood that in the field of deep learning, the performance of a neural network primarily depends on the accuracy of its parameters, which can also be called weights. Therefore, during the training process of a neural network, optimization algorithms are used to perform multiple iterative calculations to optimize the network's parameters.

[0079] Common optimization algorithms can be divided into first-order optimization algorithms and second-order optimization algorithms.

[0080] A first-order optimization algorithm is an algorithm that uses the first-order gradients (also called first-order derivatives) of the parameters in a neural network to minimize or maximize the loss function; specifically, it can be implemented using the formula... Update the parameters, where θ is the parameter to be updated and η is the learning rate. It is the first gradient of the loss function with respect to the parameters.

[0081] The loss function, also known as the cost function, is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event.

[0082] Second-order optimization algorithms refer to algorithms that use the second-order gradients (also known as second-order derivatives) of the parameters in a neural network to minimize or maximize the loss function; specifically, they can be implemented using formulas. Update the parameters, where θ is the parameter to be updated and η is the learning rate. G is the first-order gradient of the loss function with respect to the parameters, and G is the second-order information matrix. -1 It is the inverse of the second-order information matrix. The second-order gradient is defined differently in different second-order optimization algorithms.

[0083] Compared to first-order optimization algorithms, second-order optimization algorithms add a second-order information matrix and converge faster. Therefore, second-order optimization algorithms are usually used to optimize the parameters of neural networks.

[0084] Common second-order optimization algorithms include Newton's method and the natural gradient method. In Newton's method, the Hessian matrix is ​​used as the second-order information matrix, which can also be called the Hessian matrix. In the natural gradient method, the Fisher information matrix is ​​used as the second-order information matrix, which can also be called the Fisher matrix. The Fisher matrix will be used in the following description of the embodiments of this application.

[0085] Currently, the most commonly used second-order optimization algorithm is the natural gradient method, which will be further explained below. In the natural gradient method, the Fisher matrix is ​​the covariance matrix obtained by differentiating the maximum likelihood function, which can be the loss function. However, the true Fisher matrix (i.e., the covariance matrix obtained by differentiating the maximum likelihood function) is difficult to obtain directly. Therefore, an empirical Fisher matrix or a sampled Fisher matrix is ​​usually used instead of the true Fisher matrix.

[0086] The empirical Fisher matrix is ​​a Fisher information matrix calculated using the probability distribution of the samples. Specifically, a loss function is calculated based on the output value of the neural network and the true labels of the samples, and then backpropagation is performed based on the loss function to obtain the empirical Fisher matrix.

[0087] The true label of a sample can be understood as the label used to manually label the sample, or as the theoretical value of the neural network's output when the sample is used as input. For example, in the field of image recognition, if an image is used as a sample, and this image is a picture of a dog, and the image of this dog is labeled as 'a' manually, then 'a' is the true label of the sample; if a neural network is used to recognize this image, the neural network's recognition result will theoretically also be 'a'.

[0088] Backpropagation refers to calculating the intermediate variables of each layer of the neural network and the derivative of the loss function with respect to each parameter in the order from the output layer to the input layer. The intermediate variables can be the output values ​​of each layer of the neural network.

[0089] The sampled Fisher matrix is ​​a Fisher information matrix obtained by sampling the predicted probability distribution of the model. Specifically, the output of the neural network is first predicted using the probability distribution satisfied by the output of the neural network to obtain a reference value for the output of the neural network. Then, the loss function is calculated based on the predicted reference value and the actual output value of the neural network. Finally, backpropagation is performed based on the loss function to obtain the sampled Fisher matrix.

[0090] It should be noted that the first-order gradient in the formula of the aforementioned second-order optimization algorithm is also obtained by backpropagation based on the loss function, and the loss function used to obtain the first-order gradient is the same as the loss function used to obtain the empirical Fisher matrix, while the loss function used to obtain the first-order gradient is different from the loss function used to obtain the sampled Fisher matrix.

[0091] Thus, if the empirical Fisher matrix is ​​used as the true Fisher matrix, then during parameter update, it is necessary to first perform backpropagation based on the loss function to obtain the first-order gradient, then calculate the empirical Fisher matrix based on the first-order gradient, and finally use the first-order gradient and the empirical Fisher matrix to update the parameters. However, if the sampled Fisher matrix is ​​used as the true Fisher matrix, then during parameter update, it is necessary to first perform backpropagation based on one loss function to obtain the first-order gradient, and then perform backpropagation based on another loss function to obtain the sampled Fisher matrix, and finally use the first-order gradient and the sampled Fisher matrix to update the parameters.

[0092] Based on the above explanation, if the empirical Fisher matrix is ​​used as the true Fisher matrix, the true labels of the samples can be used directly without predicting the output value of the neural network, and only one backpropagation is required. If the sampled Fisher matrix is ​​used as the true Fisher matrix, the output value of the neural network needs to be predicted before calculating the sampled Fisher matrix, and two backpropagations are required. Therefore, the overhead of calculating the sampled Fisher matrix is ​​greater.

[0093] Therefore, in single-task learning scenarios, since the empirical Fisher matrix is ​​equivalent to the sampled Fisher matrix, the empirical Fisher matrix is ​​usually used as the real Fisher matrix in order to reduce overhead.

[0094] In this context, the single-task learning scenario refers to the scenario in which a neural network is used to learn a single task. In this scenario, the neural network typically contains an input layer and an output layer.

[0095] However, the applicant discovered that in multi-task learning scenarios, the deviation between the empirical Fisher matrix and the true Fisher matrix is ​​large, while the deviation between the sampled Fisher matrix and the true Fisher matrix is ​​small. Therefore, the sampled Fisher matrix can be used as the true Fisher matrix to optimize the parameters in the neural network, thereby avoiding the poor optimization effect caused by using the empirical Fisher matrix as the true Fisher matrix, and improving the accuracy of the optimized parameters.

[0096] Multi-task learning scenarios refer to situations where neural networks are used to learn multiple tasks. In this scenario, the neural network typically includes a backbone network and multiple branch networks. The branch networks are used to learn multiple tasks, with each branch network learning one of the multiple tasks. For example, a neural network in a multi-task learning scenario may contain one backbone network and two branch networks, with the two branch networks used to learn classification and regression tasks, respectively.

[0097] However, the applicant also found that in multi-task learning scenarios, the overhead of using a second-order optimization algorithm to optimize the parameters of a neural network is relatively large, and if the sampled Fisher matrix is ​​used as the real Fisher matrix, the overhead will be further increased.

[0098] Therefore, this application provides a method for optimizing parameters in a neural network. This method belongs to the natural gradient method in second-order optimization algorithms and can be applied to multi-task learning scenarios. The method segments the neural network to obtain a backbone network and multiple branch networks. The output of the backbone network is the input of each branch network, and each branch network is used to learn one task. For the segmented neural network, loss functions for the backbone network and each branch network are constructed separately. Then, the parameters of the backbone network are optimized based on the loss function of the backbone network, and the parameters of the branch networks are optimized based on the loss functions of the branch networks, thereby reducing the overhead in the optimization process. The process of reducing overhead will be described in detail below.

[0099] To facilitate understanding, the following will be combined with... Figure 1 The process of optimizing parameters in a neural network using the method described in the embodiments of this application is introduced.

[0100] Specifically, the embodiments of this application can be applied to Figure 1 The second-order optimization training system shown includes a hardware layer, an operating system, and deep learning software. When the deep learning software runs, it performs the following steps.

[0101] The first step is to process the training data to obtain samples. This preprocessing can involve manually labeling the training data to obtain a sample label for each sample, thereby giving the training data a relatively stable and accurate feature description.

[0102] The second step is forward propagation. Forward propagation refers to calculating the intermediate variables of each layer of the neural network sequentially, from the input layer to the output layer. These intermediate variables can be the output values ​​of each layer. After forward propagation, the loss function for the entire neural network can be calculated by comparing the output values ​​(also called predicted values) with the true labels of the samples (which can be understood as manually labeled values).

[0103] The third step is backpropagation. Please refer to the previous explanations for an understanding of backpropagation.

[0104] The fourth step is to calculate the loss function piecewise. Specifically, as follows: Figure 1 As shown, the network topology used for multi-task learning is segmented to obtain multiple network segments; the output value of a specified layer network is obtained based on the forward propagation process, where the specified layer network refers to the output value of the output layer of each network segment in the multiple network segments; then, the loss function corresponding to each network segment in the multiple network segments is calculated based on the output value of the specified layer network.

[0105] The fifth step is to calculate the second-order information matrix. Specifically, the second-order information matrix for each network segment is calculated based on the loss function for that segment, and then the inverse of the second-order information matrix for each segment is calculated.

[0106] Step 6: Calculate the second-order gradient. Specifically, calculate the second-order gradient for each network segment based on the inverse of the second-order information matrix and the first-order gradient.

[0107] Step 7: Update the weights. Specifically, update the parameters of each network segment using the second-order gradient corresponding to each segment.

[0108] Steps one through seven constitute one iterative calculation. By repeating steps one through seven, multiple iterative calculations can be performed, thereby optimizing the parameters in the neural network.

[0109] Steps four through seven are described below through specific examples.

[0110] This application provides an embodiment of a parameter optimization method, which is applied to a neural network including a backbone network and multiple branch networks. The output of the backbone network serves as the input of each of the multiple branch networks. Specifically, the output of the backbone network can be connected to the input of each of the multiple branch networks.

[0111] The neural network can be a neural network used for multi-task learning, including but not limited to BERT (Bidirectional Encoder Representations from Transformers) in the field of natural language processing and Domain-Adversarial Training of Neural Networks (DANN) in the field of transfer learning.

[0112] When this neural network is used for multi-task learning, each branch of the network can be used to learn one task.

[0113] It should be noted that the backbone network and multiple branch networks can constitute the entire neural network or a part of the entire neural network; this application does not limit this.

[0114] Figure 2 This illustrates a scenario where a backbone network and multiple branch networks can constitute the entire neural network. For example... Figure 2 As shown, the neural network includes a backbone network and two branch networks. The backbone network includes an input layer and three hidden layers, and each of the two branch networks includes an output layer and two hidden layers. This neural network can be used to learn two tasks, such as... Figure 2 As shown, the two branch networks are used to learn subtask one and subtask two, respectively.

[0115] The output of the backbone network is connected to the input of the branch networks. This can be understood as follows: along the direction from the input layer to the output layer of the neural network, the last layer of the backbone network (also known as the output layer of the backbone network) is connected to the first layer of the branch network (also known as the input layer of the branch network); correspondingly, the output value of the backbone network is the input value of multiple branch networks.

[0116] by Figure 2 Taking the neural network shown as an example, connecting the output of the backbone network to the input of the respective networks means that, along the direction from the input to the output of the neural network, the fourth layer of the backbone network is connected to the first layer of the two branch networks respectively; correspondingly, the fourth layer of the backbone network serves as the input value of the two branch networks respectively.

[0117] Based on the aforementioned neural network, such as Figure 3 As shown, this embodiment includes:

[0118] Operation 101 constructs the loss function of the neural network, which is used to calculate the first-order gradient.

[0119] Specifically, samples can be input into a neural network, and the output value of the neural network can be obtained through forward propagation. Then, the loss function of the neural network can be calculated based on the output value of the neural network and the true label of the sample. This loss function refers to the loss function of the entire neural network composed of the backbone network and multiple branch networks.

[0120] By backpropagating through the loss function of a neural network, the first-order gradient required in a second-order optimization algorithm can be calculated.

[0121] Operation 102: Calculate the first-order gradient based on the loss function of the neural network.

[0122] It should be noted that in backpropagation, the first-order gradient of the backbone network can be obtained by differentiating the loss function of the neural network with respect to the parameters.

[0123] Operation 103: Construct the loss function of the backbone network. The loss function of the backbone network is used to calculate the Fisher information matrix of the backbone network.

[0124] It should be noted that if the empirical Fisher matrix is ​​taken as the real Fisher matrix, the loss function needs to be calculated based on supervised learning; specifically, the actual output value of the neural network is compared with the true label of the sample to calculate the loss function.

[0125] This application embodiment does not specifically limit the execution order between operation 103 and operation 101.

[0126] Since the empirical Fisher matrix deviates significantly from the true Fisher matrix, this embodiment uses the sampled Fisher matrix as the true Fisher matrix. Specifically, in this embodiment, the probability distribution satisfied by the output of the backbone network is first determined, then a reference value corresponding to the actual output value of the backbone network is determined based on the probability distribution, and then the actual output value of the backbone network is compared with the reference value corresponding to the actual output value of the backbone network to calculate the loss function of the backbone network.

[0127] Specifically, as a way of implementation, such as Figure 4 As shown, operation 103 includes:

[0128] Operation 201 determines the probability distribution satisfied by the output of the backbone network.

[0129] The probability distribution satisfied by the output of the backbone network can be understood as follows: for any data input to the backbone network, the output of the backbone network usually satisfies a certain probability distribution.

[0130] There are various methods to determine the probability distribution satisfied by the output values ​​of the backbone network, and this application does not specifically limit the methods.

[0131] As one implementation method, operation 201 includes: determining the probability distribution satisfied by the output values ​​of the backbone network based on the topology of the backbone network.

[0132] The topology of the backbone network includes the number of network layers, the number of neurons in each layer, and the way neurons are connected to each other.

[0133] For example, the topology of the BERT network used for multi-task learning is as follows: Figure 2 As shown, the topology based on the backbone network can determine the probability distribution satisfied by the output value of the backbone network based on the multivariate normal distribution.

[0134] Operation 202: Construct the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network.

[0135] It is understandable that the reference value corresponding to each output of the backbone network can be determined based on the probability distribution satisfied by the output of the backbone network, and then the loss function of the backbone network can be calculated based on the reference value and the output of the backbone network.

[0136] The following describes one way to implement the loss function for constructing the backbone network. Specifically, as follows... Figure 5 As shown, operation 202 may include:

[0137] Operation 301: Using at least one sample as input to the neural network, obtain the output of the backbone network.

[0138] Specifically, samples are input into a neural network to obtain the output value of the backbone network. For example, in the field of image recognition, the data of a certain image is used as a sample and input into the backbone network. The processing result of the backbone network on the image data is the output of the backbone network.

[0139] based on Figure 1 According to the relevant explanations, the output of the backbone network can be obtained during the forward propagation process.

[0140] One can input a single sample into a neural network to obtain the output value of the backbone network for that single sample, or multiple samples can be input into the neural network to obtain the output value of the backbone network for those multiple samples. To improve the accuracy of the constructed loss function, multiple samples are typically input into the neural network.

[0141] Operation 302: Obtain the reference value corresponding to the output of the backbone network based on the probability distribution satisfied by the output of the backbone network.

[0142] It is understandable that, since the output of the backbone network satisfies a certain probability distribution, the reference value corresponding to the output of the sample can be determined based on the probability distribution satisfied by the output of the backbone network. This reference value can also be called the sample value.

[0143] For example, input 10 samples into Figure 2 In the neural network shown, during the forward propagation process, the output values ​​of the backbone network for these 10 samples can be obtained by specifying the output of a layer, thus obtaining the... Figure 2 The output value of the 4th layer in the backbone network shown is for 10 samples; the sampled value corresponding to the output of these 10 samples can be obtained based on the probability distribution satisfied by the output value of the backbone network.

[0144] Operation 303 constructs the loss function of the backbone network based on the output of the backbone network and the reference value of the output of the backbone network.

[0145] In this embodiment, the reference value of the backbone network output is equivalent to the true label of the sample; the process of constructing the loss function based on the sample output value and the reference value of the backbone network output is the same as the process of constructing the loss function based on the sample output value and the true label of the sample, so it will not be described in detail here.

[0146] There can be various loss functions, and this application does not limit them in the embodiments; for example, the loss function can be the mean squared error (MSE) loss function.

[0147] Operation 104: Calculate the Fisher information matrix of the backbone network based on the loss function of the backbone network.

[0148] Based on formula The second-order information matrix can be calculated. In this embodiment, the second-order information matrix is ​​the Fisher information matrix; wherein, Let L(θ) = -logp(y|x, θ) represent the expectation, L(θ) = -logp(y|x, θ) represent the loss function, θ be the parameter that needs to be updated, and G be the second-order information matrix.

[0149] This application embodiment does not specifically limit the execution order between operation 104 and operation 102.

[0150] Operation 105 optimizes the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the backbone network.

[0151] Based on formula It is known that the gradient needs to be calculated in the process of calculating the second-order information matrix. The calculation process of this gradient is similar to that of the first-order gradient, and it also needs to be calculated through backpropagation. That is, in the process of optimizing the parameters of the backbone network, two backpropagations are required: one backpropagation is used to calculate the first-order gradient, and the other backpropagation is used to calculate the Fisher information matrix.

[0152] Based on the first-order gradient, the Fisher information matrix, and the formula mentioned above. This allows for the optimization of the backbone network parameters using a second-order optimization algorithm, details of which will not be elaborated here.

[0153] Operation 106: Construct the loss function of the target branch network. The loss function of the target branch network is used to calculate the Fisher information matrix of the target branch network, which is one of multiple branch networks.

[0154] This application embodiment does not specifically limit the execution order between operation 106 and operation 103, nor does it specifically limit the execution order between operation 106 and operation 101.

[0155] As can be seen from the relevant description of operation 103, since the empirical Fisher matrix deviates significantly from the true Fisher matrix, this embodiment of the application uses the sampled Fisher matrix as the true Fisher matrix. Specifically, the probability distribution satisfied by the output of the target branch network is first determined, and then a reference value corresponding to the actual output value of the target branch network is determined based on the probability distribution. The actual output value of the target branch network and the reference value corresponding to the actual output value of the target branch network are then compared to calculate the loss function of the target branch network.

[0156] Specifically, as a way of implementation, such as Figure 6 As shown, operation 106 includes:

[0157] Operation 401 determines the probability distribution satisfied by the output of the target branch network.

[0158] The probability distribution satisfied by the output of the target branch network can be understood as follows: for any data input to the target branch network, the output of the target branch network usually satisfies a certain probability distribution.

[0159] There are various methods to determine the probability distribution satisfied by the output value of the target branch network, and this application does not specifically limit this method.

[0160] As one implementation, multiple branch networks are used to learn multiple tasks, and each branch network in the multiple branch networks is used to learn one of the multiple tasks.

[0161] There can be various types of tasks, and this application does not specifically limit them in this embodiment; for example, if classified by type, tasks can include classification tasks and regression tasks.

[0162] Based on this, operation 401 may include:

[0163] Based on the type of task corresponding to the target branch network, determine the probability distribution that the output value of the target branch network satisfies.

[0164] It is understood that there can be various types of tasks, and therefore, there are multiple methods for determining the probability distribution satisfied by the output value of the target branch network for different task types. This application embodiment does not limit this. The following uses two specific task types to further explain operation 401.

[0165] As one implementation method, Operation 401 specifically includes:

[0166] Based on the fact that the task corresponding to the target branch network is a classification task, a multinomial distribution is used as the probability distribution satisfied by the output of the target branch network.

[0167] The classification task can be a multi-class classification task or a binary classification task. It can be understood that when the type of task corresponding to the target branch network is a classification task, the output value of the target branch network is discrete and follows a multinomial distribution. Therefore, the multinomial distribution can be used as the probability distribution satisfied by the output value of the target branch network.

[0168] Taking the BERT network, used for multi-task learning, as an example, the topology of the BERT network is as follows: Figure 2 As shown, the BERT network includes two branch networks. One of the branch networks learns the task of predicting whether the next sentence is an NSP (nextsentence prediction) task. This task is a binary classification task, so the binomial distribution (which belongs to the multinomial distribution) can be used as the probability distribution satisfied by the output value of the target branch network.

[0169] As another implementation method, Operation 401 specifically includes:

[0170] Since the task corresponding to the target branch network is a regression task, the multivariate normal distribution is used as the probability distribution satisfied by the output of the target branch network.

[0171] It is understandable that regression tasks involve predicting multiple independent continuous variables. According to the law of large numbers, the output of regression tasks follows a multivariate normal distribution. Therefore, if the task type is a regression task, the multivariate normal distribution can be used as the probability distribution satisfied by the output value of the target branch network.

[0172] For example, taking the BERT network used for multi-task learning as an example, another branch of the BERT network learns the task of masked language modeling (MLM), which is a regression task. Therefore, the multivariate normal distribution can be used as the probability distribution that the output value of the target branch network satisfies.

[0173] Operation 402: Construct the loss function of the target branch network based on the probability distribution satisfied by the output values ​​of the target branch network.

[0174] It is understandable that a reference value corresponding to each output of the target branch network can be determined based on the probability distribution satisfied by the output of the target branch network, and then the loss function of the target branch network can be calculated based on the reference value and the output of the target branch network.

[0175] The following describes one way to implement the loss function for constructing the target branch network. Specifically, as follows: Figure 7 As shown, operation 402 may include:

[0176] Operation 501: Using at least one sample as input to the neural network, obtain the output of the target branch network.

[0177] Specifically, samples are input into a neural network to obtain the output value of the target branch network. For example, in the field of image recognition, the data of a certain image is used as a sample and input into the target branch network. The processing result of the target branch network on the image data is the output of the target branch network.

[0178] based on Figure 1 According to the relevant explanations, the output of the target branch network can be obtained during the forward propagation process.

[0179] One can input a single sample into a neural network to obtain the output value of the target branch network for that single sample, or multiple samples can be input into the neural network to obtain the output value of the target branch network for that multiple samples. To improve the accuracy of the constructed loss function, multiple samples are usually input into the neural network.

[0180] Operation 502: Obtain the reference value corresponding to the output of the target branch network based on the probability distribution satisfied by the output of the target branch network.

[0181] It is understandable that, since the output of the target branch network satisfies a certain probability distribution, the reference value corresponding to the output of the sample can be determined based on the probability distribution satisfied by the output of the target branch network. This reference value can also be called the sampled value.

[0182] For example, input 10 samples into Figure 2 In the neural network shown, during the forward propagation process, the output values ​​of the target branch network for these 10 samples can be obtained by specifying the output of a layer. Figure 2 The output value of the fourth layer of the target branch network shown is given for 10 samples; the sampled values ​​corresponding to the output of these 10 samples can be obtained based on the probability distribution satisfied by the output value of the target branch network.

[0183] Operation 503 constructs the loss function of the target branch network based on the output of the target branch network and the reference value corresponding to the output of the target branch network.

[0184] In this embodiment, the reference value corresponding to the output of the target branch network is equivalent to the true label of the sample; the process of constructing the loss function based on the output value of the sample and the reference value corresponding to the output of the target branch network is the same as the process of constructing the loss function based on the output value of the sample and the true label of the sample, so it will not be described in detail here.

[0185] There can be various loss functions constructed through operation 603. This application does not limit the types of loss functions. The specific types of loss functions can be related to the task corresponding to the target branch network.

[0186] For example, if the task corresponding to the target branch network is a cloze test (masked language modeling, MLM) task, the constructed loss function can be the mean squared error (MSE) loss function; if the task corresponding to the target branch network is a next sentence prediction (NSP) task, the constructed loss function can be the cross-entropy loss function.

[0187] Operation 107: Calculate the Fisher information matrix of the target branch network based on the loss function of the target branch network.

[0188] Based on formula The second-order information matrix can be calculated. In this embodiment, the second-order information matrix is ​​the Fisher information matrix; wherein, Let L(θ) = -logp(y|x, θ) represent the expectation, L(θ) = -logp(y|x, θ) represent the loss function, θ be the parameter that needs to be updated, and G be the second-order information matrix.

[0189] This application embodiment does not specifically limit the execution order between operation 107 and operation 104, nor does it specifically limit the execution order between operation 107 and operation 102.

[0190] Operation 108 optimizes the parameters of the target branch network based on the first-order gradient and the Fisher information matrix of the target branch network.

[0191] Based on formula It is known that the gradient needs to be calculated in the process of calculating the second-order information matrix. The calculation process of this gradient is similar to that of the first-order gradient, and it also needs to be calculated through backpropagation. That is, in the process of optimizing the parameters of the target branch network, two backpropagations are required: one backpropagation is used to calculate the first-order gradient, and the other backpropagation is used to calculate the Fisher information matrix.

[0192] Based on the first-order gradient, the Fisher information matrix, and the formula mentioned above. This allows for the optimization of the parameters of the target branch network using a second-order optimization algorithm, which will not be detailed here.

[0193] In this embodiment, the neural network is divided into a backbone network and multiple branch networks, and loss functions for the neural network, the backbone network, and the branch networks are constructed. Thus, the parameters of the backbone network can be optimized based on the loss functions of the backbone network and the neural network, and the parameters of the branch networks can be optimized based on the loss functions of the branch networks and the neural network. Since optimizing the parameters of the backbone network requires calculating the Fisher information matrix of the backbone network based on its loss function, and this calculation involves obtaining the derivative of the backbone network's loss function, this embodiment only needs to obtain the derivative of one loss function, except for the first-order gradient.

[0194] If we don't segment the neural network, but instead consider the backbone network and a branch network as a subnetwork, then for a neural network containing multiple branch networks, there are multiple subnetworks, and a loss function can be constructed for each subnetwork. Based on multiple loss functions for multiple subnetworks, the parameters of the neural network can be optimized using the following two methods.

[0195] The process of optimizing the parameters of a neural network using the first method includes: optimizing the parameters of the branch networks in the subnetworks using the loss functions of the subnetworks; constructing a composite loss function for the backbone network based on the loss functions of multiple subnetworks; and optimizing the parameters of the backbone network using the composite loss function of the backbone network.

[0196] In the process of optimizing the parameters of the backbone network, it is necessary to use the composite loss function of the backbone network for backpropagation, that is, to obtain the derivative of the composite loss function of the backbone network. Since the composite loss function of the backbone network is composed of multiple loss functions of the sub-networks, in the process of optimizing the parameters of the neural network using the first method, in addition to the first gradient, it is also necessary to obtain the derivatives of multiple loss functions to achieve the optimization of the parameters of the backbone network.

[0197] The process of optimizing the parameters of the neural network using the second method includes: using the loss function of each sub-network to perform backpropagation on each sub-network to optimize the parameters of the neural network. That is, for each sub-network, backpropagation is performed once from the branch network to the backbone network according to the loss function.

[0198] During the backpropagation process for each subnetwork, it is necessary to obtain the derivative of the loss function corresponding to the subnetwork. Therefore, in the process of optimizing the parameters of the neural network using the second method, in addition to the first gradient, it is also necessary to obtain the derivatives of multiple loss functions to optimize the parameters of the backbone network.

[0199] As can be seen from the above description, if the first or second method is used to optimize the parameters of the neural network, it is necessary to obtain the derivatives of multiple loss functions to optimize the parameters of the backbone network; however, if the method provided in the embodiments of this application is used to optimize the parameters of the backbone network, it is only necessary to obtain the derivative of one loss function. Therefore, the method provided in the embodiments of this application can reduce the overhead.

[0200] To facilitate understanding, the following will be combined with... Figure 2 and Figure 8 The second method described above will be explained in detail, wherein, Figure 8 The topology of the neural network shown is similar to Figure 2 The neural networks shown have the same topology.

[0201] by Figure 8 Taking the neural network shown as an example, if the neural network is not segmented, the first branch network and the backbone network will be regarded as the first sub-network, and the loss function of the first sub-network will be constructed. Then, the loss function of the first sub-network will be used to backpropagate the first sub-network. During the backpropagation process, it is necessary to obtain the derivative of the loss function of the first sub-network.

[0202] Similarly, the second branch network and the backbone network are regarded as the second sub-network, and the loss function of the second sub-network is constructed. Then, the loss function of the second sub-network is used to backpropagate the first sub-network. During the backpropagation process, it is necessary to obtain the derivative of the loss function of the second sub-network.

[0203] Therefore, if the second method is used... Figure 8 To optimize the parameters of the neural network shown, it is necessary to obtain the derivatives of the loss functions of the two sub-networks in order to optimize the parameters of the backbone network.

[0204] However, if the segmentation method provided in the embodiments of this application is adopted, then as follows: Figure 2 As shown, the backbone network is backpropagated based on its loss function. During the backpropagation process, only the derivative of the backbone network's loss function needs to be obtained. Therefore, compared to the second method, the method of this embodiment of the application requires obtaining fewer derivatives of the loss function to update the parameters of the neural network, thereby reducing overhead.

[0205] The foregoing is an introduction to the parameter optimization method provided in the embodiments of this application. Using this method and through multiple iterative calculations, the parameters of a neural network can be optimized. The following section combines... Figure 9 An iterative calculation using the method provided in the embodiments of this application will be described.

[0206] In one iterative calculation, forward propagation can be performed first, followed by backpropagation. The optimization module obtains the loss function of the backbone network based on the network topology and the output values ​​of specified layers of the neural network, and then calculates the second-order information matrix based on the backbone network's loss function. Subsequently, the second-order gradient is calculated using the first-order gradient and the second-order information matrix calculated during backpropagation, and finally, the parameters are updated using the second-order gradient. In this embodiment, the second-order information matrix refers to the Fisher information matrix.

[0207] It should be noted that the loss function of the backbone network calculated in one iteration can be used in subsequent iterations or multiple iterations. For example, if the parameter update process of the neural network includes 20 iterations, the loss function of the backbone network can be calculated in the first iteration, and then the second-order information matrix can be calculated using this loss function. The second-order information matrix can be used for parameter updates in the first to fourth iterations. The loss function of the backbone network can be calculated in the fifth iteration, and then the second-order information matrix can be calculated using this loss function. The second-order information matrix can be used for parameter updates in the fifth to eighth iterations. And so on, until all 20 iterations are completed.

[0208] Similarly, in one iteration of computation, forward propagation can be performed first, followed by back propagation. Furthermore, the optimization module obtains the loss function of the target branch network based on the network topology of the neural network and the output values ​​of the specified layer of the neural network. Then, it calculates the second-order information matrix based on the loss function of the target branch network. After that, it calculates the second-order gradient using the first-order gradient and the second-order information matrix calculated during the back propagation process, and finally uses the second-order gradient to update the parameters.

[0209] In this embodiment, the second-order information matrix refers to the Fisher information matrix.

[0210] Based on the above description, the process of optimizing the parameters of a neural network using the method provided in this application embodiment can be summarized as follows: the neural network is divided into a backbone network and multiple branch networks, and the parameters of the backbone network and each branch network are optimized respectively, thereby decoupling the parameter optimization of the backbone network and the parameter optimization of the branch networks and improving the concurrency of the neural network training system.

[0211] The process of optimizing the parameters of the backbone network and each branch network includes: constructing the loss function of the backbone network and the loss function of each branch network respectively; calculating the Fisher information matrix of the backbone network based on the loss function of the backbone network, and updating the parameters of the backbone network using the Fisher information matrix of the backbone network; calculating the Fisher information matrix of the branch network based on the loss function of the branch network, and updating the parameters of the branch network using the Fisher information matrix of the branch network.

[0212] like Figure 10 As shown, the process of constructing the loss function of the backbone network and the loss functions of each branch network can be summarized as follows: the loss function of the backbone network and the loss function of each branch network are obtained by sampling the output of the specified layer network based on the network topology of the neural network.

[0213] Specifically, for the backbone network, the output value of the output layer of the backbone network is obtained, the sampled value corresponding to the output value of the backbone network is obtained, and the loss function is calculated based on the output value of the backbone network and the sampled value of the backbone network, so as to obtain the loss function of the backbone network through sampling.

[0214] For the branch network, obtain the output value of the output layer of the branch network, obtain the sampled value corresponding to the output value of the branch network, and calculate the loss function based on the output value and sampled value of the branch network, so as to obtain the loss function of the branch network through sampling.

[0215] To demonstrate the effectiveness of the method provided in the embodiments of this application, the applicant used the method, LAMB optimizer, and THOR optimizer provided in the embodiments of this application to optimize the parameters of the same neural network in two different hardware environments, and the results are compared in the table below.

[0216]

[0217] As shown in the table above, in the first hardware environment, the method provided in this application embodiment for parameter optimization requires fewer iterations than the LAMB optimizer and the THOR optimizer. Although the average iteration time of the method provided in this application embodiment for parameter optimization is the longest, the total training time of the method provided in this application embodiment for parameter optimization is the shortest.

[0218] In the first hardware environment, the parameter optimization method provided in this application requires fewer iterations than the parameter optimization using the LAMB optimizer and the THOR optimizer. Although the average iteration time of the parameter optimization method provided in this application is shorter than that of the LAMB optimizer and longer than that of the THOR optimizer, the total training time of the parameter optimization method provided in this application is the shortest.

[0219] Therefore, in this embodiment, since the sampled Fisher matrix is ​​used as the real Fisher matrix, the calculated second-order information matrix can be guaranteed to be more accurate, thereby improving the parameter optimization effect of the neural network and ensuring that the parameters of the neural network are more accurate. Furthermore, since the neural network is segmented, the number of backpropagations to the backbone network is reduced, reducing overhead and shortening the optimization time. In addition, the method of this embodiment belongs to the second-order optimization algorithm, which has fewer iterations and faster convergence speed than the first-order optimization algorithm, thus shortening the optimization time.

[0220] like Figure 11 As shown in the illustration, this application also provides a parameter optimization device applied to a neural network including a backbone network and multiple branch networks, wherein the output of the backbone network is the input of each of the multiple branch networks. The device includes:

[0221] The computing unit 601 is used to calculate the first gradient based on the loss function of the neural network;

[0222] The computing unit 601 is also used to calculate the Fisher information matrix of the backbone network based on the loss function of the backbone network;

[0223] The optimization unit 602 is used to optimize the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the backbone network.

[0224] The computing unit 601 is also used to calculate the Fisher information matrix of the target branch network based on the loss function of the target branch network, where the target branch network is one of multiple branch networks;

[0225] The optimization unit 602 is also used to optimize the parameters of the target branch network based on the first-order gradient and the Fisher information matrix of the target branch network.

[0226] As one implementation, the device also includes a first building unit 603 for building a loss function for the neural network.

[0227] In one implementation, the device also includes a second construction unit 604 for constructing the loss function of the backbone network.

[0228] As one implementation, the second building unit 604 is also used to construct the loss function of the target branch network.

[0229] As one implementation, the first building unit 603 is specifically used to construct the loss function of the neural network based on the output of the neural network, wherein the output of the neural network includes the outputs of all branch networks.

[0230] This implementation provides a specific scheme for constructing the loss function of a neural network.

[0231] As one implementation, the first building unit 604 is used to construct the loss function of the backbone network based on the output of the backbone network.

[0232] As one implementation, the first building unit 604 is used to construct the loss function of the target branch network based on the output of the target branch network, wherein the output of the target branch network is part of the output of the neural network.

[0233] As one implementation, the first building unit 604 is used to construct the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network.

[0234] As one implementation, the first building unit 604 is used to obtain the output of the backbone network by taking at least one sample as the input of the neural network;

[0235] The reference value of the backbone network output is obtained based on the probability distribution satisfied by the backbone network output.

[0236] The loss function of the backbone network is constructed based on the output of the backbone network and the reference value of the output of the backbone network.

[0237] As one implementation, the first building unit 604 is also used to determine the probability distribution satisfied by the output of the backbone network based on the topology of the backbone network.

[0238] As one implementation, the first building unit 604 is also used to construct the loss function of the target branch network based on the probability distribution satisfied by the output of the target branch network.

[0239] As one implementation, the first building unit 604 is also used to obtain the output of the target branch network by taking at least one sample as the input of the neural network;

[0240] The reference value corresponding to the output of the target branch network is obtained based on the probability distribution satisfied by the output of the target branch network.

[0241] The loss function of the target branch network is constructed based on the output of the target branch network and the reference value corresponding to the output of the target branch network.

[0242] As one implementation, multiple branch networks are used to learn multiple tasks, and each branch network in the multiple branch networks is used to learn one of the multiple tasks; the first building unit 604 is also used to determine the probability distribution satisfied by the output of the target branch network based on the type of task learned by the target branch network.

[0243] As one implementation, the first building unit 604 is also used to take the multinomial distribution as the probability distribution satisfied by the output of the target branch network, based on the fact that the type of task learned by the target branch network is a classification task.

[0244] As one implementation, the first building unit 604 is also used to take the multivariate normal distribution as the probability distribution satisfied by the output of the target branch network, based on the fact that the type of task learned by the target branch network is a regression task.

[0245] For details on the implementation, related explanations, and technical effects of each of the above units, please refer to the description of the first aspect of the embodiments of this application.

[0246] This application also provides a computer device, such as a terminal device. Figure 12 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0247] Figure 12 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 12 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a central processing unit (CPU) 1080, a graphics processing unit (GPU) 1090, and a power supply, among other components. Those skilled in the art will understand that... Figure 12 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0248] The following is combined with Figure 12 A detailed introduction to each component of a mobile phone:

[0249] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it for the CPU 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0250] The memory 1020 can be used to store software programs and modules. The CPU 1080 executes various functions and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0251] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the CPU 1080, and can also receive and execute commands sent by the CPU 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0252] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 1041. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the CPU 1080 to determine the type of touch event. Subsequently, the CPU 1080 provides corresponding visual output on the display panel 1041 based on the type of touch event. Although in Figure 12 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0253] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0254] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and output to the CPU 1080 for processing. The audio data is then transmitted via the RF circuit 1010 to, for example, another mobile phone, or output to the memory 1020 for further processing.

[0255] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 12 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0256] The CPU 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020, and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the CPU 1080 may include one or more processing units; preferably, the CPU 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the CPU 1080.

[0257] GPU1090 is mainly used to render data in memory 1020 into images and display them in display unit 1040.

[0258] The mobile phone also includes a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the CPU1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0259] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0260] In this embodiment of the application, the CPU 1080 included in the terminal may also have the following functions:

[0261] Construct the loss function for the backbone network;

[0262] The second-order information matrix corresponding to the backbone network is obtained based on the loss function of the backbone network.

[0263] The parameters of the backbone network are optimized based on the second-order information matrix corresponding to the backbone network.

[0264] In this embodiment of the application, the CPU 1080 included in the terminal may also have the following functions:

[0265] Construct the loss function for the target branch network, which is one of multiple branch networks;

[0266] The second-order information matrix corresponding to the target branch network is obtained based on the loss function of the target branch network;

[0267] The parameters of the target branch network are optimized based on the second-order information matrix corresponding to the target branch network.

[0268] This application also provides a chip including one or more processors. Part or all of the processors are used to read and execute computer programs stored in a memory to perform the methods of the foregoing embodiments.

[0269] Optionally, the chip includes a memory, which is connected to the processor via a circuit or wire. Further optionally, the chip also includes a communication interface, to which the processor is connected. The communication interface is used to receive data and / or information that needs to be processed. The processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing result through the communication interface. The communication interface can be an input / output interface.

[0270] In some implementations, some of the processors may implement some steps of the above method through dedicated hardware. For example, the processing of neural network models may be implemented by a dedicated neural network processor or graphics processor.

[0271] The method provided in this application embodiment can be implemented by a single chip or by multiple chips working together.

[0272] This application also provides a computer storage medium for storing computer software instructions for use with the aforementioned computer device, including instructions for executing programs designed for the computer device.

[0273] The computer device can be as described above. Figure 11 The described parameter optimization device.

[0274] This application also provides a computer program product, which includes computer software instructions that can be loaded by a processor to implement the processes in the methods shown in the foregoing embodiments.

[0275] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0276] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0277] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0278] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0279] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A parameter optimization method, characterized in that, The method, applied to a neural network comprising a backbone network and multiple branch networks, wherein the output of the backbone network is the input of each of the multiple branch networks, and the output of the backbone network is the processing result of the backbone network on image data, includes: The first-order gradient is calculated based on the loss function of the neural network; The Fisher information matrix of the backbone network is calculated based on the loss function of the backbone network. The parameters of the backbone network are optimized based on the first-order gradient and the Fisher information matrix of the backbone network. The Fisher information matrix of the target branch network is calculated based on the loss function of the target branch network, where the target branch network is one of the plurality of branch networks; The parameters of the target branch network are optimized based on the first-order gradient and the Fisher information matrix of the target branch network.

2. The method according to claim 1, characterized in that, Before calculating the Fisher information matrix of the backbone network based on the loss function of the backbone network, the method further includes: The loss function of the backbone network is constructed based on the probability distribution satisfied by the output of the backbone network.

3. The method according to claim 2, characterized in that, The loss function of the backbone network, constructed based on the probability distribution satisfied by the output of the backbone network, includes: Using at least one sample as input to the neural network, the output of the backbone network is obtained; The reference value of the output of the backbone network is obtained based on the probability distribution satisfied by the output of the backbone network. The loss function of the backbone network is constructed based on the output of the backbone network and the reference value of the output of the backbone network.

4. The method according to claim 2 or 3, characterized in that, Before constructing the loss function of the backbone network based on the probability distribution satisfied by the output of the backbone network, the method includes: The probability distribution satisfied by the output of the backbone network is determined based on the topology of the backbone network.

5. The method according to any one of claims 1 to 3, characterized in that, Before calculating the Fisher information matrix of the target branch network based on the loss function of the target branch network, the method further includes: The loss function of the target branch network is constructed based on the probability distribution satisfied by the output of the target branch network.

6. The method according to claim 5, characterized in that, The loss function for constructing the target branch network based on the probability distribution satisfied by the output of the target branch network includes: Using at least one sample as input to the neural network, the output of the target branch network is obtained; Based on the probability distribution satisfied by the output of the target branch network, obtain the reference value corresponding to the output of the target branch network; The loss function of the target branch network is constructed based on the output of the target branch network and the reference value corresponding to the output of the target branch network.

7. The method according to claim 5, characterized in that, The plurality of branch networks are used to learn multiple tasks, and each branch network in the plurality of branch networks is used to learn one of the multiple tasks; Before constructing the loss function of the target branch based on the probability distribution satisfied by the output of the target branch network, the method includes: Based on the type of task learned by the target branch network, the probability distribution satisfied by the output of the target branch network is determined.

8. The method according to claim 7, characterized in that, Determining the probability distribution satisfied by the output of the target branch network based on the type of task learned by the target branch network includes: Based on the fact that the type of task learned by the target branch network is a classification task, a multinomial distribution is used as the probability distribution satisfied by the output of the target branch network.

9. The method according to claim 7, characterized in that, Determining the probability distribution satisfied by the output of the target branch network based on the type of task learned by the target branch network includes: Since the type of task learned by the target branch network is a regression task, the multivariate normal distribution is used as the probability distribution satisfied by the output of the target branch network.

10. A parameter optimization device, characterized in that, An apparatus applied to a neural network comprising a backbone network and multiple branch networks, wherein the output of the backbone network is the input of each of the multiple branch networks, and the output of the backbone network is the processing result of the backbone network on image data, the apparatus comprising: A computational unit is used to calculate the first-order gradient based on the loss function of the neural network; The computing unit is also used to calculate the Fisher information matrix of the backbone network based on the loss function of the backbone network; An optimization unit is used to optimize the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the backbone network. The computing unit is also used to calculate the Fisher information matrix of the target branch network based on the loss function of the target branch network, wherein the target branch network is one of the plurality of branch networks; The optimization unit is further configured to optimize the parameters of the backbone network based on the first-order gradient and the Fisher information matrix of the target branch network.

11. A computer device, characterized in that, include: One or more processors and a memory; wherein the memory stores computer-readable instructions; The one or more processors read the computer-readable instructions to cause the computer device to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 9.

Images