Parameter updating method of neural network and related device

Abstract

The application provides a neural network parameter updating method and device, electronic equipment and a storage medium. The neural network parameter updating method comprises the following steps: initializing network parameters of a neural network and activation parameters of an activation function, the neural network comprising a plurality of network layers, and the network layers comprising at least the activation function; selecting training data from a training set and inputting the training data into the neural network to obtain output results of the network layers, and calculating error components of the network layers based on the output results, and taking a sum of all the error components as a cost function; updating the network parameters based on the cost function to complete one iteration training; repeatedly performing the iteration training, and updating the activation parameters of the network layers based on error components of the same network layers in adjacent two iteration trainings; and performing multiple iteration trainings on the neural network until a value of the cost function is less than a preset value or a predetermined number of iteration trainings is reached. The application can improve the speed and efficiency of parameter updating in the neural network training process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for updating parameters of a neural network. Background Technology

[0002] A neural network is a parametric model that simulates the brain's learning process and performs distributed parallel processing of information. By changing the parameters in a neural network, linear and nonlinear transformations can be applied to the original data to obtain higher-level and more abstract feature representations. Neural networks are widely used in research fields such as computer vision, speech recognition, and natural language processing.

[0003] Before using neural networks to solve practical problems, the backpropagation algorithm is usually used to train the neural network to update the values ​​of the parameters in the neural network. The backpropagation algorithm is a method of updating parameters using gradient descent on all parameters. However, when the number of layers in the neural network is deep, this method is prone to causing the parameter values ​​to fall into local minima and fail to obtain the global optimum. At the same time, as the number of training iterations increases, the convergence speed of the network slows down, and even gradient vanishing may occur, causing training to be interrupted. The speed and efficiency of parameter updates during neural network training are not high. Summary of the Invention

[0004] In view of the above, it is necessary to propose a parameter update method and related equipment for neural networks to solve the technical problem of how to improve the speed and efficiency of parameter update during neural network training. The related equipment includes a parameter update device for neural networks, electronic equipment, and storage medium.

[0005] This application provides a parameter update method for a neural network, the method comprising:

[0006] S10, Initialize the network parameters of the neural network and the activation parameters of all activation functions. The neural network includes multiple network layers, and each network layer includes at least the activation functions.

[0007] S11, randomly select at least one training data from the training set and input it into the neural network to obtain the output result of each network layer, and calculate the error component of each network layer based on the output result, and use the sum of all error components as the cost function. The training set includes multiple training data.

[0008] S12, Update the network parameters based on the cost function and backpropagation algorithm to complete one iteration of training;

[0009] S13, repeat steps S11 and S12, and update the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations;

[0010] S14, return to step S11, iteratively train the neural network until the value of the cost function is less than a preset value or the predetermined number of iterations is reached.

[0011] In some embodiments, the network layer includes a linear processing unit and an activation function, the activation function including activation parameters;

[0012] The activation function maps the output values ​​of linear processing units in the network layer to activation values, where the activation values ​​are the outputs of the network layer. The activation function satisfies the following relationship:

[0013]

[0014] Among them, O k-1 f is the output of network layer k-1. wk λ is a linear processing unit of network layer k. k f represents the activation parameters of the activation function in network layer k. Wk ( k-1 ) represents the output value of the linear processing unit in network layer k, O k This is the output of network layer k.

[0015] In some embodiments, the step of randomly selecting at least one training data point from the training set and inputting it into the neural network to obtain the output of each network layer, calculating the error component of each network layer based on the output, and using the sum of all error components as the cost function includes:

[0016] A1, obtain the Kth label data of the training data by labeling, wherein the Kth label data is the expected output of network layer K after the training data is input into the neural network, and network layer K is the last network layer of the neural network;

[0017] A2, Calculate the error components of network layer K based on the Kth label data and the output of network layer K. The relationship between the error components of network layer K is as follows:

[0018]

[0019] Where M is the number of randomly selected training data points. The output of network layer K after the i-th training data is input into the neural network. For the Kth label data corresponding to the i-th training data, Loss K This refers to the error component of the network layer K;

[0020] A3, calculate the (K-1)th label data based on the Kth label data and the network parameters of the network layer K, wherein the (K-1)th label data is the expected output of the network layer K-1 after the training data is input into the neural network, and the network layer K-1 is the network layer adjacent to the network layer K;

[0021] A4. Calculate the error components of network layer K-1 based on the K-1 label data and the output of network layer K-1;

[0022] A5. Repeat steps A3 to A4 to traverse all network layers in the neural network to calculate the error component of each network layer.

[0023] A6. Calculate the sum of the error components of all network layers as the cost function, which satisfies the following relationship:

[0024]

[0025] Among them, Loss k Let $\frac{k}{k}$ be the error component of the k-th network layer, $K$ be the total number of network layers, and $Loss$ be the cost function.

[0026] In some embodiments, calculating the (K-1)th label data based on the Kth label data and the network parameters of the network layer K includes:

[0027] The linear processing unit of network layer K is obtained based on the network parameters of network layer K.

[0028] The Kth label data is input into the inverse function of the linear processing unit of the network layer K to obtain the (K-1)th label data, which satisfies the following relation:

[0029]

[0030] in, Linear processing unit of network layer K inverse function, Let λ be the Kth label data corresponding to the i-th training data. K These are the activation parameters in the activation function of the network layer K. Let K be the (K-1)th label data corresponding to the i-th training data.

[0031] In some embodiments, updating the network parameters based on the cost function and the backpropagation algorithm includes:

[0032] The gradient of any network parameter in the neural network is calculated based on the cost function.

[0033] The network parameters are updated along the gradient descent direction based on the gradient and the preset learning rate, and the update process satisfies the following relationship:

[0034]

[0035] Where η is the preset learning rate, Let θ be the gradient of the network parameter. Let θ1 and θ2 be the values ​​of the network parameter θ before and after the update, respectively, representing the direction of the gradient descent.

[0036] In some embodiments, updating the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations includes:

[0037] Use any network layer as the target layer;

[0038] Calculate the change in the error component of the target layer between two consecutive training iterations, where the change in the error component satisfies the following relationship:

[0039]

[0040] in, This refers to the error component of the target layer m in the first iteration of the two training iterations. Let ΔE() be the error component of the target layer m in the latter iteration of the two training iterations, and let ΔE() be the change in the error component of the target layer m.

[0041] If ΔE() is less than the first threshold and If the value is greater than the second threshold, the activation parameter of the activation function in the target layer is increased according to the preset rules; otherwise, the activation parameter of the activation function in the target layer is set to 1.

[0042] Iterate through all network layers to update the activation parameters of the activation function in each network layer.

[0043] In some embodiments, increasing the value of the activation parameter of the activation function in the target layer according to a preset rule includes:

[0044] The activation parameter is increased by adding a preset step size to the value of the activation parameter.

[0045] This application embodiment also provides a parameter update device for a neural network, the device comprising:

[0046] An initialization unit is used to initialize the network parameters of the neural network and the activation parameters of all activation functions. The neural network includes multiple network layers, and each network layer includes at least the activation functions.

[0047] The cost function unit is used to randomly select at least one training data from the training set and input it into the neural network to obtain the output result of each network layer, and calculate the error component of each network layer based on the output result, and use the sum of all error components as the cost function. The training set includes multiple training data.

[0048] The first update unit is used to update the network parameters based on the cost function and the backpropagation algorithm to complete one iteration of training.

[0049] The second update unit is used to repeat the cost function unit and the first update unit, and update the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations.

[0050] An iterative training unit is used to return the cost function unit and iteratively train the neural network until the value of the cost function is less than a preset value or the predetermined number of iterations is reached.

[0051] This application embodiment also provides an electronic device, the electronic device comprising:

[0052] Memory, storing at least one instruction;

[0053] The processor executes instructions stored in the memory to implement the parameter update method of the neural network.

[0054] This application also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the parameter update method of the neural network.

[0055] In summary, this application sets an activation function with activation parameters at the end of each network layer of the neural network. During training, the network parameters of each network layer are updated using the backpropagation algorithm, and the activation parameters of the activation function in each network layer are updated based on the error components of the same network layer in two adjacent training iterations. This effectively increases the range of the saturation region where the gradient is not zero in the activation function, ensuring that the network parameters of each network layer can be updated, and improving the speed and efficiency of parameter updates during neural network training. Attached Figure Description

[0056] Figure 1 This is a flowchart of a preferred embodiment of the parameter update method for the neural network involved in this application.

[0057] Figure 2 This is a schematic diagram of the activation functions corresponding to the different activation parameters involved in this application.

[0058] Figure 3 This is a functional block diagram of a preferred embodiment of the parameter update device for the neural network involved in this application.

[0059] Figure 4 This is a schematic diagram of the structure of an electronic device that is a preferred embodiment of the parameter update method for the neural network involved in this application. Detailed Implementation

[0060] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0061] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] This application provides a parameter update method for a neural network, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0064] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0065] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0066] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0067] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of a neural network parameter update method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The neural network parameter update method provided in this application can be applied to any scenario requiring neural network training, and to products applicable to these scenarios, such as using neural networks for speech recognition or image classification in the financial insurance field.

[0068] S10, Initialize the network parameters of the neural network and the activation parameters of all activation functions. The neural network includes multiple network layers, and each network layer includes at least the activation functions.

[0069] In an optional embodiment, the neural network consists of multiple network layers connected in series, with the output of the current network layer serving as the input of the next network layer. The neural network can be a fully connected neural network, a convolutional neural network, or a recurrent neural network; this application does not impose any limitations. The network parameters include the trainable parameters of each network layer in the neural network, which are related to the specific structure of the neural network. For example, if the neural network is a fully connected neural network, then the network parameters are the weights and biases of each fully connected layer in the fully connected neural network; if the neural network is a convolutional neural network, then the network parameters are the parameters of the convolutional kernel of each convolutional layer in the convolutional neural network.

[0070] In an optional embodiment, the network layer includes linear processing units and an activation function, the activation function including activation parameters; the activation function is used to map the output values ​​of the linear processing units in the network layer to activation values, the activation values ​​being the output of the network layer, and the activation function satisfies the following relation:

[0071]

[0072] Among them, O k-1 This is the output of network layer k-1. λ is a linear processing unit of network layer k. k Let be the activation parameters of the activation function in network layer k. For the output value of the linear processing unit in network layer k, O k This is the output of network layer k. It should be noted that... With network parameters W of network layer k k This is related to the training process of the neural network, when the network parameters W... k Once the value of is determined, the linear processing unit of network layer k can be determined. In an optional embodiment, the parameters of the neural network include the network parameters and the activation parameters of all activation functions. The network parameters of the neural network and the activation parameters of all activation functions are initialized. Existing initialization methods such as Kemming initialization or random initialization can be used for the network parameters of the neural network; this application is not limited to these methods. For the activation parameters of all activation functions, all activation parameters can be set to 1 to complete the initialization of the activation parameters of all activation functions.

[0073] For example, in an image classification scenario, the neural network can be an existing convolutional neural network such as ResNet or MobileNet. The network layers of this convolutional neural network include convolutional layers and fully connected layers, and the network parameters include trainable parameters for each convolutional layer and each fully connected layer. In the convolutional layer k, linear processing units... For convolution operations, the network parameters W k The parameters are the convolution kernel parameters; in the fully connected layer k, the linear processing unit... For the weighted summation operation, the network parameter W k The weights and biases are used for weighted summation. The network parameters are initialized using Kemin initialization, and the activation parameters in the activation functions at the end of each network layer are set to 1 to complete the initialization.

[0074] In this way, the initialization of network parameters and all activation parameters in the neural network is completed, and initial values ​​are assigned to the network parameters and all activation parameters, providing a basis for the training of the neural network.

[0075] S11, randomly select at least one training data from the training set and input it into the neural network to obtain the output result of each network layer, and calculate the error component of each network layer based on the output result, and use the sum of all error components as the cost function. The training set includes multiple training data.

[0076] In an optional embodiment, the step of randomly selecting at least one training data point from the training set and inputting it into the neural network to obtain the output of each network layer, calculating the error component of each network layer based on the output, and using the sum of all error components as the cost function includes:

[0077] A1, obtain the Kth label data of the training data by labeling, wherein the Kth label data is the expected output of network layer K after the training data is input into the neural network, and network layer K is the last network layer of the neural network;

[0078] A2, Calculate the error components of network layer K based on the Kth label data and the output of network layer K. The relationship between the error components of network layer K is as follows:

[0079]

[0080] Where M is the number of randomly selected training data points. The output of network layer K after the i-th training data is input into the neural network. For the Kth label data corresponding to the i-th training data, Loss K This refers to the error component of the network layer K;

[0081] A3, calculate the (K-1)th label data based on the Kth label data and the network parameters of the network layer K, wherein the (K-1)th label data is the expected output of the network layer K-1 after the training data is input into the neural network, and the network layer K-1 is the network layer adjacent to the network layer K;

[0082] A4. Calculate the error components of network layer K-1 based on the K-1 label data and the output of network layer K-1;

[0083] A5. Repeat steps A3 to A4 to traverse all network layers in the neural network to calculate the error component of each network layer.

[0084] A6. Calculate the sum of the error components of all network layers as the cost function, which satisfies the following relationship:

[0085]

[0086] Among them, Loss k Let $\frac{k}{k}$ be the error component of the k-th network layer, $K$ be the total number of network layers, and $Loss$ be the cost function.

[0087] It should be noted that the smaller the value of the cost function, the closer the output of the neural network is to the expected output, and the more accurate the output of the neural network.

[0088] In this optional embodiment, the training data and the Kth label data are related to the application scenario of the neural network. For example, if the application scenario of the neural network is an image classification task, then the training data and the label data are the image to be classified and the classification result of the image to be classified, respectively; if the application scenario of the neural network is a regression analysis task, then the training data and the label data are the data to be analyzed and the regression result of the data to be analyzed, respectively.

[0089] In this optional embodiment, the training data is input into the neural network to obtain the output of the first network layer. Since the neural network is composed of multiple network layers connected in series, the output of the first network layer is used as the input of the second network layer to obtain the output of the second network layer. The output of the second network layer satisfies the following relationship:

[0090]

[0091] Where O1 is the output of the first network layer, f W2 This represents a linear processing unit in the second network layer. The output of the current network layer is continuously used as the input to the next network layer until the output of each network layer in the neural network is obtained. The output of the last network layer K is the final output of the neural network.

[0092] For example, in an image classification scenario, the training data is the image data to be classified, and the true classification result of the image data to be classified is used as the Kth label. The neural network is a convolutional neural network, and its network layers include convolutional layers and fully connected layers. The image data to be classified is input into the convolutional neural network, and the first convolutional layer uses a convolutional kernel of a preset size to perform a convolution operation on the image data Y0 to be classified to obtain the first image feature. The first image feature is input into an activation function with activation parameter λ1 to obtain the output Y1 of the first convolutional layer. Output Y1 characterizes the activation degree of local features at different locations in the image data for the classification result. Feature extraction is continuously performed on the image data using cascaded convolutional layers, and the output of the last convolutional layer is used as the input to a fully connected layer. The fully connected layer processes the output of the last convolutional layer using linear processing units and an activation function to obtain its output. The output of the fully connected layer characterizes the activation degree of global features in the image data for the classification result, and is the predicted classification result for the image data. The Kth label data is used as the expected output of network layer K. The error component of network layer K represents the error between the predicted classification result and the expected result; a smaller error component indicates a more accurate predicted classification result. Similarly, the error component of each network layer reflects the error between the output result and the expected result; a smaller sum of all error components indicates a more accurate classification result output by the neural network.

[0093] In an optional embodiment, calculating the (K-1)th label data based on the Kth label data and the network parameters of network layer K includes:

[0094] The linear processing unit of network layer K is obtained based on the network parameters of network layer K.

[0095] The Kth label data is input into the inverse function of the linear processing unit of the network layer K to obtain the (K-1)th label data, which satisfies the following relation:

[0096]

[0097] in, Linear processing unit of network layer K inverse function, Let λ be the Kth label data corresponding to the i-th training data. K These are the activation parameters in the activation function of the network layer K. Let K be the (K-1)th label data corresponding to the i-th training data.

[0098] It should be noted that when the input of the network layer K is the (K-1)th label data Y K-1 At that time, the output of the network layer K is the Kth label data Y. K , i.e. Y K-1 and Y K The following relationship is satisfied between them:

[0099]

[0100] During the training process, Y K , λ K Since all of these are known quantities, the K-1th label data can be calculated based on the above relationship.

[0101] Thus, after inputting the training data into the neural network, the error component of each network layer is calculated based on the output result and expected output of each network layer in the neural network, and the sum of the error components of all network layers is calculated to obtain the cost function, which is used to constrain the training of the neural network.

[0102] S12, update the network parameters based on the cost function and backpropagation algorithm to complete one iteration of training.

[0103] In an optional embodiment, the backpropagation algorithm can optimize the values ​​of network parameters in the neural network using gradient descent, so that the values ​​of the network parameters are updated in the direction of decreasing the cost function.

[0104] In an optional embodiment, updating the network parameters based on the cost function and the backpropagation algorithm includes:

[0105] The gradient of any network parameter in the neural network is calculated based on the cost function.

[0106] The network parameters are updated along the gradient descent direction based on the gradient and the preset learning rate, and the update process satisfies the following relationship:

[0107]

[0108] Where η is the preset learning rate, Let θ be the gradient of the network parameter. Let θ1 and θ2 be the values ​​of the network parameter θ before and after the update, respectively, and let θ1 and θ2 be the directions of the gradient descent. The preset learning rate η is 0.01.

[0109] In this optional embodiment, the gradient of the network parameter θ can be obtained by taking the partial derivative of the cost function with respect to the network parameter θ. The gradient of the network parameter θ satisfies the following relationship:

[0110]

[0111] In this optional embodiment, the backpropagation algorithm is used to update each network parameter in the neural network along the direction of gradient descent to complete one iteration of training. The direction of gradient descent is the direction in which the cost function decreases.

[0112] In this way, the gradient of each network parameter in each neural network is calculated based on the cost function, and each network parameter is updated along the gradient descent direction, so that the value of the cost function is reduced, thus completing one iteration of training.

[0113] S13, repeat steps S11 and S12, and update the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations.

[0114] In an optional embodiment, the network parameters in the neural network are updated once after each iteration of training. Steps S11 and S12 are repeated to complete another iteration of training, and the activation parameters of the activation function in each network layer are updated based on the error components of the same network layer in two adjacent iterations of training.

[0115] In an optional embodiment, updating the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations includes:

[0116] Use any network layer as the target layer;

[0117] Calculate the change in the error component of the target layer between two consecutive training iterations, where the change in the error component satisfies the following relationship:

[0118]

[0119] in, This refers to the error component of the target layer m in the first iteration of the two training iterations. Let ΔE() be the error component of the target layer m in the latter iteration of the two training iterations, and let ΔE() be the change in the error component of the target layer m.

[0120] If ΔE() is less than the first threshold and If the value is greater than the second threshold, the activation parameter of the activation function in the target layer is increased according to the preset rules; otherwise, the activation parameter of the activation function in the target layer is set to 1.

[0121] Iterate through all network layers to update the activation parameters of the activation function in each network layer.

[0122] Wherein, the first threshold is a constant close to 0, with a value of 0.001, and the second threshold has a value of 0.005; if ΔE() is less than the first threshold, then ΔE(m) is considered close to 0. If the error component is greater than the second threshold, it is considered to be large. The larger the value, the greater the error between the output of the target layer m and the expected output, and the less accurate the output of the neural network. ΔE() can reflect the speed of parameter updates in the neural network. The smaller the value of ΔE(), the slower the parameter updates. When ΔE(m) = 0, it means that the parameters in the neural network have stopped updating.

[0123] In this optional embodiment, please refer to Figure 2 , Figure 2 This is a schematic diagram of the activation functions corresponding to different activation parameters involved in this application. As the activation parameter gradually increases, the range of the saturation region in the corresponding activation function curve gradually expands, while the range of the flat region gradually shrinks. The saturation region is the region in the curve where the gradient is not zero, and the flat region is the region in the curve where the gradient is zero. When ΔE() is less than the first threshold and When the value is greater than the second threshold, it indicates that the value of the network parameter falls within the flat region, and the gradient of the network parameter is 0, resulting in gradient vanishing. This prevents the network parameter from being updated. Therefore, it is necessary to increase the value of the activation parameter according to the preset rules to increase the range of the saturation region, so that the value of the network parameter falls within the saturation region where the gradient is not 0. This can effectively solve the gradient vanishing problem, ensure that the network parameter can be updated, and improve the speed and efficiency of network parameter updates during neural network training.

[0124] In an optional embodiment, increasing the activation parameter of the activation function in the target layer according to a preset rule includes: adding a preset step size to the activation parameter to increase the activation parameter of the activation function in the target layer. The preset step size is 1.

[0125] In this way, the activation parameters of the activation function in each network layer are updated based on the error components of the same network layer in two adjacent training iterations. This effectively increases the range of the saturation region where the gradient is not zero in the activation function, ensuring that the network parameters of each network layer can be updated, and improving the speed and efficiency of parameter updates during neural network training.

[0126] S14, return to step S11, iteratively train the neural network until the value of the cost function is less than a preset value or the predetermined number of iterations is reached.

[0127] In an optional embodiment, the preset value is 0.005, and the predetermined number of training iterations is 2000. When the value of the cost function is less than the preset value or the predetermined number of training iterations is reached, the parameter update of the neural network is stopped, and the trained neural network is obtained.

[0128] In this way, the network parameters and activation parameters of all activation functions in the neural network are updated, the training process of the neural network is completed, and the trained neural network is obtained.

[0129] As can be seen from the above technical solutions, this application sets an activation function with activation parameters at the end of each network layer of the neural network. During the training process, the network parameters of each network layer are updated using the backpropagation algorithm, and the activation parameters of the activation function in each network layer are updated based on the error components of the same network layer in two adjacent training iterations. This effectively increases the range of the saturation region where the gradient is not zero in the activation function, ensuring that the network parameters of each network layer can be updated, and improving the speed and efficiency of parameter updates during neural network training.

[0130] Please see Figure 3 , Figure 3 This is a functional block diagram of a preferred embodiment of a parameter update device for a neural network according to this application. The parameter update device 11 for a neural network includes an initialization unit 110, a cost function unit 111, a first update unit 112, a second update unit 113, and an iterative training unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0131] In an optional embodiment, the initialization unit 110 is used to initialize the network parameters of the neural network and the activation parameters of all activation functions. The neural network includes multiple network layers, and each network layer includes at least the activation functions.

[0132] In an optional embodiment, the network layer includes a linear processing unit and an activation function, the activation function including activation parameters;

[0133] The activation function maps the output values ​​of linear processing units in the network layer to activation values, where the activation values ​​are the outputs of the network layer. The activation function satisfies the following relationship:

[0134]

[0135] Among them, O k-1 f is the output of network layer k-1. Wk λ is a linear processing unit of network layer k. k f represents the activation parameters of the activation function in network layer k. Wk ( k-1 ) represents the output value of the linear processing unit in network layer k, O k This is the output of network layer k.

[0136] In an optional embodiment, the cost function unit 111 is used to randomly select at least one training data from the training set and input it into the neural network to obtain the output result of each network layer, and calculate the error component of each network layer based on the output result, and use the sum of all error components as the cost function, wherein the training set includes multiple training data.

[0137] In an optional embodiment, the step of randomly selecting at least one training data point from the training set and inputting it into the neural network to obtain the output of each network layer, calculating the error component of each network layer based on the output, and using the sum of all error components as the cost function includes:

[0138] A1, obtain the Kth label data of the training data by labeling, wherein the Kth label data is the expected output of network layer K after the training data is input into the neural network, and network layer K is the last network layer of the neural network;

[0139] A2, Calculate the error components of network layer K based on the Kth label data and the output of network layer K. The relationship between the error components of network layer K is as follows:

[0140]

[0141] Where M is the number of randomly selected training data points. The output of network layer K after the i-th training data is input into the neural network. For the Kth label data corresponding to the i-th training data, Loss K This refers to the error component of the network layer K;

[0142] A3, calculate the (K-1)th label data based on the Kth label data and the network parameters of the network layer K, wherein the (K-1)th label data is the expected output of the network layer K-1 after the training data is input into the neural network, and the network layer K-1 is the network layer adjacent to the network layer K;

[0143] A4. Calculate the error components of network layer K-1 based on the K-1 label data and the output of network layer K-1;

[0144] A5. Repeat steps A3 to A4 to traverse all network layers in the neural network to calculate the error component of each network layer.

[0145] A6. Calculate the sum of the error components of all network layers as the cost function, which satisfies the following relationship:

[0146]

[0147] Among them, Loss k Let $\frac{k}{k}$ be the error component of the k-th network layer, $K$ be the total number of network layers, and $Loss$ be the cost function.

[0148] In an optional embodiment, calculating the (K-1)th label data based on the Kth label data and the network parameters of network layer K includes:

[0149] The linear processing unit of network layer K is obtained based on the network parameters of network layer K.

[0150] The Kth label data is input into the inverse function of the linear processing unit of the network layer K to obtain the (K-1)th label data, which satisfies the following relation:

[0151]

[0152] in, Linear processing unit of network layer K inverse function, Let λ be the Kth label data corresponding to the i-th training data. K These are the activation parameters in the activation function of the network layer K. Let K be the (K-1)th label data corresponding to the i-th training data.

[0153] In an optional embodiment, the first update unit 112 is used to update the network parameters based on the cost function and the backpropagation algorithm to complete one iteration of training.

[0154] In an optional embodiment, updating the network parameters based on the cost function and the backpropagation algorithm includes:

[0155] The gradient of any network parameter in the neural network is calculated based on the cost function.

[0156] The network parameters are updated along the gradient descent direction based on the gradient and the preset learning rate, and the update process satisfies the following relationship:

[0157]

[0158] Where η is the preset learning rate, Let θ be the gradient of the network parameter. Let θ1 and θ2 be the values ​​of the network parameter θ before and after the update, respectively, representing the direction of the gradient descent.

[0159] In an optional embodiment, the second update unit 113 is used to repeat the cost function unit and the first update unit, and update the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations.

[0160] In an optional embodiment, updating the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations includes:

[0161] Use any network layer as the target layer;

[0162] Calculate the change in the error component of the target layer between two consecutive training iterations, where the change in the error component satisfies the following relationship:

[0163]

[0164] in, This refers to the error component of the target layer m in the first iteration of the two training iterations. Let ΔE(m) be the error component of the target layer m in the latter iteration of the two training iterations, and let ΔE(m) be the change in the error component of the target layer m.

[0165] If ΔE(m) is less than the first threshold and If the value is greater than the second threshold, the activation parameter of the activation function in the target layer is increased according to the preset rules; otherwise, the activation parameter of the activation function in the target layer is set to 1.

[0166] Iterate through all network layers to update the activation parameters of the activation function in each network layer.

[0167] In an optional embodiment, increasing the value of the activation parameter of the activation function in the target layer according to a preset rule includes:

[0168] The activation parameter is increased by adding a preset step size to the value of the activation parameter.

[0169] In an optional embodiment, the iterative training unit 114 is used to return the cost function unit and iteratively train the neural network until the value of the cost function is less than a preset value or a predetermined number of iterations is reached.

[0170] As can be seen from the above technical solutions, this application sets an activation function with activation parameters at the end of each network layer of the neural network. During the training process, the network parameters of each network layer are updated using the backpropagation algorithm, and the activation parameters of the activation function in each network layer are updated based on the error components of the same network layer in two adjacent training iterations. This effectively increases the range of the saturation region where the gradient is not zero in the activation function, ensuring that the network parameters of each network layer can be updated, and improving the speed and efficiency of parameter updates during neural network training.

[0171] Please see Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 uses the computer-readable instructions stored in the memory to implement the parameter update method of the neural network described in any of the above embodiments.

[0172] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a parameter update program for a neural network.

[0173] Figure 4 Only electronic device 1 with memory 12 and processor 13 is shown. It will be understood by those skilled in the art that... Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0174] Combination Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement a parameter update method for a neural network, and the processor 13 can execute the plurality of instructions to implement:

[0175] S10, Initialize the network parameters of the neural network and the activation parameters of all activation functions. The neural network includes multiple network layers, and each network layer includes at least the activation functions.

[0176] S11, randomly select at least one training data from the training set and input it into the neural network to obtain the output result of each network layer, and calculate the error component of each network layer based on the output result, and use the sum of all error components as the cost function. The training set includes multiple training data.

[0177] S12, Update the network parameters based on the cost function and backpropagation algorithm to complete one iteration of training;

[0178] S13, repeat steps S11 and S12, and update the activation parameters of the activation function in each network layer based on the error components of the same network layer in two adjacent training iterations;

[0179] S14, return to step S11, iteratively train the neural network until the value of the cost function is less than a preset value or the predetermined number of iterations is reached.

[0180] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0181] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. Electronic device 1 can be a bus-type structure or a star-type structure. Electronic device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, electronic device 1 may also include input / output devices, network access devices, etc.

[0182] It should be noted that electronic device 1 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0183] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a neural network parameter update program, but also to temporarily store data that has been output or will be output.

[0184] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing parameter update programs for neural networks) and calls data stored in the memory 12 to perform various functions and process data in the electronic device 1.

[0185] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-described embodiments of the parameter update methods for various neural networks, for example... Figure 1 The steps are shown.

[0186] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an initialization unit 110, a cost function unit 111, a first update unit 112, a second update unit 113, and an iterative training unit 114.

[0187] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the neural network parameter update methods described in the various embodiments of this application.

[0188] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0189] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0190] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0191] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0192] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 4 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0193] This application also provides a computer-readable storage medium (not shown) storing computer-readable instructions, which are executed by a processor in an electronic device to implement the parameter update method of the neural network described in any of the above embodiments.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0195] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0197] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for updating parameters of a neural network, the method comprising: The method comprises: S10, initializing network parameters of a neural network and activation parameters of all activation functions, the neural network comprising a plurality of network layers, the network layers at least comprising the activation functions, wherein the neural network is a convolutional neural network, and the convolutional neural network is used for image classification; S11, randomly selecting at least one training data from a training set and inputting the training data into the neural network to obtain output results of each network layer, and calculating error components of each network layer based on the output results, and taking a sum of all error components as a cost function, comprising: A1, obtaining Kth label data of the training data in a labeled manner, the Kth label data being an expected output of a Kth network layer after inputting the training data into the neural network, the Kth network layer being a last network layer of the neural network; A2, calculating an error component of the Kth network layer based on the Kth label data and an output result of the Kth network layer, the error component of the Kth network layer satisfying a relationship: wherein M is the number of randomly selected training data, is the output result of the network layer K after the i-th training data is input into the neural network, is the K-th label data corresponding to the i-th training data, is the error component of the network layer K. A3, calculating K-1th label data based on the Kth label data and network parameters of the Kth network layer, the K-1th label data being an expected output of a K-1th network layer after inputting the training data into the neural network, the K-1th network layer being a network layer adjacent to the Kth network layer; A4, calculating an error component of the K-1th network layer based on the K-1th label data and an output result of the K-1th network layer; A5, repeatedly performing steps A3 to A4 to calculate error components of all network layers in the neural network; A6, calculating a sum of all error components as the cost function, the cost function satisfying a relationship: wherein, is the error component for the kth network layer, K is the number of all network layers, is the cost function; The training set comprises a plurality of training data, wherein the training data is an image to be classified, and the label data is a classification result of the image to be classified; S12, updating the network parameters based on the cost function and a back propagation algorithm to complete one iteration training; S13, repeatedly performing steps S11 and S12, and updating activation parameters of activation functions in each network layer based on error components of the same network layer in adjacent two iteration trainings; S14, returning to step S11 to iteratively train the neural network until a value of the cost function is less than a preset value or a predetermined number of iteration trainings is reached. 2.The method of Claim 1, wherein The network layer comprises a linear processing unit and an activation function, and the activation function comprises activation parameters; The activation function is used to map an output value of the linear processing unit in the network layer into an activation value, the activation value being an output result of the network layer, and the activation function satisfying a relationship: wherein, is the output result of the network layer k-1, is a linear processing unit of the network layer k, is an activation parameter of an activation function in the network layer k, is an output value of the linear processing unit in the network layer k, is the output result of the network layer k. 3.The method of Claim 1, wherein The calculation of the K-1th label data based on the Kth label data and the network parameters of the Kth network layer comprises: a linear processing unit of the network layer K is obtained based on the network parameter of the network layer K ; inputting the Kth label data into an inverse function of the linear processing unit of the Kth network layer to obtain the K-1th label data, the K-1th label data satisfying a relationship: wherein, is a linear processing unit of network layer K is an inverse function of is the Kth label data corresponding to the ith training data, is an activation parameter in the activation function of the network layer K, is the K-1th label data corresponding to the ith training data. 4.The method of Claim 1, wherein The updating of the network parameters based on the cost function and the back propagation algorithm comprises: calculating gradients of any network parameter in the neural network based on the cost function; updating values of the network parameters along a gradient descent direction based on the gradient and a preset learning rate, the updating process satisfying a relationship: wherein, is a preset learning rate, is a network parameter of a gradient, is a direction of the gradient descent, and are values of the network parameter before and after the update, respectively. 5.The method of Claim 1, wherein updating the activation parameter of the activation function in each network layer based on error components of the same network layer in adjacent two iteration trainings, including: taking any network layer as a target layer; calculating a variation of the error component of the target layer in adjacent two iteration trainings, the variation of the error component satisfying a relationship: wherein is an error component of the target layer m in the former one of the two iteration trainings, is an error component of the target layer m in the latter one of the two iteration trainings, is a variation of the error component of the target layer m. If less than the first threshold value and greater than the second threshold value, the value of the activation parameter of the activation function in the target layer is increased according to a preset rule, otherwise, the value of the activation parameter of the activation function in the target layer is set to 1. updating the activation parameter of the activation function in each network layer by traversing all network layers. 6.The method of Claim 5, wherein the preset rule includes: adding a preset step to the value of the activation parameter to increase the activation parameter of the activation function in the target layer.

7. A parameter update apparatus of a neural network, characterized by, The device comprises: an initialization unit configured to initialize network parameters of a neural network and activation parameters of all activation functions, the neural network comprising a plurality of network layers, the network layers comprising at least the activation functions, wherein the neural network is a convolutional neural network, and the convolutional neural network is used for image classification; a cost function unit configured to input at least one training data from a training set into the neural network to obtain output results of each network layer, and calculate error components of each network layer based on the output results, and take a sum of all error components as a cost function, including: A1, obtaining Kth label data of the training data in a labeled manner, the Kth label data being an expected output of a Kth network layer after inputting the training data into the neural network, the Kth network layer being a last network layer of the neural network; A2, calculating an error component of the Kth network layer based on the Kth label data and an output result of the Kth network layer, the error component of the Kth network layer satisfying a relationship: wherein M is the number of randomly selected training data, is the output result of the network layer K after the i-th training data is input into the neural network, is the K-th label data corresponding to the i-th training data, is the error component of the network layer K. A3, calculating K-1th label data based on the Kth label data and network parameters of the Kth network layer, the K-1th label data being an expected output of a K-1th network layer after inputting the training data into the neural network, the K-1th network layer being an adjacent network layer of the Kth network layer; A4, calculating an error component of the K-1th network layer based on the K-1th label data and an output result of the K-1th network layer; A5, repeatedly performing steps A3 to A4 to calculate error components of all network layers in the neural network; A6, calculating a sum of all error components as a cost function, the cost function satisfying a relationship: wherein, is the error component for the kth network layer, K is the number of all network layers, is the cost function; the training set comprises a plurality of training data, wherein the training data is an image to be classified, and the label data is a classification result of the image to be classified; a first updating unit configured to update the network parameters based on the cost function and a back propagation algorithm to complete one iteration training; a second updating unit configured to repeatedly perform the cost function unit and the first updating unit, and update the activation parameter of the activation function in each network layer based on error components of the same network layer in adjacent two iteration trainings. An iteration training unit is configured to return the cost function unit and iteratively train the neural network until a value of the cost function is less than a preset value or a predetermined number of iterations is reached.

8. An electronic device, comprising: The electronic device comprises: a memory storing computer readable instructions; and a processor executing the computer readable instructions stored in the memory to implement the parameter updating method of the neural network according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the parameter updating method of the neural network according to any one of claims 1 to 6.

Images