Method and device for neural network, electronic equipment, medium and product

Abstract

The embodiment of the invention discloses a method and device for a neural network, electronic equipment, a medium and a product. The method comprises the steps that the neural network is acquired; performing binarization forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result; performing binarization reverse calculation on the neural network based on the forward output result to update parameters of the neural network; wherein a tangent function associated with parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network in the process of binarization reverse calculation. According to the technical scheme of the embodiment of the invention, the problems of high binarization gradient mismatch degree, large error and poor neural network precision in neural network data calculation in the prior art are solved, the binarization gradient mismatch degree in the neural network data calculation process can be reduced, the convergence process of network data calculation is accelerated, and the calculation efficiency is improved. And the precision of neural network data calculation is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a method and apparatus for neural network, electronic device, medium and product. BACKGROUND

[0002] In the learning and computing process of neural networks such as convolutional neural network (CNN), recurrent neural network (RNN) and recursive neural network (RecNN), the consumption of computing resources is serious. In order to save computing resources and storage resources, quantization calculation is usually used in the neural network data computing process to save a large amount of memory and calculation and facilitate the deployment of the network on resource-limited devices.

[0003] However, in the process of parameter binarization quantization of the neural network, an obvious gradient mismatch problem is introduced, which makes the learning and computing process of the neural network difficult to converge, and the network accuracy needs to be improved. SUMMARY

[0004] The present disclosure provides a method and apparatus for neural network, electronic device, medium and product, which can reduce the binarization gradient mismatch degree in the neural network data computing process, accelerate the convergence process of the neural network data computing, and improve the accuracy of the neural network data computing.

[0005] In a first aspect, the embodiments of the present disclosure provide a method for neural network, which comprises:

[0006] obtaining a neural network;

[0007] performing binarization forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result;

[0008] performing binarization backward calculation on the neural network based on the forward output result to update parameters of the neural network;

[0009] wherein, in the process of the binarization backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0010] In a second aspect, the embodiments of the present disclosure further provide an apparatus for neural network, which comprises:

[0011] a data obtaining module configured to obtain a neural network;

[0012] a first data calculating module configured to perform binarization forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result;

[0013] a second data calculation module configured to perform binary reverse calculation on the neural network based on the forward output result, to update parameters of the neural network;

[0014] wherein, in the binary reverse calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0015] In a third aspect, the embodiments of the present disclosure further provide an electronic device, which comprises:

[0016] one or more processors;

[0017] a storage device configured to store one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for neural network according to any of the embodiments of the present disclosure.

[0019] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions for performing the method for neural network according to any of the embodiments of the present disclosure when executed by a computer processor.

[0020] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product comprising a computer program which, when executed by a processor, implements the method for neural network according to any of the embodiments of the present disclosure.

[0021] In the embodiments of the present disclosure, a neural network is obtained; binary forward calculation is performed on the neural network based on at least one of image data, text data and voice data, to obtain a forward output result; binary reverse calculation is performed on the neural network based on the forward output result, to update parameters of the neural network; and in the binary reverse calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network. The technical solution of the embodiments of the present disclosure solves the problem of high binary gradient mismatch degree, large error and poor neural network precision in the data quantization calculation process of the neural network in the prior art, can reduce the binary gradient mismatch degree in the data calculation process of the neural network, accelerate the convergence speed of the neural network data quantization calculation, and improve the data calculation precision of the neural network with the determined parameters after the quantization calculation. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings. The same or similar components have the same or similar reference numbers regardless of the figure number.

[0023] Figure 1 is a flowchart of a method for a neural network provided by an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of another method for a neural network provided by an embodiment of the present disclosure;

[0025] Figure 3 is a diagram of function curve and gradient change of each function provided by an embodiment of the present disclosure;

[0026] Figure 4 is a diagram of comparison of convolutional neural network structures provided by an embodiment of the present disclosure;

[0027] Figure 5 is a diagram of comparison of another convolutional neural network structure provided by an embodiment of the present disclosure;

[0028] Figure 6 is a diagram of a structure of an apparatus for a neural network provided by an embodiment of the present disclosure;

[0029] Figure 7 is a diagram of a structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the present disclosure. It is to be understood that the drawings and descriptions are illustrative embodiments and are not intended to limit the scope of the present disclosure, and that the present disclosure is capable of various modifications in the drawings and descriptions.

[0031] It is understood that each of the steps recited in the method embodiments of the present disclosure can be performed in a different order, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this regard.

[0032] As used herein, the term "includes" and its variants are to be read to be analogous to "comprises," or "comprising." The term "based on" is to be read as "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related definitions will be given in the description below.

[0033] It should be noted that the terms "first", "second", and the like in the present disclosure are merely used to distinguish different devices, modules or units, and do not imply the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the terms "one", "multiple" in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0035] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0036] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0037] As an optional but not limited implementation manner, in response to receiving the active request of the user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0038] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0039] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions.

[0040] Figure 1As shown in a flowchart of a method for a neural network provided by an embodiment of the present disclosure, the embodiment of the present disclosure is applicable to a scenario of determining parameters of a neural network for data calculation, in particular, to reduce the amount of calculation in the data calculation process of the neural network and to reduce the consumption of computing resources of a hardware device computer. The method can be executed by a device for a neural network, which can be implemented in the form of software and / or hardware. Optionally, the device for a neural network can be implemented by an electronic device, which can be a mobile terminal, a PC terminal or a server, and the like, in particular, some mobile terminals, embedded systems and the like with fixed configuration of computing resources and memory and limited resource space.

[0041] As shown in Figure 1 The method for a neural network includes the following steps.

[0042] S110, obtaining a neural network.

[0043] Specifically, the obtained neural network is a neural network whose network parameters need to be determined through a quantization calculation process. The purpose is to reduce the data storage requirement and the calculation complexity in the parameter calculation process of the neural network, and to obtain relatively high neural network calculation accuracy as much as possible.

[0044] The quantization calculation method includes quantization-aware calculation (Quantification Aware Training, QAT), post-quantization (Post-Quantization), mixed precision quantization (Mixed Precision Quantization), knowledge distillation (Knowledge Distillation) and dynamic quantization (Dynamic Quantization) and the like. Among them, the quantization-aware calculation can introduce quantization-related constraints in the data calculation stage of the neural network, which can better solve the problem of accuracy decline in the quantization calculation process of the neural network.

[0045] In the embodiment, the type of the neural network is not limited, and can be any one or a combination of multiple neural network structures.

[0046] S120, performing binary forward calculation on the neural network based on at least one of image data, text data and voice data, to obtain a forward output result.

[0047] At least one of the image data, the text data, and the speech data is a learning object of the neural network. According to a deployment application scenario of the neural network, the learning object can be image data, and then after learning, the neural network can be used for image recognition, image segmentation, or image special effect processing; the learning object can be text data, and then after learning, the neural network can be used for text semantic recognition, text matching, text information extraction, or text generation, etc.; the learning object can also be speech data, and then after learning, the neural network can be used for speech recognition, speech text conversion, or speech generation, etc.

[0048] In the process of learning the parameters of the neural network based on any of the above learning objects, at least one of the image data, the text data, and the speech data is input into the neural network for forward calculation.

[0049] In this embodiment, in the process of determining the data calculation of the neural network parameters, the binarized neural network can be used in combination with the quantization calculation algorithm. By binarizing and quantizing the neural network to be determined, more efficient calculation and smaller neural network size can be achieved without significantly reducing the performance of the neural network. Such technology is particularly useful on resource-constrained devices (such as small mobile terminals, embedded systems, and chips on chips), and can improve the running efficiency and real-time performance of the neural network.

[0050] The neural network can be binarized for forward calculation based on at least one of the image data, the text data, and the speech data. In the calculation process, the weights and activation values in the neural network are limited to binary values (0 or 1), which can save a lot of memory and calculation, and greatly facilitate the deployment of the neural network on devices with limited storage and computing resources (such as small mobile terminals, embedded systems, and chips on chips). Through binarization, the number of parameters and the amount of calculation in the data calculation process of the neural network can be significantly reduced, thereby realizing the compression and acceleration of the neural network.

[0051] The binarized forward calculation process based on at least one of the image data, the text data, and the speech data input into the neural network can be a process of binarizing the weight values and / or activation values of each convolutional layer. For example, binarization is performed by a sign function, to obtain binarized weight values and binarized activation values, and by multiplying them by a floating-point scaling coefficient to approximate the weight values and activation values, the forward output result is the product of the data values input to each input neuron of the output layer and the corresponding binarized weight values and binarized activation values. Each convolutional layer includes one or more convolutional kernels.

[0052] S130, binaryzation back propagation is performed based on the forward output result, so as to update the parameters of the neural network, wherein, in the process of the binaryzation back propagation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network.

[0053] Due to the non-differentiable and discontinuous forward binaryzation function in the binaryzation neural network calculation process, there is a gradient mismatch problem caused by approximation error in the back propagation process. In the process of back gradient calculation according to the forward calculation derivation process, the differentiable approximation of the formula function sign(.) is needed when the gradient value is approximated. Generally, the "Straight Through Estimator" method is used to perform differentiable approximation of the formula function sign(.), and a fixed value is taken as the approximation result, but the estimation accuracy is not high, and it cannot provide sufficient accuracy or adaptability.

[0054] In this embodiment, in order to improve the accuracy of gradient approximation in the binaryzation back propagation process, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network.

[0055] The parameters can include weight values and activation values. The binaryzation back propagation process updates the corresponding parameters by calculating the gradient value of the loss function on the parameters.

[0056] The tangent function can be a parameterized tangent function, which is high in accuracy and continuously differentiable, and can be used to approximate the gradient value of the sign function sign(.) to replace the sign function sign(.) to obtain the gradient value of the parameters.

[0057] Finally, the gradient descent algorithm, such as the stochastic gradient descent algorithm, is used to update the parameters of the network according to the gradient value, so as to reduce the value of the loss function.

[0058] The technical scheme of the embodiment of the present disclosure comprises the following steps: obtaining a neural network; performing binaryzation forward calculation on the neural network based on at least one of image data, text data and voice data, to obtain a forward output result; performing binaryzation back propagation on the neural network based on the forward output result, to update the parameters of the neural network; wherein, in the process of the binaryzation back propagation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network. The technical scheme of the embodiment of the present disclosure solves the problem of high binaryzation gradient mismatch degree, large error and poor network accuracy in the data calculation of determining the parameters of the neural network in the prior art, can reduce the binaryzation gradient mismatch degree in the neural network data calculation process, accelerate the convergence process of the neural network data calculation, and improve the accuracy of the neural network data calculation.

[0059] Figure 2 Another flowchart of the method for neural network provided by the embodiments of the present disclosure is shown in the above embodiments, and the process of binarized forward calculation and binarized backward calculation is further explained in the process of implementing the method flow for neural network. The method can be executed by a device for neural network, which can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc.

[0060] As shown in the above embodiments, the method for neural network comprises: Figure 2

[0061] S210, obtaining a neural network.

[0062] S220, inputting at least one of image data, text data and voice data to the neural network, and in the calculation of each convolution layer in the neural network, performing binarized approximation on the weight value corresponding to the input node associated with each neuron node to obtain the corresponding binarized approximation weight value. The node, also known as neuron, is the basic calculation unit in each convolution layer of the neural network. Each node receives input signals from other nodes, and the node sending the input signals to each node is the input node associated with each neuron node. According to the corresponding weight value, the corresponding binarized approximation weight value is obtained.

[0063] The following description is the process of calculating the output activation value of each corresponding neuron node of each layer, which is also the process of layer-by-layer forward calculation for the structure of the neural network:

[0064] For example, for a convolutional neural network containing L convolution layers, the l(th) (greater than or equal to 1 and less than or equal to L) layer convolution layer of the neural network can be represented by a triple <W l ,I l-1 ,*>. Wherein, W l ∈R k×k×Cin×Cout represents the floating-point weight value of the layer, Cin represents the input channel, Cout represents the output channel, I l-1 ∈R w×h×Cin represents the input activation value (output activation value of the previous layer) of the layer, and * represents the convolution operation based on multiply-accumulate. W l,k(k＝1...Cout) ∈R k×k×Cin represents the floating-point weight value of the k(th) convolution kernel of the l(th) layer. The process of binarizing W l is as follows:

[0065] Suppose there are binarized weights B l,k ∈{-1,+1} k×k×Cin and floating-point scaling coefficients a l,k ∈R​+ , then the floating-point weight value W l,k can be binarized as The solving process can be converted into an optimization problem Solving B l,k = Sign(W l,k ), where n = k x k x Cin, representing the number of elements of the floating-point weight W l,k . Accordingly, the binarized approximation of the weight value of the l-th layer can be represented as where B l = [B l,1 ,…,B l,Cout ], a l = [a l,1 ,…,a l,Cout ].

[0066] S230, the input activation value of each input node is operated with the corresponding binarized approximation weight value to obtain the output activation value of each neuron node of each layer until the forward output result of the neural network is obtained.

[0067] The binarization process of the activation value can refer to the above-mentioned binarization process of the weight value. For the input activation value I l-1 ∈ R w×h×Cin of the l-th layer, the corresponding binarized approximation activation value can be represented as where Q l-1,i ∈ {-1, +1} w×h×Cin , and β l-1 ∈ R + , i takes the value range (1,…, Cout).

[0068] After the binarization approximation process of the weight value and the activation value, the output activation value of each input node is multiplied by the corresponding binarized approximation weight value to obtain the forward output result (output activation value) of the l-th layer, which can be represented as: O l = W l *I l-1 .

[0069] Optionally, the output activation value of each node of each layer is binarized and approximated.

[0070] Specifically, by substituting the input activation value and the corresponding binarized approximation weight value into , the binarized approximation value of the output activation value is

[0071] S240, calculating a loss function value of the neural network according to the forward output result and an expected output result corresponding to at least one of the image data, the text data and the voice data, and calculating a gradient value of an output node of the neural network according to the loss function value.

[0072] Specifically, in the neural network calculation process, the loss function value of the neural network can be calculated according to the forward output result and the expected output result corresponding to at least one of the image data, the text data and the voice data input to the neural network. The loss function can measure the difference between the expected output result and the forward output result of the neural network. The smaller the loss function value is, the better the prediction effect of the neural network is, and the higher the calculation accuracy is.

[0073] Further, the gradient value of the output node of the neural network is calculated by deriving the forward output result according to the calculated loss function.

[0074] S250, according to the chain rule, the gradient value of the output node is conducted layer by layer, and in the gradient conducting process, the tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation is determined to determine the weight gradient value and / or activation value gradient value in the binary backward calculation.

[0075] In the neural network, the output activation value of each node is calculated by its corresponding input value and weight value and activation function. According to the chain rule, the gradient is propagated from the output layer to the input layer by deriving each component function in turn. Therefore, the gradient can be propagated between different convolution layers of the neural network, so as to calculate the contribution degree of the parameter to the loss function.

[0076] The chain rule in this embodiment plays a key role in the back propagation of the neural network. The gradient value of the output node is conducted layer by layer by the chain rule to calculate the gradient value of the parameter, so as to update the parameter of the neural network and minimize the loss function.

[0077] Specifically, taking the neural network in step S220 as an example, the loss function is represented by L, and the weight gradient value is wherein, At this time, it is necessary to solve In the prior art, the sign(.) function used in the forward calculation is approximated by the "Straight Through Estimator (STE)", for example, STE can be represented as y = STE(x) = 1 when |x| < 1 else 0, then However, since the approximate gradient solved by STE is quite different from the real gradient distribution of the sign(.) function, it causes serious forward and backward gradient adaptation problems, which leads to convergence difficulty in the neural network calculation process and low calculation result accuracy.

[0078] To solve the above problem, in this embodiment, the tangent function value corresponding to each binarized weight value and / or binarized activation value is used to determine the weight gradient value and / or activation value gradient value in the binarization reverse calculation.

[0079] In an optional embodiment, during the gradient propagation process, the weight gradient value and / or activation value gradient value in the binarized reverse calculation are determined based on the tangent function value corresponding to each binarized weight value and / or binarized activation value in the binarized forward calculation. This may be achieved by calculating the product of each binarized weight value and / or binarized activation value and the parameter to be updated during the gradient propagation process, and using the tangent function value of the product as the weight gradient value and / or activation value gradient value of the corresponding binarized weight value and / or binarized activation value. The parameter to be updated is learned and updated during the calculation process of the neural network.

[0080] The tangent function can be expressed as Where ρ is the parameter to be updated. Since the activation gradient is similar to the weight gradient, we take the weight gradient as an example. Then, we substitute the binary weight into the tangent function to replace x, and the tangent function value of the product with the parameter to be updated ρ is tanh(ρW l ), then the gradient weight calculation of the lth layer is known to be Substitution as well as The weight gradient value can be obtained.

[0081] The comparison chart of using tangent function for gradient approximation and using STE function for gradient calculation is as follows Figure 3 As shown, the left side is the function curve and the right side is the gradient change. The black curve represents the sign(.) function and the gradient change. Its function value is 1 when it is greater than 0 and -1 when it is less than 0. The gradient value of the sign(.) function is a convex function and changes suddenly at zero point and cannot be used for calculation. The red curve represents the forward function of the STE function and the corresponding gradient change. Its function value is -1 when it is less than -1, and changes linearly between -1 and +1. It is 1 when it is greater than +1. It can be seen that it is very different from the gradient of the sign(.) function, which will cause the forward and reverse gradients to be incompatible. Figure 3 The different shades of blue curves represent the function and gradient for different values ​​of the parameter to be updated ρ. It can be seen that as the value of ρ increases, the function curve becomes more and more similar to the sign(.) function, and the gradient also approximates a peak distribution, making the gradient approximation more accurate. However, an excessively high gradient may lead to computational instability. As the value of ρ decreases, the gradient becomes more and more similar to the STE function. At this time, the gradient approximation error becomes larger, but the computation is stable. Therefore, in an optional implementation, to balance the accuracy of the gradient approximation and computational stability, the initial value of the parameter to be updated is set to 2.

[0082] S260. Update the parameters of the neural network according to the weight gradient value and / or the activation value gradient value.

[0083] The parameters of the neural network are updated through the gradient descent algorithm according to the weight gradient value and / or activation value gradient value to obtain a target neural network with better calculation effect.

[0084] The technical solution of the embodiment of the present disclosure is to obtain a neural network; perform a binarized forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result; calculate the loss function value of the neural network according to the forward output result and the expected output result corresponding to at least one of the image data, text data and voice data, and calculate the gradient value of the output node of the neural network according to the loss function value; conduct the gradient value of the output node layer by layer according to the chain rule, and determine the weight gradient value and / or activation value gradient value in the binarized reverse calculation according to the tangent function value corresponding to each binarized weight value and / or binarized activation value in the binarized forward calculation during the gradient conduction process; update the parameters of the neural network according to the weight gradient value and / or the activation value gradient value. Wherein, in the process of the binarized reverse calculation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network. The technical solution of the embodiment of the present disclosure solves the problems of high binarization gradient mismatch, large error and poor neural network accuracy in network data calculation in the prior art. The degree of binarization gradient mismatch in the network data calculation process can be further reduced by the tangent function value corresponding to each binarized weight value and / or binarized activation value, thereby accelerating the convergence process of network data calculation and improving the accuracy of network data calculation.

[0085] On the basis of the above embodiment, when the neural network being calculated is a convolutional neural network, unlike the conventional convolutional neural network structure, the pooling layer in the neural network can be set before the batch normalization layer.

[0086] Specifically, refer to Figure 4 Schematic diagram of the structural comparison of convolutional neural networks in Figure 4 The left picture in the middle is the traditional convolutional neural network structure, and the right picture is the convolutional neural network structure of this embodiment. In the traditional convolutional neural network structure design, the structure is composed of structural units of convolution layer-batch normalization layer-pooling layer. Under this structure, the gradient adaptation problem caused by the convolution layer will be further amplified by the batch normalization layer. Therefore, to solve this problem, this embodiment sets the pooling layer before the batch normalization layer. By performing the pooling operation in advance, this embodiment can effectively alleviate the problem of low accuracy caused by gradient mismatch and improve the accuracy of the neural network.

[0087] Reference is made to Figure 5 The structure comparison diagram of the convolutional neural network shown in the figure, the left side is the structure of the traditional convolutional neural network, and the right side is the structure of the convolutional neural network of the embodiment. The traditional convolutional neural network structure is a convolution layer (Conv)-a bias calculation layer (BiasAdd)-a batch normalization layer (BN)-a pooling layer (Pool). In the embodiment, the network structure further includes a binarization convolution layer (BinConv)-a bias calculation layer (BiasAdd)-a pooling layer (Pool)-a batch normalization layer (BN) during the binarization quantization calculation of the convolutional neural network.

[0088] The embodiment can effectively alleviate the gradient mismatch caused by binarization processing, reduce the error of the whole neural network, and improve the precision of the neural network by performing the pooling operation in advance.

[0089] Figure 6 The device for neural network provided by the embodiment of the present disclosure is suitable for the scenario of network calculation, especially the case of weak gradient mismatch network calculation. The device for neural network can be realized in the form of software and / or hardware, and can be configured in an electronic device, which can be a mobile terminal, a PC terminal, or a server, etc.

[0090] As Figure 6 The network calculation device includes a data acquisition module 310, a first data calculation module 320, and a second data calculation module 330.

[0091] The data acquisition module 310 is configured to acquire a neural network.

[0092] The first data calculation module 320 is configured to perform binarization forward calculation on the neural network based on at least one of image data, text data, and voice data, to obtain a forward output result.

[0093] The second data calculation module 330 is configured to perform binarization reverse calculation on the neural network based on the forward output result, to update parameters of the neural network.

[0094] In the binarization reverse calculation process, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolution layer in the neural network.

[0095] The technical scheme of the embodiment of the present disclosure is that a neural network is acquired; at least one of image data, text data and voice data is used to perform binary forward calculation on the neural network to obtain a forward output result; the neural network is used to perform binary backward calculation based on the forward output result to update parameters of the neural network; and in the process of the binary backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolution layer in the neural network. The technical scheme of the embodiment of the present disclosure solves the problem of high binary gradient mismatch degree, large error and poor network precision in the prior art, can reduce the binary gradient mismatch degree in the network calculation process, speeds up the convergence process of the network calculation and improves the precision of the network calculation.

[0096] In an optional implementation, the second data calculation module 330 is specifically configured to:

[0097] According to the forward output result and the expected output result corresponding to at least one of the image data, the text data and the voice data, a loss function value of the neural network is calculated, and a gradient value of an output node of the neural network is calculated according to the loss function value; the gradient value of the output node is transmitted layer by layer according to the chain rule, and in the gradient transmission process, a tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation is used to determine a weight gradient value and / or an activation value gradient value in the binary backward calculation; and the parameters of the neural network are updated according to the weight gradient value and / or the activation value gradient value.

[0098] In an optional implementation, the second data calculation module 330 is further specifically configured to:

[0099] In the gradient transmission process, the product of each binary weight value and / or binary activation value in the binary forward calculation and a parameter to be updated in the neural network is calculated, and the tangent function value of the product is used as the weight gradient value and / or the activation value gradient value of the corresponding binary weight value and / or binary activation value; wherein the parameter to be updated is learned and updated in the calculation process of the neural network.

[0100] In an optional implementation, the initial value of the parameter to be updated is set to 2.

[0101] In an optional implementation, the first data calculation module 320 is specifically configured to:

[0102] input at least one of the image data, the text data and the voice data to the neural network, and in the calculation of each convolution layer in the neural network, the weight value corresponding to each input node associated with a neuron node is binarily approximated to obtain a corresponding binary approximation weight value; the input activation value of each input node is operated with the corresponding binary approximation weight value to obtain the output activation value of the corresponding neuron node of each layer until the forward output result of the neural network is obtained.

[0103] In an optional implementation, the first data calculation module 320 is further configured to:

[0104] The output activation value of each node of each layer is binarily approximated.

[0105] In an optional implementation, when the neural network is a convolutional neural network, the pooling layer in the neural network is arranged before the batch normalization layer.

[0106] The apparatus for neural network provided by the embodiments of the present disclosure can perform the method for neural network provided by any of the embodiments of the present disclosure, and has the corresponding function modules and advantages of performing the method.

[0107] It is worth noting that each unit and module included in the apparatus is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for easy mutual distinction, and does not limit the protection scope of the embodiments of the present disclosure.

[0108] Figure 7 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure. Hereinafter, the structural schematic diagram of the electronic device (for example, a terminal device or a server) 600 suitable for implementing the embodiments of the present disclosure is shown. Figure 7 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure. Figure 7 The terminal device in the embodiments of the present disclosure can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 7 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0109] As shown in FIG. 6, the electronic device 600 can include a communication interface 610, a processor 620, a memory 630 and the like. Figure 7As shown, the electronic device 600 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 605.

[0110] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although Figure 7 The electronic device 600 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0111] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0112] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0113] The electronic device provided by the embodiments of the present disclosure and the method for a neural network provided by the above embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above embodiments, and the present embodiments have the same beneficial effects as the above embodiments.

[0114] The embodiments of the present disclosure also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for a neural network provided by the above embodiments.

[0115] Note that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, a computer readable signal medium can include a computer readable program code carried in a baseband or as a part of a carrier wave, in which the computer readable program code can be used by or in connection with an instruction execution system, apparatus or device. Such a propagated computer readable signal medium can take various forms, including but not limited to electro-magnetic, optical or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus or device, except for the computer readable storage media described above. The program code carried by the computer readable media can be transmitted in any suitable media, including but not limited to wire, cable, fiber optic, RF (radio frequency), or any suitable combination of the foregoing.

[0116] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0117] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and not be assembled in the electronic device.

[0118] The computer readable medium described above carries one or more programs, which when executed by the electronic device, cause the electronic device to:

[0119] obtain a neural network;

[0120] perform binary forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result;

[0121] perform binary backward calculation on the neural network based on the forward output result to update parameters of the neural network;

[0122] wherein, in the process of the binary backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0123] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0124] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0125] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of a unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0126] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical wire, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the method for a neural network provided by any one of the embodiments of the present disclosure.

[0129] The computer program product, during implementation, may be written in one or more programming languages, or a combination thereof, for performing the operations of the present disclosure and may include computer program code written in object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0131] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0132] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for a neural network, characterized in that include: Get the neural network; Performing a binarized forward calculation on the neural network based on at least one of the image data, the text data, and the voice data to obtain a forward output result; Performing a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network; In the process of the binarization reverse calculation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolution layer in the neural network.

2. The method according to claim 1, characterized in that The performing a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network includes: Calculating a loss function value of the neural network based on the forward output result and an expected output result corresponding to at least one of the image data, text data, and voice data, and calculating a gradient value of an output node of the neural network based on the loss function value; The gradient value of the output node is transmitted layer by layer according to the chain rule, and during the gradient transmission process, the weight gradient value and / or activation value gradient value in the binarization reverse calculation is determined according to the tangent function value corresponding to each binarized weight value and / or binarized activation value in the binarization forward calculation; Update the parameters of the neural network according to the weight gradient value and / or the activation value gradient value.

3. The method according to claim 2, characterized in that Determining the weight gradient value and / or activation value gradient value in the binarization reverse calculation according to the tangent function value corresponding to each binarization weight value and / or binarization activation value in the binarization forward calculation during the gradient conduction process includes: During the gradient conduction process, the product of each binarized weight value and / or binarized activation value in the binarized forward calculation and the parameter to be updated in the neural network is calculated, and the tangent function value of the product is used as the weight gradient value and / or activation value gradient value of the corresponding binarized weight value and / or binarized activation value.

4. The method according to claim 3, characterized in that The initial value of the parameter to be updated is set to 2.

5. The method according to any one of claims 1 to 4, characterized in that: The performing binarization forward calculation on the neural network based on at least one of the image data, text data and voice data to obtain a forward output result includes: Inputting at least one of the image data, text data, and speech data into the neural network, and in the calculation of each convolutional layer in the neural network, performing binarization approximation on the weight value corresponding to the input node associated with each neuron node to obtain a corresponding binarized approximate weight value; The input activation value of each input node is calculated with the corresponding binarized approximate weight value to obtain the output activation value of the corresponding neuron node in each layer until the forward output result of the neural network is obtained.

6. The method according to claim 5, characterized in that The method further comprises: The output activation value of each node in each layer is binarized and approximated.

7. The method according to claim 1, characterized in that When the neural network is a convolutional neural network, the pooling layer in the neural network is arranged before the batch normalization layer.

8. A device for a neural network, characterized in that include: A data acquisition module, used to acquire a neural network; A first data calculation module is used to perform a binarization forward calculation on the neural network based on at least one of the image data, text data and voice data to obtain a forward output result; A second data calculation module is used to perform a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network; In the process of the binarization reverse calculation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolution layer in the neural network.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for a neural network according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for a neural network according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for a neural network according to any one of claims 1 to 7.

Images