Method and apparatus for neural network, and electronic device, medium and product

Abstract

Provided in the embodiments of the present disclosure are a method and apparatus for a neural network, and an electronic device, a medium and a product. The method comprises: obtaining a neural network; on the basis of at least one piece of image data, text data and speech data, performing binarization forward computation on the neural network, so as to obtain a forward output result; and on the basis of the forward output result, performing binarization backward calculation on the neural network, so as to update parameters of the neural network, wherein during the process of binarization backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolution layer in the neural network. The technical solution in the embodiments of the present disclosure solves the problems of a high degree of binarization gradient mismatch and large errors in neural network data computation and poor neural network precision, and can reduce the degree of binarization gradient mismatch during the process of neural network data computation, accelerate the convergence process of network data computation, and improve the precision of neural network data computation.

Description

Method, device, electronic device, medium and product for neural network

[0001] This application claims priority to Chinese Patent Application No. 202410433164.7, filed on April 10, 2024, the disclosure of which is incorporated herein in its entirety as part of the present application. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to a method, device, electronic device, medium and product for neural network. BACKGROUND

[0003] In the learning calculation process of neural networks such as convolutional neural network (CNN), recurrent neural network (RNN) and recursive neural network (RecNN), the consumption of computing resources is serious. In order to save computing resources and storage resources, quantization calculation is usually used in the neural network data calculation process to save a large amount of memory and calculation and facilitate the deployment of the network on resource-limited devices.

[0004] However, in the process of parameter binarization quantization of the neural network, an obvious gradient mismatch problem is introduced, which makes the learning calculation process of the neural network difficult to converge, and the network accuracy needs to be improved. SUMMARY

[0005] The present disclosure provides a method, device, electronic device, medium and product for neural network, which can reduce the binarization gradient mismatch degree in the neural network data calculation process, accelerate the convergence process of the neural network data calculation, and improve the accuracy of the neural network data calculation.

[0006] In a first aspect, the embodiments of the present disclosure provide a method for neural network, the method comprising:

[0007] obtaining a neural network;

[0008] performing binarization forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result;

[0009] performing binarization backward calculation on the neural network based on the forward output result to update parameters of the neural network;

[0010] wherein, in the process of the binarization backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0011] In a second aspect, the embodiments of the present disclosure further provide a device for neural network, the device comprising:

[0012] The data acquisition module is configured to acquire a neural network.

[0013] The first data calculation module is configured to perform binary forward calculation on the neural network based on at least one of the image data, the text data, and the voice data, to obtain a forward output result.

[0014] The second data calculation module is configured to perform binary backward calculation on the neural network based on the forward output result, to update parameters of the neural network.

[0015] In the binary backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0016] In a third aspect, the present disclosure provides an electronic device, which includes:

[0017] one or more processors;

[0018] a storage device configured to store one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for neural network according to any of the embodiments of the present disclosure.

[0020] In a fourth aspect, the present disclosure provides a storage medium containing computer executable instructions for performing the method for neural network according to any of the embodiments of the present disclosure when executed by a computer processor.

[0021] In a fifth aspect, the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the method for neural network according to any of the embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals are used to represent the same or similar elements. It is to be understood that the drawings are schematically showing the elements and not necessarily to scale.

[0023] FIG. 1 is a flow diagram of a method for neural network according to an embodiment of the present disclosure;

[0024] FIG. 2 is a flow diagram of another method for neural network according to an embodiment of the present disclosure;

[0025] FIG. 3 is a function curve and gradient change diagram of various functions according to an embodiment of the present disclosure;

[0026] FIG. 4 is a convolutional neural network structure comparison diagram according to an embodiment of the present disclosure;

[0027] FIG. 5 is another convolutional neural network structure comparison diagram according to an embodiment of the present disclosure;

[0028] FIG. 6 is a structure diagram of a device for a neural network according to an embodiment of the present disclosure; and

[0029] FIG. 7 is a structure diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings of the present disclosure are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0031] It should be understood that the various steps of the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "comprising" and variations thereof as used herein are open-ended, that is, "including, but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related terms are defined in the following description.

[0033] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0035] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0036] For example, in response to receiving the active request of the user, the prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can autonomously select whether to provide the personal information to the software or hardware such as the electronic device, application program, server or storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0037] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending the prompt information to the user may, for example, be the manner of a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0038] It can be understood that the above notification and obtaining of the authorization of the user are only illustrative and do not limit the implementation manners of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0039] It can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the present technical solutions should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0040] FIG. 1 is a flow diagram of a method for a neural network provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to a scenario of performing data calculation for determining parameters of a neural network, in particular, to a situation of reducing the amount of calculation in the data calculation process of the neural network and reducing the consumption of the computing resources of a hardware device. The method can be performed by a device for a neural network. The device for a neural network can be implemented in the form of software and / or hardware. Optionally, the device for a neural network can be implemented by an electronic device. The electronic device can be a mobile terminal, a PC terminal or a server, and in particular, some mobile terminals with fixed configuration of computing resources and memory, such as mobile terminals, embedded systems, etc.

[0041] As shown in FIG. 1, the method for the neural network comprises the following steps.

[0042] S110, obtaining a neural network.

[0043] Specifically, the obtained neural network is a neural network whose network parameters need to be determined through a quantization calculation process. The purpose is to reduce the data storage requirement and calculation complexity in the parameter calculation process of the neural network, and to obtain relatively high neural network calculation accuracy as much as possible.

[0044] The quantization calculation method includes quantization aware calculation (QAT), post-quantization, mixed precision quantization, knowledge distillation, and dynamic quantization, etc. Among them, quantization aware calculation can introduce quantization related constraints in the neural network data calculation stage, which can better solve the problem of accuracy decline in the neural network quantization calculation process.

[0045] In this embodiment, the type of neural network is not limited, and can be any one or a combination of multiple neural network structures.

[0046] In S120, a binary forward calculation is performed on the neural network based on at least one of the image data, the text data, and the voice data, to obtain a forward output result.

[0047] At least one of the image data, the text data, and the voice data is a learning object of the neural network. According to the deployment application scenario of the neural network, the learning object can be image data, so that after learning, the neural network can be used for image recognition, image segmentation, or image special effect processing; the learning object can be text data, so that after learning, the neural network can be used for text semantic recognition, text matching, text information extraction, or text generation, etc. The learning object can also be voice data, so that after learning, the neural network can be used for voice recognition, voice text conversion, or voice generation, etc.

[0048] In the process of learning the parameters of the neural network based on any of the above learning objects, at least one of the image data, the text data, and the voice data is input into the neural network for forward calculation.

[0049] In this embodiment, in the process of determining the data calculation of the neural network parameters, the binary neural network can be combined with the quantization calculation algorithm. By binarizing and quantizing the neural network whose network parameters need to be determined, more efficient calculation and smaller neural network size can be achieved without significantly reducing the performance of the neural network. Such technology is particularly useful on resource-constrained devices (such as small mobile devices, embedded systems, on-chip chips, etc.), and can improve the running efficiency and real-time performance of the neural network.

[0050] The binarized forward calculation of the neural network based on at least one of the image data, the text data and the voice data can limit the weights and the activation values in the neural network to binary values (0 or 1) in the calculation process, which can save a large amount of memory and calculation, and greatly facilitate the deployment of the neural network on devices with limited storage and calculation resources (such as small mobile terminals, embedded systems, and chips on a chip). Through binarization, the number of parameters and the amount of calculation in the data calculation process of the neural network can be significantly reduced, thereby realizing compression and acceleration of the neural network.

[0051] The binarized forward calculation process of the neural network based on at least one of the image data, the text data and the voice data input to the neural network can be a process of binarizing the weight values and / or the activation values of each convolutional layer. For example, the weight values and the activation values are binarized by a sign function to obtain binarized weight values and binarized activation values, and the forward output result, i.e., the output activation value, is the product of the data values input to each input neuron of the output layer and the corresponding binarized weight values and binarized activation values multiplied by a floating-point scaling coefficient. Each convolutional layer includes one or more convolutional kernels.

[0052] S130, binarized backward calculation of the neural network based on the forward output result to update the parameters of the neural network, wherein the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network in the binarized backward calculation process.

[0053] Due to the non-differentiable / discontinuous forward binarization function in the binarized neural network calculation process, there is a gradient mismatch problem caused by approximation error in the backward propagation process. In the backward gradient calculation process according to the forward calculation derivation process, the gradient value needs to be approximated when the formula function sign. is approximated. Generally, the formula function sign. is approximated by the "Straight Through Estimator" method, which takes a fixed value as the approximation result, but the estimation accuracy is not high, and cannot provide sufficient accuracy or adaptability.

[0054] In this embodiment, in order to improve the accuracy of gradient approximation in the binarized backward calculation process, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolutional layer in the neural network.

[0055] The parameters can include weight values and activation values. The binarized backward calculation process updates the corresponding parameters by calculating the gradient values of the parameters with respect to the loss function.

[0056] The tangent function can be a parameterized tangent function, which is high in accuracy and continuously derivable, and can be used to approximate the gradient value of the sign function, and replace the sign function to obtain the gradient value of the parameter.

[0057] Finally, the gradient descent algorithm, such as the stochastic gradient descent algorithm, is used to update the parameters of the network according to the gradient value, so as to reduce the value of the loss function.

[0058] The technical scheme of the embodiment of the present disclosure comprises the following steps: obtaining a neural network; performing binary forward calculation on the neural network based on at least one of image data, text data, and voice data to obtain a forward output result; performing binary backward calculation on the neural network based on the forward output result to update the parameters of the neural network; and using a tangent function associated with the parameters of the neural network as a gradient approximation function of each convolutional layer in the neural network during the binary backward calculation. The technical scheme of the embodiment of the present disclosure solves the problem of high binary gradient mismatch degree, large error, and poor network accuracy in data calculation for determining the parameters of the neural network, can reduce the binary gradient mismatch degree in the data calculation process of the neural network, accelerate the convergence process of the data calculation of the neural network, and improve the accuracy of the data calculation of the neural network.

[0059] FIG. 2 is a flowchart of another method for a neural network provided by an embodiment of the present disclosure. The method can be performed by a device for a neural network, which can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, such as a mobile terminal, a PC terminal, or a server.

[0060] As shown in FIG. 2, the method for a neural network comprises the following steps:

[0061] S210, obtaining a neural network.

[0062] S220, inputting at least one of image data, text data, and voice data to the neural network, and performing binary approximation value taking on the weight value corresponding to each neuron node in the calculation of each convolutional layer in the neural network to obtain a corresponding binary approximation weight value. The node, also known as a neuron, is a basic calculation unit in each convolutional layer of the neural network. Each node receives an input signal from another node, and the node that sends the input signal to each node is the input node associated with each neuron node. The corresponding binary approximation weight value is obtained by performing binary approximation value taking on the corresponding weight value.

[0063] The following description is a process of calculating the output activation value of each neuron node of each layer, and is also a process of performing forward calculation layer by layer for the structure of the neural network:

[0064] For example, for a convolutional neural network including L convolutional layers, the l-th (greater than or equal to 1 and less than or equal to L) convolutional layer of the neural network can be represented by a triple <W l ,I l-1 ,*>. Among them, W l ∈R k×k×Cin×Cout represents the floating-point weight value of the layer, Cin represents the input channel, Cout represents the output channel, I l-1 ∈R w×h×Cin represents the input activation value (output activation value of the previous layer) of the layer, and * represents a convolution operation based on multiplication and accumulation. W l,k(k＝1...Cout) ∈R k×k×Cin represents the floating-point weight value of the k-th convolution kernel of the l-th layer. The binarization process of W l is as follows:

[0065] Suppose there are binarized weights B l,k ∈{-1,+1} k×k×Cin and floating-point scaling coefficients a l,k ∈R + , then the floating-point weight value W l,k can be approximated by the binarized weights and the floating-point scaling coefficients as The solving process can be converted into an optimization problem Solving B l,k =Sign(n(W l,k ), where n=k×k×Cin represents the number of elements of the floating-point weight W l,k . Correspondingly, the binarized approximation weight value of the l-th layer weight value can be represented as where B l =[B l,1 ,…,B l,Cout ], a l =[a l,1 ,…,a l,Cout ].

[0066] S230, operate the input activation value of each input node with the corresponding binarized approximation weight value to obtain the output activation value of each neuron node of each layer, until the forward output result of the neural network is obtained.

[0067] The binarization process of the activation value can refer to the above-mentioned binarization process of the weight value, and the input activation value I l-1 ∈R w×h×CinThe corresponding binary approximation activation value can be represented as wherein Q l-1,i ∈{-1,+1} w×h×Cin , and β l-1 ∈R + , and i ranges from (1, …, Cout).

[0068] After the binary approximation process of the weight values and the activation values, the output activation value of each input node is multiplied by the corresponding binary approximation weight value, and the forward output result (output activation value) of the lth layer can be represented as: O l = W l *I l-1 .

[0069] Optionally, the output activation value of each node of each layer is binary approximated.

[0070] Specifically, the input activation value and the corresponding binary approximation weight value are substituted into O l = W l *I l-1 , and the binary approximation value of the output activation value is

[0071] S240, according to the forward output result and the expected output result corresponding to at least one of the image data, the text data and the voice data, the loss function value of the neural network is calculated, and the gradient value of the output node of the neural network is calculated according to the loss function value.

[0072] Specifically, during the neural network calculation process, the loss function value of the neural network can be calculated according to the forward output result and the expected output result corresponding to at least one of the image data, the text data and the voice data input to the neural network. The loss function can measure the difference between the expected output result of the neural network and the forward output result. The smaller the loss function value, the better the prediction effect of the neural network, and the higher the calculation accuracy.

[0073] Further, according to the calculated loss function, the gradient value of the output node of the neural network is calculated by deriving the forward output result.

[0074] S250, according to the chain rule, the gradient value of the output node is conducted layer by layer, and during the gradient conduction process, the tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation is determined to determine the weight gradient value and / or activation value gradient value in the binary backward calculation.

[0075] In a neural network, the output activation value of each node is calculated by its corresponding input value and weight value and activation function. According to the chain rule, the gradient is propagated from the output layer to the input layer by deriving each component function in turn. Thus, the gradient can be propagated between different convolutional layers of the neural network, so as to calculate the contribution of the parameters to the loss function.

[0076] The chain rule plays a key role in the backpropagation of the neural network in this embodiment. The gradient value of the output node is transmitted layer by layer through the chain rule to calculate the gradient value of the parameter, so as to update the parameters of the neural network and minimize the loss function.

[0077] Specifically, taking the neural network in step S220 as an example, the loss function is represented by L, and the weight gradient value is wherein, At this time, it is necessary to solve The sign( ) function used in forward calculation is approximated by the "Straight Through Estimator (STE)". . For example, STE can be represented as y = STE(x) = 1 when |x| < 1 else 0, and else 0. However, since the approximate gradient solved by STE is quite different from the true gradient distribution of the sign( ) function, it causes serious forward and backward gradient adaptation problems, leading to convergence difficulties in the neural network calculation process and low calculation result accuracy.

[0078] To solve the above problems, in this embodiment, the tangent function value corresponding to each binary weight value and / or binary activation value is used to determine the weight gradient value and / or activation value gradient value in the binary backward calculation.

[0079] In an alternative embodiment, the tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation is used to determine the weight gradient value and / or activation value gradient value in the binary backward calculation during the gradient transmission process. The product of each binary weight value and / or binary activation value and the parameter to be updated can be calculated during the gradient transmission process, and the tangent function value of the product is used as the weight gradient value and / or activation value gradient value of the corresponding binary weight value and / or binary activation value. Wherein, the parameter to be updated is learned and updated in the calculation process of the neural network.

[0080] The tangent function can be represented as wherein, p is the parameter to be updated. Since the calculation method of the activation value gradient value and the weight gradient value is similar, taking the calculation of the weight gradient value as an example, the binary weight value is substituted into the tangent function to replace x, and the tangent function value of the product of the binary weight value and the parameter to be updated p is tanh(pWl ), then the gradient weight of the l-th layer can be calculated as Substitute And The weight gradient value can be obtained.

[0081] The comparison between the gradient approximation using the tangent function and the gradient calculation using the STE function is shown in FIG. 3. The left side is the function curve, and the right side is the gradient change. The black curve represents the sign(.) function and the gradient change, the function value of which is 1 when greater than 0, and -1 when less than 0. The gradient value of the sign(.) function is a convex function, which suddenly changes at zero point and cannot be used for calculation. The red curve represents the forward function of the STE function and the corresponding gradient change. The function value is -1 when less than -1, linearly changes between -1 and +1, and is 1 when greater than +1. It can be seen that the gradient of the STE function is very different from that of the sign(.) function, which will cause the gradient of the forward and backward to be not adaptive. The blue curves of different shades in FIG. 3 represent the function and gradient of different values of the to-be-updated parameter p. It can be seen that as the value of p increases, the function curve is more and more similar to the sign(.) function, and the gradient is also similar to the sharp peak distribution. The gradient approximation is more accurate, but the too high gradient may cause unstable calculation. As the value of p decreases, the gradient is more and more similar to the STE function, and the gradient approximation error becomes large at this time, but the calculation is stable. Therefore, in an optional implementation, to balance the accuracy of the gradient approximation and the calculation stability, the initial value of the to-be-updated parameter is set to 2.

[0082] S260, updating the parameters of the neural network according to the weight gradient value and / or the activation value gradient value.

[0083] The parameters of the neural network are updated by the gradient descent algorithm according to the weight gradient value and / or the activation value gradient value, so that the target neural network with better calculation effect is obtained.

[0084] The technical scheme of the embodiment of the present disclosure comprises the following steps: obtaining a neural network; performing binary forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result; calculating a loss function value of the neural network according to the forward output result and an expected output result corresponding to at least one of the image data, the text data and the voice data, and calculating a gradient value of an output node of the neural network according to the loss function value; conducting the gradient value of the output node layer by layer according to the chain rule, and determining a weight gradient value and / or an activation value gradient value in binary backward calculation according to a tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation in the process of gradient conduction; and updating parameters of the neural network according to the weight gradient value and / or the activation value gradient value. In the process of the binary backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network. The technical scheme of the embodiment of the present disclosure solves the problem of high binary gradient mismatch degree, large error and poor neural network precision in network data calculation, can further reduce the binary gradient mismatch degree in the network data calculation process through the tangent function value corresponding to each binary weight value and / or binary activation value, accelerate the convergence process of network data calculation, and improve the precision of network data calculation.

[0085] On the basis of the above-mentioned embodiment, when the calculated neural network is a convolutional neural network, the pooling layer in the neural network can be arranged before the batch normalization layer, which is different from the conventional convolutional neural network structure.

[0086] Specifically, FIG. 4 is a contrast diagram of a convolutional neural network structure provided by the embodiment of the present disclosure. In FIG. 4, the left side is a traditional convolutional neural network structure, and the right side is the convolutional neural network structure of the present embodiment. In the design of the traditional convolutional neural network structure, the structure is composed of a convolutional layer, a batch normalization layer and a pooling layer. Under this structure, the gradient adaptation problem caused by the convolutional layer is further amplified by the batch normalization layer. Therefore, to solve this problem, the pooling layer is arranged before the batch normalization layer in the present embodiment. The present embodiment can effectively alleviate the problem of low precision caused by gradient mismatch by performing a pooling operation in advance, thereby improving the precision of the neural network.

[0087] FIG. 5 is a contrast diagram of another convolutional neural network structure provided by the embodiment of the present disclosure, in which the left side is a traditional convolutional neural network structure, and the right side is the convolutional neural network structure of the embodiment. The traditional convolutional neural network structure is a convolution layer (Conv) - a bias calculation layer (BiasAdd) - a batch normalization layer (BN) - a pooling layer (Pool). In the embodiment, the network structure further includes a binarization convolution layer (BinConv) - a bias calculation layer (BiasAdd) - a pooling layer (Pool) - a batch normalization layer (BN) in the process of binarization quantization calculation of the convolutional neural network.

[0088] The embodiment can effectively alleviate the gradient mismatch caused by binarization processing, cause the whole neural network to have errors, and improve the accuracy of the neural network by performing the pooling operation in advance.

[0089] FIG. 6 is a structural diagram of a device for a neural network provided by the embodiment of the present disclosure, which is suitable for a network computing scenario, especially a weak gradient mismatch network computing scenario. The device for a neural network can be realized in the form of software and / or hardware, and can be configured in an electronic device, which can be a mobile terminal, a PC terminal, or a server.

[0090] As shown in FIG. 6, the network computing device includes a data acquisition module 310, a first data calculation module 320, and a second data calculation module 330.

[0091] The data acquisition module 310 is configured to acquire a neural network.

[0092] The first data calculation module 320 is configured to perform binarization forward calculation on the neural network based on at least one of image data, text data, and voice data, to obtain a forward output result.

[0093] The second data calculation module 330 is configured to perform binarization reverse calculation on the neural network based on the forward output result, to update parameters of the neural network.

[0094] In the process of the binarization reverse calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolution layer in the neural network.

[0095] The technical scheme of the embodiment of the present disclosure is that a neural network is acquired; at least one of image data, text data and voice data is used to perform binary forward calculation on the neural network to obtain a forward output result; the neural network is used to perform binary backward calculation based on the forward output result to update parameters of the neural network; and in the process of the binary backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolution layer in the neural network. The technical scheme of the embodiment of the present disclosure solves the problem of high binary gradient mismatch degree, large error and poor network precision in network calculation, can reduce the binary gradient mismatch degree in the network calculation process, accelerate the convergence process of the network calculation and improve the precision of the network calculation.

[0096] In an optional implementation, the second data calculation module 330 is specifically configured to:

[0097] According to the forward output result and the expected output result corresponding to at least one of the image data, the text data and the voice data, a loss function value of the neural network is calculated, and a gradient value of an output node of the neural network is calculated according to the loss function value; the gradient value of the output node is transmitted layer by layer according to the chain rule, and in the gradient transmission process, a tangent function value corresponding to each binary weight value and / or binary activation value in the binary forward calculation is used to determine a weight gradient value and / or an activation value gradient value in binary backward calculation; and the parameters of the neural network are updated according to the weight gradient value and / or the activation value gradient value.

[0098] In an optional implementation, the second data calculation module 330 is further specifically configured to:

[0099] In the gradient transmission process, the product of each binary weight value and / or binary activation value in the binary forward calculation and a parameter to be updated in the neural network is calculated, and the tangent function value of the product is used as the weight gradient value and / or the activation value gradient value of the corresponding binary weight value and / or binary activation value; and the parameter to be updated is learned and updated in the calculation process of the neural network.

[0100] In an optional implementation, the initial value of the parameter to be updated is set to 2.

[0101] In an optional implementation, the first data calculation module 320 is specifically configured to:

[0102] input at least one of the image data, the text data and the voice data to the neural network, and in the calculation of each convolution layer in the neural network, the weight value corresponding to each input node associated with a neuron node is binarily approximated to obtain a corresponding binary approximation weight value; the input activation value of each input node is operated with the corresponding binary approximation weight value to obtain the output activation value of the corresponding neuron node of each layer until the forward output result of the neural network is obtained.

[0103] In an optional implementation, the first data calculation module 320 is further configured to:

[0104] The output activation value of each node of each layer is binarily approximated.

[0105] In an optional implementation, when the neural network is a convolutional neural network, the pooling layer in the neural network is arranged before the batch normalization layer.

[0106] The apparatus for neural network provided by the embodiments of the present disclosure can perform the method for neural network provided by any of the embodiments of the present disclosure, and has the corresponding function modules and advantages of performing the method.

[0107] It is worth noting that each unit and module included in the apparatus is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for easy to distinguish from each other, and does not limit the protection scope of the embodiments of the present disclosure.

[0108] FIG. 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. Referring to FIG. 7, a structural schematic diagram of an electronic device (for example, a terminal device or a server in FIG. 7) 600 suitable for implementing the embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 7 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0109] As shown in FIG. 7, the electronic device 600 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0110] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 7 shows the electronic device 600 with various devices, it should be understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0111] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0112] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0113] The electronic device provided by the embodiments of the present disclosure and the method for neural network provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiments can be referred to the above-mentioned embodiments, and the present embodiments have the same beneficial effects as the above-mentioned embodiments.

[0114] The embodiments of the present disclosure also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for neural network provided by the above-mentioned embodiments.

[0115] Note that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, a computer readable signal medium can include a computer readable program code carried in a baseband or as a part of a carrier wave, in which the computer readable program code can be used by or in connection with an instruction execution system, apparatus or device. Such a propagated computer readable signal medium can take various forms, including but not limited to electro-magnetic, optical or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus or device, except for the computer readable storage media described above. The program code carried by the computer readable media can be transmitted in any suitable media, including but not limited to wire, cable, fiber optic, RF (radio frequency), or any suitable combination of the foregoing.

[0116] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0117] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and can not be assembled in the electronic device.

[0118] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to:

[0119] obtain a neural network;

[0120] perform binarized forward calculation on the neural network based on at least one of image data, text data and voice data to obtain a forward output result;

[0121] perform binarized backward calculation on the neural network based on the forward output result to update parameters of the neural network;

[0122] wherein, in the process of the binarized backward calculation, a tangent function associated with the parameters of the neural network is used as a gradient approximation function of each convolutional layer in the neural network.

[0123] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0124] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0125] The units described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of a unit does not constitute a limitation on the unit itself. For example, the first obtaining unit can also be described as a unit for obtaining at least two Internet protocol addresses.

[0126] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical wire, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the method for a neural network provided by any one of the embodiments of the present disclosure.

[0129] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0130] The specific embodiments described hereinabove are illustrative of certain embodiments of the present disclosure and the present disclosure is not limited to these specific embodiments. Numerous variations and modifications will occur to those skilled in the art. In particular, it is within the scope of the disclosure to employ, in the construction of the above described technical solution, not necessarily the recited technical features alone, but a combination of the technical features indicated in the above description or shown in the attached drawings. For example, features from one embodiment can be combined with features from another embodiment. In addition, where necessary, the functions of the different means can be performed by the same means providing sequential performance of equivalent functions. The embodiments described above are merely possible implementations provided to enable those skilled in the art to better understand the present disclosure and thereby implement the present disclosure in alternative forms. Accordingly, the claims are not intended to be limited to the preferred and exemplary embodiments described herein.

[0131] Further, although operations have been described in a particular, sequential order, this should not be understood as requiring or implying that any particular order of operation is essential. On the contrary, in certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementations have been described in the context of particular examples, other implementations can also be possible. Specifically, any feature in the singular can also be implemented in the plural, vice versa; one or more features can be combined with another one or more features; any listed implementation of two or more symptoms can be distinguishable from each other or can be combined with one another.

[0132] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for a neural network, comprising: Get the neural network; Performing a binarized forward calculation on the neural network based on at least one of the image data, the text data, and the voice data to obtain a forward output result; Performing a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network; In the process of the binarization reverse calculation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolution layer in the neural network.

2. The method according to claim 1, wherein The performing a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network includes: Calculating a loss function value of the neural network based on the forward output result and an expected output result corresponding to at least one of the image data, text data, and voice data, and calculating a gradient value of an output node of the neural network based on the loss function value; The gradient value of the output node is transmitted layer by layer according to the chain rule, and during the gradient transmission process, the weight gradient value and / or activation value gradient value in the binarization reverse calculation is determined according to the tangent function value corresponding to each binarized weight value and / or binarized activation value in the binarization forward calculation; Update the parameters of the neural network according to the weight gradient value and / or the activation value gradient value.

3. The method according to claim 2, wherein: Determining the weight gradient value and / or activation value gradient value in the binarization reverse calculation according to the tangent function value corresponding to each binarization weight value and / or binarization activation value in the binarization forward calculation during the gradient conduction process includes: During the gradient conduction process, the product of each binarized weight value and / or binarized activation value in the binarized forward calculation and the parameter to be updated in the neural network is calculated, and the tangent function value of the product is used as the weight gradient value and / or activation value gradient value of the corresponding binarized weight value and / or binarized activation value.

4. The method according to claim 3, wherein: The initial value of the parameter to be updated is set to 2.

5. The method according to any one of claims 1 to 4, wherein: The performing binarization forward calculation on the neural network based on at least one of the image data, text data and voice data to obtain a forward output result includes: Inputting at least one of the image data, text data, and speech data into the neural network, and in the calculation of each convolutional layer in the neural network, performing binarization approximation on the weight value corresponding to the input node associated with each neuron node to obtain a corresponding binarized approximate weight value; The input activation value of each input node is calculated with the corresponding binarized approximate weight value to obtain the output activation value of the corresponding neuron node in each layer until the forward output result of the neural network is obtained.

6. The method according to claim 5, further comprising: The output activation value of each node in each layer is binarized and approximated.

7. The method according to any one of claims 1 to 6, wherein: When the neural network is a convolutional neural network, the pooling layer in the neural network is arranged before the batch normalization layer.

8. An apparatus for a neural network, comprising: a data acquisition module configured to acquire a neural network; A first data calculation module is configured to perform a binarization forward calculation on the neural network based on at least one of the image data, the text data, and the voice data to obtain a forward output result; a second data calculation module, configured to perform a binary reverse calculation on the neural network based on the forward output result to update the parameters of the neural network; In the process of the binarization reverse calculation, the tangent function associated with the parameters of the neural network is used as the gradient approximation function of each convolution layer in the neural network.

9. An electronic device comprising: one or more processors; a storage device configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method for a neural network according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method for a neural network according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the computer program implements the method for a neural network according to any one of claims 1 to 7.

Images