Method for training a neural network and neural network training system

Abstract

A method and a neural network training system for training a neural network are described. The method includes providing at least one continuously differentiable model of a neural network. The at least one continuously differentiable model is specific to hardware of the neural network. The method also includes training the neural network in an iterative manner using the at least one continuously differentiable model to provide at least one output for the neural network. Each iteration uses at least one output of a previous iteration and a current continuously differentiable model of the at least one continuously differentiable model.

Description

[0001] [Cross-reference to related applications]

[0002] This application claims the rights to the provisional patent application No. 62 / 664,142, filed on April 28, 2018, entitled “A HARDWARE-AWARE ALGORITHM FOR OFF-LINE TRAINING OF NEURAL NETS”, and the provisional patent application No. 62 / 664,102, filed on April 28, 2018, entitled “A HARDWARE-AWARE ALGORITHM FOR OFF-LINE TRAINING OF NEURAL NETS”, assigned to the assignee of this application, and the non-provisional patent application No. 16 / 122,789, filed on September 5, 2018, and incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to a method for training neural networks, and more specifically, to improving the inference accuracy of neural networks. Background Technology

[0004] Applications involving deep learning neural networks (NNs) or neuromorphic computing (e.g., image recognition, natural language processing, and more generally, various pattern matching or classification tasks) are rapidly becoming as important as general computing. The basic computational elements of a neural network or neuron include multiple inputs and an output. Associated with each input are numbers or weights. Neuron activation is computed by performing a weighted sum of the inputs (using the weights), followed by processing the weighted sum using an activation function. This activation function is typically a thresholding function. Therefore, neurons generally perform vector-matrix product operations or multiply-accumulate (MAC) operations, followed by thresholding of these operations.

[0005] Weights, defined by the mathematical description of a neural network, are real numbers and therefore continuous. However, many hardware constructions of neural networks use, or claim to use, low-precision discrete approximations of the true values ​​of the weights. For example, some recent neural networks are XNOR networks or gated XNOR (GXNOR) networks, which use only two (binary) discrete layers or three (ternary) discrete layers. Such neural networks can use -1 and 1 (binary) weights or -1, 0 and 1 (ternary) weights. Other hardware constructions can use different numbers of discrete weights. While this reduced precision of weights is attractive from a hardware perspective, there is a potential loss in achievable inference accuracy. This is especially true in off-chip training, where training is performed on a different system than the one actually used for inference.

[0006] The extent of the loss inference accuracy depends on the details of the weights and the training algorithm used. A straightforward approach is to perform standard training offline using floating-point weights, and then select discrete "windows" to place mathematical weights within them. An improved form of this algorithm treats the window size as a hyperparameter, which is optimized for validation data to achieve the best accuracy. However, even with this improved form, neural networks using low-precision weights can still suffer a considerable loss of inference accuracy.

[0007] The goal is to improve the inference accuracy of neural networks that use lower-precision weights (even if such networks are trained offline). Summary of the Invention

[0008] This disclosure relates to a method for training a neural network, the method comprising: providing at least one continuously differentiable model of the neural network, the at least one continuously differentiable model being dedicated to the hardware of the neural network; and training the neural network iteratively using the at least one continuously differentiable model to provide at least one output for the neural network, each iteration using at least one output from the previous iteration and the current continuously differentiable model from the at least one continuously differentiable model.

[0009] This disclosure relates to a method for training a neural network, the method comprising: providing at least one continuously differentiable model of the neural network, the at least one continuously differentiable model being specifically designed for hardware of the neural network, the neural network using a plurality of discrete weights; iteratively training the neural network using the at least one continuously differentiable model to provide at least one output for the neural network, each iteration using at least one output from a previous iteration and a current continuously differentiable model among the at least one continuously differentiable models, the current continuously differentiable model providing a closer approximation of the hardware of the neural network than a previous continuously differentiable model among the at least one continuously differentiable models, the iterative training step further comprising: in a... In each iteration, the software model of the neural network is used as the first continuously differentiable model; backpropagation is performed using at least one output from each iteration to obtain at least one weight for each iteration; for each iteration, the following function (f) is applied to the at least one weight from the previous iteration, which is the current continuously differentiable model multiplied by at least one input plus a partial term: f(g(ω,X,α)X+b)).

[0010]

[0011] Where ω is the weight, and Δ is the discretization step order of the at least one discrete weight, ω sc It is the transition scale between steps, ε n It is the offset that makes the continuously differentiable model pass through the origin, and σ is 1 / (1+e -ω / ωsc If the at least one weight is not greater than at least one threshold that is different from at least one previous weight, then the iterative repetition ends. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an exemplary embodiment of a system for training a neural network using a continuously differentiable model.

[0013] Figure 2 This is another block diagram of an exemplary embodiment of a system for training a neural network using a continuously differentiable model.

[0014] Figure 3This is a flowchart illustrating an exemplary embodiment of a method for training a neural network using a continuously differentiable model.

[0015] Figure 4 This is a flowchart illustrating another exemplary embodiment of a method for training a neural network using a continuously differentiable model.

[0016] Figure 5 It is a graph illustrating an exemplary embodiment of a continuously differentiable model.

[0017] Figure 6 It is a drawing A graph of another exemplary embodiment of the continuously differentiable model.

[0018] Figure 7 This is a graph illustrating another exemplary embodiment of a continuously differentiable model.

[0019] Figure 8 This is a graph illustrating an exemplary embodiment of weights calculated using a continuously differentiable model.

[0020] Figure 9 This is a graph illustrating an exemplary embodiment of weights calculated using a continuously differentiable model.

[0021] [Explanation of Symbols]

[0022] 100: System / Training System;

[0023] 110: Training Engine;

[0024] 112: Continuously differentiable models / components / models;

[0025] 112-1: Continuously Differentiable Model / First Continuously Differentiable Model / Model;

[0026] 112-2: Continuously Differentiable Model / Second Continuously Differentiable Model / Model;

[0027] 112-3: Continuously differentiable model / Last continuously differentiable model / Model;

[0028] 114: Training algorithm / component / training function;

[0029] 120: Neural Network;

[0030] 130: Training data / components;

[0031] 150: Processor;

[0032] 160: Data storage;

[0033] 170: Input / output device;

[0034] 200, 210: Methods;

[0035] 202, 204, 212, 214, 216, 218, 220, 222: Steps;

[0036] 300, 310, 320, 330, 340: Curve graph;

[0037] 302, 312, 322: Dashed line / curve / model;

[0038] 304, 314, 324, 334, 344, 354: solid lines;

[0039] 332, 342, 352: dotted lines;

[0040] 336, 346, 356: Minimum values;

[0041] 350: Illustration / Graph. Detailed Implementation

[0042] Exemplary embodiments relate to training neural networks that can be employed in various fields, including but not limited to machine learning, artificial intelligence, neuromorphic computing, and neural networks. The methods and systems are extendable to other applications in which logic devices are used. The following description is provided to enable those skilled in the art to make and use the invention, and is provided in the context of a patent application and its claims. Various modifications to the exemplary embodiments set forth herein, as well as the general principles and features, will be apparent. The exemplary embodiments are primarily described with respect to the specific methods and systems provided in particular embodiments. However, the methods and systems will also function effectively in other embodiments.

[0043] For example, phrases such as "exemplary embodiment," "one embodiment," and "another embodiment" can refer to the same or different embodiments, as well as multiple embodiments. Embodiments will be described with respect to systems and / or apparatuses having certain components. However, systems and / or apparatuses may include more or fewer components than those shown in the figures, and the arrangement and type of components may vary without departing from the scope of the invention. Exemplary embodiments will also be described in the context of specific methods having certain steps. However, the methods and systems described will also function effectively for other methods with different and / or additional steps and steps in a different order, without contradicting the exemplary embodiments. Therefore, the invention is not intended to be limited to the embodiments shown in the figures, but is accorded the broadest scope consistent with the principles and features described herein.

[0044] The terms “a” and “an” and “the”, as well as similar references, used in the context of setting forth this invention (especially in the context of the foregoing claims) are intended to be considered to cover both the singular and the plural, unless otherwise specified herein or obviously contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are intended to be considered as open-ended terms (i.e., meaning “including but not limited to”).

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should be noted that, unless otherwise specified, the use of any and all instances or exemplary terms provided herein is intended only to better illustrate the invention and not to limit its scope. Furthermore, unless otherwise defined, all terms defined in common dictionaries should not be over-interpreted.

[0046] This invention describes a method and system for training a neural network. The method includes providing at least one continuously differentiable model of the neural network. The at least one continuously differentiable model is dedicated to the hardware of the neural network. The method further includes iteratively training the neural network using the at least one continuously differentiable model to provide at least one output to the neural network. Each iteration uses at least one output from the previous iteration and the currently continuously differentiable model from the at least one continuously differentiable model.

[0047] Figure 1 and Figure 2 This is a block diagram illustrating an exemplary embodiment of a system 100 for training a neural network (NN) (e.g., a deep learning neural network). For simplicity, only a portion of system 100 is shown. The training system 100 is used to train a neural network 120. In other words, the training system 100 performs training on the neural network 120 and provides the training results (e.g., weights) to the neural network 120. Although in Figure 1 Neural network 120 is shown; however, in at least some embodiments, neural network 120 is not part of training system 100. Therefore, training system 100 may perform off-chip (or offline) training on neural network 120. Thus, neural network 120 may be implemented as an inference-only neural network.

[0048] Training system 100 includes training data 130 and training engine 110. Training engine 110 includes training algorithm 114 and continuously differentiable models 112-1, 112-2, and 112-3 (collectively referred to as continuously differentiable models 112). Although three continuously differentiable models 112 are shown in the figure, fewer or more models may be used. Training system 100 also includes processor 150, data store 160, and input / output (I / O) devices (170). Data store 160 can store components 112, 114, and 130. Processor 150 can execute training algorithm 114 and continuously differentiable models 112.

[0049] Training system 100 uses one or more of continuously differentiable models 112 to train neural network 120. The continuously differentiable models 112 approximate the behavior of neural network 120. Therefore, the continuously differentiable models 112 take into account the hardware of neural network 120. For example, neural network 120 may be a binary network (with weights of -1 and 1), a ternary network (with weights of -1, 0, and 1), a two-bit weighted network (with weights of -2, -1, 0, 1, and 2), or other neural networks using discrete weights. In these embodiments, the continuously differentiable models 112 can approximate the transitions (e.g., step functions) between weights while maintaining differentiability. This allows gradients to be computed during training. Similarly, neural network 120 may use discrete activations. The continuously differentiable models 112 can provide an approximation of the activations in a manner similar to the approximation provided for the weights. Therefore, the continuously differentiable models 112 represent neural network 120 more accurately while maintaining differentiability.

[0050] Traditional off-chip training does not use continuously differentiable models 112 and may result in poor inference accuracy for neural networks 120. It is believed that the accuracy loss in off-chip training occurs because conventional training methods provide a poor approximation to the hardware of neural networks 120. For example, discrete neural networks (e.g., neural networks using discrete weights or activations) have gradients equal to zero at all points. Such neural networks cannot perform standard backpropagation. Conventional training systems (not shown) can therefore use direct discretization of floating-point weights to obtain binary or ternary (i.e., discrete) weights. This approach introduces weight uncertainty near the discretization boundary. For floating-point trained networks, small weight uncertainties generally have little effect due to the partial elimination of errors occurring in neurons. However, after discretization, the weights near the boundary are located on one side or the other. This can lead to error amplification and potentially a large loss in inference accuracy. In other words, since training is not performed with the knowledge that the weights will be discretized or will change due to hardware limitations, inference accuracy may be significantly reduced.

[0051] In contrast, the continuously differentiable model 112 provides an approximation of the behavior of the hardware of the neural network 120 while maintaining computable gradients. By using the continuously differentiable model (112), the training system 100 becomes aware of the hardware of the neural network 120 being trained. Because this model is differentiable, gradient-based techniques (e.g., backpropagation) can be employed. The training can therefore take into account the discrete nature of the hardware of the neural network 120 while employing accepted training techniques in the training algorithm 114. Thus, the training system 100 can provide hardware-aware training and the inference accuracy of the neural network 120 can be improved.

[0052] Figure 3 This is a flowchart illustrating an exemplary embodiment of a method 200 for training a neural network using a continuously differentiable model of the trained neural network. For simplicity, some steps may be omitted, performed in a different order, and / or combined. Method 200 is also described in the context of training system 100 and neural network 120. However, method 200 may be used in conjunction with another training system and / or another neural network.

[0053] In step 202, one or more continuously differentiable models 112 are provided for the neural network 120 to be trained. The continuously differentiable models 112 are specifically designed for the hardware of the neural network 120. In some embodiments, the continuously differentiable models 112 include software models that are used for floating-point weights and do not take into account aspects of the hardware of the neural network 120 (e.g., discrete weights). Such software models can be used for the first iteration (e.g., continuously differentiable model 112-1). However, subsequent iterations use other continuously differentiable models 112 that provide a closer approximation of the hardware (e.g., continuously differentiable models 112-2 and / or 112-3). Alternatively, all continuously differentiable models 112 approximate the hardware of the trained neural network 120.

[0054] In step 204, iterative training of the neural network 120 is performed using the continuously differentiable model 112. Each training iteration provides an output to the neural network 120. Each iteration uses the output of the previous iteration and the current continuously differentiable model to provide a new output. As described above, in some embodiments, the first iteration is performed using a conventional floating-point / continuous model that does not include discretization or other hardware aspects of the neural network 120. This iteration provides a first approximation of the input (e.g., weights) for subsequent hardware-aware iterations. In other embodiments, all iterations use a hardware-specific continuously differentiable model 112. In these embodiments, the first iteration may use input from a floating-point model or input obtained in another manner.

[0055] The iterative training in step 204 may include using the output to perform backpropagation to obtain the weights for each neuron. These weights are then combined with the next continuously differentiable model 112 for the next iteration. Iterative training continues until the desired result is achieved. For example, training may terminate in response to the weights obtained from the current iteration being within a threshold relative to the weights from the previous iteration.

[0056] For example, step 204 may use a continuously differentiable model 112-1, training data 130, and predetermined weights or activations from the first iteration. The output of the first iteration may undergo backpropagation or other processing to obtain a second set of weights for the second iteration. The second iteration may be trained using training data 130, a continuously differentiable model 112-2, and the second set of weights. The continuously differentiable model 112-2 may better approximate the hardware of the neural network 120 than the first continuously differentiable model 112-1. For example, the transitions between discrete weights may be more abrupt. Based on the output of the second iteration, a third set of weights may be computed. The third iteration may be trained using training data 130, a continuously differentiable model 112-3, and the third set of weights. The continuously differentiable model 112-3 may better approximate the hardware of the neural network 120 than the second continuously differentiable model 112-2. This iterative training in step 204 may continue until certain conditions are met. In some embodiments, the condition is that all computed new weights are within a specific threshold of the previous weights. In this scenario, because the weights may converge to their final values ​​more quickly, it may not be necessary to use all continuously differentiable models 112. In other embodiments, different conditions may be used. For example, it may be necessary to use all available continuously differentiable models 112. Once training is complete, the results (e.g., weights) can be fed into the neural network 120 for use.

[0057] Using method 200, a significant improvement in the inference accuracy of neural network 120 can be achieved. Since the continuously differentiable model 112 provides an approximation of the behavior of the hardware of neural network 120, the training system 100 is aware of the hardware of the neural network 120 being trained. Because this model 112 is differentiable, gradient-based techniques can be employed during training. The training can take into account the discrete nature of the hardware of neural network 120 while employing well-established training techniques. Therefore, the inference accuracy of neural network 120 is improved. Because off-chip training is possible, neural network 120 can be smaller and more efficient. In some applications, only a few iterations are required. For example, acceptable accuracy can be achieved using only one or a few iterations. Therefore, the beneficial effects described herein can be achieved by moderately increasing the training central processing unit (CPU) time.

[0058] Figure 4 This is a flowchart illustrating another exemplary embodiment of method 210 for training a neural network using a continuously differentiable model of the trained neural network. For simplicity, some steps may be omitted, performed in a different order, and / or combined. Method 210 is also described in the context of training system 100 and neural network 120. However, method 210 may be used in conjunction with another training system and / or another neural network.

[0059] In step 212, one or more continuously differentiable models 112 are provided for the neural network 120 to be trained. Therefore, step 212 is similar to step 202. The continuously differentiable models 112 are specifically designed for the hardware of the neural network 120. In some embodiments, the continuously differentiable models 112 include a software model that can be used for the first iteration. However, subsequent iterations use other continuously differentiable models 112 that provide a closer approximation to the hardware (e.g., continuously differentiable models 112-2 and / or 112-3). Alternatively, all continuously differentiable models 112 approximate the hardware of the trained neural network 120.

[0060] For example, a continuously differentiable model 112 can be represented as g(ω,X,α), where ω is the weights, X is the input from the training data 130, and b is the partial term. In some embodiments, each continuously differentiable model g n (where n is the number of discrete levels) is represented as:

[0061]

[0062] Where ω is the weight, and Δ is the discretization step order of the at least one discrete weight, ω sc It is the transition scale between steps, ε n It is the offset that makes each of the at least one continuously differentiable model pass through the origin, and σ is a stretched sigmoid function. In some embodiments, the weight set is characterized by the number of non-negative discrete weights in the set minus one (0 for binary, 1 for ternary, etc.). By reducing the weight parameter ω sc The weighting function becomes an increasingly accurate approximation of the step function. The stretched sigmoid function can be expressed as:

[0063] σ=1 / (1+e -ω / ωsc )

[0064] The continuously differentiable model 112 described above can be used for discrete neural networks. In another embodiment, other hardware models can be used to reflect other aspects of the neural network 120. For example, if the neural network 120 is a pruned analog neural network, the continuously differentiable model 112 can be represented as:

[0065] g prune (ω,Δ,ω sc )=2ωΔ·[σ(ω,Δ,ω sc )+σ(ω,Δ,-ω sc )]

[0066] Where (-Δ, Δ) defines the zero window. In step 212, the range of allowable weights can also be set. This range can be chosen such that the initial values ​​are equivalent to purely floating-point-based training, while the final values ​​are a good approximation of the hardware of the neural network 120.

[0067] In step 214, training is performed to produce a model 112-1 containing the first continuously differentiable term. For example, the standard activation of a software training method (e.g., training function 114) can be a = f(ωX + b), where f is the activation weight, ω is the (floating-point) weight, X is the input, and b is the partial term. A similar form of this function for a continuously differentiable model can be used in method 210. Therefore, for method 210, the activation can be expressed as the activation function:

[0068] a=f(g n (ω,X,α i )X+b)

[0069] Where g n Let be the aforementioned continuously differentiable model 112, where 'a' is the reality parameter of the current continuously differentiable model, 'i' is the iteration number, and 'b' is a partial term. The partial term 'b' and its weights can be determined iteratively. Therefore, the partial term used for the next iteration can be determined using the current continuously differentiable model. Alternatively, the partial term can be determined using a different mechanism that includes a continuously differentiable model (e.g., the previous or last one) that differs from the current continuously differentiable model.

[0070] For the first iteration in step 214, the standard (unmodified) activation a = f(ωX + b) can be used. In other embodiments, the output of the standard activation can be used to compute the weights for the first iteration in step 214. For method 210, it is assumed that the first iteration uses the weights from the activation described above as initial weights. Therefore, the first iteration in step 214 can use a = f(g(ω,X,α1)X + b), where g(ω,X,α1) is the continuously differentiable model 112-1. Step 214 may also include determining the weights, for example, using backpropagation or a similar method employing gradients. Since the continuously differentiable model 112-1 has gradients, these methods can be used.

[0071] Step 216 may optionally validate the result of the first iteration. Validation may include performing it using the weights determined in step 214 and the last continuously differentiable model 112-3. In some embodiments, a model with noise from the training process may also be applied as part of the validation step. In other embodiments, other validation mechanisms may be used and / or validation may be performed at the end of method 210.

[0072] Steps 214 and (optionally) 216 are iteratively repeated via step 218. In each iteration, the weights from the previous iteration are used, along with the next continuously differentiable models 112-2 and 112-3. Each continuously differentiable model 112-2 and 112-3 approximates the hardware of the neural network 120 more accurately than the previous continuously differentiable models 112-1 and 112-2, respectively. Therefore, the second iteration can use the weights determined in the first iteration and a = f(g(ω,X,α2)X+b), where g(ω,X,α2) is the continuously differentiable model 112-2. The third iteration can use the weights determined in the second iteration and a = f(g(ω,X,α3)X+b), where g(ω,X,α3) is the continuously differentiable model 112-3.

[0073] For example, Figure 5 , Figure 6 and Figure 7 These are graphs 300, 310, and 320 illustrating exemplary embodiments of the continuously differentiable model 112. Graphs 300, 310, and 320 are for illustrative purposes only and are not intended to be limiting or to illustrate specific results. Each graph 300, 310, and 320 includes a dashed line representing a real number. Graph 300 corresponds to model 112-1, graph 310 to model 112-2, and graph 320 to model 112-3. The dotted lines 302, 312, and 322 are functions g(ω,X,α1), g(ω,X,α2), and g(ω,X,α3) of the discrete weights for the continuously differentiable models 112-1, 112-2, and 112-3, respectively. Therefore, the dotted lines 302, 312, and 322 are the first, second, and third approximations of the discrete weights. Each model 112-1, 112-2, and 112-3 is continuously differentiable. This is illustrated by solid lines 304, 314, and 324, which represent the gradients of curves 302, 312, and 322, respectively. Figures 5 to 7 As can be seen, each of the models 112-1 / 302, 112-2 / 312 and 112-3 / 322 improves the approximation of the hardware of the neural network 120 that utilizes discrete weights.

[0074] By step 220, training ends when an acceptable result is achieved. In some embodiments, this occurs when fewer iterations have been performed than for the continuously differentiable model 112. For example, for system 100, two or three iterations may be performed, depending on how quickly an acceptable result is achieved. In some embodiments, training ends when the weights determined in a particular iteration do not exceed a threshold relative to the weights determined in the immediately preceding iteration. In some such embodiments, training ends when the weights determined in a particular iteration are less than a threshold relative to the weights determined in the immediately preceding iteration. In other embodiments, other termination conditions may be used.

[0075] Through step 222, once acceptable weights have been determined, the results are provided to neural network 120. Thus, off-chip training of neural network 120 can be completed. For example, Figure 8 and Figure 9 Graphs 330 and 340, and inset 350, are used to illustrate the difference between floating-point modeling and training using method 200. Graphs 330, 340, and 350 are for illustrative purposes only and are not intended to represent specific neural networks. In graphs 330, 340, and 350, dashed lines 332, 342, and 352 represent weights calculated using method 210, while solid lines 334, 344, and 354 represent weights calculated using the conventional method. The boundaries between the discrete values ​​of the weights are represented by the minimum values ​​336, 346, and 356, respectively. Figures 8 to 9 As can be seen from the above, the method described above can divide the weights into discrete values ​​of the weights.

[0076] Using method 210, a significant improvement in the inference accuracy of neural network 120 can be achieved. Since the continuously differentiable model 112 provides an approximation of the behavior of the hardware of neural network 120, the training system 100 is aware of the hardware of the neural network 120 being trained. For example, the training incorporates the discrete nature of the weights used in neural network 120. Because this model 112 is differentiable, gradient-based techniques can be employed. Therefore, the inference accuracy of neural network 120 is improved. Because off-chip training can be performed, neural network 120 can be smaller and more efficient. In some applications, only a fewer number of iterations are required.

[0077] Therefore, using methods 200 and 210, hardware-aware training can be performed on the neural network 120. This results in improved inference accuracy. The methods and systems described herein have been illustrated with reference to exemplary embodiments shown in the figures, and those skilled in the art will readily recognize that various changes can be made to the embodiments, and any such changes are within the spirit and scope of the methods and systems. Therefore, many modifications can be made by those skilled in the art without departing from the spirit and scope of the appended claims.

Claims

1. A method for training a neural network using low-precision weights, wherein the neural network is trained off-chip via a training system including a processor and memory, the method comprising: Provide at least one continuously differentiable model of the neural network, the at least one continuously differentiable model being dedicated to the hardware of the neural network, wherein the neural network performs image recognition, natural language processing, pattern matching, or pattern classification tasks. The neural network is trained iteratively by perceiving the hardware of the neural network using the at least one continuously differentiable model, providing at least one output to the neural network, and using the at least one output to perform backpropagation to obtain at least one weight, the at least one weight being used for the next iteration, each iteration using at least one output from the previous iteration and the current continuously differentiable model in the at least one continuously differentiable model, such that the current continuously differentiable model in the at least one continuously differentiable model is closer to the hardware of the neural network than the previous continuously differentiable model in the at least one continuously differentiable model; The activation function of the iterative training step is a function of the current continuously differentiable model and the partial term, wherein the partial term is determined in each iteration, and the current continuously differentiable model used in the activation function is different for each iteration; The at least one continuously differentiable model is represented as: in These are weights, and (-Δ, Δ) defines a zero window. It is the transition scale between steps, and It is a stretched S-shaped function; The iterative training step for the neural network is performed off-chip, and the method further includes: At least one final output is provided to the neural network.

2. The method according to claim 1, wherein the first continuously differentiable model in the at least one continuously differentiable model is a software model.

3. The method of claim 1, wherein each of the current continuously differentiable models of the neural network provides a closer approximation of the hardware of the neural network than the previous continuously differentiable model.

4. The method of claim 1, wherein the step of training in an iterative manner further comprises: The process ends if the at least one weight is not greater than at least one threshold relative to at least one previous weight.

5. The method of claim 1, wherein the continuously differentiable model is based on the at least one weight and at least one input of the iterative training step.

6. The method of claim 5, wherein the activation function in the iterative training step is a function of the at least one input and the realism parameters of the currently continuously differentiable model.

7. The method of claim 1, wherein the step of training in an iterative manner further comprises: The partial term is determined using the currently continuously differentiable model.

8. The method of claim 1, wherein the step of training in an iterative manner further comprises: The partial term is determined using at least one other continuously differentiable model that is different from the currently continuously differentiable model.

9. The method of claim 5, wherein the neural network utilizes at least one discrete weight.

10. The method of claim 1, wherein the step of training in an iterative manner further comprises: After each iteration, the last continuously differentiable model among the at least one continuously differentiable models is used to verify the at least one weight.

11. The method of claim 10, wherein the verification step includes applying a process noise model.

12. The method of claim 1, wherein the step of training in an iterative manner further comprises: The currently continuously differentiable model is applied to at least one activation and at least one weight.

13. A method for training a neural network using low-precision weights, wherein the neural network is trained off-chip via a training system including a processor and memory, the method comprising: Provide at least one continuously differentiable model of the neural network, the at least one continuously differentiable model being dedicated to the hardware of the neural network, wherein the neural network performs image recognition, natural language processing, pattern matching, or pattern classification tasks. The neural network is trained iteratively by using the at least one continuously differentiable model to perceive the hardware for the neural network, performing backpropagation using the at least one output to obtain at least one weight for the next iteration, the at least one weight being used for the next iteration to provide at least one output for the neural network, each iteration using at least one output from the previous iteration and the current continuously differentiable model in the at least one continuously differentiable model, such that the current continuously differentiable model in the at least one continuously differentiable model is closer to the hardware of the neural network than the previous continuously differentiable model in the at least one continuously differentiable model; The activation function of the iterative training step is a function of the current continuously differentiable model and the partial term, wherein the partial term is determined in each iteration and the current continuously differentiable model used in the activation function is different for each iteration; The iterative training step further includes: The partial term is determined using at least one other continuously differentiable model that is different from the current continuously differentiable model; Each of the at least one continuously differentiable models It is represented as: in It represents the weights, where Δ is the discretization step order of at least one discrete weight. It is the transition scale between steps. It is the offset of each of the at least one continuously differentiable models through the origin, and It is a stretched S-shaped function.

14. The method of claim 13, wherein the stretched S-shaped function is expressed as .

15. A method for training a neural network using low-precision weights, wherein the neural network is trained off-chip via a training system including a processor and memory, the method comprising: Provide at least one continuously differentiable model of the neural network, the at least one continuously differentiable model being dedicated to the hardware of the neural network, the neural network using multiple discrete weights, wherein the neural network performs image recognition, natural language processing, pattern matching, or pattern classification tasks. The neural network is trained iteratively using the hardware of the at least one continuously differentiable model-aware system to provide at least one output. Each iteration uses at least one output from the previous iteration and the current continuously differentiable model from the at least one continuously differentiable model, such that the current continuously differentiable model is closer to the hardware of the neural network than the previous continuously differentiable model from the at least one continuously differentiable model. The iterative training step further includes: The software model of the neural network is used as the first continuously differentiable model in the first iteration; Backpropagation is performed using at least one output from each iteration to obtain at least one weight for each iteration; For each iteration, the following function (f) is applied to the at least one weight from the previous iteration: : Multiply the currently continuously differentiable model by at least one input and add a partial term, where g is the currently continuously differentiable model. X is the at least one weight in the current iteration, X is the at least one input, and b is a partial term. It is a realism parameter representing the proximity to the hardware of the neural network, and wherein the currently continuously differentiable model is in It represents the weights, where Δ is the discretization step order of at least one discrete weight. It is the transition scale between steps. It is the offset that makes the continuously differentiable model pass through the origin, and yes ;and If the at least one weight is not greater than at least one threshold relative to at least one previous weight, then the iterative training step ends.

16. A neural network training system using low-precision weights built off-chip using at least one computing device, said at least one computing device including at least one processor and memory, the training system comprising: At least one continuously differentiable model of the neural network, each of the continuously differentiable models being dedicated to the hardware of the neural network, wherein the neural network performs image recognition, natural language processing, pattern matching, or pattern classification tasks. A training subsystem iteratively uses at least one continuously differentiable model of the neural network and at least one input. The training subsystem is configured to train the neural network iteratively by using the at least one continuously differentiable model to be aware of the hardware for the neural network, providing at least one output to the neural network, performing backpropagation using the at least one output to obtain at least one weight, which is used for the next iteration such that each iteration uses at least one output from the previous iteration and the current continuously differentiable model among the at least one continuously differentiable models is closer to the hardware of the neural network than the previous continuously differentiable model among the at least one continuously differentiable models. The activation function of the iterative training step is a function of the current continuously differentiable model and a bias term, wherein the bias term is determined in each iteration and the current continuously differentiable model used in the activation function is different for each iteration. The model that is continuously differentiable It is represented as: in It represents the weights, where Δ is the discretization step order of at least one discrete weight. It is the transition scale between steps. It is the offset that makes the continuously differentiable model pass through the origin, and It is by Let n represent the stretched sigmoid function, where n is the number of iterations.

Images