A deep neural network training method and device

Abstract

The present invention discloses a deep neural network training method and device, the method comprising: establishing a deep neural network framework; obtaining data to be processed in full-precision floating-point format; using the data to be processed in full-precision floating-point format to perform preheating training on the deep neural network framework; converting the data to be processed in full-precision floating-point format into Posit format data; and using the Posit format data to train the deep neural network framework after preheating training. Compared with the prior art, this embodiment uses the data to be processed in full-precision floating-point format to perform preheating training on the deep neural network framework, then converts the data to be processed in full-precision floating-point format into Posit format data, and uses the Posit format data to train the deep neural network framework after preheating training, which can reduce the occupation of storage resources, shorten the training time, and improve the training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a deep neural network training method and device. Background Art

[0002] Deep learning is a new field in machine learning research. Its motivation is to build and simulate neural networks that analyze and learn in the human brain. It imitates the mechanism of the human brain to interpret data, such as images, sounds, and text. Deep learning is a type of unsupervised learning. The concept of deep learning originated from the study of artificial neural networks. For example, a multi-layer perceptron with multiple hidden layers is a deep learning structure. Deep learning combines low-level features to form more abstract high-level representations of attribute categories or features to discover distributed feature representations of data. With the rapid development of deep learning technology, deep convolutional neural networks play an important role in image classification, speech recognition and other fields. However, deep neural network models have become increasingly large and complex in the process of development. Both training and inference processes have put forward high requirements for computing resources and storage space, which is a huge challenge for the efficient deployment of models. At present, deep neural network training is mainly completed using floating-point format data. Using full-precision 32-bit floating-point arithmetic units to train deep neural networks not only occupies a large amount of storage resources, but also takes a long time to train and is inefficient. Summary of the invention

[0003] The purpose of the present invention is to provide a deep neural network training method and device to solve the problem that the prior art uses floating-point arithmetic units to train deep neural networks, which not only occupies a large amount of storage resources, but also takes a long time to train and has low efficiency.

[0004] On the one hand, the present invention provides a deep neural network training method, which includes: establishing a deep neural network framework; obtaining data to be processed in full-precision floating-point format; using the data to be processed in full-precision floating-point format to perform preheating training on the deep neural network framework; converting the data to be processed in full-precision floating-point format into Posit format data; and using the Posit format data to train the deep neural network framework after preheating training.

[0005] In combination with one aspect, in a first possible implementation method, converting the data to be processed in the full-precision floating-point format into Posit format data includes: calculating the clamped input value of the data to be processed in the full-precision floating-point format according to the bit width n and the exponent bit width es; extracting the sign, exponent value and mantissa value of the clamped input value; calculating the sign, organization value, exponent value and mantissa value of the Posit format data according to the sign, exponent value and mantissa value of the clamped input value; generating Posit format data according to the sign, organization value, exponent value and mantissa value of the Posit format data.

[0006] In combination with the first possible implementation manner, in a second possible implementation manner, converting the to-be-processed data in the full-precision floating-point format into Posit format data further includes:

[0007] The exponent bit width es is obtained according to the data distribution state of the deep neural network framework after the preheating training.

[0008] In combination with the first possible implementation method, in a third possible implementation method, calculating the clamped input value of the data to be processed in the full-precision floating-point format according to the bit width n and the exponent bit width es includes: calculating the used value according to the exponent bit width es; calculating the maximum positive number and the minimum integer according to the used value and the bit width n; calculating the clamped input value according to the maximum positive number and the minimum integer.

[0009] In combination with the third possible implementation method, in a fourth possible implementation method, calculating the sign, organization value, exponent value and mantissa value of the Posit format data according to the sign, exponent value and mantissa value of the clamped input value includes: taking the sign of the clamped input value as the sign of the Posit format data; calculating the organization value of the Posit format data; calculating the organization bit width, exponent bit width and mantissa bit width according to the organization value of the Posit format data; calculating the exponent value and mantissa value of the Posit format data according to the exponent bit width and the mantissa bit width.

[0010] In combination with one aspect, in a fifth possible implementation, after preheating the deep neural network framework using the data to be processed in the full-precision floating-point format, the method also includes: extracting the data distribution state of the deep neural network framework after the preheating training; calculating the scaling factor sf added to the tensor X during the conversion according to the data distribution state; scaling the deep neural network framework after the preheating training according to the scaling factor sf; and training the scaled deep neural network framework using the Posit format data.

[0011] In combination with the fifth possible implementation manner, in a sixth possible implementation manner, calculating the scaling factor sf added to the tensor X during conversion according to the distribution state includes: obtaining the distribution center and a constant correction amount of the tensor X in the log domain; and calculating the scaling factor sf according to the distribution center and the constant correction amount.

[0012] In combination with one aspect, in a seventh possible implementation, the deep neural network framework includes: a model structure, a training algorithm, and model hyperparameters of a deep neural network.

[0013] In combination with the seventh possible implementation, in an eighth possible implementation, the model structure of the deep neural network is a ResNet-18 deep residual network model, the training algorithm is a stochastic gradient descent algorithm, and the model hyperparameters include an initial learning rate and a total number of iterations.

[0014] A second aspect of the embodiments of the present disclosure provides a deep neural network training device, including:

[0015] A framework building unit, used to build a deep neural network framework;

[0016] A data acquisition unit, used for acquiring the data to be processed in full-precision floating point format;

[0017] A preheating training unit, used for performing preheating training on the deep neural network framework using the to-be-processed data in the full-precision floating-point format;

[0018] A format conversion unit, used for converting the to-be-processed data in the full-precision floating-point format into Posit format data;

[0019] A training unit is used to train the deep neural network framework after the preheating training using the Posit format data.

[0020] It can be seen from the above embodiments that a deep neural network framework is established; data to be processed in full-precision floating-point format is obtained; the deep neural network framework is preheated and trained using the data to be processed in full-precision floating-point format; the data to be processed in full-precision floating-point format is converted into Posit format data; and the deep neural network framework after preheating and training is trained using the Posit format data. Compared with the prior art, this embodiment uses the data to be processed in full-precision floating-point format to preheat and train the deep neural network framework after preheating and training, thereby reducing the usage of storage resources, shortening the training time, and improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of an embodiment of a deep neural network training method of the present invention;

[0023] Figure 2 A flow chart for converting the to-be-processed data in full-precision floating-point format into data in Posit format according to the present invention;

[0024] Figure 3 This is a framework diagram of the deep neural network training device of the present invention;

[0025] Figure 4 A schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1 , is a flowchart of an embodiment of a deep neural network training method of the present invention, the method comprising the following steps:

[0028] Step 101: Establish a deep neural network framework. A neural network is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for processing imprecise and fuzzy information processing problems that require simultaneous consideration of many factors and conditions. The embodiment of the present invention can be used for the training of deep convolutional neural networks. The deep neural network framework may include: a model structure of a deep neural network, a training algorithm, and model hyperparameters.

[0029] Commonly used deep neural network frameworks include: feedforward neural network, radial basis neural network, Hopfield network, etc. The model structure of the deep neural network of the embodiment of the present invention can be a ResNet-18 deep residual network model. The depth of the classic convolutional neural network is crucial to the performance of the model to a certain extent. When the number of network layers is increased, the network can extract more complex feature patterns, so when the model is deeper, better results can be achieved in theory. However, the deep network has the problem of gradient disappearance or explosion, which makes it difficult to train deep learning models. For traditional convolutional neural networks, directly stacked networks are equivalent to doing affine transformation-nonlinear transformation layer by layer, and the affine transformation step is mainly matrix multiplication. So in general, the directly stacked network is equivalent to the calculation of multiplication properties. In ResNet, compared with the directly stacked network, because of the emergence of shortcut, the nature of the calculation changes from multiplication to addition, and the calculation becomes more stable. The ResNet deep residual network model can solve the degradation phenomenon well, and the error is also lower.

[0030] The training algorithm may be a stochastic gradient descent algorithm, and the model hyperparameters may include an initial learning rate and a total number of iterations. The gradient descent algorithm includes a variety of different algorithms, including a batch gradient algorithm, a stochastic gradient algorithm, a compromise gradient algorithm, and the like. For the stochastic gradient descent algorithm, it continuously judges and selects the optimal path under the current goal, so that the optimal result can be achieved under the shortest path. We can take a person going down a hill as an example. If we want to reach the bottom of the hill faster, the easiest way is to go down the hill in the steepest direction at the current position, and then continue to look for the steepest direction after reaching another position. In this way, the method of stopping to observe the lowest route every step is the essence of the stochastic gradient descent algorithm.

[0031] Step 102: Obtain the data to be processed in full-precision floating-point format. The floating-point numbers in the computer adopt the IEEE754 standard. Floating-point numbers are divided into single-precision floating-point numbers (32 bits) and double-precision floating-point numbers (64 bits). Although the range of floating-point numbers is very wide, due to the existence of precision loss and the amplification effect of the power, a floating-point number actually represents a rational number interval around it. If the floating-point numbers are plotted on a number axis, intuitively, the floating-point numbers appear denser near zero. The closer to infinity, the sparser the distribution of floating-point numbers, and a floating-point value represents a piece of data around it. As shown in the figure below. In this sense, floating-point numbers should not be directly compared for equality. They represent a data range. Therefore, using full-precision floating-point format to represent the data to be processed is more likely to occupy a large amount of storage resources. Directly using full-precision floating-point format data to train deep neural networks is likely to cause long training time and low training efficiency.

[0032] Step 103: Preheat training the deep neural network framework using the data to be processed in the full-precision floating-point format. The purpose of preheating the established deep neural network framework using the data to be processed in the full-precision floating-point format is to stabilize the initial data distribution and extract the data distribution state of the deep neural network framework after the preheat training. Depending on the task and model structure, the preheat training needs to be performed 1-5 iterations. In the subsequent iteration process, the distribution range of the tensors in the model remains basically stable.

[0033] After preheating the deep neural network framework using the data to be processed in the full-precision floating-point format, the method may also include: extracting the data distribution state of the deep neural network framework after the preheating training; calculating the scaling factor sf added to the tensor X during the conversion according to the data distribution state; scaling the deep neural network framework after the preheating training according to the scaling factor sf; and training the scaled deep neural network framework using the Posit format data.

[0034] Specifically, according to the distribution state of each layer of data at the end of the warm-up training, the scaling factor sf added to each tensor X during the conversion is determined to ensure that as much data as possible obtains a higher representation accuracy. Calculating the scaling factor sf added to the tensor X during the conversion according to the distribution state includes: obtaining the distribution center and constant correction amount of the tensor X in the log domain; calculating the scaling factor sf according to the distribution center and the constant correction amount. The calculation formula of the scaling factor sf is as follows:

[0035] sf=2 c+σ , where c = round(mean(log2(|X|))), c is the distribution center of tensor X in the log domain. The data near c is scaled to have higher precision. σ is a constant positive integer used for correction to slightly shift the distribution toward larger values. According to the distribution state of each layer of data at the end of warm-up training, the exponential bit width es of the posit corresponding to each tensor during conversion is determined to ensure that its dynamic range can meet the requirements of the numerical distribution range and avoid overflow.

[0036] Step 104: Convert the data to be processed in the full-precision floating-point format into Posit format data. Posit is a new data format, which is different from the traditional floating-point number and contains four components: a sign bit (s), an organization bit (k), an exponent bit (e), and a mantissa bit (f). The organization bit adopts variable-length coding, and k consecutive 1s plus 1 0 represent a value of k-1, and k consecutive 0s plus 1 1 represent a value of -k. For a posit number (p) with a bit width of n and an exponent bit width of es, the real value it represents is:

[0037] in used can be changed by adjusting the size of es to achieve a good balance between its dynamic range and representation accuracy.

[0038] Specifically, Figure 2 As shown, converting the to-be-processed data in the full-precision floating-point format into Posit format data includes the following steps:

[0039] Step 401: Calculate the clamped input value of the to-be-processed data in the full-precision floating-point format according to the bit width n and the exponent bit width es.

[0040] The bit width n can be customized, and the exponential bit width es can be obtained based on the data distribution state of the deep neural network framework after the preheat training. According to the obtained exponential bit width es, the formula Get used. According to the obtained used and bit width n, the range that the Posit format data can represent, namely the maximum positive integer maxpos and the minimum positive integer minpos, can be determined. The expression formula of the maximum positive integer maxpos is as follows: maxpos = used n-2 , the minimum positive integer minpos is expressed as follows: minpos = used 2-n Finally, determine whether the input data is 0. If so, the final output is 0. Otherwise, the input value is clamped according to the maximum positive integer maxpos and the minimum positive integer minpos. The formula for clamping the input value is: y=min{max{abs(x),minpos},maxpos}.

[0041] Step 402: extract the sign s, exponent value and mantissa value of the clamped input value, wherein the exponent value is represented by: exp=floor(log2y), and the mantissa value is represented by: mant=y / exp-1.

[0042] Step 403: Calculate the sign, organization value, exponent value and mantissa value of the Posit format data according to the sign s, exponent value exp and mantissa value mant of the clamped input value. The sign of the Posit format data is the sign s of the clamped input value, and the organization value of the Posit format data is represented by: k = floor (exp / 2 es ), the width of the organization bits, exponent bits, and mantissa bits can be determined accordingly.

[0043] The width kb of the organization bit can be expressed as:

[0044] The width of the exponent bit can be expressed as: eb = min{n-1-kb,es};

[0045] The width of the mantissa bits can be expressed as: fb=min{n-1-kb-rb,0}.

[0046] Correspondingly, the exponent value e and the mantissa value f can be calculated.

[0047] The calculation formula of the exponential value e is: e = floor ((exp-k·2 es )·2 eb-es )·2 es-eb ;

[0048] The calculation formula for the tail value f is: f = floor ((man·2 fb )·2 -fb .

[0049] Step 404: Generate Posit format data according to the sign, organization value, exponent value and mantissa value of the Posit format data.

[0050] Finally, according to the obtained symbol s, organization value k, exponent value e and tail value f, the Posit expression formula is used:

[0051] Get the corresponding Posit format data after conversion.

[0052] Step 105: Use the Posit format data to train the preheated deep neural network framework. Replace the full-precision floating-point number operation with the Posit number operation form converted in step 104, and train according to the original full-precision floating-point number benchmark strategy (i.e., use the same hyperparameter settings). The variables that need to be replaced include the activation values, weights, and gradients of the convolutional layer, the fully connected layer, and the batch normalization layer.

[0053] It can be seen from the above embodiments that a deep neural network framework is established; data to be processed in full-precision floating-point format is obtained; the deep neural network framework is preheated and trained using the data to be processed in full-precision floating-point format; the data to be processed in full-precision floating-point format is converted into Posit format data; and the deep neural network framework after preheating and training is trained using the Posit format data. Compared with the prior art, this embodiment uses the data to be processed in full-precision floating-point format to preheat and train the deep neural network framework after preheating and training, thereby reducing the usage of storage resources, shortening the training time, and improving the training efficiency.

[0054] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0055] In addition, as an implementation of the above embodiments, the present disclosure also provides a deep neural network training device, which is located in a terminal, such as Figure 3 As shown, the device includes: a framework building unit 10, a data acquisition unit 20, a warm-up training unit 30, a format conversion unit 40 and a training unit 50. Among them:

[0056] The framework building unit 10 is configured to build a deep neural network framework;

[0057] The data acquisition unit 20 is configured to acquire the data to be processed in a full-precision floating point format;

[0058] The preheating training unit 30 is configured to perform preheating training on the deep neural network framework using the to-be-processed data in the full-precision floating point format;

[0059] The format conversion unit 40 is configured to convert the to-be-processed data in the full-precision floating point format into Posit format data;

[0060] The training unit 50 is configured to train the preheated deep neural network framework using the Posit format data.

[0061] The deep neural network training device provided by the disclosed embodiment establishes a deep neural network framework; obtains the data to be processed in full-precision floating-point format; performs preheating training on the deep neural network framework using the data to be processed in full-precision floating-point format; converts the data to be processed in full-precision floating-point format into Posit format data; and trains the deep neural network framework after preheating training using the Posit format data. Compared with the prior art, the present embodiment uses the data to be processed in full-precision floating-point format to perform preheating training on the deep neural network framework, converts the data to be processed in full-precision floating-point format into Posit format data, and uses the Posit format data to train the deep neural network framework after preheating training, which can reduce the occupancy of storage resources, shorten the training time, and improve the training efficiency.

[0062] Figure 4 8 is a block diagram of a terminal 800 according to an exemplary embodiment. For example, the terminal 800 may be a message transceiving device, a tablet device, a computer, etc.

[0063] Reference Figure 4 The terminal 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power supply component 806 , an input / output interface 808 , and a communication component 810 .

[0064] The processing component 802 generally controls the overall operation of the terminal 800, such as display, data communication, recording operation and other related operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components.

[0065] The memory 804 is configured to store various types of data to support operations at the terminal 800. The power component 806 provides power to various components of the terminal 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal 800. The input-output interface 808 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.

[0066] The communication component 810 is configured to facilitate wired or wireless communication between the terminal 800 and other devices. The terminal 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 810 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 810 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0067] In an exemplary embodiment, terminal 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0068] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0069] The above-described embodiments of the present invention do not constitute a limitation on the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A deep neural network training device, characterized in that: include: A data acquisition unit, used for acquiring the data to be processed in full-precision floating point format; A format conversion unit is used to convert the data to be processed in the full-precision floating-point format into Posit format data; wherein the clamped input value of the data to be processed in the full-precision floating-point format is calculated according to the bit width n and the exponent bit width es, including: The used value is calculated according to the exponent bit width es, and the formula is: ; Calculate the maximum positive integer maxpos and the minimum positive integer minpos according to the used value and the bit width n; The maximum positive integer maxpos is represented by: maxpos = used n-2 ; The minimum positive integer minpos is represented by: minpos = used 2-n ; Among them, the bit width n is customized; Determine whether the data to be processed in the full-precision floating-point format is 0, if so, the final output is 0, otherwise calculate the clamped input value according to the maximum positive integer maxpos and the minimum positive integer minpos; The formula for clamping the input value is: y = min{max{abs(x),minpos},maxpos}; Extracting the sign, exponent value and mantissa value of the clamped input value; calculating the sign, organization value, exponent value and mantissa value of the Posit format data according to the sign, exponent value and mantissa value of the clamped input value; generating Posit format data according to the sign, organization value, exponent value and mantissa value of the Posit format data; A pre-training unit is used to perform pre-training on a deep neural network framework using the to-be-processed data in the full-precision floating-point format, extract the data distribution state of the pre-trained deep neural network framework; calculate the scaling factor sf added to the tensor X during the conversion according to the data distribution state; scale the pre-trained deep neural network framework according to the scaling factor sf to obtain a scaled deep neural network framework; A training unit is used to train the scaled deep neural network framework using the Posit format data to obtain a training result.

2. The device according to claim 1, characterized in that Converting the to-be-processed data in full-precision floating-point format into Posit format data further includes: The exponent bit width es is obtained according to the data distribution state of the deep neural network framework after the preheating training.

3. The device according to claim 1, characterized in that Calculating the sign, organization value, exponent value, and mantissa value of the Posit format data according to the sign, exponent value, and mantissa value of the clamped input value includes: Using the sign of the clamped input value as the sign of the Posit format data; Calculate the organization value of the Posit format data; Calculate the organization bit width, exponent bit width and mantissa bit width according to the organization value of the Posit format data; The exponent value and the mantissa value of the Posit format data are calculated according to the exponent bit width and the mantissa bit width.

4. The device according to claim 1, characterized in that Calculating the scaling factor sf added to the tensor X during conversion according to the distribution state includes: Get the distribution center and constant correction of tensor X in the log domain; The scaling factor sf is calculated based on the distribution center and the constant correction amount.

5. The device according to claim 1, characterized in that The deep neural network framework includes: a deep neural network model structure, a training algorithm, and model hyperparameters.

6. The device according to claim 5, characterized in that The model structure of the deep neural network is the ResNet-18 deep residual network model, the training algorithm is the stochastic gradient descent algorithm, and the model hyperparameters include the initial learning rate and the total number of iterations.

Images