Convolutional neural network training method, device, electronic device and readable storage medium

Abstract

The embodiments of the present disclosure disclose a convolutional neural network training method, apparatus, electronic device, and readable storage medium. The convolutional neural network training method includes: inputting a training image into the convolutional neural network; based on the training image, alternatingly performing a forward propagation process and a backward propagation process of the convolutional neural network to adjust the parameters of the convolutional neural network, wherein, during the backward propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the output of a specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format based on the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a convolutional neural network training method, device, electronic device, and readable storage medium. Background Art

[0002] In recent years, convolutional neural networks have achieved great success in image retrieval, object detection, video understanding, and other fields. However, the huge computational cost of deploying or training a deep convolutional neural network model has brought great challenges to many applications, as the training process involves a large number of floating-point operations, resulting in huge computational overhead and resource consumption. Summary of the Invention

[0003] In order to solve the problems in the related art, the embodiments of the present disclosure provide a convolutional neural network training method, device, electronic device and readable storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a convolutional neural network training method.

[0005] Specifically, the convolutional neural network training method includes:

[0006] Inputting a training image into the convolutional neural network;

[0007] Based on the training image, alternately performing the forward propagation process and the back propagation process of the convolutional neural network to adjust the parameters of the convolutional neural network,

[0008] In the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network.

[0009] In combination with the first aspect, in a first implementation manner of the first aspect of the present disclosure, the designated layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer.

[0010] In combination with the first aspect, in a second implementation of the first aspect of the present disclosure, the fixed-point format includes any one of INT16, INT8, and INT4.

[0011] In combination with the first aspect, in a third implementation of the first aspect of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0012] In combination with the first aspect, in a fourth implementation of the first aspect of the present disclosure, converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameter of the specified layer in the convolutional neural network from a floating-point value to a fixed-point value according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer includes:

[0013] Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image;

[0014] For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

[0015] In combination with the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect of the present disclosure, determining the conversion parameters based on the current gradient distribution of the loss function of the convolutional neural network relative to the output of a specified layer in the convolutional neural network on each channel of the training image includes:

[0016] For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

[0017] In combination with the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect of the present disclosure, determining the conversion parameter based on the current gradient distribution of the channel so that, on the channel, the loss function of the convolutional neural network caused by the conversion is minimized relative to the error of the current gradient of the output of a specified layer in the convolutional neural network in the channel dimension of the training image, includes:

[0018] When the distribution of the current gradient of the loss function of the convolutional neural network with respect to the output of the specified layer in the convolutional neural network on the channel of the training image is Gaussian, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network with respect to the output of the specified layer;

[0019] When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

[0020] In combination with the first aspect, in a seventh implementation of the first aspect of the present disclosure, the method further includes:

[0021] The current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the weight parameter of the specified layer is converted into the floating point format.

[0022] In combination with the first aspect, in an eighth implementation of the first aspect of the present disclosure, the method further includes:

[0023] During the back-propagation process, at least a portion of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer is converted from a floating-point format to a fixed-point format using a conversion parameter determined based on the maximum absolute value of the elements in the corresponding matrix of the at least a portion of the data.

[0024] In combination with the eighth implementation of the first aspect, in a ninth implementation of the first aspect of the present disclosure,

[0025] At least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0026] In combination with the ninth implementation of the first aspect, the present disclosure further includes, in a tenth implementation of the first aspect:

[0027] Convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the input data of the specified layer into the floating point format.

[0028] In a second aspect, an embodiment of the present disclosure provides a data processing method, including:

[0029] Inputting the image data to be processed into a convolutional neural network trained according to any one of the methods of the first aspect to the tenth implementation manner of the first aspect;

[0030] An output of the convolutional neural network is obtained as a processing result of the image data.

[0031] In a third aspect, an embodiment of the present disclosure provides a convolutional neural network training device.

[0032] Specifically, the convolutional neural network training device includes:

[0033] A first input module is configured to input a training image into the convolutional neural network;

[0034] An adjustment module is configured to alternately perform a forward propagation process and a backward propagation process of the convolutional neural network based on the training image to adjust the parameters of the convolutional neural network,

[0035] In the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network.

[0036] In combination with the third aspect, in a first implementation of the third aspect of the present disclosure, the designated layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer.

[0037] In combination with the third aspect, in a second implementation of the third aspect of the present disclosure, the fixed-point format includes any one of INT16, INT8, and INT4.

[0038] In combination with the third aspect, in a third implementation of the third aspect of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0039] In combination with the third aspect, in a fourth implementation of the third aspect of the present disclosure, converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameter of the specified layer in the convolutional neural network from a floating-point value to a fixed-point value based on the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer includes:

[0040] Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image;

[0041] For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

[0042] In combination with the fourth implementation manner of the third aspect, in a fifth implementation manner of the third aspect of the present disclosure, determining the conversion parameters based on the current gradient distribution of the loss function of the convolutional neural network relative to the output of a specified layer in the convolutional neural network on each channel of the training image includes:

[0043] For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

[0044] In combination with the fifth implementation manner of the third aspect, in a sixth implementation manner of the third aspect of the present disclosure, determining the conversion parameter based on the current gradient distribution of the channel so that, on the channel, the loss function of the convolutional neural network caused by the conversion is minimized relative to the error of the current gradient of the output of a specified layer in the convolutional neural network in the channel dimension of the training image, includes:

[0045] When the distribution of the current gradient of the loss function of the convolutional neural network with respect to the output of the specified layer in the convolutional neural network on the channel of the training image is Gaussian, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network with respect to the output of the specified layer;

[0046] When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

[0047] In combination with the third aspect, in a seventh implementation of the third aspect of the present disclosure, the apparatus further includes:

[0048] The first conversion module is configured to convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem relative to the weight parameters of the specified layer into the floating point format.

[0049] In combination with the third aspect, in an eighth implementation of the third aspect of the present disclosure, the apparatus further includes:

[0050] The second conversion module is configured to convert, during the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer from a floating-point format to a fixed-point format using a conversion parameter determined according to the maximum absolute value of the elements in the corresponding matrix of the at least part of the data.

[0051] In combination with the eighth implementation of the third aspect, in a ninth implementation of the third aspect of the present disclosure,

[0052] At least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0053] In combination with the ninth implementation of the third aspect, the present disclosure further includes, in a tenth implementation of the third aspect:

[0054] Convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the input data of the specified layer into the floating point format.

[0055] In a fourth aspect, an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0056] A second input module is configured to input the image data to be processed into a convolutional neural network trained according to any one of the methods of the first aspect to the tenth implementation manner of the first aspect;

[0057] An acquisition module is configured to acquire the output of the convolutional neural network as a processing result of the image data.

[0058] In a fifth aspect, an embodiment of the present disclosure provides an electronic device comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement the method as described in any one of the first to second aspects.

[0059] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the methods described in the first to second aspects are implemented.

[0060] According to the technical solution provided by the embodiment of the present disclosure, a training image is input into the convolutional neural network; based on the training image, the forward propagation process and the backward propagation process of the convolutional neural network are alternately performed to adjust the parameters of the convolutional neural network, wherein, during the backward propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating point format to a fixed point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network. According to the embodiment of the present disclosure, by converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from a floating point format to a fixed point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network during the backward propagation process, the training accuracy and precision can be ensured while significantly reducing the amount of calculation and resource consumption.

[0061] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0063] Figure 1 A flowchart of a convolutional neural network training method according to an embodiment of the present disclosure is shown.

[0064] Figure 2 The figure schematically shows the gradient representation of the loss function of the convolutional neural network with respect to the convolutional layer output Y.

[0065] Figure 3 The comparison results of the method of the embodiment of the present disclosure and the existing methods on the Imagenet dataset are shown.

[0066] Figure 4 A structural block diagram of a convolutional neural network training device according to an embodiment of the present disclosure is shown.

[0067] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown.

[0068] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0069] Figure 7 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0070] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0071] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, components, parts, or combinations thereof exist or are added.

[0072] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] In this disclosure, the acquisition of user information or user data is an operation authorized and confirmed by the user, or actively selected by the user.

[0074] As mentioned above, in recent years, convolutional neural networks have achieved great success in image retrieval, object detection, video understanding, and other fields. However, the huge computational cost of deploying or training a deep convolutional neural network model has brought great challenges to many applications, because the training process involves a large number of floating-point operations, which consumes a lot of computational overhead and resources.

[0075] The present disclosure provides a convolutional neural network training method, including:

[0076] Inputting a training image into the convolutional neural network;

[0077] Based on the training image, alternately performing the forward propagation process and the back propagation process of the convolutional neural network to adjust the parameters of the convolutional neural network,

[0078] In the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network.

[0079] According to an embodiment of the present disclosure, by converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer during the back-propagation process, the training accuracy and precision can be ensured while significantly reducing the amount of calculation and resource consumption.

[0080] Figure 1 FIG. 1 is a flow chart of a convolutional neural network training method according to an embodiment of the present disclosure. Figure 1 As shown, the convolutional neural network training method includes the following steps S101-S102:

[0081] In step S101, a training image is input into the convolutional neural network;

[0082] In step S102, based on the training image, the forward propagation process and the backward propagation process of the convolutional neural network are alternately performed to adjust the parameters of the convolutional neural network, wherein, in the backward propagation process, according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer is converted from a floating point format to a fixed point format.

[0083] Converting neural network weights and / or input data from floating-point to fixed-point format can be used to accelerate the inference phase (i.e., forward propagation) with minimal impact on accuracy. Converting gradients from floating-point to fixed-point format during backward propagation can also accelerate the backward propagation of training. Because backward propagation consumes more computational resources than forward propagation, format conversion can effectively reduce computational resource consumption.

[0084] According to an embodiment of the present disclosure, by converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network from a floating-point format to a fixed-point format based on the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network during the back-propagation process, the training accuracy and precision can be guaranteed while significantly reducing the amount of calculation and resource consumption.

[0085] According to an embodiment of the present disclosure, the specified layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer, and the fixed-point format includes any one of FP16, INT16, FP8, INT8, and INT4.

[0086] Among low-bit-width formats (such as FP16, INT16, FP8, INT8, INT4, etc.), INT8 is twice as fast as FP16 / INT16, has more architectural support than FP8, and is more expressive than INT4 or other lower bit widths (less than 4 bits).

[0087] Although it is attractive to convert gradients to fixed-point format (especially INT8 format) to accelerate training, it is still a big challenge because the conversion error of gradients can easily mislead the direction of convergence. Therefore, it is hoped that fixed-point acceleration of the training process can be used while maintaining accuracy.

[0088] According to an embodiment of the present disclosure, a format conversion from floating point to fixed point can be performed on any one or more layers in a convolutional neural network. For example, for a specified layer, the input data of the calculation can be converted from floating point format to fixed point format, and the calculation result can be converted from fixed point format to floating point format and input to the next layer. If the next layer also uses a fixed-point format for calculation, the calculation result in the fixed-point format can be directly input to the next layer. According to an embodiment of the present disclosure, since the convolution layer generally consumes the most computing resources among all layers, the floating point-fixed point-floating point format conversion can be performed only on the convolution layer.

[0089] According to an embodiment of the present disclosure, a symmetric uniform quantization method is used to convert the format from floating point to fixed point, which is hardware-friendly. For a given floating point data x and conversion parameter s, the INT8 conversion is defined as:

[0090]

[0091] Where clamp(x,s) is defined as:

[0092]

[0093] It can limit x to the range of [-s, s]. Accordingly, the formula for converting x from INT8 fixed-point format to floating-point format is: x = q(x) * s / 127.

[0094] According to an embodiment of the present disclosure, when converting gradients, round(y) is rounded randomly, i.e., one of two integers adjacent to y is randomly selected to reduce the misleading effect of format conversion errors on the convergence direction. For example, if y = 0.5, round(y) is randomly 0 or 1. When converting the input data and parameters of a specified layer, round(y) is rounded to the nearest integer. For example, if y = 0.3, round(y) is equal to 0, and if y = -1.6, round(y) is equal to -2.

[0095] According to an embodiment of the present disclosure, when the fixed-point format is INT4, 127 in the above formula may be replaced by 7; when the fixed-point format is INT16, 127 in the above formula may be replaced by 32767.

[0096] The principles of the present disclosure are described below using a convolutional layer as a designated layer. However, it is understood that the embodiments of the present disclosure are also applicable to layers other than the convolutional layer.

[0097] Without loss of generality, assume that the input of the convolutional layer is X:(N, C_{in}, H_{in}, W_{in}), W:(C_{out}, C_{in}, k_1, k_2) is the weight parameter, and the output is Y:(N, C_{out}, H_{out}, W_{out}), where N is the number of samples, C_{in} is the number of input image channels (such as RGB channels, YCbCr channels, etc.), C_{out} is the number of output image channels, H_{in} is the input image height, W_{in} is the input image width, H_{out} is the output image height, W_{out} is the output image width, k_1 is the convolution kernel height, and k_2 is the convolution kernel width.

[0098] Figure 2 The figure schematically shows the gradient representation of the loss function of the convolutional neural network with respect to the convolutional layer output Y.

[0099] During the backward propagation, the gradient of the loss function of the convolutional neural network with respect to the convolutional layer output Y is denoted as G. During the training process, the G of the convolutional layer is observed from the dimensions of samples and channels, as Figure 2 As shown. By observing that the channel dimension can clearly divide the distribution of G into two categories: Gaussian distribution and sharp shape with a long tail (called Inverted-T distribution). No similar phenomenon is observed from the sample dimension, which may be caused by the specific properties contained in the channel dimension. According to the embodiments of the present disclosure, different conversion parameters are used for these two distributions.

[0100] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0101] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0102] According to an embodiment of the present disclosure, during the back-propagation process, at least a portion of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer is converted from a floating-point format to a fixed-point format using a conversion parameter determined based on the maximum absolute value of the elements in the corresponding matrix of the at least a portion of the data.

[0103] During the backward propagation process, the current gradient G of the loss function of the convolutional neural network relative to the specified output can be output from the previous layer of the specified layer (for example, the convolutional layer) (that is, the layer after the convolutional layer in the forward propagation process). Therefore, the current gradient G of the loss function of the convolutional neural network relative to the weight parameter W of the specified layer is W And the current gradient G of the convolutional neural network loss function with respect to the input data X of the specified layer X It can be calculated as follows:

[0104] G X =G⊙W

[0105]

[0106] That is, G X Equal to the deconvolution of G and W, G W Equivalent to the dilated convolution of X' and G'. X When , since there is no obvious difference in the distribution of G in the sample dimension, a global conversion method can be used to convert the formats of G and W, where the conversion parameter of G is the maximum absolute value of the elements in the G matrix, and the conversion parameter of W is the maximum absolute value of the elements in the W matrix.

[0107] According to an embodiment of the present disclosure, the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer is converted into the floating point format. For example, the fixed point format q(G X ) to floating-point format.

[0108] In calculating G W When , we can do a size transformation on X and G to make the shapes match: X→X':(C_{in},N,H_{in},W_{in}), G→G':(C_{out},N,H_{out},W_{out}).

[0109] X' can be transformed globally, with the transformation parameter s x is the maximum absolute value of the elements in the X matrix.

[0110] Let G' = [G'_1,...,G'_i,...] T, (i=1~C_{out}), G'_i represents the component of G' in the i-th channel. Perform format conversion on each G'_i, and the conversion parameters are recorded as s i , so we have:

[0111]

[0112]

[0113] in:

[0114] According to an embodiment of the present disclosure, the current gradient of the loss function of the convolutional neural network obtained by solving the problem relative to the weight parameters of the specified layer is converted into the floating-point format.

[0115] For example, when solving the fixed-point format q(G w ) and then, q(G w ) from fixed-point format to floating-point format

[0116]

[0117]

[0118] It's important to note that although gradient-format conversion has C (the number of channels) conversion parameters, it has essentially the same computational complexity as global conversion. Taking the search for conversion parameters as an example, global conversion requires searching N*C*H*W floating-point numbers to find the optimal conversion parameter s, while gradient-format conversion only requires searching N*H*W floating-point numbers to find each conversion parameter, resulting in essentially the same overall computational complexity.

[0119] According to an embodiment of the present disclosure, converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from floating-point values ​​to fixed-point values ​​according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network includes:

[0120] Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image;

[0121] For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

[0122] According to an embodiment of the present disclosure, determining the conversion parameters based on the current gradient distribution of the output of a specified layer in the convolutional neural network on each channel of the training image relative to the loss function of the convolutional neural network includes:

[0123] For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

[0124] According to an embodiment of the present disclosure, determining the conversion parameter based on the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of a specified layer in the convolutional neural network on the channel dimension of the training image is minimized includes:

[0125] When the distribution of the current gradient of the loss function of the convolutional neural network with respect to the output of the specified layer in the convolutional neural network on the channel of the training image is Gaussian, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network with respect to the output of the specified layer;

[0126] When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the channel of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

[0127] Compared with small gradients, large gradient conversion errors have more adverse effects on training. To this end, the magnitude contribution is introduced into the calculation of the conversion error E as follows:

[0128]

[0129] Where g represents the gradient, represents the inverse quantized gradient. p(g) represents the distribution of the gradient, and f(g) is defined as follows:

[0130] f(g)=e α|g|

[0131] The non-negative number α is a hyperparameter that reflects the importance of the magnitude to the quantization error.

[0132] When solving the conversion parameter s of the i-th channel iWhen , g is G'_i.

[0133] For a channel with a Gaussian gradient, we can find: The conversion parameters for this channel.

[0134] For a channel with an Inverted-T gradient, we can find:

[0135] is the current conversion parameter of the channel, The conversion parameter used for the last back propagation of this channel. The conversion parameter during the first back propagation is the maximum absolute value of the elements in the g matrix, |g| max，t is the maximum absolute value of the current G'_i element, k and A are both hyperparameters.

[0136] An embodiment of the present disclosure provides a data processing method, comprising: inputting image data to be processed into a convolutional neural network trained according to the above method; and obtaining the output of the convolutional neural network as a processing result of the image data.

[0137] Figure 3 The comparison results of the method of the embodiment of the present disclosure and the existing methods on the Imagenet dataset are shown.

[0138] In order to verify the performance of the present invention, a detailed comparison is conducted with existing methods on the Imagenet dataset. The comparison results are shown in the figure below. Figure 3 The ImageNet project is a large-scale visual database used for research in visual object recognition software. ImageNet contains over 14 million image URLs that have been manually annotated to indicate the objects in the images.

[0139] exist Figure 3 In the table, the Model column is the model used, the Method column is the format conversion method, the Fp32% column is the prediction accuracy of the model trained using 32-bit floating-point format data, the Int8% column is the prediction accuracy of the model trained by converting the floating-point format to Int8 fixed-point format data, and the Δ / % column is the difference between Int8% and Fp32%.

[0140] from Figure 3 It can be seen that, in general, among the seven commonly used models (Model), the model trained using the method disclosed in the present invention (Ours) has better accuracy than other methods (including UI8, AFP, DoReFa-Net, and FP8 training), and is basically on par with the accuracy of the FP32 format.

[0141] In terms of computing speed, in TensorCore mode, the training method according to the embodiment of the present disclosure is more than 100% faster than training using data in FP32 format, and is far superior to other methods in terms of efficiency improvement.

[0142] Figure 4 A block diagram of a convolutional neural network training device according to an embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0143] like Figure 4 As shown, the convolutional neural network training device 400 includes a first input module 410 and an adjustment module 420, wherein:

[0144] The first input module 410 is configured to input a training image into the convolutional neural network;

[0145] The adjustment module 420 is configured to alternately perform the forward propagation process and the back propagation process of the convolutional neural network based on the training image to adjust the parameters of the convolutional neural network.

[0146] In the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network.

[0147] According to an embodiment of the present disclosure, the designated layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer.

[0148] According to an embodiment of the present disclosure, the fixed-point format includes any one of INT16, INT8, and INT4.

[0149] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0150] According to an embodiment of the present disclosure, converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from floating-point values ​​to fixed-point values ​​according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network includes:

[0151] Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image;

[0152] For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

[0153] According to an embodiment of the present disclosure, determining the conversion parameters based on the current gradient distribution of the output of a specified layer in the convolutional neural network on each channel of the training image relative to the loss function of the convolutional neural network includes:

[0154] For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

[0155] According to an embodiment of the present disclosure, determining the conversion parameter based on the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of a specified layer in the convolutional neural network on the channel dimension of the training image is minimized includes:

[0156] When the distribution of the current gradient of the loss function of the convolutional neural network with respect to the output of the specified layer in the convolutional neural network on the channel of the training image is Gaussian, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network with respect to the output of the specified layer;

[0157] When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

[0158] According to an embodiment of the present disclosure, the convolutional neural network training device 400 further includes:

[0159] The first conversion module 430 is configured to convert the current gradient of the loss function of the convolutional neural network with respect to the weight parameter of the specified layer into the floating point format.

[0160] According to an embodiment of the present disclosure, the convolutional neural network training device 400 further includes:

[0161] The second conversion module 440 is configured to convert, during the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer from a floating-point format to a fixed-point format using a conversion parameter determined according to the maximum absolute value of the elements in the corresponding matrix of the at least part of the data.

[0162] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0163] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0164] According to an embodiment of the present disclosure, the convolutional neural network training device 400 further includes:

[0165] The third conversion module 450 is configured to convert the current gradient of the loss function of the convolutional neural network with respect to the input data of the specified layer into the floating point format.

[0166] Figure 5 A structural block diagram of a data processing device according to an embodiment of the present disclosure is shown, wherein the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0167] like Figure 5 As shown, the data processing device 500 includes a second input module 510 and an acquisition module 520, wherein:

[0168] The second input module 510 is configured to input the image data to be processed into the convolutional neural network trained according to the convolutional neural network training method described above;

[0169] The acquisition module 520 is configured to acquire the output of the convolutional neural network as the processing result of the image data.

[0170] The present disclosure also discloses an electronic device, Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0171] like Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602, wherein the memory 601 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 602 to implement the convolutional neural network training method and / or data processing method according to an embodiment of the present disclosure.

[0172] The convolutional neural network training method comprises:

[0173] Inputting a training image into the convolutional neural network;

[0174] Based on the training image, alternately performing the forward propagation process and the back propagation process of the convolutional neural network to adjust the parameters of the convolutional neural network,

[0175] In the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network.

[0176] According to an embodiment of the present disclosure, the designated layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer.

[0177] According to an embodiment of the present disclosure, the fixed-point format includes any one of INT16, INT8, and INT4.

[0178] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0179] According to an embodiment of the present disclosure, converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from floating-point values ​​to fixed-point values ​​according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network includes:

[0180] Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image;

[0181] For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

[0182] According to an embodiment of the present disclosure, determining the conversion parameters based on the current gradient distribution of the output of a specified layer in the convolutional neural network on each channel of the training image relative to the loss function of the convolutional neural network includes:

[0183] For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

[0184] According to an embodiment of the present disclosure, determining the conversion parameter based on the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of a specified layer in the convolutional neural network on the channel dimension of the training image is minimized includes:

[0185] When the distribution of the current gradient of the loss function of the convolutional neural network with respect to the output of the specified layer in the convolutional neural network on the channel of the training image is Gaussian, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network with respect to the output of the specified layer;

[0186] When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

[0187] According to an embodiment of the present disclosure, the convolutional neural network training method further includes:

[0188] The current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the weight parameter of the specified layer is converted into the floating point format.

[0189] According to an embodiment of the present disclosure, the convolutional neural network training method further includes:

[0190] During the back-propagation process, at least a portion of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer is converted from a floating-point format to a fixed-point format using a conversion parameter determined based on the maximum absolute value of the elements in the corresponding matrix of the at least a portion of the data.

[0191] According to an embodiment of the present disclosure, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

[0192] According to an embodiment of the present disclosure, the convolutional neural network training method further includes:

[0193] Convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the input data of the specified layer into the floating point format.

[0194] According to an embodiment of the present disclosure, the data processing method includes:

[0195] Inputting the image data to be processed into the convolutional neural network trained according to the convolutional neural network training method described above;

[0196] An output of the convolutional neural network is obtained as a processing result of the image data.

[0197] Figure 7 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0198] like Figure 7 As shown, the computer system 700 includes a processing unit 701, which can execute various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The processing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0199] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom can be installed into the storage section 708 as needed. Among them, the processing unit 701 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0200] In particular, according to embodiments of the present disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a computer-readable storage medium, the computer program comprising program code for executing the methods described above. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 709 and / or installed from a removable medium 711.

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0202] The units or modules involved in the embodiments described in this disclosure may be implemented by software or programmable hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves.

[0203] As another aspect, the present disclosure further provides a computer-readable storage medium. This computer-readable storage medium may be included in the electronic device or computer system described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the present disclosure.

[0204] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims

1. A convolutional neural network training method, comprising: Inputting a training image into the convolutional neural network; Based on the training image, alternately performing the forward propagation process and the back propagation process of the convolutional neural network to adjust the parameters of the convolutional neural network, wherein, during the back-propagation process, at least a portion of data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer in the convolutional neural network is converted from a floating point format to a fixed point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the specified layer; In which, in the process of converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from a floating-point format to a fixed-point format, the error of the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network on the channel dimension of the training image caused by the conversion is minimized by the following method: when the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is Gaussian distributed on each channel of the training image, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer.

2. The method according to claim 1, wherein The designated layer includes any one of a fully connected layer, a pooling layer, a convolutional layer, a deconvolution layer, and a batch normalization layer.

3. The method according to claim 1, wherein The fixed-point format includes any one of INT16, INT8, and INT4.

4. The method according to claim 1, wherein At least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer includes current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

5. The method according to claim 1, wherein The step of converting at least a portion of data required for solving a current gradient of a loss function of the convolutional neural network relative to a weight parameter of the specified layer from a floating-point value to a fixed-point value according to a current gradient distribution of an output of the convolutional neural network relative to a specified layer in the convolutional neural network includes: Determining conversion parameters for the corresponding channels based on a current gradient distribution of a loss function of the convolutional neural network relative to an output of a specified layer in the convolutional neural network on each channel of the training image; For each channel, the at least portion of the data for the channel is converted from a floating point format to a fixed point format using the conversion parameters.

6. The method according to claim 5, wherein: Determining the conversion parameters according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of a specified layer in the convolutional neural network on each channel of the training image includes: For each channel, the conversion parameters are determined according to the current gradient distribution of the channel so that the error of the loss function of the convolutional neural network caused by the conversion on the channel relative to the current gradient of the output of the specified layer in the convolutional neural network is minimized.

7. The method according to claim 6, wherein: The step of determining the conversion parameters based on the current gradient distribution of the channel so that, on the channel, the error of the loss function of the convolutional neural network caused by the conversion relative to the current gradient of the output of a specified layer in the convolutional neural network on the channel dimension of the training image is minimized further includes: When the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed in an inverted T-shaped distribution on the channel of the training image, the conversion parameter of the channel is determined according to the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer and the conversion parameter of the channel during the last back propagation.

8. The method according to claim 1, further comprising: The current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the weight parameter of the specified layer is converted into the floating point format.

9. The method according to claim 1, further comprising: During the back-propagation process, at least a portion of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer is converted from a floating-point format to a fixed-point format using a conversion parameter determined based on the maximum absolute value of the elements in the corresponding matrix of the at least a portion of the data.

10. The method according to claim 9, wherein: At least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer includes: the current weight parameters of the specified layer; and / or the current gradient data of the loss function of the convolutional neural network relative to the output of the specified layer.

11. The method according to claim 10, further comprising: Convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem with respect to the input data of the specified layer into the floating point format.

12. A data processing method, comprising: Inputting the image data to be processed into a convolutional neural network trained by the method according to any one of claims 1 to 11; An output of the convolutional neural network is obtained as a processing result of the image data.

13. A convolutional neural network training device, comprising: A first input module is configured to input a training image into the convolutional neural network; An adjustment module is configured to alternately perform a forward propagation process and a backward propagation process of the convolutional neural network based on the training image to adjust the parameters of the convolutional neural network, Wherein, during the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is converted from a floating-point format to a fixed-point format according to the current gradient distribution of the loss function of the convolutional neural network relative to the output of the weight parameters of the specified layer; wherein, in the process of converting at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the weight parameters of the specified layer from a floating-point format to a fixed-point format, the error of the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network on the channel dimension of the training image caused by the conversion is minimized by the following method: when the current gradient of the loss function of the convolutional neural network relative to the output of the specified layer in the convolutional neural network is distributed as a Gaussian distribution on each channel of the training image, the conversion parameter of the channel is the maximum absolute value of the element corresponding to the channel in the current gradient matrix of the loss function of the convolutional neural network relative to the output of the specified layer.

14. The apparatus according to claim 13, further comprising: The first conversion module is configured to convert the current gradient of the loss function of the convolutional neural network obtained by solving the problem relative to the weight parameters of the specified layer into the floating point format.

15. The apparatus according to claim 13, further comprising: The second conversion module is configured to convert, during the back-propagation process, at least part of the data required to solve the current gradient of the loss function of the convolutional neural network relative to the input data of the specified layer from a floating-point format to a fixed-point format using a conversion parameter determined according to the maximum absolute value of the elements in the corresponding matrix of the at least part of the data.

16. A data processing device comprising: A second input module is configured to input image data to be processed into a convolutional neural network trained by the method according to any one of claims 1 to 11; An acquisition module is configured to acquire the output of the convolutional neural network as a processing result of the image data.

17. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 12.

18. A readable storage medium having computer instructions stored thereon, wherein the computer instructions implement the method according to any one of claims 1 to 12 when executed by a processor.

Images