An image recognition method and system based on multiple shift-and-add quantization

Abstract

The application discloses an image recognition method and system based on multiple shift and addition quantization, and belongs to the technical field of image recognition. The method comprises the following steps: acquiring an image to be recognized; inputting the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolution layer; the convolution layer adopts multiple shift and addition quantization to process the image to be recognized, and converts floating-point multiplication in the convolution layer into shift, addition and negation operations on fixed-point numbers; and according to the output of the image recognition model, an identification result of the image to be recognized is generated. The method can greatly improve the operation efficiency of an edge intelligent computing terminal for the image recognition model. The multiple shift and addition quantization method simultaneously solves the training problem of the model and can effectively avoid gradient explosion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to an image recognition method and system based on multiple shift add quantization. BACKGROUND

[0002] The safe and reliable operation of a substation is the basis for the stable operation of a power system. With the development of intelligent technology, a large number of intelligent visual devices have been deployed in substations to monitor the safe operation of substations from various dimensions. For example, a large number of intelligent monitoring cameras installed in the station area can effectively identify abnormal targets to assist operation and maintenance personnel in preventing and responding to external damage; for example, a substation inspection robot can patrol switch cabinets and perform switching operations. Currently, the implementation of these intelligent applications mainly relies on intelligent visual / image recognition models deployed on the cloud. With the increase in sensors, robots and other terminals, a large amount of image data will be uploaded in the future, which will not only bring a huge transmission pressure to the communication link, causing transmission delay and even packet loss, but also bring a heavy computing and storage burden to the cloud. To solve this problem, a power intelligent architecture system is proposed, which adds an edge intelligent terminal between the cloud computing center and the terminal-side visual device, plays a role of connecting the upper and lower, realizes on-site processing of image data, and greatly reduces the amount of data uploaded to the cloud.

[0003] In order to truly realize on-site processing of image data in substations and ensure the effectiveness of various applications, sufficient computing power needs to be provided for various visual applications on the edge side. In addition to increasing edge intelligent computing devices, image recognition models can also be optimized to ensure real-time computing while minimizing costs. The main component of an image recognition model is a floating-point multiplication operation, as it is the basis for convolution operations. Current multiplication implementation mainly relies on complex logic, shift, and addition operations. If this operation process can be simplified, or if a suitable quantization method is applied to convert large-bit-width floating-point weight parameters into small-bit-width fixed-point numbers, the number of bottom operation operations can be effectively reduced, the large calculation delay caused by complex calculation process can be shortened, the energy consumption caused by data transfer can be reduced, and the demand of image recognition model for computing power can be optimized. SUMMARY

[0004] To solve the above problems, the present application provides an image recognition method and system based on multiple shift add quantization. The method can be a multiple shift add (MSA) quantization method for substation visual detection models. The method can greatly improve the running efficiency of edge intelligent computing terminals for image recognition models without affecting the recognition accuracy of visual detection (image recognition) models.

[0005] The purpose of the present application is achieved at least by the following technical solutions:

[0006] An image recognition method based on multiple shift-addition quantization includes:

[0007] Acquire the image to be recognized;

[0008] The image to be recognized is input into the trained image recognition model; the image recognition model includes at least one convolutional layer; the convolutional layer processes the image to be recognized by multiple shift-addition quantizations, converting floating-point multiplication in the convolutional layer into shift, addition and inversion operations on fixed-point numbers;

[0009] Based on the output of the image recognition model, the recognition result of the image to be recognized is generated.

[0010] As a further improvement of the present invention, the convolutional layer employs multiple shift-addition quantization to process the image to be recognized, transforming the floating-point multiplication in the convolutional layer into shift, addition, and inversion operations on fixed-point numbers, including:

[0011] Convolution kernel in convolutional layer , ,in and These are the number of channels in the convolution kernel's output and input, respectively. If the size of the convolution kernel is given, then the convolution calculation result is... for:

[0012]

[0013] in, The tensor of the input image for the convolutional layer. For convolution operations; the result of convolution of floating-point numbers. yes and Tensor corresponding element and The product is obtained by multiplying and summing, where , ;

[0014] Then perform multiple shift-add quantization steps, specifically including:

[0015] For input / activation value tensors Perform the conversion to change it from a floating-point number to a fixed-point number;

[0016] weight The absolute value of is transformed into the form of a sum of multiple powers of 2, let:

[0017]

[0018] in It is the weight after multiple shifts and additions for quantization. is with the result of multiplication, is the number of bits shifted, the calculation formula of is:

[0019]

[0020] wherein,

[0021]

[0022] the calculation formula of is:

[0023]

[0024] wherein is a down rounding function, that is rounded down;

[0025] The multiplication operation is realized by shifting operation on , and the shifting operation adopts the following way:

[0026]

[0027] wherein represents left shifting, represents right shifting.

[0028] As a further improvement of the present application, it also includes the step of limiting the bit width of , which specifically includes:

[0029]

[0030] wherein, is the storage bit width occupied by the number of bits shifted , and the highest bit is a sign bit used to indicate left shift, sign bit > 0; right shift, sign bit < 0; or no shift, sign bit = 0; is a truncation function.

[0031] As a further improvement of the present application, it also includes introducing a sign function to obtain the sign of , and the formula is as follows:

[0032]

[0033] When is a negative number , is a positive number , is 0 ;

[0034] The expression of the weight sign is realized by using the NOT operation and the addition operation, and the following flip function is used:

[0035]

[0036] The flip function has three mapping relationships, respectively, keeping unchanged, setting to 0, or changing the sign of ; when, the sign of is flipped, and the corresponding underlying hardware implementation is to obtain the complement value; the complement calculation is composed of the NOT operation and the addition operation. As a further improvement of the application, the operation operation of the convolution layer is:

[0037] wherein is an input / activation value tensor composed of, is a convolution calculation result after multiple shift and addition quantization, and is a ConvMSA operator.

[0038]

[0039] wherein is an input / activation value tensor composed of, is a convolution calculation result after multiple shift and addition quantization, and is a ConvMSA operator. As a further improvement of the application, the image recognition model further comprises a calculation method of a bias parameter B, and the calculation method of the bias parameter B comprises: The bias parameter B is converted to convert a floating point number to a fixed point number.

[0040] A training method of an image recognition model based on multiple shift and addition quantization comprises:

[0041] Obtain a sample image with a label;

[0042] Input the sample image into the constructed image recognition model, the image recognition model comprising at least one convolution layer, replace all convolution operation operators in the image recognition model with multiple shift and addition quantization operators, and train the image recognition model using multiple shift and addition quantization;

[0043] When the preset condition is met, the training of the convolutional neural network is completed, and an image recognition model based on multiple shift and addition quantization is obtained; the convolution layer in the image recognition model based on multiple shift and addition quantization can process the image to be recognized using multiple shift and addition quantization, and can convert the floating point multiplication in the convolution layer to shift, addition and NOT operation on fixed point numbers.

[0044]

[0045] When the preset condition is met, the training of the convolutional neural network is completed, and an image recognition model based on multiple shift and addition quantization is obtained; the convolution layer in the image recognition model based on multiple shift and addition quantization can process the image to be recognized using multiple shift and addition quantization, and can convert the floating point multiplication in the convolution layer to shift, addition and NOT operation on fixed point numbers.

[0046] ​​​​​​​As a further improvement of the present invention, the backpropagation formula of the image recognition model based on multiple shift-addition quantization is as follows:

[0047]

[0048]

[0049]

[0050] in, It is the output value of the model's loss function. It is the gradient of the model loss with respect to the input. It is the gradient of the model loss with respect to the weights. It is the gradient of the model loss with respect to the output. It is the gradient of the model loss with respect to the bias.

[0051] As a further improvement of the present invention, the convolutional neural network is trained using multiple shift-addition quantization, including the following steps:

[0052] Input tensor Weighted tensor Number of shifts Shift width Learning rate ;

[0053] Randomly initialize weight tensors Or load the pre-trained model to initialize the weight tensor. ;

[0054] when Calculate the number of shifts. ,Will Cut off to [ ]scope;

[0055] Calculate the quantized weights ;

[0056] By quantifying weights Complete the forward inference of the model and obtain the model loss. ; Calculate loss right gradient ; Calculate loss right gradient Update weights ;

[0057] Update bias ;

[0058] If the training round number is greater than or equal to the total training round number, the training is ended; if the training round number is less than the total training round number, the method returns , .

[0059] As a further improvement of the present application, the calculation of the shift number , comprises:

[0060]

[0061] Wherein,

[0062]

[0063] The calculation formula is:

[0064]

[0065] Wherein is a floor function, that is rounded down.

[0066] As a further improvement of the present application, the calculation of the shift number is truncated to the range of , which specifically comprises:

[0067]

[0068] Wherein, is the storage bit width occupied by the shift bit number , and the highest bit is a sign bit used to indicate left shift, sign bit > 0; right shift, sign bit < 0; or no shift, sign bit = 0; is a truncation function.

[0069] As a further improvement of the present application, the calculation of the quantized weight ; comprising:

[0070] .

[0071] An image recognition system based on multiple shift addition quantization, comprising:

[0072] An acquisition module for acquiring an image to be recognized;

[0073] A processing module for inputting the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolution layer; the convolution layer uses multiple shift addition quantization to process the image to be recognized, and converts the floating point multiplication in the convolution layer into shift, addition and negation operations on fixed point numbers;

[0074] an output module configured to generate an identification result of the image to be identified according to an output of the image identification model.

[0075] A training system of an image identification model based on multiple shift-add quantization, comprising:

[0076] an obtaining module configured to obtain a sample image with a label;

[0077] a training module configured to input the sample image into a constructed image identification model, the image identification model comprising at least one convolutional layer, replace all convolution operation operators in the image identification model with multiple shift-add quantization operators, and train the image identification model using multiple shift-add quantization.

[0078] a completion module configured to complete the training of the convolutional neural network to obtain the image identification model based on multiple shift-add quantization when a preset condition is met, the convolutional layer in the image identification model based on multiple shift-add quantization being capable of processing an image to be identified using multiple shift-add quantization and converting floating-point multiplication in the convolutional layer into shift, addition and negation operations on fixed-point numbers.

[0079] An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the image identification method based on multiple shift-add quantization or the steps of the training method of the image identification model based on multiple shift-add quantization when executing the computer program.

[0080] A computer-readable storage medium storing a computer program, the computer program being executable by a processor to implement the steps of the image identification method based on multiple shift-add quantization or the steps of the training method of the image identification model based on multiple shift-add quantization.

[0081] Compared with the prior art, the present application has the following advantages:

[0082] The multiple shift-add quantization method proposed in the present application can improve the identification efficiency without affecting the image identification accuracy, and can convert a large number of floating-point multiplication operations involved in the model into addition, shift and negation operation operations on fixed-point numbers, adapt to the hardware bottom layer calculation characteristics, and greatly improve the running efficiency of the edge intelligent computing terminal for the image identification model.

[0083] The multiple shift-addition quantization method solves the training problem of the model and can effectively avoid gradient explosion. The offline quantization model of the recognition model can achieve an accuracy close to that of the original model, and the calibration data set does not need to participate in the quantization process, which is of great significance for application scenarios that only need to perform inference tasks. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise of the drawings.

[0085] Figure 1 A flow chart of an image recognition method based on multiple shift-addition quantization is given in the present application;

[0086] Figure 2 A training method flow chart of an image recognition model based on multiple shift-addition quantization is given in the present application;

[0087] Figure 3 An image recognition method system based on multiple shift-addition quantization is provided in the present application;

[0088] Figure 4 A training method system of an image recognition model based on multiple shift-addition quantization is provided in the present application;

[0089] Figure 5 An electronic device schematic diagram is provided in the present application. DETAILED DESCRIPTION

[0090] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numbers represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0091] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right and the like, is based on the orientation or position relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0092] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0093] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0094] Term explanation:

[0095] Image recognition refers to the use of computers to process, analyze and understand images to identify various different patterns of targets and objects, and is a practical application of deep learning algorithm.

[0096] The traditional recognition process of image is divided into four steps: image acquisition→image preprocessing→feature extraction→image recognition. Most of the image recognition models are built by convolutional neural networks.

[0097] Shift quantization, also known as quantization, quantizes numerical values from floating-point numbers to integers or fixed-point numbers, but it is different from linear quantization. Two adjacent numbers are uniformly distributed in the logarithmic domain with 2 as the base, which makes it possible to directly implement fast reasoning through shift operation, and also has an exponentially growing dynamic range with the increase of bit number.

[0098] Shift quantization can only quantize weights (shift activation values), or only quantize activations (shift weights), or both quantize weights and activations. Because it involves underlying bit shift operations, various approximate bit shift or parallel bit shift techniques can be designed, and these schemes are mostly more suitable for FPGA.

[0099] There is a method of converting complex floating-point multiplication into simple shift, addition and negation operations of fixed-point numbers by single shift quantization. The application of fixed-point numbers equal to 2 raised to the power of N replaces the floating-point weight, effectively improving the calculation efficiency of the model. However, since 2 raised to the power of N can represent a very limited fixed-point number, it is inevitable to make the model judgment accuracy decline when it is applied to replace the floating-point weight, and it can only be used to process simple classification problems and cannot be responsible for complex image recognition tasks. In addition, when applying this method to train, since it involves a large number of approximation operations, it is easy to cause gradient explosion.

[0100] Therefore, the present application provides a multiple shift add (MSA) quantization method for a substation visual detection model. This method can convert a large number of floating-point multiplication operations involved in the model into addition, shift and negation operations of fixed-point numbers without affecting the recognition accuracy of the visual detection (image recognition) model. It is suitable for the hardware bottom layer calculation characteristics and greatly improves the running efficiency of the edge intelligent computing terminal for image recognition models. The multiple shift add quantization method also solves the training problem of the model, which can effectively avoid gradient explosion. In addition, since the conversion process of multiplication operation is actually a quantization process of model weight parameters, the proposed multiple shift add quantization method can effectively compress the recognition model size and save the storage resources on the edge intelligent computing terminal.

[0101] The content of the present application will be described in detail in the following parts.

[0102] 1. Multiple shift add quantization

[0103] As shown in Figure 1 The first object of the present application is to provide an image recognition method based on multiple shift add quantization, comprising:

[0104] Obtaining an image to be recognized;

[0105] Inputting the image to be recognized into a trained image recognition model; the image recognition model includes at least one convolutional layer; the convolutional layer uses multiple shift add quantization to process the image to be recognized, and converts floating-point multiplication in the convolutional layer into shift, addition and negation operations of fixed-point numbers.

[0106] According to the output of the image recognition model, the recognition result of the image to be recognized is generated.

[0107] The multiple shift and addition quantization method can convert a large number of floating point multiplication operations involved in the model into addition, shift and negation operation operations on fixed point numbers without affecting the identification accuracy of the visual detection model of the substation, adapt to the hardware bottom layer calculation characteristics, and greatly improve the running efficiency of the edge intelligent computing terminal for the image recognition model.

[0108] The image recognition model adopted by the application is a conventional recognition model with a convolution layer, other layers in the neural network are processed by using a conventional calculation method, and the application is only directed to floating point multiplication operation in the convolution layer. The image recognition model can be AlexNet, VGG19, ResNet_152, InceptionV4, DenseNet, etc. It can be used for on-site processing of image data in a substation or transmission line abnormal target detection.

[0109] The convolution layer adopts multiple shift and addition quantization to process the image to be recognized, and converts floating point multiplication in the convolution layer into shift, addition and negation operation on fixed point numbers, including:

[0110] The convolution kernel in the convolution layer , , and are the number of channels of the convolution kernel output and input, respectively, is the size of the convolution kernel, and the convolution calculation result is:

[0111]

[0112] wherein, is the tensor of the input image of the convolution layer, is the convolution operation; the convolution calculation result of the floating point number is and tensor corresponding position elements and are multiplied and summed, wherein , ;

[0113] Then multiple shift and addition quantization is performed, specifically including:

[0114] The input / activation value tensor is converted from a floating point number to a fixed point number;

[0115] The absolute value of the weight is converted into the form of the addition of multiple 2's n power, and let:

[0116]

[0117] wherein is the weight after quantization by multiple shift-addition, is the result of multiplication with , is the number of bits of the first shift, the calculation formula of is:

[0118]

[0119] wherein,

[0120]

[0121] the calculation formula of

[0122]

[0123] wherein is a floor function, that is, rounded down;

[0124] the multiplication operation is realized by shift operation on , and the shift operation adopts the following way:

[0125]

[0126] wherein represents left shift, represents right shift.

[0127] The method further comprises a step of limiting the bit width of , which specifically comprises:

[0128]

[0129] wherein, is the storage bit width occupied by the shift bit number , and the highest bit is a sign bit used to indicate left shift, sign bit > 0; right shift, sign bit < 0; or no shift, sign bit = 0; is a truncation function.

[0130] The multiple shift-addition quantization method simultaneously solves the training problem of the model, and can effectively avoid gradient explosion. In addition, since the conversion process of the multiplication operation is actually a quantization process of the model weight parameter, the multiple shift-addition quantization method proposed in the application can effectively compress the size of the recognition model and save the storage resources on the edge intelligent computing terminal.

[0131] As a specific example of the embodiment of the application, the identification method of the application further comprises introducing a sign function to obtain The formula is as follows:

[0132]

[0133] When is negative , is positive , is 0 ;

[0134] The expression of the weight sign is realized by using negation and addition operations, and the following flip function is used:

[0135]

[0136] The flip function has three mapping relationships, respectively keeping unchanged, setting to 0, or changing the sign of ; when , the sign of is flipped, and the corresponding bottom hardware implementation is to obtain the complement value; the complement value calculation is composed of negation and addition operations.

[0137] As a specific example of the embodiment of the application, the operation of the convolution layer is:

[0138]

[0139] Wherein is an input / activation value tensor composed of , is a convolution calculation result after multiple shift-add quantization, is a ConvMSA operator.

[0140] The image recognition model further comprises a calculation method of a bias parameter B, and the calculation method of the bias parameter B comprises:

[0141] Converting the bias parameter B to convert a floating-point number to a fixed-point number.

[0142] The following takes image recognition applied in a substation as an example for description. Since the image recognition model is mainly composed of convolution operations, taking one convolution operation as an example, the principle of the multiple shift-add quantization method is described in detail.

[0143] Suppose a convolution kernel , , wherein and is the number of channels of the input and output of the convolution kernel respectively, is the size of the convolution kernel, and the result of the convolution calculation is should be:

[0144] (1)

[0145] wherein is the input tensor of the convolution layer, is the convolution operation (convolution operator). The result of the convolution calculation of the floating point number is and the corresponding position elements of the tensors are multiplied and summed to obtain, wherein , , . The parameters involved in the multiplication operation of and are quantized by multiple shift-add operations, as follows:

[0146] First, the input / activation value tensor needs to be converted from a floating point number to a fixed point number. This conversion process is a commonly used method and will not be described here.

[0147] Then, without considering the positive and negative of the weight, the absolute value of the weight is converted into the form of multiple 2's power of n addition, that is, let:

[0148] (2)

[0149] wherein is the weight quantized by multiple shift-add operations, is the result of the multiplication of and , is the number of bits of the th shift. The calculation formula of is:

[0150] (3) wherein,

[0151]

[0152] (4)

[0153] The calculation formula of is:

[0154] (5)

[0155] wherein is the floor function, That is rounded down.

[0156] In this way, the multiplication operation can be implemented by means of a shift operation, which follows the following manner:

[0157] (6)

[0158] wherein represents left shift, represents right shift.

[0159] Considering the shift bit number If it is too large, it will cause greater power consumption and memory occupation, and therefore the bit width of needs to be limited, according to:

[0160] (7)

[0161] wherein, is the storage bit width occupied by the shift bit number , and the highest bit is a sign bit used to indicate left shift (sign bit > 0), right shift (sign bit < 0) or no shift (sign bit = 0). is a truncation function.

[0162] Further, the positive and negative of needs to be considered, because for a neural network, a negative weight is also very important, especially in a target detection task, and a convolution kernel with positive and negative weights has important significance for extracting edge features of a target. Therefore, the present application introduces a sign function to obtain the sign of , and the formula is as follows:

[0163] (8)

[0164] When is negative, , is positive, , is 0. .

[0165] Then the negation and addition operations can be used to realize the expression of the weight sign, and the specific implementation should be based on the following flip function:

[0166] (9)

[0167] The flip function has three mapping relationships, respectively making remain unchanged, making ​is set to 0, or the sign is changed When , the sign of is flipped, which corresponds to the implementation of the underlying hardware, that is, the value of the complement. Like the shift calculation, the complement calculation also consumes very few resources, which consists of negation and addition operations.

[0168] In summary, the final expression of is as follows:

[0169] (10)

[0170] Since the actual convolution operation is a tensor calculation, this is further generalized to the tensor dimension, and the following can be obtained:

[0171] (11)

[0172] Then, the entire convolution operation becomes:

[0173] (12)

[0174] Where is the input / activation value tensor composed of , and is the convolution calculation result after multiple shift and addition quantizations. In order to distinguish from the ordinary convolution operation operator , the present application names as ConvMSA operator.

[0175] In addition, there is a bias parameter B in the recognition model. For B, the present application adopts the same method as processing the input / activation value to process it, that is, converting it from a floating-point number to a fixed-point number.

[0176] After multiple shift and addition quantizations, the model during inference converts the complex traditional floating-point multiplication into simple shift, addition and negation operations on fixed-point numbers, greatly reducing the number of data transfers and calculation times. Therefore, it can greatly improve the inference and training speed, and reduce the energy consumption during inference and training.

[0177] 2. Multiple shift and addition quantization model training method

[0178] As shown in Figure 2 , the present application also provides a multiple shift and addition quantization based image recognition model training method, comprising:

[0179] obtaining a sample image with a label;

[0180] input the sample image into a constructed image recognition model, the image recognition model comprising at least one convolution layer, replace all convolution operation operators in the image recognition model with multiple shift-add quantization operators, and train the image recognition model by using multiple shift-add quantization;

[0181] when the preset condition is met, the training of the convolutional neural network is completed, and an image recognition model based on multiple shift-add quantization is obtained; the convolution layer in the image recognition model based on multiple shift-add quantization can process the image to be recognized by using multiple shift-add quantization, and can convert the floating point multiplication in the convolution layer into shift, addition and negation operations on fixed point numbers.

[0182] The training method of the application is very suitable for being deployed on an edge intelligent computing terminal, and can greatly reduce the data amount of the uplink cloud and effectively improve the image recognition efficiency.

[0183] As a specific example of the embodiment of the application, the formula of the back propagation of the multiple shift-add quantization model is as follows:

[0184]

[0185]

[0186]

[0187] wherein, is the output value of the model loss function, is the gradient of the model loss to the input, is the gradient of the model loss to the weight, is the gradient of the model loss to the output, is the gradient of the model loss to the bias.

[0188] The training of the convolutional neural network by using multiple shift-add quantization comprises the following steps:

[0189] input tensor , weight tensor , shift number , shift bit width , learning rate ;

[0190] randomly initialize the weight tensor , or load a pre-trained model to initialize the weight tensor ;

[0191] when , calculate the shift number , and Cut off to [ ]scope;

[0192] Calculate the quantized weights ;

[0193] By quantifying weights Complete the forward inference of the model and obtain the model loss. ; Calculate loss right gradient ; Calculate loss right gradient Update weights ;

[0194] Update bias ;

[0195] If the number of training epochs is greater than or equal to the total number of training epochs, then terminate; otherwise, return. , .

[0196] Specifically, the training process of this invention includes: firstly, constructing a multi-shift addition quantization model (MSA quantization model) for visual detection in substations, i.e., using the ConvMSA operator. Replace all ordinary convolution operators in the visual inspection model for substations. The formula for backpropagation of the MSA quantization model is as follows:

[0197] (13)

[0198] (14)

[0199] (15)

[0200] in, It is the output value of the model's loss function. It is the gradient of the model loss with respect to the input. It is the gradient of the model loss with respect to the weights. It is the gradient of the model loss with respect to the output. It is the gradient of the model loss with respect to the bias.

[0201] It is worth noting that, due to for The approximate value (approximated by multiple shifts and additions) is actually... and Not all are equal, thus leading to Discontinuous. That is to say... The derivative of the point is 0, and the derivative value of the remaining points is 0, which may cause The update cannot be completed in the back propagation process. Therefore, the training method of the MSA quantization model needs to be redesigned.

[0202] Because and The value is close, the present application directly uses the derivative of instead of the derivative of , that is:

[0203] (16)

[0204] Bring it into equation (14), we can get:

[0205] (17)

[0206] In summary, the MSA quantization model training process includes the following steps:

[0207]

[0208] In summary, the MSA quantization method proposed by the present application can not only realize online training of the model, but also has excellent offline quantization capability, that is, it can directly quantize floating-point weights. The offline quantization model can achieve an accuracy close to the original model, and the quantization process does not require the participation of a calibration data set, which is of great significance for application scenarios that only need to perform inference tasks.

[0209] In practical applications, the steps of the image recognition method based on multiple shift-add quantization or the training method of the image recognition model based on multiple shift-add quantization given by the present application can be used on edge computing terminals, and the common processor module / AI computing module is mainly used on the FPGA platform.

[0210] Simulation proves that the method provided by the present application has better recognition accuracy compared with the full-precision model, the power consumption of convolution calculation is reduced by about 15.25~19 times, and the inference speed is improved by about 7.42~15.21 times.

[0211] For existing recognition models, multiple shift-add quantization can effectively simplify the model calculation process and reduce the model size, so the overall logic circuit structure during deployment can be simplified, the data transfer frequency can be reduced, and the logic circuit energy consumption can be reduced, that is, the inference efficiency of the model can be effectively improved, the model calculation and deployment cost can be reduced, and more applications can be run simultaneously on the edge intelligent computing terminal.

[0212] In addition, the multiple shift-add quantization can also be applied to the training process of the model, efficiently supporting the training of the model on a cloud server or even an edge side computing terminal. Due to the simplified calculation process involved in the training, the multiple shift-add quantization can greatly reduce the complexity of the operation and the number of data transfer in the training, improve the training speed, and reduce the power consumption.

[0213] As shown in Figure 3 The third object of the embodiment of the present application is to provide a multiple shift-add quantization based image recognition method system, which comprises:

[0214] An acquisition module is configured to acquire a to-be-recognized image.

[0215] A processing module is configured to input the to-be-recognized image into a trained image recognition model. The image recognition model comprises at least one convolution layer. The convolution layer adopts multiple shift-add quantization to process the to-be-recognized image, and converts the floating-point multiplication in the convolution layer into shift, addition and negation operations on fixed-point numbers.

[0216] An output module is configured to generate a recognition result of the to-be-recognized image according to the output of the image recognition model.

[0217] As shown in Figure 4 The fourth object of the embodiment of the present application is to provide a multiple shift-add quantization based image recognition model training system, which comprises:

[0218] An obtaining module is configured to obtain a sample image with a label.

[0219] A training module is configured to input the sample image into a constructed image recognition model. The image recognition model comprises at least one convolution layer. A multiple shift-add quantization operator is used to replace all convolution operation operators in the image recognition model, and the multiple shift-add quantization is used to train the image recognition model.

[0220] A completion module is configured to complete the training of the convolutional neural network to obtain a multiple shift-add quantization based image recognition model when a preset condition is met. The convolution layer in the multiple shift-add quantization based image recognition model can process a to-be-recognized image by using the multiple shift-add quantization, and can convert floating-point multiplication in the convolution layer into shift, addition and negation operations on fixed-point numbers.

[0221] As shown in Figure 5 The fourth object of the embodiment of the present application is to provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multiple shift-add quantization based image recognition method or the steps of the multiple shift-add quantization based image recognition model training method are implemented.

[0222] The image recognition method based on multiple shift-add quantization comprises:

[0223] Obtaining an image to be recognized;

[0224] Inputting the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolution layer; the convolution layer adopts multiple shift-add quantization to process the image to be recognized, and converts floating-point multiplication in the convolution layer into shift, addition and inversion operations on fixed-point numbers;

[0225] Generating a recognition result of the image to be recognized according to the output of the image recognition model.

[0226] The training method of the image recognition model based on multiple shift-add quantization comprises:

[0227] Obtaining a sample image with a label;

[0228] Inputting the sample image into a constructed image recognition model; the image recognition model comprises at least one convolution layer; all convolution operation operators in the image recognition model are replaced with multiple shift-add quantization operators, and the image recognition model is trained by using multiple shift-add quantization;

[0229] When a preset condition is met, the training of the convolutional neural network is completed, and an image recognition model based on multiple shift-add quantization is obtained; the convolution layer in the image recognition model based on multiple shift-add quantization can process an image to be recognized by using multiple shift-add quantization, and can convert floating-point multiplication in the convolution layer into shift, addition and inversion operations on fixed-point numbers.

[0230] A fifth object of the embodiment of the application is to provide a computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image recognition method based on multiple shift-add quantization or the steps of the training method of the image recognition model based on multiple shift-add quantization.

[0231] The image recognition method based on multiple shift-add quantization comprises:

[0232] Obtaining an image to be recognized;

[0233] Inputting the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolution layer; the convolution layer adopts multiple shift-add quantization to process the image to be recognized, and converts floating-point multiplication in the convolution layer into shift, addition and inversion operations on fixed-point numbers;

[0234] Generating a recognition result of the image to be recognized according to the output of the image recognition model.

[0235] The training method of the image recognition model based on multiple shift-add quantization comprises:

[0236] Obtaining a sample image with a label;

[0237] Inputting the sample image into the constructed image recognition model, wherein the image recognition model comprises at least one convolution layer, all convolution operation operators in the image recognition model are replaced by multiple shift-add quantization operators, and the image recognition model is trained by using multiple shift-add quantization;

[0238] When a preset condition is met, the training of the convolutional neural network is completed, and the image recognition model based on multiple shift-add quantization is obtained; the convolution layer in the image recognition model based on multiple shift-add quantization can process the to-be-recognized image by using multiple shift-add quantization, and can convert the floating-point multiplication in the convolution layer into shift, addition, and inversion operations on fixed-point numbers.

[0239] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0240] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0241] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product comprising instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0242] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide the function of implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a plurality of blocks.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limit the technical solutions of the present application, and although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application. Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0244] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the function specified in the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a plurality of blocks.

[0245] These computer program instructions can also be stored in a computer readable storage medium capable of guiding the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction means, which implements the function specified in the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or a plurality of blocks.

[0246] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0247] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the technical solutions of the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. An image recognition method based on multiple shift-and-add quantization, characterized by, The method comprises the following steps: obtaining an image to be recognized; inputting the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolutional layer; the convolutional layer adopts multiple shift-add-quantization to process the image to be recognized, and converts floating-point multiplication in the convolutional layer into shift, addition and negation operations on fixed-point numbers; generating a recognition result of the image to be recognized according to the output of the image recognition model; the convolutional layer adopts multiple shift-add-quantization to process the image to be recognized, and converts floating-point multiplication in the convolutional layer into shift, addition and negation operations on fixed-point numbers, which comprises the following steps: Convolution kernel in convolution layer , where and are the number of channels of the output and input of the convolution kernel respectively, is the size of the convolution kernel, then the convolution calculation result is: in, The tensor of the input image for the convolutional layer. For convolution operations; the result of convolution of floating-point numbers. yes and Tensor corresponding element and The product is obtained by multiplying and summing, where , ; further performing multiple shift-add-quantization, specifically comprising the following steps: to an input / activation value tensor transforming it from a floating point number to a fixed point number; The absolute values of the weights are converted into the form of the addition of N powers of 2 of n, denoted: wherein is the weight after quantization by multiple shift-addition, is multiplied by , is the number of bits of the th shift, the calculation formula of is: wherein, The calculation formula is: wherein is a floor function, i.e. floor; By performing a shift operation on The multiplication is performed by a shift operation in the following way: wherein represents a left shift, represents a right shift.

2. The image recognition method based on quantization of multiple shift-addition according to claim 1, characterized in that, Also included is a step of limiting the bit width of the , which specifically includes: wherein, is the number of shift bits is the storage bit width occupied, the most significant bit of which is the sign bit used to indicate is left shift if the sign bit > 0, right shift if the sign bit < 0, or no shift if the sign bit = 0 is the truncate function. 3.The image recognition method based on the multi-shift-addition quantization of claim 1, wherein, Also included is the introduction of a sign function to obtain the sign of the convolution kernel, is the weight element before quantization corresponding to the ij position in the convolution kernel, whose formula is as follows: When is negative , is positive , is zero ; the negation and addition operations are used to express the weight sign, and the following flip function is adopted: There are three mapping relationships of the flipping function, which respectively make unchanged, set to 0, or change the sign of ; when , the sign of is flipped, which corresponds to the implementation of the underlying hardware, i.e., the value of the complement; the complement calculation is composed of negation and addition operations.

4. The image recognition method based on quantization of multiple shift-addition according to claim 1, characterized in that, the operation of the convolutional layer is as follows: wherein is an input / activation value tensor composed of , is a convolution calculation result after multiple shift-add quantization, is a ConvMSA operator.

5. The image recognition method based on quantization of multiple shift-addition according to claim 1, characterized in that, the image recognition model further comprises a calculation method of a bias parameter B, and the calculation method of the bias parameter B comprises the following steps: performing bias parameter B conversion to convert floating-point numbers into fixed-point numbers. 6.A method for training an image recognition model based on multiple shift-and-add quantization, characterized in that, The method comprises the following steps: obtaining a sample image with a label; inputting the sample image into a constructed image recognition model, the image recognition model comprising at least one convolutional layer, replacing all convolution operation operators in the image recognition model with multiple shift-add-quantization operators, and training the image recognition model by using multiple shift-add-quantization; when a preset condition is met, training of the convolutional neural network is completed, and an image recognition model based on multiple shift-add-quantization is obtained; the convolutional layer in the image recognition model based on multiple shift-add-quantization can process an image to be recognized by using multiple shift-add-quantization, and can convert floating-point multiplication in the convolutional layer into shift, addition and negation operations on fixed-point numbers; the formula of back propagation of the image recognition model based on multiple shift-add-quantization is as follows: wherein, is an output value of the model loss function, is a gradient of the model loss with respect to the input, is a gradient of the model loss with respect to the weight, is a gradient of the model loss with respect to the output, is a gradient of the model loss with respect to the bias; training the convolutional neural network by using multiple shift-add-quantization comprises the following steps: input tensor , weight tensor , number of shifts , shift bit width , learning rate ; Randomly initializing weight tensors Or loading a pretrained model to initialize weight tensors ; When the number of shifted bits is calculated , m times truncated to the range ] Computing quantized weights ; quantizing weights complete forward inference of the model to obtain model loss ; calculate loss the gradient of ; calculate loss the gradient of ; update weights ; Update bias ; determining whether the training round number is greater than or equal to the total training round number; if the training round number is greater than or equal to the total training round number, then ending; if the training round number is less than the total training round number, then returning , .

7. The method of claim 6, wherein the method further comprises: Computing the shift number of bits comprises: wherein, The calculation formula is: wherein is a floor function, i.e. is rounded down. 8.The method of claim 6, wherein, will be described below. truncated to the range of [0, 1], which specifically includes: ] range, which specifically includes: where, is the number of bits shifted is the storage bit width occupied, the most significant bit of which is the sign bit used to indicate left shift if the sign bit > 0; right shift if the sign bit < 0; or no shift if the sign bit = 0; is the truncate function.

9. The training method of the image recognition model based on multiple shift-add-quantization according to claim 6, characterized in that, the computed quantized weights ; The method comprises the following steps: 。 10. An image recognition system based on multiple shift-and-add quantization, based on the image recognition method based on multiple shift-and-add quantization according to any one of claims 1 to 5, characterized in that, The method comprises the following steps: an acquisition module, configured to acquire an image to be recognized; a processing module, configured to input the image to be recognized into a trained image recognition model; the image recognition model comprises at least one convolutional layer; the convolutional layer adopts multiple shift-add-quantization to process the image to be recognized, and converts floating-point multiplication in the convolutional layer into shift, addition and negation operations on fixed-point numbers; an output module, configured to generate a recognition result of the image to be recognized according to the output of the image recognition model.

11. A training system of an image recognition model based on multiple shift-and-add quantization, based on the training method of an image recognition model based on multiple shift-and-add quantization according to any one of claims 6 to 9, characterized in that, The method comprises the following steps: an obtaining module, configured to obtain a sample image with a label; a training module, configured to input the sample image into a constructed image recognition model, the image recognition model comprising at least one convolutional layer, replacing all convolution operation operators in the image recognition model with multiple shift-add-quantization operators, and training the image recognition model by using multiple shift-add-quantization; A completion module is configured to complete the training of the convolutional neural network to obtain the image recognition model based on the multiple times of shift-add quantization when a preset condition is met. 12.An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image recognition method based on the multiple times of shift-add quantization according to any one of claims 1-5 or the training method of the image recognition model based on the multiple times of shift-add quantization according to any one of claims 6-9. 13.A computer readable storage medium, storing a computer program, wherein the computer program is executed by a processor to implement the image recognition method based on the multiple times of shift-add quantization according to any one of claims 1-5 or the training method of the image recognition model based on the multiple times of shift-add quantization according to any one of claims 6-9.

Images