Image data processing method, device, equipment and medium

Abstract

The present invention discloses an image data processing method, device, equipment and medium. The method comprises: receiving an image to be processed; processing the image to be processed using an image processing convolutional neural network to obtain a processing result of the image to be processed; wherein the image processing convolutional neural network is a network after updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor; for each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of the multiple first sub-tensors of the weight tensor respectively, and the weight tensor is a tensor of the original convolutional neural network obtained and corresponding to the fixed-point tensor; the multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension. According to an embodiment of the present invention, it is possible to improve the image data processing accuracy on the basis of having the applicability of the hardware platform.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image data processing method, device, equipment and medium. Background Art

[0002] Deep learning has brought the performance of many computer vision tasks to an unprecedented level. Complex deep learning models have good performance, but the high storage space and computing resource consumption make it difficult to effectively apply them on various hardware platforms.

[0003] Model compression is a commonly used improvement method in the current business deployment service of deep learning models, which can greatly improve the inference speed of the model. However, due to the different platforms of different business scenarios, the model compression expression methods on different hardware platforms are not the same. Currently, most commonly used model compression expression methods are aimed at a single hardware platform or greatly reduce the accuracy of the model in order to adapt to multiple hardware platforms, and cannot obtain a high-precision compressed model that is suitable for multiple hardware platforms.

[0004] Therefore, if the deep learning model obtained by the current model compression method is used to process image data, it is impossible to improve the accuracy of image data processing on the basis of applicability to multiple hardware platforms. Summary of the invention

[0005] The embodiment of the present invention provides an image data processing method, which can improve the image data processing accuracy on the basis of applicability to multiple hardware platforms.

[0006] In a first aspect, an embodiment of the present invention provides an image data processing method, the method comprising:

[0007] receiving an image to be processed;

[0008] Using an image processing convolutional neural network to process the image to be processed, and obtaining a processing result of the image to be processed;

[0009] The image processing convolutional neural network is a network after updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor;

[0010] For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of the multiple first sub-tensors of the weight tensor, where the weight tensor is a tensor of the obtained original convolutional neural network and corresponds to the fixed-point tensor;

[0011] The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor.

[0012] In one embodiment, the image to be processed is processed using an image processing convolutional neural network to obtain a processing result of the image to be processed, including:

[0013] Each fixed-point tensor of the image processing convolutional neural network is used to obtain different feature information of the image to be processed;

[0014] Based on different feature information of the image to be processed, a processing result of the image to be processed is obtained.

[0015] In one embodiment, the image to be processed is a face image to be processed, and the image processing convolutional neural network is a face recognition convolutional neural network;

[0016] or,

[0017] The image to be processed is a pedestrian image to be processed, and the image processing convolutional neural network is a pedestrian detection convolutional neural network.

[0018] In one embodiment, the image data processing method is applied to a field-programmable gate array (FPGA), a neural-network processing unit (NPU), an application-specific integrated circuit (ASIC), or a central processing unit (CPU).

[0019] In one embodiment, the image processing convolutional neural network is a network obtained by updating a weight tensor corresponding to each fixed-point tensor based on each fixed-point tensor, and updating an initial scaling factor of a first sub-tensor corresponding to each fixed-point sub-tensor based on a final scaling factor of each fixed-point sub-tensor;

[0020] Among them, for each fixed-point sub-tensor, the final scaling factor of the fixed-point sub-tensor is obtained based on the fixed-point weight scaling factor corresponding to the fixed-point sub-tensor.

[0021] In one embodiment, for any weight tensor of the original convolutional neural network, a quantization method for any first sub-tensor of the weight tensor is determined based on the degree of data discreteness of the first sub-tensor.

[0022] In one embodiment, the quantization result of the weight tensor W is represented by Q dot multiplication by alpha, where Q is a fixed-point tensor and alpha is a scaling vector.

[0023] In one embodiment, the original convolutional neural network includes at least one convolutional layer and at least one fully connected layer;

[0024] The at least one fixed-point tensor includes a fixed-point tensor corresponding to the weight tensor of each convolutional layer and a fixed-point tensor corresponding to the weight tensor of each fully connected layer.

[0025] In a second aspect, an embodiment of the present invention provides an image data processing device, the device comprising:

[0026] An image receiving module, used for receiving an image to be processed;

[0027] An image processing module is used to process the image to be processed using an image processing convolutional neural network to obtain a processing result of the image to be processed;

[0028] The image processing convolutional neural network is a network after updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor;

[0029] For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of the multiple first sub-tensors of the weight tensor, where the weight tensor is a tensor of the obtained original convolutional neural network and corresponds to the fixed-point tensor;

[0030] The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor.

[0031] In a third aspect, an embodiment of the present invention provides an image data processing device, the device comprising: a processor and a memory storing computer program instructions;

[0032] When the processor executes the computer program instructions, the image data processing method provided by the embodiment of the present invention is implemented.

[0033] In one embodiment, the processor is an FPGA, an NPU, an ASIC, or a CPU.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, an image data processing method as provided in an embodiment of the present invention is implemented.

[0035] In the image data processing method in the embodiment of the present invention, since the fixed-point tensor in the image processing convolutional neural network and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each first sub-tensor of the weight tensor corresponding to the fixed-point tensor, the precision loss of the weight tensor in the original convolutional neural network can be reduced, thereby obtaining a high-precision image processing convolutional neural network. By using a high-precision image processing convolutional neural network to process the image to be processed, the accuracy of image processing can be improved. And since no offset is involved in the quantization process of the weight tensor of the original convolutional neural network, the image processing convolutional neural network can be deployed on different hardware platforms, thereby improving the wide applicability of the image data processing method on multiple hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following briefly introduces the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of a flow chart of an image data processing method provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of a weight tensor and a fixed-point tensor provided in an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of an image data processing device provided by an embodiment of the present invention;

[0040] Figure 4 FIG. 4 is a structural diagram of an exemplary hardware architecture of an image data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0042] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0043] When training a deep learning model, forward and backward propagation must be guaranteed, and each gradient update is very small, which requires relatively high precision. Therefore, floating-point data is generally used for model training. However, in the deployment phase of the deep learning model, that is, when the deep learning model is deployed on a hardware platform, in order to meet the real-time requirements of the deep learning model for data processing, the trained deep learning model needs to be compressed and then deployed to the hardware. However, the current model compression method cannot obtain a high-precision deep learning model that is suitable for a variety of hardware platforms, which results in the inability to improve the accuracy of image data processing on the basis of applicability to multiple hardware platforms when using the compressed image processing model for image processing.

[0044] Based on this, an embodiment of the present invention provides an image data processing method. Figure 1 As shown, the image data processing method 100 provided by the embodiment of the present invention includes:

[0045] S110, receiving an image to be processed.

[0046] S120, using the image processing convolutional neural network to process the image to be processed to obtain a processing result of the image to be processed.

[0047] The image processing convolutional neural network is a network obtained by updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor.

[0048] For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of the multiple first sub-tensors of the weight tensor, where the weight tensor is a tensor of the original convolutional neural network obtained and corresponds to the fixed-point tensor.

[0049] The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor.

[0050] Among them, the original convolutional neural network is a convolutional neural network that has been trained for image processing but has not undergone parameter update. The data type of the weight tensor of the original convolutional neural network is generally floating-point data.

[0051] In the image data processing method provided by the embodiment of the present invention, since the fixed-point tensor in the image processing convolutional neural network and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each first sub-tensor of the weight tensor corresponding to the fixed-point tensor, the data precision loss of the weight tensor in the original convolutional neural network can be reduced, thereby obtaining a high-precision image processing convolutional neural network. By using a high-precision image processing convolutional neural network to process the image to be processed, the accuracy of image processing can be improved.

[0052] Moreover, no offset is involved in the quantization process of the weight tensor of the original convolutional neural network, so the image processing convolutional neural network can be deployed on different hardware platforms, thereby improving the wide applicability of the image data processing method on multiple hardware platforms, that is, the accuracy of image data processing is also improved on the basis of applicability to multiple hardware platforms.

[0053] The image processing convolutional neural network provided in the embodiment of the present invention can be deployed on different hardware platforms such as FPGA, NPU, ASIC or CPU. Therefore, the image data processing method provided in the embodiment of the present invention has wide applicability on different hardware platforms.

[0054] In the embodiment of the present invention, since the data type of the fixed-point tensor is fixed-point data, such as integer data, in the process of processing the image to be processed, the image processing convolutional neural network adopts fixed-point calculation, thereby improving the image processing accuracy and the speed of image processing.

[0055] In some embodiments of the present invention, the image to be processed may be a face image to be processed, and the image processing convolutional neural network is a face recognition convolutional neural network. In other embodiments, the image to be processed is a pedestrian image to be processed, and the image processing convolutional neural network is a pedestrian detection convolutional neural network. The image processing convolutional neural network is not limited to the above two convolutional neural network models.

[0056] Taking the face recognition convolutional neural network as an example, the following introduces in detail the update process of the face recognition convolutional neural network with high accuracy and wide hardware usage.

[0057] First, the first original convolutional neural network for face recognition is trained using floating-point data. Since floating-point data is generally used in the training process of the first original convolutional neural network, the weight tensor of the trained first original convolutional neural network is floating-point data. In order to improve the reasoning speed of the first original convolutional neural network and meet the real-time requirements of data processing, the first original convolutional neural network needs to be updated to obtain a high-precision face recognition convolutional neural network with a wide range of hardware platform applications.

[0058] In an embodiment of the present invention, updating the first original convolutional neural network means converting the data type of the model parameters of the first original convolutional neural network from floating point to fixed point, that is, converting the weight tensor of the first original convolutional neural network into a fixed-point tensor.

[0059] In an embodiment of the present invention, before updating the first original convolutional neural network, it is first necessary to obtain at least one weight tensor of the first original convolutional neural network that has been trained from the memory. Each weight tensor of the first original convolutional neural network is used to extract feature information of different levels of the face image. If the weight tensor of the first original convolutional neural network has been stored in the memory, at least one weight tensor of the first original convolutional neural network can be directly obtained from the memory.

[0060] In an embodiment of the present invention, a tensor is an organized multi-dimensional array. For example, a scalar is a 0-dimensional tensor, a vector is a 1-dimensional tensor, a matrix is ​​a 2-dimensional tensor, and a matrix array is a 3-dimensional tensor. Tensors are used to push vectors and matrices to higher dimensions.

[0061] In an embodiment of the present invention, a convolutional neural network is a type of neural network that includes convolution calculations and has a deep structure. A convolutional neural network generally includes an input layer, a hidden layer, and an output layer. Among them, the hidden layer generally includes a convolution layer, a pooling layer, and a fully connected layer.

[0062] In an embodiment of the present invention, the weight tensor of the first original convolutional neural network is the model parameter of the trained first original convolutional neural network, which is a tensor of floating-point data. As an example, the first original convolutional neural network includes at least one convolutional layer and at least one fully connected layer. Then all weight tensors of the first original convolutional neural network include the weight tensor corresponding to each convolutional layer and the weight tensor corresponding to each fully connected layer.

[0063] Each convolution layer corresponds to a set of convolution kernels, and each convolution kernel includes multiple weight values. For each convolution layer, all the convolution kernels corresponding to the convolution layer together constitute the weight tensor of the convolution layer. The fully connected layer can be regarded as a special convolution layer.

[0064] In an embodiment of the present invention, the weight tensor of the first original convolutional neural network can be represented as a four-dimensional tensor. As an example, the dimension of the weight tensor corresponding to the convolution layer of the first original convolutional neural network can be represented by (N, C, H, W), where N represents the number of convolution kernels in the convolution layer (the number of channels of the facial feature map output by the convolution layer), C represents the number of channels of the facial image input to the convolution layer), and H and W represent the height and width of the convolution kernel, respectively.

[0065] As an example, a convolution layer of the first original convolutional neural network includes 64 convolution kernels, each of which has a size of 3*3*3, and the dimension of the weight tensor of the convolution layer can be expressed as (64, 3, 3, 3). Among them, NCHW represents the number of weight values ​​in the weight tensor of the convolution layer of the first original convolutional neural network. That is to say, the number of weight values ​​in the weight tensor of the convolution layer in the above example is 64*3*3*3.

[0066] In an embodiment of the present invention, since the amount of computation involved in the convolutional layer and the fully connected layer in the convolutional neural network is relatively large, when updating the convolutional neural network, the weight tensor of the convolutional layer and the weight tensor of the fully connected layer are mainly quantized to compress the convolutional neural network.

[0067] In an embodiment of the present invention, in order to reduce the loss of data accuracy in the weight tensor during the quantization process, after obtaining at least one weight tensor of the first original convolutional neural network, for each weight tensor, the processor quantizes each of the multiple first sub-tensors of the weight tensor respectively to obtain a fixed-point tensor and a scaling vector corresponding to the fixed-point tensor.

[0068] Among them, multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor.

[0069] That is to say, if all weighted tensors of the first original convolutional neural network include the weight tensor of each convolutional layer and the weight tensor of each fully connected layer, then the at least one fixed-point tensor obtained after quantization includes the fixed-point tensor corresponding to the weight tensor of each convolutional layer and the fixed-point tensor corresponding to the weight tensor of each fully connected layer.

[0070] Taking the convolution layer in the first original convolutional neural network as an example, the predetermined dimension may be the convolution kernel dimension, that is, the weight tensor of the convolution layer is decomposed according to the convolution kernel. In other words, a convolution kernel in the convolution layer is used as a first sub-tensor. As an example, Figure 2 A schematic diagram showing a 4-dimensional weight tensor of a convolutional layer including 5 convolution kernels. Figure 2The weight tensor in includes 5 three-dimensional first sub-tensors, each of which corresponds to an 8*3*3 convolution kernel. Figure 2 Each cube on the left side of the figure represents a first subtensor.

[0071] For a convolution layer of the first original convolutional neural network, a convolution kernel (i.e., the first sub-tensor) in the convolution layer is used to focus on specific local feature information of the face image input to the convolution layer, and each convolution kernel focuses on different local feature information. When the convolution layer performs convolution on the input face image, each convolution kernel performs convolution on the input face image to obtain different local feature information of the face image. For example, the first convolution kernel is used to extract texture features of the face image, the second convolution kernel is used to extract contour features of the face image, the third convolution kernel is used to extract posture features in the face image, and so on.

[0072] That is to say, different first sub-tensors of the weight tensor of the convolution layer of the first original convolution neural network are used to extract different local feature information of the input face image. If the convolution layer of the first original convolution neural network performs convolution calculation on the input face image using M convolution kernels, it can output M feature images of faces, that is to say, the number of channels of the face feature map output by the convolution layer is the same as the number of convolution kernels. Therefore, a first sub-tensor of the weight tensor of the convolution layer of the first original convolution neural network can also be regarded as a channel of the weight tensor.

[0073] The next convolutional layer of the first original convolutional neural network abstracts and combines the feature information of the face image extracted by the previous convolutional layer to obtain feature information of a higher level. In other words, each weight tensor of the first original convolutional neural network is used to extract feature information of different levels of the face image.

[0074] It should be noted that the data type in the weight tensor is floating-point data, and the data type in the fixed-point tensor can be integer data, such as int8 data.

[0075] In an embodiment of the present invention, for any weight tensor in the first original convolutional neural network, by separately quantizing different first sub-tensors of the weight tensor, a fixed-point sub-tensor corresponding to each first sub-tensor and a fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor can be obtained.

[0076] Continue to see Figure 2 , Figure 2 Each first sub-tensor in is quantized to obtain a fixed-point sub-tensor corresponding to the first sub-tensor. Figure 2 Each cube on the right side of represents a fixed-point sub-tensor. And each fixed-point sub-tensor has a corresponding fixed-point weight scaling factor ( Figure 2). The dimension of the fixed-point sub-tensor is the same as the dimension of its corresponding first sub-tensor, and the difference is the data type.

[0077] In an embodiment of the present invention, the quantization result of the weight tensor W of the first original convolutional neural network can be represented by Q point multiplication by alpha, where Q is a fixed-point tensor corresponding to the weight tensor W, and alpha is a scaling vector corresponding to the fixed-point tensor Q.

[0078] The weight tensor W includes m first sub-tensors, the fixed-point tensor Q includes a fixed-point sub-tensor corresponding to each first sub-tensor, the scaling vector alpha includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor, and m is a positive integer. In an embodiment of the present invention, decomposing the weight tensor W into Q points multiplied by alpha is to quantize the weight tensor W.

[0079] In an embodiment of the present invention, by quantizing different sub-tensors of the weight tensor respectively, Q point multiplication by alpha can better express W. That is, the difference between the result of W and Q point multiplication by alpha can be made smaller, that is, the precision loss of the weight tensor can be reduced, thereby obtaining a high-precision face recognition convolutional neural network. In addition, there is no offset in the process of quantizing each first sub-tensor of the weight tensor, but the weight tensor is expressed as the product of a fixed-point tensor and a scaling vector, so it can be applied on different hardware platforms and has a wider range of applications.

[0080] In an embodiment of the present invention, the quantization method for each first sub-tensor of the floating-point weight tensor may be the same or different. The embodiment of the present invention does not specifically limit the specific quantization method. For example, the first sub-tensor of the weight tensor may be quantized using the nearest neighbor method.

[0081] In an embodiment of the present invention, for any weight tensor of the first original convolutional neural network, different quantization methods can be selected according to the data discreteness of each first sub-tensor of the weight tensor to calculate the fixed-point weight scaling coefficient corresponding to each first sub-tensor. By utilizing the uniformity of the data distribution of the first sub-tensor and configuring a suitable quantization method, the weight tensor of the convolutional neural network can be expressed more accurately, further reducing the loss of data accuracy of the weight tensor.

[0082] In an embodiment of the present invention, after obtaining the fixed-point tensor and scaling vector corresponding to each weight tensor of the first original convolutional neural network, the first original convolutional neural network is updated using each fixed-point tensor and each scaling vector to obtain a face recognition convolutional neural network.

[0083] In an embodiment of the present invention, a face recognition convolutional neural network is a network obtained by updating the weight tensor corresponding to each fixed-point tensor based on each fixed-point tensor, and updating the initial scaling coefficient of the first sub-tensor corresponding to each fixed-point sub-tensor based on the final scaling coefficient of each fixed-point sub-tensor.

[0084] Among them, for each fixed-point sub-tensor, the final scaling factor of the fixed-point sub-tensor is obtained based on the fixed-point weight scaling factor corresponding to the fixed-point sub-tensor.

[0085] In an embodiment of the present invention, for any weight tensor of the first original convolutional neural network, each first sub-tensor of the weight tensor has a corresponding original scaling parameter. As an example, each convolution layer of the first original convolutional neural network is connected to a scaling layer (scale layer). For each scale layer, the number of channels of the scale layer is the same as the number of convolution kernels in the corresponding convolution layer, that is, the same as the number of the first sub-tensor of the weight tensor of the convolution layer, and each channel of the scale layer is used to scale each facial feature output by the convolution layer. Figure 1 Since each facial feature map output by the convolution layer has its own corresponding convolution kernel, that is, each has a corresponding first sub-tensor, that is to say, each first sub-tensor of the weight tensor of the convolution layer has a corresponding original scaling parameter. Among them, the channels of the scaling layer connected after each convolution layer correspond one-to-one to the channels of the weight tensor of the convolution layer.

[0086] Specifically, for each fixed-point sub-tensor in any fixed-point tensor, the processor uses the product of the fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor and the obtained initial scaling parameter of the first sub-tensor corresponding to the fixed-point sub-tensor as the final scaling parameter corresponding to the fixed-point sub-tensor.

[0087] It should be noted that the initial scaling factor corresponding to the first sub-tensor is the initial scaling factor corresponding to the fixed-point sub-tensor corresponding to the first sub-tensor, and the initial scaling factor can be pre-acquired from the memory.

[0088] That is to say, after each first sub-tensor of the weight tensor of the first original convolutional neural network is quantized respectively, the corresponding weight tensor is replaced by the fixed-point tensor. Then, according to the correspondence between the channels of the scaling layer and the channels of the weight tensor, for each fixed-point sub-tensor, the fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor is multiplied with the initial scaling parameter corresponding to the fixed-point sub-tensor for merge optimization, so that after the face recognition convolutional neural network obtained after updating the first original convolutional neural network is deployed on the hardware platform, the amount of calculation can be reduced and the inference speed of the face recognition convolutional neural network can be improved.

[0089] Among them, for any fixed-point sub-tensor, the final scaling coefficient corresponding to the fixed-point sub-tensor is used to scale the facial feature map output after convolution calculation using the fixed-point sub-tensor.

[0090] After obtaining the face recognition convolutional neural network, the face convolutional neural network can be deployed on multiple different hardware platforms. As an example, the hardware platform can be any one of the following hardware: ASIC, FPGA, NPU and CPU.

[0091] After the hardware platform on which the face recognition convolutional neural network is deployed receives the face image to be processed, the deployed face recognition convolutional neural network can be used to process the face image to be processed to obtain a face recognition result of the face image.

[0092] In the process of the face recognition convolutional neural network processing the received face image to be processed, each fixed-point tensor in the face recognition convolutional neural network is used to obtain feature information of different levels of the face image to be processed. By using the feature information of different levels of the face image to be processed, the face recognition result of the face image to be processed can be obtained.

[0093] Since floating-point operations are very expensive and resources are relatively scarce, due to cost and efficiency issues, face recognition convolutional neural networks are run on FPGAs and ASICs using fixed-point operations.

[0094] When a hardware platform on which an image processing convolutional neural network is deployed receives an image to be processed, the image processing convolutional neural network on the hardware platform uses fixed-point calculation, so that the reasoning speed of the convolutional neural network can be improved based on the high-precision image processing convolutional neural network.

[0095] Figure 3 FIG. 2 shows a schematic diagram of the structure of an image data processing device provided according to an embodiment of the present invention. Figure 3 As shown, the image data processing device 300 includes:

[0096] The image receiving module 310 is used to receive an image to be processed.

[0097] The image processing module 320 is used to process the image to be processed using an image processing convolutional neural network to obtain a processing result of the image to be processed.

[0098] The image processing convolutional neural network is a network obtained by updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor.

[0099] For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of the multiple first sub-tensors of the weight tensor, where the weight tensor is a tensor of the original convolutional neural network obtained and corresponds to the fixed-point tensor.

[0100] The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor.

[0101] In one embodiment, the image processing module 320 is specifically used for:

[0102] Each fixed-point tensor of the image processing convolutional neural network is used to obtain different feature information of the image to be processed;

[0103] Based on different feature information of the image to be processed, a processing result of the image to be processed is obtained.

[0104] In one embodiment, the image to be processed is a face image to be processed, and the image processing convolutional neural network is a face recognition convolutional neural network;

[0105] or,

[0106] The image to be processed is a pedestrian image to be processed, and the image processing convolutional neural network is a pedestrian detection convolutional neural network.

[0107] In one embodiment, the image data processing apparatus may be applied to FPGA, NPU, ASIC or CPU.

[0108] In one embodiment, the image processing convolutional neural network is a network obtained by updating the weight tensor corresponding to each fixed-point tensor based on each fixed-point tensor, and updating the initial scaling coefficient of the first sub-tensor corresponding to each fixed-point sub-tensor based on the final scaling coefficient of each fixed-point sub-tensor; wherein, for each fixed-point sub-tensor, the final scaling coefficient of the fixed-point sub-tensor is obtained based on the fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor.

[0109] In one embodiment, for any weight tensor of the original convolutional neural network, a quantization method for any first sub-tensor of the weight tensor is determined based on the degree of data discreteness of the first sub-tensor.

[0110] In one embodiment, the quantization result of the weight tensor W is represented by Q dot multiplication by alpha, where Q is a fixed-point tensor and alpha is a scaling vector.

[0111] In one embodiment, the original convolutional neural network includes at least one convolutional layer and at least one fully connected layer, and the at least one fixed-point tensor includes a fixed-point tensor corresponding to the weight tensor of each convolutional layer and a fixed-point tensor corresponding to the weight tensor of each fully connected layer.

[0112] By using the image processing convolutional network to process image data, it is possible to improve the processing accuracy of image data on the basis of applicability to a wide range of hardware platforms. In addition, the image processing convolutional neural network can use fixed-point calculations to process the image data, so the reasoning speed of the convolutional neural network can be improved on the basis of a high-precision convolutional neural network.

[0113] Combination Figure 1 and Figure 3 The image data processing method and apparatus described in the embodiments of the present invention may be implemented by an image data processing device. Figure 4 FIG. 4 is a schematic diagram showing a hardware structure 400 of an image data processing device according to an embodiment of the present invention.

[0114] like Figure 4 As shown, the image data processing device 400 includes an input device 401, an input interface 402, a central processing unit 403, a memory 404, an output interface 405, and an output device 406. The input interface 402, the central processing unit 403, the memory 404, and the output interface 405 are connected to each other via a bus 410, and the input device 401 and the output device 406 are connected to the bus 410 via the input interface 402 and the output interface 405, respectively, and are further connected to other components of the image data processing device 400.

[0115] Specifically, the input device 401 receives input information from the outside and transmits the input information to the central processing unit 403 through the input interface 402; the central processing unit 403 processes the input information based on the computer executable instructions stored in the memory 404 to generate output information, temporarily or permanently stores the output information in the memory 404, and then transmits the output information to the output device 406 through the output interface 405; the output device 406 outputs the output information to the outside of the image data processing device 400 for user use.

[0116] That is to say, Figure 4 The image data processing device shown can also be implemented as comprising: a memory storing computer executable instructions; and a processor, which can realize the combination of Figure 1 and Figure 3 The invention describes an image data processing method and device.

[0117] An embodiment of the present invention further provides a computer storage medium, on which computer program instructions are stored; when the computer program instructions are executed by a processor, the image data processing method provided by the embodiment of the present invention is implemented.

[0118] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0119] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0120] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for processing image data, characterized in that: The method comprises: receiving an image to be processed; Processing the image to be processed using an image processing convolutional neural network to obtain a processing result of the image to be processed; The image processing convolutional neural network is a network obtained by updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor; For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of a plurality of first sub-tensors of a weight tensor, respectively, where the weight tensor is a tensor of the original convolutional neural network obtained and corresponding to the fixed-point tensor; The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor; The image processing convolutional neural network is a network obtained by updating a weight tensor corresponding to each fixed-point tensor based on each fixed-point tensor, and updating an initial scaling coefficient of a first sub-tensor corresponding to each fixed-point sub-tensor based on a final scaling coefficient of each fixed-point sub-tensor; Among them, for each fixed-point sub-tensor in any fixed-point tensor, the product of the fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor and the initial scaling coefficient of the first sub-tensor corresponding to the fixed-point sub-tensor is taken as the final scaling coefficient corresponding to the fixed-point sub-tensor, and the initial scaling coefficient is the scaling coefficient pre-acquired from the memory.

2. The method according to claim 1, characterized in that The using the image processing convolutional neural network to process the image to be processed to obtain a processing result of the image to be processed includes: Using each fixed-point tensor of the image processing convolutional neural network to respectively obtain different feature information of the image to be processed; Based on different feature information of the image to be processed, a processing result of the image to be processed is obtained.

3. The method according to claim 1, characterized in that The image to be processed is a face image to be processed, and the image processing convolutional neural network is a face recognition convolutional neural network; or, The image to be processed is a pedestrian image to be processed, and the image processing convolutional neural network is a pedestrian detection convolutional neural network.

4. The method according to claim 1, characterized in that The method is applied to a field programmable gate array FPGA, an embedded neural network processor NPU, an application specific integrated circuit ASIC or a central processing unit CPU.

5. The method according to claim 1, characterized in that For any weight tensor of the original convolutional neural network, a quantization method of any first sub-tensor of the weight tensor is determined based on the degree of data discreteness of the first sub-tensor.

6. The method according to claim 1, characterized in that The quantization result of the weight tensor W is represented by multiplying Q by alpha, where Q is the fixed-point tensor and alpha is the scaling vector.

7. The method according to claim 1, characterized in that The original convolutional neural network includes at least one convolutional layer and at least one fully connected layer; The at least one fixed-point tensor includes a fixed-point tensor corresponding to the weight tensor of each convolutional layer and a fixed-point tensor corresponding to the weight tensor of each fully connected layer.

8. An image data processing device, characterized in that: The device comprises: An image receiving module, used for receiving an image to be processed; An image processing module, used for processing the image to be processed by using an image processing convolutional neural network to obtain a processing result of the image to be processed; The image processing convolutional neural network is a network obtained by updating the original convolutional neural network based on each fixed-point tensor in at least one fixed-point tensor and a scaling vector corresponding to each fixed-point tensor; For each fixed-point tensor, the fixed-point tensor and the scaling vector corresponding to the fixed-point tensor are obtained by quantizing each of a plurality of first sub-tensors of a weight tensor, respectively, where the weight tensor is a tensor of the original convolutional neural network obtained and corresponding to the fixed-point tensor; The multiple first sub-tensors are obtained by decomposing the weight tensor along a predetermined dimension, the fixed-point tensor includes a fixed-point sub-tensor corresponding to each first sub-tensor, and the scaling vector includes a fixed-point weight scaling coefficient corresponding to each fixed-point sub-tensor; The image processing convolutional neural network is a network obtained by updating a weight tensor corresponding to each fixed-point tensor based on each fixed-point tensor, and updating an initial scaling coefficient of a first sub-tensor corresponding to each fixed-point sub-tensor based on a final scaling coefficient of each fixed-point sub-tensor; Among them, for each fixed-point sub-tensor in any fixed-point tensor, the product of the fixed-point weight scaling coefficient corresponding to the fixed-point sub-tensor and the initial scaling coefficient of the first sub-tensor corresponding to the fixed-point sub-tensor is taken as the final scaling coefficient corresponding to the fixed-point sub-tensor, and the initial scaling coefficient is the scaling coefficient pre-acquired from the memory.

9. An image data processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image data processing method according to claims 1-7 is implemented.

10. The device according to claim 9, characterized in that The processor is a field programmable gate array FPGA, an embedded neural network processor NPU, an application specific integrated circuit ASIC or a central processing unit CPU.

11. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image data processing method according to any one of claims 1 to 7 is implemented.

Images