Integer Tensor Network Data Processing Method

By tensor decomposing and quantizing the weight parameters of the neural network, the problem of redundant parameters of deep neural networks is solved, and more efficient storage and computing performance is achieved.

CN113095486BActive Publication Date: 2024-11-29TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110436569.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-22
Publication Date
2024-11-29
Estimated Expiration
2041-04-22

AI Technical Summary

Technical Problem

The existing tensor decomposition algorithms have limitations in improving the performance of deep neural networks, especially CP decomposition and Tucker decomposition cannot flexibly characterize the correlation between data of different dimensions of tensors, resulting in a large number of neural network parameters and large consumption of storage and computing resources. , and TT decomposition or TR decomposition has limited effect on performance improvement.

Method used

Multiple kernel tensors are obtained by tensor decomposing the weight parameters of the neural network, and quantizing them into fixed-point data with preset bit widths, reducing the amount of parameter data, reducing storage space, improving compression rate, and speeding up calculation speed.

Benefits of technology

It effectively reduces the amount of data in neural network parameters, reduces the storage space requirement, improves the compression rate and calculation speed of neural networks, and is suitable for image and video processing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113095486B_ABST
    Figure CN113095486B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, the method comprising: obtaining a neural network for image processing, the neural network comprising a plurality of network layers, the initial first weight parameter of each network layer comprising a multidimensional tensor and being floating point data; performing tensor decomposition on the first weight parameter of the network layer to obtain a plurality of core tensors; quantizing the plurality of core tensors according to a preset bit width to obtain a second weight parameter, the second weight parameter comprising fixed-point data of a preset bit width; calling the neural network to process a target image to obtain an image processing result. By performing tensor decomposition on the weight parameters of the neural network to obtain a plurality of core tensors, and quantizing the plurality of core tensors into fixed-point data of a preset bit width, the present disclosure can reduce the data volume of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and accelerate the calculation speed of the neural network, and has a wide range of applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Neural network-based image data processing algorithms or video data processing algorithms require deep neural network (DNN) models to extract data features in order to complete the final classification, detection, and recognition tasks. In the process of executing tasks, it is inevitable to introduce large-scale vectors or matrices and other tensors for multiplication and addition operations. Considering the limitations of computing speed and storage resources, it is usually necessary to decompose the high-dimensional tensors in the neural network operation process.

[0003] In related technologies, tensor decomposition algorithms are used to decompose high-dimensional tensors in the neural network operation process, that is, by decomposing the multi-dimensional kernel to reduce the number of trainable parameters in the neural network and weaken the high redundancy of data in the DNN. Common tensor decomposition algorithms include CANDECOMP / PARAFAC (CP) decomposition, Tucker decomposition, Tensor-Train (TT) decomposition, and Tensor-Ring (TR) decomposition. Among them, CP decomposition, also known as canonical decomposition (CANDECOMP) or parallel factor analysis (PARAFAC), can decompose a tensor into the sum of several rank-1 tensors; Tucker decomposition can decompose a tensor into a core tensor and a factor matrix along each dimension of the core tensor; TT decomposition can decompose a tensor into a series of three-dimensional core tensors and the product of two matrices (i.e., two-dimensional core tensors), and the core tensors are in a chain structure; TR decomposition uses a three-dimensional core tensor to replace the two matrices in the TT decomposition, and can decompose a tensor into a series of three-order core tensors, and the core tensors are in a ring structure.

[0004] However, due to the limitation of dimensionality, CP decomposition and Tucker decomposition cannot flexibly characterize the correlation between different dimensional data of tensors, have certain limitations, and require more storage resources; while in related technologies, TT decomposition or TR decomposition of high-dimensional tensors can improve the performance of neural networks to a certain extent, but for neural networks (especially deep neural networks), there are many neural network parameters involved in the storage and calculation process, and only TT decomposition or TR decomposition of high-dimensional tensors has limited effect on improving the performance of neural networks.

[0005] Therefore, how to further improve the performance of neural networks, especially deep neural networks, is a problem worthy of study. Summary of the invention

[0006] In view of this, the present disclosure proposes an image processing method, device, electronic device and storage medium, which can reduce the data amount of neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and accelerate the calculation speed of the neural network, and are widely used.

[0007] According to one aspect of the present disclosure, there is provided an image processing method, the method comprising: obtaining a neural network, the neural network being used for image processing, the neural network comprising a plurality of network layers, the initial first weight parameter of each network layer comprising a multidimensional tensor and being floating-point data; for any network layer, performing tensor decomposition on the first weight parameter of the network layer to obtain a plurality of core tensors, the plurality of core tensors comprising a two-dimensional tensor and / or a three-dimensional tensor; quantizing the plurality of core tensors according to a preset bit width to obtain a quantized second weight parameter, the second weight parameter comprising fixed-point data of a preset bit width; calling the neural network to process a target image to obtain an image processing result of the target image.

[0008] In a possible implementation, calling the neural network to process the target image to obtain the image processing result of the target image includes: for the i-th network layer, convolving the i-1th level processing result output by the i-1th network layer according to the second weight parameter of the i-th network layer to obtain the i-th convolution result, 1≤i≤N, N is the number of network layers of the neural network, wherein the 0th level processing result is the target image; batch normalizing the i-th convolution result to obtain the i-th normalized result; activating and quantizing the i-th normalized result to obtain the i-th processing result, and determining the N-th processing result as the image processing result of the target image.

[0009] In a possible implementation, the multiple core tensors are quantized according to a preset bit width to obtain a quantized second weight parameter, including: according to the preset bit width, the multiple core tensors are respectively quantized to obtain multiple quantized first tensors; for the kth first tensor, the kth first tensor is merged with the k-1th second tensor to obtain the kth third tensor, 2≤k≤K, K is the number of the first tensors, wherein the 1st second tensor is the 1st first tensor; according to the preset bit width, the kth third tensor is quantized to obtain the kth second tensor; and the Kth second tensor is determined as the second weight parameter.

[0010] In a possible implementation, performing tensor decomposition on the first weight parameter of the network layer to obtain a plurality of core tensors includes: performing tensor decomposition on the first weight parameter by a tensor row-column TT decomposition method to obtain the plurality of core tensors.

[0011] In a possible implementation, the batch normalization of the i-th convolution result to obtain the i-th normalized result includes: quantizing the first batch of normalized parameters according to the preset bit width to obtain the quantized second batch of normalized parameters, the second batch of normalized parameters including fixed-point data of the preset bit width; and batch normalizing the i-th convolution result according to the second batch of normalized parameters to obtain the i-th normalized result.

[0012] In one possible implementation, the activating and quantizing the i-th standardized result to obtain the i-th processed result includes: activating the i-th standardized result to obtain the i-th activated result; and quantizing the i-th activated result according to the preset bit width to obtain the i-th processed result, wherein the i-th processed result includes fixed-point data of a preset bit width.

[0013] In a possible implementation, the preset bit width is configured according to the dimensions of the multiple core tensors, core tensors of different dimensions are quantized into first tensors of different bit widths, and the preset bit width is greater than or equal to a bit width threshold.

[0014] According to another aspect of the present disclosure, an image processing device is provided, the device comprising: an acquisition module for acquiring a neural network, the neural network being used for image processing, the neural network comprising multiple network layers, the initial first weight parameter of each network layer comprising a multidimensional tensor and being floating-point data; a tensor decomposition module for performing tensor decomposition on the first weight parameter of any network layer to obtain multiple core tensors, the multiple core tensors comprising a two-dimensional tensor and a three-dimensional tensor; a quantization module for quantizing the multiple core tensors according to a preset bit width to obtain a quantized second weight parameter, the second weight parameter comprising fixed-point data of a preset bit width; and a calling module for calling the neural network to process a target image to obtain an image processing result of the target image.

[0015] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above method.

[0016] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0017] By performing tensor decomposition on the weight parameters of a neural network to obtain multiple core tensors, and quantizing the multiple core tensors into fixed-point data with a preset bit width, according to various aspects of the present disclosure, the data volume of the neural network parameters can be reduced, the storage space of the neural network can be reduced, the compression rate of the neural network can be improved, the calculation speed of the neural network can be accelerated, and it has a wide range of applications.

[0018] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0020] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A schematic diagram of tensor decomposition according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A schematic diagram showing an exemplary image processing method according to an embodiment of the present disclosure.

[0023] Figure 4a and Figure 4b A schematic diagram of the core tensor data distribution of an embodiment of the present disclosure is shown.

[0024] Figure 5 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0025] Figure 6 A block diagram of a terminal for data processing according to an exemplary embodiment is shown.

[0026] Figure 7 A block diagram of a server for data processing according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0027] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0029] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0030] Relying on the rapid development of computing clusters such as GPU (Graphics Processing Unit), Convolutional Neural Networks (CNN) have been successfully applied to various fields, especially in the field of deep learning. Despite many advances, training and reasoning large-scale convolutional neural networks still require huge computing and storage resources, and are difficult to be widely used in portable devices with limited resources.

[0031] In the related art, tensor decomposition can be used to decompose high-dimensional tensors into multiple low-dimensional tensors without much impact on the overall performance of convolutional neural networks, thereby reducing the number of parameters of the neural network, thereby compressing the neural network and improving the performance of the neural network. However, due to the curse of dimensionality (or limitation), the common CP decomposition and Tucker decomposition have key limitations, and the related art of only performing TT decomposition or TR decomposition on high-dimensional tensors has limited effect on improving the performance of neural networks. Therefore, it is necessary to further improve the performance of neural networks, especially deep neural networks.

[0032] In view of this, the present disclosure proposes an image processing method, device, electronic device and storage medium, which obtains multiple core tensors by performing tensor decomposition on the weight parameters of the neural network, and quantizes the multiple core tensors into fixed-point data with a preset bit width. The present disclosure can reduce the data volume of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and speed up the calculation speed of the neural network, and has a wide range of applications.

[0033] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown.

[0034] like Figure 1 As shown, in a possible implementation, the image processing method includes:

[0035] Step S101, obtaining a neural network, wherein the neural network is used for image processing, the neural network includes multiple network layers, and an initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating-point data.

[0036] In a possible implementation, an electronic device may be used to obtain a neural network. The neural network may be a neural network model. The electronic device may be a terminal or a server. The terminal includes a mobile phone, a tablet computer, a desktop computer, etc. The server may be a single server, a server cluster consisting of several servers, or a cloud computing service center. The electronic device may include a neural network chip for storing neural network parameters and performing neural network calculations.

[0037] In one possible implementation, the neural network can be used for image or video processing. For example, an application for processing a target image can be installed on the electronic device to obtain the neural network, thereby using the neural network to extract features of the input target image to obtain an image processing result of the target image. The neural network includes but is not limited to any type of convolutional neural network, artificial neural network, recurrent neural network, long short-term memory neural network, deep reinforcement learning network, graph neural network and tensor neural network. It should be understood by those skilled in the art that the present disclosure does not limit the type of neural network.

[0038] In a possible implementation, the neural network may include multiple network layers, and an initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating-point data.

[0039] Among them, tensor is a data structure commonly used in neural networks, which can describe data from the perspective of dimension and how elements in the tensor are arranged and combined, and may include various types of data such as scalar, vector and matrix. For example, a scalar can be a 0-dimensional tensor, a vector can be a 1-dimensional tensor, and a matrix can be a 2-dimensional tensor. In the disclosed embodiment, a tensor of 3 dimensions or more can be called a high-dimensional tensor, also known as a multi-dimensional tensor. Floating-point data or fixed-point data can describe data from the perspective of how the data is stored. For example, a decimal can be described by a floating-point type with an unfixed decimal point, or by a fixed-point type with a fixed decimal point, even if there may be a loss of precision when a fixed-point type is used for description. It is worth noting that, despite the different description angles, tensors or floating-point data can describe the first weight parameter, and the floating-point data can also be called a floating-point tensor.

[0040] Taking a convolutional neural network as an example, a convolutional neural network may include an input layer, a hidden layer, and an output layer. Among them, the hidden layer may include a convolutional layer, a pooling layer, and a fully connected layer, and each layer in the convolutional neural network may include multiple neurons. In the process of performing a convolution operation, each convolutional layer corresponds to a set of convolution kernels, and each convolution kernel includes multiple weight values. The multiple weight values ​​in each convolutional layer can constitute the weight tensor of the convolutional layer, that is, the initial first weight parameter. For the sake of simplicity of description, the first weight parameter can also be directly called a weight value or weight.

[0041] Furthermore, the first weight parameter can be a multidimensional tensor. For example, the four dimensions of the first weight parameter can be represented by the number of convolution kernels in the convolution layer (i.e., the number of channels of the feature map output by the convolution layer), the number of channels of the image input to the convolution layer, the height of the convolution kernel, and the width of the convolution kernel. For another example, a convolution layer includes 32 convolution kernels, and the size of the convolution kernel is 4*4*4. The dimension of the first weight parameter of the convolution layer can be expressed as (32, 4, 4, 4), and the number of weight values ​​in the first weight parameter is 32*4*4*4.

[0042] Furthermore, the first weight parameter may be a floating point tensor. For example, each data (ie, element) in the first weight parameter may be data of type float32 or float16.

[0043] In a possible implementation, the neural network is used for image processing and may include two processes: a forward propagation (FP) process and a back propagation (BP) process. The forward propagation process may perform multiplication and accumulation operations on the input image of the neural network and the pre-configured weights to obtain a preliminary output result of the neural network; the back propagation process may compare the preliminary output result of the neural network with the actual true value, calculate the error between the preliminary output result of the neural network and the true value, and adjust the weight of the neural network according to the error, so that the output result of the neural network approaches the actual true value. In addition, the forward propagation process can be used for neural network reasoning, also known as the reasoning process; the back propagation process can be used for neural network training, also known as the training process.

[0044] In one possible implementation, the forward propagation process may include a batch normalization process. Batch Normalization (BN) can be used in the process of forward propagation of a neural network to improve the performance and stability of the neural network, so as to overcome the problem that the number of neural network layers is difficult to train. Among them, the batch normalization process can use a specific function to transform the output of a certain layer of the neural network to obtain a normalized value. For example, batch normalization can be placed before the activation function to regularize the output of the convolutional layer into a normal distribution with a mean of 0 and a variance of 1, and then the regularized data is input into the activation function for activation.

[0045] In a possible implementation, the network parameters of the neural network may include at least one of the first weight parameter, neuron value, first batch of standardized parameters, activation function value, back propagation error value, neuron gradient value and weight update value. Wherein, corresponding to the process of image processing by the neural network, the network parameters in the forward propagation process may include the first weight parameter, neuron value, first batch of standardized parameters and activation function value, etc.; the network parameters in the back propagation process may include back propagation error value, neuron gradient value and weight update value, etc. It is worth noting that in addition to the first weight parameter, other network parameters (for example, the first batch of standardized parameters) may also include multi-dimensional tensors and are floating-point data, and may be tensor decomposed or quantized according to the image processing method provided in the embodiment of the present disclosure. In addition, the network parameters may also be fixed-point data. It should be understood by those skilled in the art that there are many parameters involved in the neural network calculation process. For example, the network parameters may also include multiple types such as matrices and 3D tensors generated by processes such as data conversion. The embodiment of the present disclosure takes the first weight parameter in the network parameters as an example for explanation, and does not constitute a limitation on the type of network parameters.

[0046] For example, during the forward propagation of a neural network, the neuron value of the input layer of the neural network can be a column vector, that is, a one-dimensional tensor; the neuron value can be multiplied and added with the pre-configured weight value, and then the multiplication and addition results are sent to the batch normalization layer, and the results are regularized using the first batch of normalization parameters to obtain the output of the batch normalization layer. Finally, the output of the batch normalization layer is applied to the activation function for nonlinear processing to obtain the activation function value, which can be a scalar or a multi-dimensional tensor.

[0047] For another example, during the back propagation of a neural network, after obtaining the preliminary output result of the neural network, the preliminary output result of the neural network can be compared with the actual true value to obtain an error value (i.e., a back propagation error value), and reverse iteration is performed based on the error value to update the weight value so that the output result of the neural network is closer to the true value, wherein the value for updating or adjusting the weight value during the forward propagation process can be the weight update value. In addition, during the back propagation of a neural network, a loss function can be used to evaluate the error between the output result of the neural network and the true value. In order to minimize the loss function, a gradient descent algorithm can be used to calculate the gradient, and the minimum value of the error is solved along the direction of the gradient descent, and then the weight value is updated, that is, the value obtained after calculating the gradient of the network parameters (such as weight values ​​or biases) of the neural network can be the neuron gradient value.

[0048] Step S102: for any network layer, perform tensor decomposition on a first weight parameter of the network layer to obtain a plurality of core tensors, wherein the plurality of core tensors include a two-dimensional tensor and / or a three-dimensional tensor.

[0049] In a possible implementation, for any network layer, a first weight parameter of the network layer is subjected to tensor decomposition to obtain multiple core tensors. Among them, the first weight parameter has a high dimension and can be a high-dimensional tensor; the core tensor obtained after decomposition has a low dimension and can be a low-dimensional tensor, that is, tensor decomposition can decompose a high-dimensional tensor into a series of low-dimensional core tensors. The core tensor is also called a core tensor or a factor matrix, and can be, for example, a two-dimensional matrix or a three-dimensional tensor. For example, when the first weight parameter is subjected to tensor decomposition by using the TT decomposition method, the multiple core tensors may include a two-dimensional tensor and a three-dimensional tensor; when the first weight parameter is subjected to tensor decomposition by using the TR decomposition method, the multiple core tensors may include a three-dimensional tensor. It is worth noting that, similar to matrix decomposition, the decomposed core tensors are not completely equal to the high-dimensional tensor after merging (or restoring), and there will be a certain degree of loss in the process of tensor decomposition, that is, the decomposed core tensors are approximately equal to the high-dimensional tensor after merging.

[0050] In a possible implementation, performing tensor decomposition on the first weight parameter of the network layer to obtain a plurality of core tensors may include: performing tensor decomposition on the first weight parameter by a tensor row-column TT decomposition method to obtain the plurality of core tensors.

[0051] In a possible implementation, the first weight parameter is tensor decomposed by a tensor row-column TT decomposition method (i.e., Tensor-Train decomposition algorithm) to obtain the multiple core tensors. Among them, the head core tensor and the tail core tensor in the multiple core tensors can be two-dimensional matrices (i.e., two-dimensional tensors), and the remaining core tensors except the head core tensor and the tail core tensor can be three-dimensional tensors. It should be understood by those skilled in the art that although the embodiment of the present disclosure uses the tensor row-column TT decomposition method as an example to illustrate the image processing method, in other embodiments of the present disclosure, other types of tensor decomposition methods such as the Tensor-Ring decomposition method may also be used to perform tensor decomposition on the first weight parameter, and the present disclosure does not limit the method of tensor decomposition.

[0052] The first weight parameter is decomposed into a tensor by using a tensor row-column TT decomposition method. Compared with CP decomposition and Tucker decomposition, the embodiment of the present disclosure is insensitive to the dimension and simple structure of the tensor, has fewer limiting factors, is highly versatile, and has a wider range of applications.

[0053] Figure 2 A schematic diagram of tensor decomposition according to an embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, in one possible implementation, the tensor It can be the first weight parameter in the network parameters. Figure 2 , tensor Can be a 5-dimensional tensor, K H , K W , K D , M and N can be tensors respectively 5 dimensions, for example, K H , K W and K D can represent the height, width and depth of the convolution kernel respectively, and M and N can represent the number of input channels and output channels respectively. It is worth noting that when tensors are Before decomposing the tensor, Figure 2 The M and N in are decomposed according to the following formula: After decomposition, m k n k Can be used to represent a dimension of a kernel tensor. For example, for the second kernel tensor For example, there can be three dimensions r1, r2 and m1n1, where m1n1 is a dimension of the core tensor.

[0055] In a possible implementation, the first weight parameter is decomposed into a tensor by a tensor row-column TT decomposition method to obtain the multiple core tensors. Figure 2 , the TT decomposition method can be used to decompose the 5-dimensional tensor Decompose to obtain d lower-dimensional tensors (i.e., two-dimensional tensors or three-dimensional tensors), that is, the original d-dimensional tensor can be decomposed into a contraction of multiple core tensors. And the tail tensor It can be a two-dimensional tensor (i.e., a two-dimensional matrix), and the remaining decomposed tensors can be three-dimensional tensors. In terms of representation, one dimension of the head tensor is r0 and one dimension of the tail tensor is r d+1 The value of is 1, indicating that the first tensor actually has only two dimensions. Therefore, for the first tensor In terms of is a two-dimensional tensor; for the second three-dimensional tensor In terms of is a three-dimensional tensor. Figure 2 The TT decomposition in can also be schematically expressed by the following formula:

[0056]

[0057] in, Can represent d-dimensional tensors, which are generally of higher dimensions; It can represent a d-dimensional tensor The tensor obtained after tensor decomposition generally has a lower dimension. It can also be viewed as a tensor sequence. Taking the TT decomposition method as an example, Can be a 3D tensor, and Can be a 2D tensor, All of them can be called core tensors (or TT cores); the operator symbol × 1 It can represent the product of multiple decomposed core tensors. Multiple decomposed core tensors can be multiplied by the operator × 1 For example, the core tensor Can be used with core tensors The corresponding elements can be multiplied and then merged to obtain the merged tensor. It is worth noting that the operator × 1 The superscript 1 in the upper right corner is for illustration only and does not affect the actual meaning of the operator symbol for contraction or merging. For example, the operator symbol × can be used without the superscript, or the superscript can be placed in the lower right corner and ×1 can be used to represent merging.

[0058] Step S103, according to the preset bit width, quantize the multiple core tensors to obtain a quantized second weight parameter, wherein the second weight parameter includes fixed-point data of the preset bit width. The preset bit width can be pre-configured, such as 8 bits (i.e., bits) or 15 bits, and the present disclosure does not limit the specific number of bits of the preset bit width. It should be understood by those skilled in the art that in addition to quantizing the core tensor after tensor decomposition, the embodiments of the present disclosure can also quantize other neural network parameters, for example, it can also quantize the image data input to the convolutional layer, or quantize the first batch of standardized parameters, and the present disclosure does not limit the neural network parameters to be quantized.

[0059] In a possible implementation, the plurality of core tensors may be floating point data or fixed point data. The fixed point data of the preset bit width may include integer data, such as int8 type data.

[0060] In a possible implementation, the electronic device may quantize (also called discretize) the multiple core tensors by the following formula to obtain a quantized first tensor, that is, the multiple core tensors may be converted into fixed-point data of a preset bit width based on the following formula:

[0061]

[0062] Among them, x can be the core tensor, and the core tensor can be floating point data; round(·) is a rounding function; bits is the target bit width of x (i.e., the preset bit width), which can be pre-configured. It should be understood by those skilled in the art that the rounding function in formula (2) can also be an upward rounding function, such as a ceil(·) function, or a downward rounding function, such as a floor(·) function, and the present disclosure does not limit the type of the rounding function.

[0063] It is worth noting that x may also be a network parameter such as the first batch of standardized parameters. In addition, in addition to converting a floating-point tensor into fixed-point data of a preset bit width, formula (2) may also convert a fixed-point tensor into fixed-point data of a preset bit width, which is not limited in the present disclosure.

[0064] By performing tensor decomposition on the first weight parameter of the neural network and quantizing the core tensor obtained after the decomposition, the embodiment of the present disclosure can compress the neural network model in image processing or video processing, reduce the storage of neural network parameters, and improve the calculation speed of the neural network.

[0065] In a possible implementation, the quantization may include inference quantization and training quantization. Inference quantization may be the process of quantizing network parameters related to inference when using a neural network for inference, corresponding to the forward propagation process of the neural network, and the network parameters for inference quantization may include the first weight parameter, neuron value, the first batch of standardized parameters and activation function value, etc.; training quantization may be the process of quantizing network parameters related to training when using a neural network for training, corresponding to the back propagation process of the neural network, and the network parameters for training quantization may include back propagation error value, neuron gradient value and weight update value, etc.

[0066] In the following, the embodiment of the present disclosure will take the inference quantization (i.e., the quantization of each parameter in the forward propagation process) as an example to introduce the quantization process. It should be understood by those skilled in the art that the inference quantization and training quantization processes can be based on the Quan (·) function in formula (2), and the quantization process of the network parameters in the reverse propagation process can be similar to the quantization process of the network parameters in the forward propagation process, which will not be described in detail.

[0067] In one possible implementation, the multiple core tensors are quantized according to a preset bit width to obtain a quantized second weight parameter, which may include: quantizing the multiple core tensors separately according to the preset bit width to obtain multiple quantized first tensors; for the kth first tensor, fusing the kth first tensor with the k-1th second tensor to obtain the kth third tensor, 2≤k≤K, K is the number of the first tensors, wherein the 1st second tensor is the 1st first tensor; quantizing the kth third tensor according to the preset bit width to obtain the kth second tensor; and determining the Kth second tensor as the second weight parameter.

[0068] In a possible implementation, the multiple core tensors are quantized respectively according to a preset bit width to obtain multiple quantized first tensors. The multiple core tensors are quantized respectively based on formula (2). For example, the multiple core tensors may be multiple core tensors obtained by performing tensor decomposition on the first weight parameter in the network parameter, and the multiple core tensors may be quantized respectively based on formula (2) to obtain multiple quantized first tensors.

[0069] In a possible implementation, for the kth first tensor, the kth first tensor is fused with the k-1th second tensor to obtain the kth third tensor, 2≤k≤K, K is the number of the first tensors, and the first second tensor is the first first tensor. For example, when k=2, for the second first tensor, the second first tensor can be fused with the first second tensor (i.e., the first first tensor) to obtain the second third tensor.

[0070] In a possible implementation, the electronic device may quantize the third tensor according to a preset bit width by using the following formula to obtain a second tensor, wherein the second tensor may be fixed-point data of a preset range and a preset bit width:

[0071] QuanK(x,bits)=clip(Quan(x,bits),-1+Δ,1-Δ)) (3)

[0072] in,

[0073] In formula (3), x can be the third tensor, that is, the tensor obtained by fusing the first tensor with the second tensor, and the third tensor can be floating point data or fixed point data; the clip(·) function can project the third tensor x to a preset minimum and maximum interval, that is, the clip(·) function can be a clipping function for controlling the representation range of the second tensor; bits can be a preset bit width; Δ can represent the minimum interval between the elements of the third tensor, and can be set according to the preset bit width based on formula (4). It is worth noting that the Kth second tensor can be determined as the second weight parameter, and the second weight parameter can be used as a convolution kernel for convolution operation.

[0074] In a possible implementation, when the bit width of the first tensor is a preset bit width of bits, the bit width of the second tensor may be the same as the bit width of the first tensor, and the bit width of the third tensor may be (2*bits-1). Referring to formula (2), for two first tensors with a preset bit width of bits, since the first tensor is a function of Δ, the third tensor obtained after fusion is Δ×Δ (i.e., ), that is, the bit width of the third tensor can be (2*bits-1). For example, when the bit width of each first tensor is 8 bits, the bit width of the second tensor is also 8 bits, and the bit width of the third tensor obtained by fusing the second first tensor with the first first tensor can be 15 bits. It should be understood by those skilled in the art that the present disclosure does not limit the method of tensor fusion.

[0075] In one possible implementation, the kth third tensor is quantized according to a preset bit width to obtain the kth second tensor, and the Kth second tensor is determined as the second weight parameter. The kth third tensor can be quantized based on formula (3). For example, when k=2, for the second first tensor, the second first tensor can be fused with the first second tensor (i.e., the first first tensor) to obtain the second third tensor. For the second third tensor, it is first quantized based on formula (3) to obtain the second second tensor. When k increases by 1, the same process is repeated until the Kth second tensor is obtained. Taking the first tensor as the core tensor obtained by tensor decomposition of the first weight parameter as an example, the multiple core tensors can be quantized based on the preset bit width and formula (2) to obtain multiple first tensors, which is equivalent to the compression process of the first weight parameter; the quantized first tensor is fused with the second tensor to obtain a third tensor, and then the third tensor is quantized based on formula (3) to obtain the second weight parameter, which is equivalent to the decompression (i.e., recovery) process of the first weight parameter. In addition, taking the core tensors as 4 as an example, after quantizing the 4 core tensors to obtain 4 first tensors, the second weight parameter can be obtained after 3 fusions and quantizations.

[0076] By performing tensor decomposition on the first weight parameter to obtain multiple core tensors, quantizing the multiple core tensors to obtain multiple first tensors, and fusing and quantizing the multiple first tensors to obtain the second weight parameter, the disclosed embodiment can reduce the amount of data of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and speed up the calculation speed of the neural network. Moreover, since the data loss of the second weight parameter is very small compared to the first weight parameter, the data loss during the compression process of the neural network is also reduced. In addition, fusing multiple first tensors can obtain data with a higher bit width, reduce the time consumption of the neural network, and is simple and convenient.

[0077] Step S104, calling the neural network to process the target image to obtain an image processing result of the target image.

[0078] In a possible implementation, the electronic device may call the neural network to process the target image to obtain an image processing result of the target image. The image processing result may be used to further obtain image features of the target image.

[0079] In a possible implementation, in the process of calling the neural network to process the target image, network parameters are obtained, wherein the network parameters may include at least one of a first weight parameter, a neuron value, a first batch of standardized parameters, an activation function value, a back propagation error value, a neuron gradient value, and a weight update value.

[0080] In a possible implementation, calling the neural network to process the target image to obtain the image processing result of the target image may include: for the i-th network layer, convolving the i-1th level processing result output by the i-1th network layer according to the second weight parameter of the i-th network layer to obtain the i-th convolution result, 1≤i≤N, N is the number of network layers of the neural network, wherein the 0th level processing result is the target image; batch normalizing the i-th convolution result to obtain the i-th normalized result; activating and quantizing the i-th normalized result to obtain the i-th processing result, and determining the N-th processing result as the image processing result of the target image.

[0081] In a possible implementation, for the i-th network layer, the i-1th level processing result output by the i-1th network layer is convolved according to the second weight parameter of the i-th network layer to obtain the i-th convolution result. Wherein, 1≤i≤N, N is the number of network layers of the neural network, and the 0th level processing result is the target image. For example, when i=1, the 0th level processing result is the target image, and at this time, the second weight parameter of the 1st network layer is used as the convolution kernel to convolve the target image, thereby obtaining the 1st convolution result.

[0082] In a possible implementation, the target image may be quantized in advance based on formula (2) to obtain a target image with a preset bit width. For example, the target image may be a 4-dimensional tensor, where the 4 dimensions may be the number of samples, image height, image width, and number of channels, respectively. The preset bit width may be 8 bits, that is, the target image may be quantized in advance to a target image with a bit width of 8 bits.

[0083] In a possible implementation, the i-th convolution result is batch standardized to obtain the i-th standardized result; the i-th standardized result is activated and quantized to obtain the i-th processing result, and the N-th processing result is determined as the image processing result of the target image. The i-th convolution result can be used as the input of the batch standardization layer. For each convolution result, batch standardization can be performed through the batch standardization layer. The batch standardization layer can be located before the activation layer, that is, the output of the batch standardization layer can be used as the input of the activation layer. Then, the output of the activation layer can be quantized to obtain a processing result corresponding to the convolution result, and the processing result finally output by the activation layer is determined as the image processing result of the target image.

[0084] In one possible implementation, batch normalizing the i-th convolution result to obtain the i-th normalized result may include: quantizing the first batch of normalized parameters according to the preset bit width to obtain the quantized second batch of normalized parameters, wherein the second batch of normalized parameters include fixed-point data of the preset bit width; and batch normalizing the i-th convolution result according to the second batch of normalized parameters to obtain the i-th normalized result.

[0085] Among them, the first batch of standardized parameters may include any one of the mean of the batch standardized layer input data, the standard deviation of the batch standardized layer input data, the proportion of the batch standardized layer, and the offset, which are used to batch standardize the i-th convolution result. The first batch of standardized parameters can be either floating-point data or fixed-point data. It is worth noting that the quantization of the first batch of standardized parameters can also be based on formula (2). For example, the proportion of the batch standardized layer can be quantized into fixed-point data with a bit width of 8 bits based on formula (2), and then the i-th convolution result can be batch standardized according to the fixed-point data.

[0086] In a possible implementation, after quantizing the first batch of standardized parameters to obtain the quantized second batch of standardized parameters, the electronic device may perform batch standardization on the i-th convolution result by the following formula to obtain the i-th standardized result, and the i-th standardized result may be used as the output of the batch standardization layer:

[0087]

[0088] in,

[0089] In formula (6), x can be the i-th convolution result; μ o It can be the mean of the input data of the batch normalization layer, σ o It can be the standard deviation of the batch normalization layer input data, and ∈ can be a small value to avoid division by zero; It can be a normalization parameter, for example, using μ o , σ o After normalizing the i-th convolution result x, the normalized parameters that conform to the normal distribution are obtained by γ o and β o can be the scale and offset of the batch normalization layer, respectively, for Make adjustments.

[0090] Based on the above formulas (5) and (6), the i-th convolution result can be normalized first, and then the output of batch normalization can be adjusted using the ratio and offset. In the disclosed embodiment, the first batch of normalization parameters of the batch normalization layer can be quantized first, and the i-th convolution result can be batch normalized using the quantized second batch of normalization parameters, that is, the parameters of the batch normalization layer in formulas (5) and (6) can be the quantized second batch of normalization parameters.

[0091] By quantizing the first batch of standardized parameters to obtain the quantized second batch of standardized parameters, and batch standardizing the convolution results according to the second batch of standardized parameters, the disclosed embodiment can reduce the data volume of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and speed up the calculation speed of the neural network, and is particularly suitable for convolutional neural networks.

[0092] In one possible implementation, activating and quantizing the i-th standardized result to obtain the i-th processed result may include: activating the i-th standardized result to obtain the i-th activated result; quantizing the i-th activated result according to the preset bit width to obtain the i-th processed result, wherein the i-th processed result includes fixed-point data of a preset bit width.

[0093] In a possible implementation, the electronic device may activate the i-th normalized result (i.e., the activation function value) by the following formula to obtain the i-th activation result, and then quantize the i-th activation result according to the preset bit width to obtain the i-th processing result, wherein the i-th processing result includes fixed-point data of the preset bit width:

[0094]

[0095] in,

[0096] In formula (8), x can be the activation function value in the network parameters, that is, the i-th normalized result; the activation function can be the ReLu(·) function; It can be the output after the activation function value acts on the activation function and serves as the input of the Quan(·) function; bits can be a preset bit width. It should be understood by those skilled in the art that, in addition to the ReLu(·) function, the activation function can include multiple types, such as the Sigmoid(·) function and the Tanh(·) function, etc., and the present disclosure does not limit the type of activation function. It is worth noting that the activation layer can be located before the batch normalization layer or after the batch normalization layer, and the present disclosure does not limit the quantization order of the neural network parameters.

[0097] By activating the standardized result to obtain the activation result, and quantizing the activation result according to the preset bit width to obtain the processing result, the disclosed embodiment can reduce the data volume of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and speed up the calculation speed of the neural network. In addition, since the bit width of the activation function input increases due to the multiplication operation, quantizing the output of the activation function into low-bit-width fixed-point data can reduce the amount of calculation of the neural network and improve the calculation efficiency of the neural network.

[0098] In summary, the embodiment of the present disclosure obtains a neural network including multiple network layers for image processing, and for any network layer, performs tensor decomposition on the first weight parameter of the network layer to obtain multiple core tensors, and then quantizes the multiple core tensors according to a preset bit width to obtain the quantized second weight parameters including fixed-point data of the preset bit width, and finally calls the neural network to process the target image to obtain the image processing result of the target image. This can reduce the data volume of the neural network parameters, reduce the storage space of the neural network, improve the compression rate of the neural network, and speed up the calculation speed of the neural network, and has a wide range of applications.

[0099] Figure 3 A schematic diagram showing an exemplary image processing method according to an embodiment of the present disclosure.

[0100] like Figure 3 As shown, the image processing method of the embodiment of the present disclosure may include four stages, namely, quantization of the first weight parameter, convolution, quantization of the first batch of normalization parameters, and quantization of the activation function value. Figure 3 Core Tensor It can be obtained by performing tensor decomposition of the first weight parameter in the network parameters through the above formula (1). Those skilled in the art should understand that: Figure 3 The processing process of the parameters involved in the forward propagation of the neural network is provided by way of example, and does not constitute a limitation on other processing methods (for example, further processing of the acquired image processing results and processing of the neural network parameters in the back propagation process of the neural network).

[0101] In one possible implementation, Figure 3 As shown, FP can represent the forward propagation process of the neural network. It can be the tensor in the above formula (1) (For example, the first weight parameter in the neural network parameter) is the core tensor obtained by tensor decomposition. It can be floating-point data, and the core tensor obtained after tensor decomposition can still be floating-point data.

[0102] In one possible implementation, Figure 3As shown, the multiple core tensors can be processed according to the preset bit width. Quantize them separately to obtain multiple quantized first tensors Figure 3 There are (d+1) first tensors in total, and the bit width of each first tensor is 8 bits. The quantification can be based on the following formula:

[0103]

[0104] The Quan(·) function in formula (9) is equivalent to the Quan(·) function in formula (2), that is, formula (9) is expressed differently from formula (2), but the actual quantization process is the same.

[0105] See also Figure 3 , the first two first tensors after quantization can be and Merge (i.e., fuse) to obtain the third tensor with a width of 15 bits The third tensor can be fixed-point data. Then, based on formula (3), the third tensor Quantize to get the second tensor with the first 8-bit width Since the first tensor is also 8-bit wide, so the second tensor With the first tensor Merge to get the third tensor with a second 15-bit width Then for the third tensor Apply formula (3) to quantize and get the second tensor with a width of 8 bits. And so on, until the third tensor with a width of 15 bits is obtained. And for this third tensor Apply formula (3) to quantize and get the second tensor with the dth 8-bit width The second tensor In addition, the plurality of first tensors are fused and quantized to obtain a second weight parameter, which can also be expressed by the following set of formulas (ie, formula (10)):

[0106]

[0107]

[0108] …

[0109]

[0110] in, The second weight parameter

[0111] Continue to see Figure 3 , Figure 3 Q8 in the formula (2) or the formula (9) may represent the Quan(·) function, and 8 may represent that the preset bit width in the formula (2) or the formula (9) is 8 bits; Figure 3 Q8 k It can represent the QuanK(·) function in formula (3) or formula (10), and 8 can represent that the preset bit width in formula (3) or formula (10) is 8 bits. In the convolutional neural network, the second weight parameter It can be used as a convolution kernel to perform convolution operations with the input image that has been quantized to 8 bits. In the case of It can still be 5 dimensions (for example, the dimension is K H ×K W ×K D ×M×N).

[0112] like Figure 3 As shown, the second weight parameter As a convolution kernel, it performs convolution operation with the 8-bit input image. The output of the convolution operation can be a 15-bit wide tensor x conv . Tensor x conv Can be used as the input of the batch normalization layer, that is, the tensor x conv It can be a convolution result. It is worth noting that the neural network may include multiple network layers. Figure 3 The convolution result in can be the i-th convolution result output by the i-th network layer.

[0113] exist Figure 3 Medium, μ x Can be the mean of the input data (i.e., MEAN); σ x It can be the standard deviation (i.e., STD) of the input data; γ can be the scale (i.e., Scale) of the batch normalization layer, and β can be the offset (i.e., Offset) of the batch normalization layer. All of the above four parameters can be the first batch normalization parameters.

[0114] In a possible implementation, the mean μ can be first calculated based on formula (2): x and standard deviation σ x Quantize and get the mean value μ of the 8-bit width after quantization x q and standard deviation σ x q, then combined with formula (6), according to the mean value μ of the 8-bit width after quantization x q and standard deviation σ x q For the convolution result x conv Normalize and get the normalized parameters with a bit width of 15 bits Then, based on formula (2), the normalized parameter Quantize again and normalize the 15-bit bit width parameters Quantized to 8-bit normalized parameters Then, the ratio γ and the offset β can be quantized based on formula (2) to obtain the quantized 8-bit width ratio γ q and the offset β q , and combined with formula (5), the normalized parameter of 8-bit width is Adjust to get the output of the batch normalization layer with a width of 15 bits It is worth noting that the present disclosure does not limit the order in which the first batch of standardized parameters are quantized.

[0115] Continue to see Figure 3 , the output of the batch normalization layer can be used as the input of the activation function, that is, the output of the batch normalization layer can be combined with formula (8) Activate. The activation function can be the ReLu(·) function. The activation result (i.e., the activation function value) is a 15-bit wide x a Then, the activation result is quantized based on formula (2) to obtain a processing result with a bit width of 8 bits. It is worth noting that the neural network includes multiple network layers, and the processing results It can correspond to the i-th network layer and is the i-th processing result.

[0116] The embodiments of the present disclosure combine tensor decomposition and quantization of network parameters during the forward propagation of the neural network, including tensor decomposition and quantization of the first weight parameter and quantization of the first batch of standardized parameters and activation function values ​​into low-bit-width data. On the one hand, compared with the related art of only performing tensor decomposition on weight values, the neural network model can be further reduced, the neural network can be further compressed, and the calculation of the neural network can be accelerated, which is particularly suitable for 3D tensor convolutional neural network models. On the other hand, in the process of quantizing the network parameters, the embodiments of the present disclosure convert floating-point operations into fixed-point operations, which can reduce data storage space, speed up calculations, and reduce the storage and computing resources required for deep neural networks.

[0117] In a possible implementation, the preset bit width is configured according to the dimensions of the multiple core tensors, and the core tensors of different dimensions are quantized into first tensors of different bit widths, and the preset bit width is greater than or equal to the bit width threshold. For example, for the TT decomposition method, the multiple core tensors obtained after tensor decomposition of the first weight parameter include both two-dimensional tensors and three-dimensional tensors. Figure 2 , the edge kernel tensor and Can be two-dimensional, central core tensor It can be three-dimensional. When the plurality of kernel tensors are quantized, the edge kernel tensor can be and Quantize the first tensor to 4 bits wide and convert the central core tensor The first tensor is quantized to 8 bits. The bit width threshold may be preset to 4 bits or 8 bits.

[0118] Figure 4a and Figure 4b A schematic diagram of the core tensor data distribution of an embodiment of the present disclosure is shown.

[0119] The horizontal axis represents the value of the core tensor data, and the interval is [-1, 1]. The vertical axis represents the density of the core tensor data distribution. The upper bound represents the maximum value of the core tensor data, and the lower bound represents the minimum value of the core tensor data. The maximum and minimum values ​​are represented by dotted lines. Taking the core tensor as a two-dimensional matrix as an example, each element in the two-dimensional matrix can be normalized to the interval of [-1, 1], and then the density of each element in the interval can be counted, that is, Figure 4a and Figure 4b The distribution of each element in the matrix can be displayed.

[0120] like Figure 4a and Figure 4b As shown, Figure 4a and Figure 4b There are four TT cores in total. Figure 4a The two figures on the left show the data distribution of the first TT core when the preset bit width is 15 bits and 8 bits. Figure 4a The two figures on the right show the data distribution of the second TT core when the preset bit width is 15 bits and 8 bits. Figure 4b The two figures on the left show the data distribution of the third TT core when the preset bit width is 15 bits and 8 bits. Figure 4bThe two figures on the right show the data distribution of the fourth TT core when the preset bit width is 15 bits and 8 bits. Among them, the first TT core and the fourth TT core are edge TT cores, and the second TT core and the third TT core are central TT cores. Figure 4a and Figure 4b It can be seen that the edge TT core has a flatter data distribution than the central TT core, and the preset bit width of 15bit or 8bit has little effect on the data distribution of the TT core. This phenomenon is also called Core Distribution Shift (CDS), which is the first discovery of the inventor. Based on the data distribution shift phenomenon, different preset bit widths can be configured according to the dimensions of the multiple core tensors, and the core tensors of different dimensions can be quantized into first tensors of different bit widths, which can further reduce the bit width of the edge core tensor without significantly increasing the loss of precision, so that the neural network can be further compressed, the compression rate of the neural network can be improved, the storage space of the neural network can be further reduced, and the calculation speed of the neural network can be accelerated.

[0121] In one possible implementation, the preset bit width (also called characteristic bit width) may be greater than or equal to the bit width threshold to ensure the stability of the neural network training. For example, the first threshold may be pre-configured to 8 bits. When the preset bit width is greater than or equal to 8 bits, the stability of the neural network training process can be guaranteed; when the preset bit width is less than 8 bits (for example, 4 bits), the stability of the neural network training process cannot be guaranteed. In addition, compared to the preset bit width of 4 bits, the preset bit width is configured as 8 bits to better restore network parameters (for example, the first weight parameter). Configuring the preset bit width to 4 bits will result in excessive data loss during the neural network compression process.

[0122] The image processing method provided by the embodiments of the present disclosure, on one hand, since the Tensor-Train decomposition method and the quantization method are orthogonal, that is, the results of the Tensor-Train decomposition method and the results of the quantization method do not affect each other, the tensor Tensor-Train decomposition is combined with quantization, which can compress the neural network (for example, 2D or 3D convolutional neural network) to the greatest extent and improve the compression rate of the neural network; on the other hand, compared with the current neural network data processing algorithm, the embodiments of the present disclosure can be combined with the current mainstream deep learning acceleration chip, embed the decomposed weights, and convert the multiplication and addition operations of floating-point numbers into low-bit width data. The multiplication and addition operations can greatly reduce the chip area and energy consumption; on the other hand, the embodiments of the present disclosure are universal and can be widely used in models such as convolutional neural networks, artificial neural networks, recurrent neural networks, long short-term memory neural networks, deep reinforcement learning networks, graph neural networks and tensor neural networks in neural network data processing algorithms; on the other hand, the embodiments of the present disclosure also discovered the data distribution offset phenomenon, and configured different preset bit widths for core tensors of different dimensions, which can further reduce the bit width of the edge core tensor and reduce the precision loss; on the other hand, the embodiments of the present disclosure can enhance the stability of the image processing method by configuring the preset bit width to be greater than or equal to the bit width threshold.

[0123] At the application level, the present disclosure can be widely used in the fields of artificial neural networks, long short-term memory neural networks, reinforcement learning, graph neural networks, tensor neural networks, etc. in neural network image, video and 3D point cloud data processing to reduce the storage of neural networks, reduce the consumption of computing resources, reduce the area and energy consumption of neural network chips, and improve the computing speed of neural network chips. In addition, the reduction of computing resources and the improvement of computing speed brought about by the present disclosure will help to realize the real-time operation of neural networks on mobile phones, and will also be conducive to the deployment of cloud computing. Applications related to neural networks, such as image classification, recognition and tracking, are expected to be transferred from server CPU / GPU computing clusters with high computing costs and huge energy consumption to embedded intelligent terminals, so that artificial intelligence can better serve human society.

[0124] In a possible implementation, the image processing method may be executed on an electronic device. The electronic device may include a processor, a memory, and a communication interface. Those skilled in the art may understand that the structure of the electronic device does not constitute a limitation on the electronic device, and may include more or fewer components, or a combination of certain components, or different component arrangements. Among them:

[0125] The processor can be the control center of the electronic device, connecting the various parts of the entire electronic device using various interfaces and lines, and executing various functions of the electronic device and processing data by running or executing software programs and / or modules stored in the memory, and calling data stored in the memory, thereby controlling the electronic device as a whole. The processor can be implemented by a CPU or a graphics processing unit (GPU). The processor may also include a neural network chip.

[0126] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an acquisition module, a tensor decomposition module, a quantization module, a call module and an application required for at least one function (such as neural network training and neural network reasoning), etc.; the data storage area may store data created according to the use of the electronic device, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), programmable read-only memory (Programmable Read-Only Memory, PROM), read-only memory (Read Only Memory, ROM), magnetic memory, flash memory, disk or optical disk. Accordingly, the memory may further include a memory controller to provide the processor with access to the memory.

[0127] Among them, the processor performs the following functions by running the first acquisition module: acquiring a neural network, the neural network is used for image processing, the neural network includes multiple network layers, and the initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating-point data; the processor performs the following functions by running the tensor decomposition module: for any network layer, the first weight parameter of the network layer is tensor decomposed to obtain multiple core tensors, and the multiple core tensors include two-dimensional tensors and / or three-dimensional tensors; the processor performs the following functions by running the quantization module: according to a preset bit width, the multiple core tensors are quantized to obtain a quantized second weight parameter, and the second weight parameter includes fixed-point data of a preset bit width; the processor performs the following functions by running the calling module: calling the neural network to process the target image to obtain the image processing result of the target image.

[0128] Figure 5 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0129] The image processing device can be implemented as all or part of the electronic device through software, hardware or a combination of the two. Figure 5 As shown, in a possible implementation, the image processing device may include an acquisition module, a tensor decomposition module, a quantization module and a calling module.

[0130] The acquisition module 510 is used to acquire a neural network, where the neural network is used for image processing. The neural network includes multiple network layers, and the initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating-point data.

[0131] The tensor decomposition module 520 is used to perform tensor decomposition on the first weight parameter of any network layer to obtain multiple core tensors, where the multiple core tensors include two-dimensional tensors and / or three-dimensional tensors.

[0132] The quantization module 530 is used to quantize the multiple core tensors according to a preset bit width to obtain a quantized second weight parameter, where the second weight parameter includes fixed-point data of a preset bit width.

[0133] The calling module 540 is used to call the neural network to process the target image and obtain the image processing result of the target image.

[0134] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In practical applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0135] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0136] The present disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the steps performed by the electronic device in the above-mentioned various method embodiments. For example, the electronic device may be a terminal or a server.

[0137] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods in the above-mentioned various method embodiments are implemented.

[0138] Figure 6 A block diagram of a terminal 800 for data processing according to an exemplary embodiment is shown. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0139] Reference Figure 6 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0140] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0141] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0143] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0145] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0146] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the device 800 to perform the above method.

[0150] Figure 7 FIG. 1 is a block diagram of a server 1900 for data processing according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Figure 7, the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0151] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output (I / O) interface 1958. The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0152] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0153] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0154] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0155] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0156] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0157] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0159] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0160] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0161] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, It is characterized in that The method comprises: Acquire a neural network, the neural network is used for image processing, the neural network includes a plurality of network layers, and an initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating point data; For any network layer, performing tensor decomposition on a first weight parameter of the network layer to obtain a plurality of core tensors, wherein the plurality of core tensors include a two-dimensional tensor and / or a three-dimensional tensor; quantizing the plurality of core tensors according to a preset bit width to obtain a quantized second weight parameter, wherein the second weight parameter includes fixed-point data of a preset bit width; Calling the neural network to process the target image to obtain an image processing result of the target image; The calling of the neural network to process the target image to obtain the image processing result of the target image includes: For the i-th network layer, convolve the i-1th level processing result output by the i-1th network layer according to the second weight parameter of the i-th network layer to obtain the i-th convolution result, 1≤i≤N, N is the number of network layers of the neural network, wherein the 0th level processing result is the target image; Perform batch normalization on the i-th convolution result to obtain an i-th normalized result; Activate and quantize the i-th standardized result to obtain the i-th processed result, Determining the Nth processing result as the image processing result of the target image; The step of quantizing the plurality of core tensors according to a preset bit width to obtain a quantized second weight parameter includes: quantizing the plurality of core tensors respectively according to a preset bit width to obtain a plurality of quantized first tensors; For the kth first tensor, fuse the kth first tensor with the k-1th second tensor to obtain the kth third tensor, 2≤k≤K, K is the number of the first tensors, wherein the first second tensor is the first first tensor; quantizing the kth third tensor according to a preset bit width to obtain a kth second tensor; The K-th second tensor is determined as the second weight parameter.

2. The method according to claim 1, It is characterized in that The step of performing tensor decomposition on the first weight parameter of the network layer to obtain a plurality of core tensors includes: The first weight parameter is subjected to tensor decomposition by means of tensor row-column TT decomposition to obtain the multiple core tensors.

3. The method according to claim 1, It is characterized in that The batch normalizing the i-th convolution result to obtain the i-th normalized result includes: quantizing the first batch of standardized parameters according to the preset bit width to obtain a quantized second batch of standardized parameters, wherein the second batch of standardized parameters includes fixed-point data of the preset bit width; The i-th convolution result is batch standardized according to the second batch standardization parameters to obtain an i-th standardized result.

4. The method according to claim 1, It is characterized in that The activating and quantifying the i-th standardized result to obtain the i-th processing result includes: Activating the i-th normalization result to obtain an i-th activation result; According to the preset bit width, the i-th activation result is quantized to obtain the i-th processing result, and the i-th processing result includes fixed-point data of the preset bit width.

5. The method according to claim 1, It is characterized in that The preset bit width is configured according to the dimensions of the multiple core tensors, core tensors of different dimensions are quantized into first tensors of different bit widths, and the preset bit width is greater than or equal to a bit width threshold.

6. An image processing device, It is characterized in that The device comprises: An acquisition module, used for acquiring a neural network, the neural network is used for image processing, the neural network includes a plurality of network layers, and an initial first weight parameter of each network layer includes a multi-dimensional tensor and is floating point data; A tensor decomposition module, used for performing tensor decomposition on a first weight parameter of any network layer to obtain a plurality of core tensors, wherein the plurality of core tensors include a two-dimensional tensor and a three-dimensional tensor; A quantization module, configured to quantize the plurality of core tensors according to a preset bit width to obtain a quantized second weight parameter, wherein the second weight parameter includes fixed-point data of a preset bit width; A calling module, used for calling the neural network to process the target image and obtain an image processing result of the target image; Wherein, the calling module is used for: For the i-th network layer, convolve the i-1th level processing result output by the i-1th network layer according to the second weight parameter of the i-th network layer to obtain the i-th convolution result, 1≤i≤N, N is the number of network layers of the neural network, wherein the 0th level processing result is the target image; Perform batch normalization on the i-th convolution result to obtain an i-th normalized result; Activate and quantize the i-th standardized result to obtain the i-th processed result, Determining the Nth processing result as the image processing result of the target image; Wherein, the quantization module is used for: quantizing the plurality of core tensors respectively according to a preset bit width to obtain a plurality of quantized first tensors; For the kth first tensor, fuse the kth first tensor with the k-1th second tensor to obtain the kth third tensor, 2≤k≤K, K is the number of the first tensors, wherein the first second tensor is the first first tensor; quantizing the kth third tensor according to a preset bit width to obtain a kth second tensor; The K-th second tensor is determined as the second weight parameter.

7. An electronic device, It is characterized in that The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 5.

8. A non-volatile computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.