Data transformation method and related equipment

Abstract

The embodiment of the invention provides a data conversion method and related equipment. The method comprises the steps that an input image is acquired, the input image comprises N pixel points, and N is a positive integer; carrying out nonlinear transformation on the values of the N pixels to obtain N first pixel values; m second pixel values are obtained according to a quantization model and the N first pixel values, and M is a positive integer; performing inverse transformation corresponding to the nonlinear transformation on the M second pixel values to obtain M third pixel values; and determining an output image according to the M third pixel values. The proposed technique can improve the quality of a quantitative model with non-uniformly distributed data, while the reasoning time of the quantitative model does not actually change.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and more specifically, to a data transformation method and related equipment. Background Art

[0002] In computer science, there are two main types of numbers: integers and floating-point numbers. Floating-point numbers are frequently used in machine learning models. For example, the input data of a machine learning model and the parameters of a machine learning model, such as the weights of a machine learning model and the input data of the model's inner layers, are all floating-point values. However, integer arithmetic runs faster than floating-point arithmetic on hardware. Therefore, a quantization technique is proposed to speed up inference while reducing memory and power consumption on hardware.

[0003] The purpose of quantization is to transform floating-point values ​​into integer values. For example, the input data of a machine learning model can be converted from floating-point values ​​to integer values, and the machine learning model can perform operations with integer values ​​instead of floating-point values. However, the input data in various tasks has its own particularities, which makes quantization difficult. Summary of the invention

[0004] The embodiment of the present application provides a data transformation method and related equipment. The technical solution can improve the quality of data quantization.

[0005] In a first aspect, an embodiment of the present application provides a data transformation method, the method comprising: obtaining an input image, wherein the input image comprises N pixels, where N is a positive integer; performing a nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values; obtaining M second pixel values ​​based on a quantization model and the N first pixel values, where M is a positive integer; performing an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values; and determining an output image based on the M third pixel values.

[0006] The proposed data transformation method is independent of the machine learning model and the quantization format and does not require model changes. The proposed data transformation method can be implemented efficiently. The proposed data transformation of the input and output does not substantially change the inference time of the quantization model. In addition, the proposed data transformation can be implemented efficiently. Therefore, the proposed technique can improve the quality of quantization models with uneven data distribution, while the inference time of the quantization model does not substantially change.

[0007] In a possible design, performing a nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values ​​includes: performing a polynomial transformation on the values ​​of the N pixels to obtain the N first pixel values.

[0008] The polynomial transformation is an efficient nonlinear transformation method. According to the polynomial transformation, the electronic device can perform nonlinear transformation more efficiently. In addition, the polynomial transformation can be performed using the Horner method. The electronic device can perform the polynomial transformation using a fused multiply-add (FMA) operation, making the nonlinear transformation more efficient and faster.

[0009] In one possible design, the bounded degree of the polynomial transformation is less than 5.

[0010] A bounded degree of less than 5 may provide a good balance between the computation time of the polynomial transform and the quality of the result.

[0011] In a possible design, performing a nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values ​​includes: performing a gamma correction on the values ​​of the N pixels to obtain the N first pixel values.

[0012] In a possible design, part or all of the parameters of the nonlinear transformation are obtained through training, and the data used to train the parameters are used to train the quantization model.

[0013] According to this design, the nonlinear transformation and the nonlinear inverse transformation can be trained together with the quantization model. Therefore, the nonlinear transformation and the nonlinear inverse transformation may be more suitable for the quantization model, and high-quality results can be obtained.

[0014] In one possible design, performing the nonlinear transformation on the values ​​of the N pixels to obtain the N first pixel values ​​includes: performing the nonlinear transformation on the values ​​of the N pixels by searching a first transformation table to obtain the N first pixel values.

[0015] Although storing the first transformation table consumes storage space of the electronic device, it is the fastest nonlinear transformation compared to the polynomial transformation and gamma-inverse gamma correction. In one possible design, one of the nonlinear transformation or the nonlinear inverse transformation can be given by a lookup table, and the other can be given by the polynomial transformation / gamma-inverse gamma correction. For example, the forward transformation can be given by a lookup table, and the inverse transformation can be given by a polynomial. Any combination options are possible.

[0016] In a possible design, a bit width of the quantization model is smaller than a bit width of the input image.

[0017] In the quantization task of a model with a bit width smaller than the bit width of the input data, for example, in the 8-bit full integer quantization task of a model processing 12-bit original data, this technology can significantly improve the quality of the quantization model.

[0018] In a second aspect, an embodiment of the present application provides an electronic device, which has the function of implementing the method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the above functions.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable storage medium including instructions. When the instructions are executed on a computer, the computer is caused to execute the method in the first aspect or any possible implementation of the first aspect.

[0020] In a fourth aspect, an electronic device is provided, comprising a processor, a memory, and a communication interface. The processor is connected to the memory and the communication interface. The memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other network elements under the control of the processor. When the processor executes the instructions stored in the memory, the processor executes the method in the first aspect or any possible implementation of the first aspect.

[0021] In a fifth aspect, a chip system is provided, which includes a memory and a processor, the memory being used to store a computer program, the processor being used to call the computer program from the memory and run the computer program, so that a server provided with the chip executes the method in the first aspect or any possible implementation of the first aspect.

[0022] According to a sixth aspect, a computer program product is provided. When the computer program product runs on a server, the server executes the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of an embodiment of a data transformation method;

[0024] Figure 2 The pipeline corresponding to the data transformation method is shown;

[0025] Figure 3 The difference between the typical case of model input data distribution and the transformation results proposed in this application is shown;

[0026] Figure 4 A schematic block diagram of an electronic device provided for one embodiment of the present invention;

[0027] Figure 5 A schematic block diagram of an electronic device provided for one embodiment of the present invention;

[0028] Figure 6 A schematic diagram of a system architecture provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present application is described below in conjunction with the accompanying drawings.

[0030] In artificial intelligence (AI) models, using integer arithmetic can significantly increase speed. But it is not usually easy to get integer values ​​through scaling. The hardware is limited by the number of bits, that is, the bit width of the hardware. For example, an 8-bit signed integer can store 28 (or 256) integer values. If the bit width of the hardware is 8 bits, the hardware can support integer values ​​from –128 to 127. If two floating-point values ​​(0.1 and 0.0001) are transformed into two integer values ​​by scaling, the scaling parameter for transforming these two floating-point values ​​is 10000, and the results of the transformation are 1000 and 1. However, 1000 is greater than 128, and the hardware cannot support this integer value. Therefore, the scaling parameter 10000 cannot be used to transform hardware floating-point values.

[0031] The process of searching for scaling parameters of an AI model and adjusting the parameters to obtain integers is called quantization. The AI ​​models mentioned in this application may include machine learning models, deep learning models, neural network models, deep neural network (DNN) models, etc. Quantization is an effective way to accelerate and compress AI models. Full integer quantization of AI models means that specialized integer arithmetic is used when calculating AI models, so the first and last model layers and model inputs need to be quantized. For current hardware, using different algorithms for different network parts will bring a lot of overhead in the model calculation process, so for calculating some parts that use a larger bit rate, such as 16 or 32 bits (even if only for the first or last layer), the full integer quantization model is much more efficient than the quantized model. However, the need for absolute quantization of all layers usually leads to a decrease in quantization quality. Therefore, due to efficiency reasons, full integer quantization has high commercial value, but the high quality decline of such a quantization model significantly complicates the use of full integer quantization.

[0032] The input data distribution of AI models is often non-uniform, especially in some computer vision tasks. In some deep learning tasks, the distribution of input data and the input distribution to the inner layers are non-uniform, and most of the data values ​​may be concentrated in certain areas. In the case of quantization, when the bit width of the quantized model is smaller than the bit width of the original input data, this fact often poses a serious obstacle to the traditional uniform quantization method of existing AI models. Data in various tasks have their own particularities and may have special distributions that make quantization difficult. For example, the original data in the denoising task is represented using 12 bits, and their distribution is usually concentrated around 0, which significantly reduces the quality of traditional 8-bit quantization of models in such tasks. In addition, for example, the color distribution of night city photos has a peak around 0 and a very long tail. Due to the large quantization error of the input data, as well as the input to the inner layers of the model, the quality of the final quantized model will be significantly degraded. A special type of quantization is full integer quantization - it has high commercial value because it supports the most efficient implementation of quantized networks, but is most affected by the described problems.

[0033] The present application provides a new technique for network quantization that allows reducing the impact of the described problem. More specifically, the present application proposes a new data transformation method for the quantized model to make the distribution of the inputs of the first model layer and other model layers closer to a uniform distribution. This can reduce data quantization errors, and in the result, the inputs of the model layers will be easier to quantize.

[0034] Figure 1 is a flow chart of an embodiment of a data transformation method.

[0035] 101: The electronic device obtains an input image. The input image includes N pixels, where N is a positive integer.

[0036] In some cases, the electronic device may be a computer. For example, the electronic device may be a personal computer, a laptop computer, a mobile phone, a tablet computer, a virtual reality device, etc.

[0037] In some other cases, the electronic device may be a component in a computer. For example, the electronic device may be a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a system on chip (SoC), etc.

[0038] The input image can be a picture, a part of a picture, a frame of a video, a part of a frame, etc.

[0039] 102: The electronic device performs nonlinear transformation on the values ​​of N pixels to obtain N first pixel values.

[0040] The N first pixel values ​​correspond to the N pixels one by one. In other words, the electronic device performs a nonlinear transformation on the value of the nth pixel among the N pixels to obtain the nth first pixel value among the N first pixel points, where n=1...N.

[0041] 103: The electronic device obtains M second pixel values ​​according to the quantization model and the N first pixel values, where M is a positive integer.

[0042] For the model mentioned in step 103, the only limitation is that the model is a quantized model. The quantized model can be a full integer quantized model or any other type of quantized model. The present application does not limit the function of the quantized model or the computer vision task performed using the quantized model. In other words, the quantized model can be any model for computer vision tasks. For example, the quantized model can be a model for image enhancement, a model for image compression, a model for image segmentation, etc.

[0043] The value of M can be determined according to the task of the quantization model. For example, if the quantization model is used to denoise the input image, M can be equal to N; if the quantization model is used to increase the resolution of the input image, M can be greater than N; if the quantization model is used to extract specific elements (such as people, license plates, etc.) in the input image, M can be less than N.

[0044] 104: Perform an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values.

[0045] 105: The electronic device determines an output image according to the M third pixels.

[0046] For example, if the quantization model is used to increase the resolution of the input image, the output image may be a clearer image; if the quantization model is used to extract a specific element, the output image may be an image of the extracted specific element.

[0047] Figure 1 The method shown can be viewed as a pipeline. Figure 2 Shown with Figure 1 The pipeline corresponding to the method shown.

[0048] like Figure 2 As shown, pipeline 200 includes three units: input unit 210 , processing unit 220 , and output unit 230 .

[0049] The input unit 210 is used to obtain an input image.

[0050] The processing unit 220 includes three sub-units: a non-linear transformation sub-unit 221 , a quantization model sub-unit 222 , and a non-linear inverse transformation sub-unit 223 .

[0051] The non-linear transformation subunit 221 is used to perform non-linear transformation on the values ​​of N pixels of the input image to obtain N first pixel values.

[0052] The non-linear transformation subunit 221 is used to perform a non-linear transformation on each value of the N pixels to obtain N first pixel values. The non-linear transformation does not change the numerical type of the input of the non-linear transformation subunit 221. In other words, if the input of the non-linear transformation subunit 221 is a floating point number, the output of the non-linear transformation subunit 221 will also be a floating point number. Therefore, the numerical types of the N first pixel values ​​and the N pixel values ​​are floating points.

[0053] The quantization model subunit 222 is used to process the N first pixel values ​​to obtain M second pixel values.

[0054] The quantization model subunit 222 may first quantize the input data (i.e., N first pixel values), then use the quantization model to process the quantized input data, and finally dequantize the output data of the quantization model to obtain output data (i.e., M second pixel values). Therefore, the work of the quantization model subunit 222 can be summarized as: quantization, quantization model processing, and dequantization (dequantization).

[0055] As described above, the nonlinear transformation subunit 221 does not transform floating point numbers into integers. Therefore, the transformation of the numerical type of the N first pixel values ​​can be performed by the quantization model subunit 222. The quantization model subunit 222 can first transform the N first pixel values ​​from floating point numbers into integers (i.e., quantization). For ease of description, the result of quantization is referred to as N first integer values. The N first integer values ​​correspond one-to-one to the N first pixel values. Each first integer value in the N first integer values ​​is obtained according to the corresponding first pixel value. In other words, the quantization model subunit 222 converts the nth first pixel value in the N first pixel values ​​into the nth first integer value in the N first integer values. The key to this application is how to transform the value of the input image to obtain the input of the quantization model subunit 222 (i.e., the working principle of the nonlinear transformation subunit 221), and how to transform the output of the quantization model subunit 222 to obtain the output image (i.e., the working principle of the nonlinear inverse transformation subunit 223). Therefore, this application does not limit the method for quantizing / dequantizing floating point numbers. In other words, the quantization model subunit 222 may perform full integer quantization or any other type of quantization on the N first pixel values ​​to obtain N first integer values.

[0056] After quantization, the quantization model subunit 222 can use the quantization model to process the N first integer values. As mentioned above, the present application does not limit the function of the quantization mode. The N first integer values ​​are input to the quantization model, and the output of the quantization model is M second integer values.

[0057] After obtaining the M second integer values, the quantization model subunit 222 may convert the M second integer values ​​into M floating point values ​​(i.e., M second pixel values). Similarly, the M second integer values ​​correspond to the M second pixel values ​​one by one. Each second pixel value in the M second pixel values ​​is obtained according to the corresponding second integer value. In other words, the quantization model subunit 222 converts the mth second integer value in the M second integer values ​​into the mth second pixel value in the M second pixel values, where m=1...M.

[0058] The nonlinear inverse transformation subunit 223 is used to perform an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values. Similarly, the inverse transformation performed by the nonlinear transformation subunit 223 does not change the numerical type of the M second pixel values.

[0059] The output unit 230 is configured to determine an output image according to the M third pixel values.

[0060] The distribution of machine learning model inputs is often non-uniform, especially in some computer vision tasks. In some machine learning tasks, the distribution of input data and the distribution of input to the inner layer are non-uniform, and most of the data values ​​may be concentrated in certain areas. In the case of quantization, when the bit width of the quantized model is smaller than the bit width of the original input data, this fact often poses a serious obstacle to the traditional uniform quantization methods of existing machine learning models. Data in various tasks have their own particularities and may have special distributions, making quantization difficult. For example, the original data in the denoising task is represented using 12 bits, and their distribution is usually concentrated around 0, which significantly reduces the quality of traditional 8-bit quantization of the model in such tasks. In addition, for example, the color distribution of night city photos has a peak around 0 and a very long tail. Due to the large quantization error of the input data, as well as the input to the inner layer of the model, the quality of the final quantized model will be significantly reduced.

[0061] Figure 1 or Figure 2 The pipeline shown in provides a new technique for transforming the input data of a quantized model, which allows reducing the impact of the described problem. A new data transformation and inverse transformation method (pipeline) for calculating a quantized model makes the input distribution of the first model layer and other model layers closer to a uniform distribution. This can reduce data quantization errors, and in the result, the input of the model layer will be easier to quantize.

[0062] The technical solution of this application uses nonlinear transformation of input and output data to improve the quality of data quantization. For convenience, the technical solution proposed in this application can be called nonlinear data transformation for quantization (NDTQ). Figure 2 The pipeline shown may be referred to as an NDTQ pipeline.

[0063] Figure 3 The difference between the typical case of model input data distribution and the transformation results proposed in this application is shown.

[0064] Figure 3 The left figure in shows the distribution of the original input data with values ​​ranging from 0 to 1. Most of the data is clustered around zero, and only a few pixels have values ​​exceeding half of the dynamic range (greater than 0.5). Therefore, if a quantization range equal to [0,1.0] is chosen, most values ​​will be quantized with a low approximation, otherwise, if the range is reduced to [0,0.5], pixels belonging to this range will be quantized more accurately, but an unacceptably strong saturation of bright pixels will occur: the clipping function (during the quantization process) will chop the data without a chance to recover the lost information. Quantizing inputs to the inner layers of a machine learning model may face the same problem, especially machine learning models used for image-to-image tasks. For this type of tasks, the input image is passed through the model via skip connections. Therefore, the distribution of feature maps of these layers will be close to the distribution of feature maps of the input.

[0065] This application proposes to use transformations of the input and output of machine learning models to better utilize the entire range of data (see Figure 3 The proposed data transformation is automatically adjusted during the training phase and / or inference phase of the quantized model, which makes it possible to obtain and use the best data transformation for each quantized task. Moreover, the proposed data transformation can be implemented efficiently. Therefore, the proposed technique can improve the quality of quantized models for computer vision tasks with uneven data distribution, while the inference time of the quantized model does not actually change.

[0066] The proposed data transformation method is compatible with any quantization technique and any training method, and the proposed data transformation method is independent of the machine learning model and the quantization format and does not require changing the model. The proposed data transformation method can be effectively implemented. The proposed input and output data transformation does not actually change the inference time of the quantization model. In addition, the proposed data transformation method has a great impact in quantization tasks. In quantization tasks of models with a bit width smaller than the input data bit width, for example, in 8-bit full integer quantization tasks of models processing 12-bit raw data, this technology can significantly improve the quality of the quantization model. The proposed technology makes it easier to use effective low-bit full integer quantization.

[0067] The forward transform (i.e., the nonlinear transform performed by the nonlinear transform subunit 221) makes the distribution of the inputs of the first model layer and other model layers closer to uniform distribution, so it becomes easier to quantize the inputs to the model layers using uniform quantization. The inverse transform (i.e., the nonlinear inverse transform performed by the nonlinear inverse transform subunit 223) returns to the desired space of the model output.

[0068] As mentioned above, the key of the present application lies in how the nonlinear transformation subunit 221 and the nonlinear inverse transformation subunit 223 work. The nonlinear transformation and the nonlinear inverse transformation will be described in more detail below.

[0069] In some embodiments, the nonlinear transformation subunit 221 may perform a polynomial transformation on the values ​​of the N pixels of the input image. In other words, the nonlinear transformation performed by the nonlinear transformation subunit 221 is a polynomial transformation.

[0070] Equation 1 shows the standard form of the polynomial transformation.

[0071] P(x)=a0+a1x+a2x 2 +...+a k x k (Formula 1)

[0072] Among them, a0, a1, a2, ... a k is the parameter of the polynomial transformation, k is the bounded degree of the polynomial transformation, and k is a positive integer.

[0073] Corresponding to the present application, the nonlinear transformation subunit 221 may use Formula 2 to obtain the nth first pixel value among the N first pixel values:

[0074] P(x n )=a0+a1x n +a2x n 2 +...+a k x n k(Formula 2)

[0075] Among them, x n is the value of the nth pixel among N pixels, P(x n ) is the nth first pixel value among N first pixel values, a0, a1, a2, ..., a k is the parameter of the polynomial transformation, and k is the bounded degree of the polynomial transformation.

[0076] The bounded degree of the polynomial transform (i.e., k in Formulas 1 and 2) is related to the computation time of the polynomial transform and the quality of the results of the computer vision task of the input image. In general, the larger the value of the bounded degree, the better the quality of the result, but the longer the computation time; the smaller the value of the bounded degree, the shorter the computation time, but the worse the quality of the result. Therefore, the value of the bounded degree can be selected based on a good balance between the computation time of the polynomial transform and the quality of the result, that is, the value of the bounded degree can be less than 5 (i.e., k<5).

[0077] In some embodiments, there is an efficient implementation of a polynomial transform using a fused multiply-add (FMA) operation. The FMA operation is of the form d=a+b×c. Therefore, the electronic device can calculate the value of the polynomial according to Horner's method. According to Horner's method, Formula 1 can be expressed as:

[0078] P(x)=a0+a1x+a2x 2 +...+a k x k =a0+x(a1+x(a2+...+x(a k-1 +a k x)...)) (Formula 3)

[0079] In some embodiments, the nonlinear transformation subunit 221 may perform gamma correction on the values ​​of the N pixels of the input image. In other words, the nonlinear transformation performed by the nonlinear transformation subunit 221 is gamma correction.

[0080] Equation 4 shows the standard form of gamma correction.

[0081]

[0082] Where p, a, and b are the parameters of gamma correction.

[0083] Corresponding to the present application, the nonlinear transformation subunit 221 may use Formula 5 to obtain the nth first pixel value among the N first pixel values:

[0084]

[0085] Among them, x nis the value of the nth pixel among N pixels, Gamma(x n ) is the nth first pixel value among the N first pixel values, and p, a and b are parameters of gamma correction.

[0086] In some embodiments, the nonlinear transformation subunit 221 can perform nonlinear transformation on the values ​​of N pixels by searching a first transformation table to obtain N first pixel values. Table 1 is an example of a first transformation table.

[0087] Table 1

[0088] The value of the pixel Nonlinear transformation results 0. 0. 0.01 0.1 0.1 0.32 0.25 0.5 0.9 0.95

[0089] The nonlinear transformation subunit 221 may find the result of the nonlinear transformation according to Table 1. For example, according to Table 1, if the value of a pixel is 0.25, the nonlinear transformation result of the pixel is 0.5.

[0090] The first conversion table may be determined based on arithmetic conversion such as polynomial conversion and gamma correction.

[0091] The nonlinear inverse transform sub-unit 223 performs a nonlinear inverse transform corresponding to the nonlinear transform performed by the nonlinear transform sub-unit 221 .

[0092] In some embodiments, the nonlinear inverse transformation performed by the nonlinear inverse transformation subunit 223 may be an accurate inverse transformation of the corresponding nonlinear transformation. An accurate inverse transformation means that all parameters used in the nonlinear inverse transformation are the same as the parameters used in the corresponding nonlinear transformation. Taking gamma-inverse gamma correction as an example, the nonlinear transformation subunit 221 performs gamma correction using the following formula:

[0093]

[0094] Wherein, p1, a1 and b1 are parameters of gamma correction. The nonlinear inverse transformation subunit 223 can perform inverse gamma correction using the following formula:

[0095]

[0096] Where p2, a2 ​​and b2 are parameters of inverse gamma correction. If the nonlinear inverse transform is an exact inverse transform, then p1 = p2, a1 = a2, b1 = b2.

[0097] In some embodiments, the nonlinear inverse transform may be an inexact inverse transform of the corresponding nonlinear transform. An inexact inverse transform means that the method used for the nonlinear inverse transform corresponds to the method used for the nonlinear transform, but the parameters of the nonlinear inverse transform may be different from the parameters of the nonlinear transform. For example, if the nonlinear transform subunit 221 performs gamma correction, the nonlinear inverse transform subunit 223 will perform inverse gamma correction, but the parameters of the inverse gamma correction may be different from the parameters of the gamma correction. Taking Formula 6 and Formula 7 as examples, if the nonlinear inverse transform is an inexact inverse transform, then p1≠p2, a1≠a2 or b1≠b2. For polynomial transforms, the bounded degree of the nonlinear transform may be different from the bounded degree of the nonlinear inverse transform. For example, the bounded degree of the nonlinear transform may be 4, while the bounded degree of the inverse transform corresponding to the nonlinear transform may be 3.

[0098] In some embodiments, the inverse transform can be not only an exact or inexact inverse transform of the nonlinear transform, but also absolutely independent of the nonlinear transform. This means that the inverse transform can not only have its own parameters independent of the nonlinear transform, but also have an absolutely independent format. For example, the forward transform can be given by a lookup table, while the inverse transform can be given by a polynomial. Any combination of options is possible.

[0099] It is understandable that the above nonlinear transformation method is only an example of nonlinear transformation, not a limitation. Nonlinear transformation can have other forms. For example, the nonlinear transformation subunit nonlinear t 221 can use an AI model to transform the pixel values ​​of the input image.

[0100] In some embodiments, the parameters of the nonlinear transformation (i.e., a0 to a1 in Formula 1) k ; p, a and b in Formula 4) can be obtained through training. For example, the parameters of the nonlinear transformation can be trained simultaneously with the quantization model. In other words, the training system can train the nonlinear transformation and the quantization model on the training input from the training data repository to determine the parameters of the nonlinear transformation and the parameters of the quantization model. Therefore, during the training phase of the quantization model, the parameters of the nonlinear transformation are variable. The nonlinear transformation in the training phase can be regarded as a trainable data transformation. This can improve the final quality of the quantization model. Once the training is completed, the parameters of the nonlinear transformation are fixed, and the nonlinear transformation with fixed parameters can be regarded as a fixed data transformation. In the inference phase, the fixed data transformation can be used to obtain the input of the quantization model. Accordingly, the parameters of the inverse transformation corresponding to the nonlinear transformation can be trained together with the parameters of the nonlinear transformation and the quantization model.

[0101] In one embodiment, all parameters of the nonlinear transformation can be obtained through training. In another embodiment, some parameters of the nonlinear transformation are obtained through training. For example, the power of the nonlinear transformation (i.e., the parameter p in Formula 4) can be obtained through training, and other parameters of the nonlinear transformation can be preset numbers.

[0102] An example of a computer vision problem is the original Joint Denoising-Demosaicking (JDD) task. For the original JDD task, the input data has a non-uniform distribution and is difficult to quantize (even to 8 bits). In the case of original JDD, values ​​that are bright enough can be quantized with lower precision because noise and color variations in bright areas are invisible to the human eye, while values ​​near 0 (darker areas) can be quantized with higher precision because quantization noise is more noticeable in dark areas.

[0103] In general, the proposed nonlinear transformations can be arbitrary, and their polynomial approximations can be used to improve efficiency in the inference phase. In the case of quantizing the original JDD model, the input to the quantized model is nonlinearly transformed so that the input distribution is wider around 0. Square root calculation (SQRT)-like functions (i.e., gamma-inverse gamma correction) help quantize dark areas more accurately than bright areas. Therefore, in the following experiments, baseline full 8-bit quantization and quantization using non-trainable square root NDTQ, as well as quantization using polynomial trainable NDTQ with approximate SQRT-like functions, are compared. In order to get the most out of NDTQ, it is sufficient to use a 4th-order polynomial for the forward transform and a 3rd-order polynomial for the inverse transform. In addition, keeping the bias parameters in the forward and inverse NDTQ formulas to zero improves quality.

[0104] The proposed NDTQ pipeline (i.e. Figure 2 The pipeline shown in ) uses two examples of original JDD models: AIRAW1 and AIRAW2 models. In these experiments, these models are fully quantized to 8 bits in SNPE format.

[0105] The quantized models are trained by using a straight-through estimator and smooth regularization for the quantized weights and activations. However, any other method can be used for quantization-aware training. The training method should be selected based on the task, and the proposed technique is compatible with any training method. To evaluate the contribution of the proposed technique, all quantized models are trained using the same training protocol.

[0106] At the beginning of training for the baseline quantized model, the baseline quantized model is initialized with a pre-trained full-precision model without NDTQ, and at the beginning of training for the quantized model with NDTQ, the quantized model is initialized with a pre-trained full-precision model with a fixed (square root) non-trainable NDTQ. The pre-trained full-precision models are trained in the same amount of time and with the same training configuration and have the same quality.

[0107] As a quality metric, the PSNR of the quantized model relative to the original full-precision model on RGB images from the test dataset is measured. In addition, example output images of the quantized model are demonstrated.

[0108] Experiments show that the proposed NDTQ technique improves the quality of the original JDD full 8-bit quantization model.

[0109] The AIRAW1 model consists of 24 convolutions. Using this model’s example, we compare the results of baseline full 8-bit quantization with full 8-bit quantization with non-trainable square root NDTQ and trainable polynomial NDTQ.

[0110] It can be observed that using NDTQ improves the quality of the full 8-bit quantization model, in particular the noise level is reduced and details and colors are improved.

[0111] The PSNRs corresponding to different full 8-bit quantization models are shown in Table 2.

[0112] Table 2

[0113] Baseline full 8 bits Full 8-bit with non-trainable square root NDTQ Full 8-bit with polynomially trainable NDTQ 34.43dB 35.25dB 37.01dB

[0114] The AIRAW2 model consists of 30 convolutions. This model solves a more complex JDD task than the AIRAW1 model. The results of the baseline full 8-bit quantization are compared with the results of full 8-bit quantization using polynomial trainable NDTQ.

[0115] The perceptual quality of the quantization model using NDTQ exceeds that of the baseline quantization model, especially the noise level is reduced, and the details and colors are improved. The PSNR corresponding to different full 8-bit quantization models is shown in Table 3.

[0116] Table 3

[0117]

[0118] Figure 4 FIG. 4 is a schematic block diagram of an electronic device 400 provided in accordance with an embodiment of the present invention. Figure 4 As shown, the computing device 400 includes: an input unit 401 , a processing unit 402 , and an output unit 403 .

[0119] The input unit 401 is used to obtain an input image, where the input image includes N pixels, and N is a positive integer.

[0120] The processing unit 402 is configured to perform nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values.

[0121] The processing unit 402 is further configured to obtain M second pixel values ​​according to the quantization model and the N first pixel values, where M is a positive integer.

[0122] The processing unit 402 is further configured to perform an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values.

[0123] The output unit 403 is configured to determine an output image according to the M third pixel values.

[0124] It should be understood that the electronic device 400 according to the embodiment of the present application may correspond to the electronic device in the above embodiment, and the above and other management operations and / or functions of each unit in the electronic device are respectively for implementing the corresponding steps of the above method. For the sake of brevity, no further description is given here.

[0125] like Figure 5 As shown in FIG. 5 , the electronic device 500 may include a transceiver 502, a processor 501 and a memory 503. The memory 503 may be used to store codes, instructions, etc. executed by the processor 502.

[0126] It should be understood that the processor 502 can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method embodiment can be completed by hardware integrated logic circuits in the processor or instructions in software form. The processor can be a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a system on chip (SoC) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any traditional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0127] It is understandable that the memory 503 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, various forms of RAM may be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0128] It should be noted that the memories in the systems and methods described herein include, but are not limited to, these memories and any other suitable types of memories.

[0129] See also Figure 6, an embodiment of the present application provides a system architecture 600. As shown in the system architecture 600, a data acquisition device 660 is used to collect training data. In an embodiment of the present application, the training data includes one or more images (i.e., training samples) and the real results corresponding to the one or more images. The training data can be stored in a database 630. The training device 620 can obtain a quantization model 601, parameters of a nonlinear transformation, and parameters of an inverse nonlinear transformation through training according to the training data maintained in the database 630. For details of the training process, reference can be made to the above embodiments, which will not be repeated here. It should be noted that, in actual applications, the training data maintained in the database 630 are not necessarily all collected by the data acquisition device 660, but may also be received from other devices. In addition, it should be noted that the training device 620 is not necessarily completely based on the training data maintained in the database 630 for training to obtain the target model / rule 601, and training data can also be obtained from the cloud or other places for model training. The above description should not be understood as a limitation on the embodiments of the present application.

[0130] The target model / rule 601 obtained by the training device 620 through training can be applied to different systems or devices, such as Figure 6 The execution device 610 shown. The execution device 610 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) device, a virtual reality (VR) device, a vehicle terminal, etc., or a server, etc. Figure 6 In the embodiment, the I / O interface 612 is configured on the execution device 610 and is used to exchange data with an external device. The user can input data to the I / O interface 612 through the client device 640. In the embodiment of the present application, the input data may include an input image. The input image may be an image collected by the execution device 610 through the data acquisition device 660, or an image in the database 630, or an image from the client device 640.

[0131] The preprocessing module 613 is used to perform preprocessing according to the input data (e.g., input image) received by the I / O interface 612. In the embodiment of the present application, the preprocessing module 613 can be used to implement one or more of the following operations: nonlinear transformation, inverse nonlinear transformation, etc.; and also used to implement other preprocessing operations.

[0132] During the related processing process in which the execution device 610 preprocesses the input data or the computing module 611 of the execution device 610 performs calculations, the execution device 610 can call the data, code, etc. in the data storage system 650 to implement the corresponding processing, and can also store the data, instructions, and the data obtained after the corresponding processing in the data storage system 650.

[0133] Finally, the I / O interface 612 returns the processing result (eg, the image processing result obtained as described above) to the client device 640 and provides the processing result to the user.

[0134] It should be noted that the training device 620 can obtain the quantization model 601 corresponding to different goals according to different training data, which can also be called different tasks. The corresponding quantization model 601 can be used to achieve the above goals or complete the above tasks to provide the user with the desired results.

[0135] exist Figure 6 In the case shown, the user can manually provide input data. Manual provision can be performed by using a screen provided on the I / O interface 612. In another case, the client device 640 can automatically send input data to the I / O interface 612. If the client device 640 is required to automatically send input data, the user can set the corresponding authority for the client device 640. The user can view the result output by the execution device 610 in the client device 640. Specifically, the result can be displayed or presented in the form of sound, action, etc. The client device 640 can also be used as a data acquisition terminal to collect input data input to the I / O interface 612 and output results output by the I / O interface 612, as shown in the figure, the input data and output results are used as new sample data, and the new sample data is stored in the database 630. Of course, the client device 640 may not be collected, and the I / O interface 612 directly stores the input data input to the I / O interface 612 and the output results output by the I / O interface 612 as new sample data in the database 630, as shown in the figure.

[0136] It should be noted that Figure 6 This is only a schematic diagram of the system architecture provided by one embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute a limitation. Figure 6 In the embodiment, the data storage system 650 is an external memory relative to the execution device 610. In another embodiment, the data storage system 650 can also be set in the execution device 610.

[0137] The embodiment of the present application also provides a system chip, the system chip comprising: an input / output interface, at least one processor, at least one memory and a bus. The at least one memory is used to store instructions, and the at least one processor is used to call the instructions of the at least one memory to perform the operations in the method in the above embodiment.

[0138] The embodiment of the present application also provides a computer storage medium, which can store program instructions for executing any of the above methods.

[0139] Optionally, the storage medium may specifically be the memory 503 .

[0140] Those skilled in the art will appreciate that, in conjunction with the various examples described in the embodiments disclosed in the specification, the present application can implement units and algorithm steps through electronic hardware or a combination of computer software and electronic hardware. Whether such functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementations should not be considered to be beyond the scope of the present application.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices and units can refer to the corresponding processes in the aforementioned method embodiments, which will not be described in detail here.

[0142] In the present application, several embodiments are provided, and it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described are merely examples. For example, the unit division is merely a logical function division and may be other divisions in the actual implementation. For example, multiple units or components may be merged or integrated into another system, or some features may be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be implemented through some interfaces. The indirect coupling or communication connection between devices or units may be implemented in electronic, mechanical or other forms.

[0143] The units described as discrete parts may be physically separated or not, and the parts shown as units may be physical units or not, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] When these functions are implemented in the form of software functional units and sold or used as independent products, these functions can be stored in computer-readable storage media. According to this understanding, the technical solution of the present application can be implemented in the form of a software product, either substantially or as part of the prior art or as part of the technical solution. The computer software product is stored in a storage medium, including several instructions for instructing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present application. The above-mentioned storage medium includes any medium that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0146] The above description is only a specific implementation of the present invention and is not intended to limit the scope of protection of this application. Within the technical scope disclosed in this application, changes or substitutions that can be thought of by those skilled in the art should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data transformation method, characterized in that: include: Acquire an input image, wherein the input image includes N pixels, where N is a positive integer; Performing nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values; Obtain M second pixel values ​​according to the quantization model and the N first pixel values, where M is a positive integer; Performing an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values; An output image is determined according to the M third pixel values.

2. The method according to claim 1, characterized in that The performing nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values ​​includes: Perform a polynomial transformation on the values ​​of the N pixels to obtain the N first pixel values.

3. The method according to claim 2, characterized in that The bounded degree of the polynomial transformation is less than 5.

4. The method according to claim 1, characterized in that: The performing nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values ​​includes: Gamma correction is performed on the values ​​of the N pixels to obtain the N first pixel values.

5. The method according to any one of claims 1 to 4, characterized in that Part or all of the parameters of the nonlinear transformation are obtained through training, and the data used to train the parameters are used to train the quantization model.

6. The method according to claim 1, characterized in that The performing nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values ​​includes: The nonlinear transformation is performed on the values ​​of the N pixels by looking up a first transformation table to obtain the N first pixel values.

7. The method according to any one of claims 1 to 6, characterized in that The bit width of the quantization model is smaller than the bit width of the input image.

8. An electronic device, characterized in that: include: An input unit, used to obtain an input image, wherein the input image includes N pixels, where N is a positive integer; A processing unit, configured to perform nonlinear transformation on the values ​​of the N pixels to obtain N first pixel values; The processing unit is further configured to obtain M second pixel values ​​according to the quantization model and the N first pixel values, where M is a positive integer; The processing unit is further configured to perform an inverse transformation corresponding to the nonlinear transformation on the M second pixel values ​​to obtain M third pixel values; An output unit is used to determine an output image according to the M third pixel values.

9. The device according to claim 8, characterized in that The processing unit is specifically used to perform polynomial transformation on the values ​​of the N pixels to obtain the N first pixel values.

10. The device according to claim 8, characterized in that The bounded degree of the polynomial transformation is less than 5.

11. The device according to claim 8, characterized in that The processing unit is specifically used to perform gamma correction on the values ​​of the N pixels to obtain the N first pixel values.

12. The device according to any one of claims 8 to 11, characterized in that Part or all of the parameters of the nonlinear transformation are obtained through training, and the data used to train the parameters are used to train the quantization model.

13. The device according to claim 8, characterized in that The processing unit is specifically configured to perform the nonlinear transformation on the values ​​of the N pixels by searching a first transformation table to obtain the N first pixel values.

14. The device according to any one of claims 8 to 13, characterized in that The bit width of the quantization model is smaller than the bit width of the input image.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions which, when executed on a server, enable the server to perform the method according to any one of claims 1 to 7.

16. A device, characterized in that It comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, so that a server provided with a chip executes the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that When the computer program product is run on a server, the server is enabled to perform the method according to any one of claims 1 to 7.

Images