Input preprocessing method and output post-processing method and device of image processing network

Abstract

The application discloses an input preprocessing method and an output postprocessing method and device of an image processing network, which can not only adapt to the requirement of constraining the bit number of the value of the element in the input matrix for reducing the calculation amount of the image processing network, but also can purposefully enhance or compress the signal of the expected area, avoid the precision loss caused by the bit number conversion, and ensure the image quality. The method comprises the following steps: acquiring a first pixel matrix of a first image to be processed, the value of the element in the first pixel matrix is represented by a first format, and the first format occupies more than 8 bits; performing nonlinear transformation on the first pixel matrix to obtain a transformation result matrix; performing quantization processing on the transformation result matrix to obtain a quantization matrix, the value of the element in the quantization matrix is represented by a second format, and the second format occupies less than or equal to 8 bits; and inputting the quantization matrix into the image processing network.

Description

Technical Field

[0001] This application relates to artificial intelligence (AI) technology for image processing, and more particularly to an input preprocessing method and output postprocessing method and apparatus for an image processing network. Background Technology

[0002] Deep learning is being used more and more widely, especially in the field of image processing. To ensure the quality of image processing, neural networks typically use high bit widths (such as int16, float16, or float32), which results in a very large computational load for the neural network, potentially limiting its application on chips or edge devices.

[0003] Related technologies employ quantization to preprocess the image data input to the neural network. Specifically, the 10-bit or 12-bit image data to be input is directly quantized to 8 bits before being fed into the neural network. Internally, the neural network uses 8-bit processing. The neural network outputs 8-bit image data, which is then post-processed, i.e., dequantized to obtain 10-bit or 12-bit image data. This method can reduce the computational load of the neural network, but it also results in a significant loss of image precision, leading to a degradation in image quality. Summary of the Invention

[0004] This application provides an input preprocessing method and an output postprocessing method and apparatus for an image processing network. It can both meet the requirement of constraining the number of bits occupied by the values ​​of elements in the input matrix in order to reduce the computational load of the image processing network, and avoid the accuracy loss caused by bit number transformation based on the enhancement of the characteristics of different regions in the first image block, thus ensuring image quality.

[0005] In a first aspect, this application provides an input preprocessing method for an image processing network, wherein the image processing network is a neural network with image processing capabilities. The method includes: acquiring a first pixel matrix of a first image to be processed, wherein the values ​​of the elements in the first pixel matrix are all represented in a first format, and the first format occupies more than 8 bits; performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix; performing quantization processing on the transformation result matrix to obtain a quantization matrix, wherein the values ​​of the elements in the quantization matrix are all represented in a second format, and the second format occupies less than or equal to 8 bits; and inputting the quantization matrix into the image processing network.

[0006] To suit various application scenarios, a training engine can be used to train a neural network with image processing capabilities. Training data can include pixel matrices and constraints corresponding to various images or image patches. This training data can be stored in a database. The training engine then trains the neural network for image processing, such as image transformation, super-resolution processing, encoding / decoding, and filtering. The neural network trained by the engine can be applied to an image processing device. For example, if the training engine trains a neural network with image transformation capabilities, the image processing device can use this neural network to perform image transformation on the image or image patch to be processed, obtaining the transformed image or image patch. Alternatively, if the training engine trains a neural network with encoding capabilities, the image processing device can use this neural network to encode the image or image patch to be processed, obtaining the bitstream of the image or image patch.

[0007] An image or image patch can be represented using a matrix, where each element in the matrix corresponds to a pixel in the image or image patch. This can also be described as the corresponding element in the matrix corresponding to a pixel in the image patch. For example, an image patch of size 64×64 indicates that its pixels are distributed in 64 rows and 64 columns, and x(i,j) represents the pixel in the i-th row and j-th column of the image patch. The corresponding matrix for this image patch also has 64 rows and 64 columns, with a total of 64×64 elements, and A(i,j) represents the element in the i-th row and j-th column of the matrix. A(i,j) corresponds to x(i,j), and the value of A(i,j) can be the brightness value, chromaticity value, or other relevant values ​​representing the characteristics of the pixel in x(i,j). The elements in the first pixel matrix correspond to the pixels in the first image patch to be processed; that is, the first image patch is represented by the first pixel matrix. The first image patch can be a complete image frame or one of the image patches after an image frame has been divided. The elements in the first pixel matrix are all represented using a first format, which occupies more than 8 bits. For example, the first format occupies 10 or 12 bits.

[0008] Nonlinear transformation can enhance and / or compress signals in different regions of an image block. Therefore, the nonlinear transformation of the first pixel matrix in this application can change the signal intensity of different regions in the original image block. In particular, when a suitable parameter value is selected, the nonlinear transformation corresponding to the parameter value can purposefully enhance or compress the signal of the desired region.

[0009] The values ​​of the elements in the quantization matrix are all represented in a second format, which occupies less than or equal to 8 bits. This second format is a data format supported by the image processing network. It can be seen that the purpose of quantization is to reduce the number of bits occupied by the values ​​of the elements in the matrix so as to meet the computational requirements of the image processing network.

[0010] This application first performs a nonlinear transformation on the pixel matrix of the image block to be processed to obtain a transformation result matrix. The nonlinear transformation can change the signal intensity of different regions in the original image block; therefore, suitable parameter values ​​can be selected to achieve the desired signal enhancement or compression in the desired region. Then, the transformation result matrix is ​​quantized to obtain a quantization matrix, which reduces the number of bits occupied by the elements in the matrix, making the quantization matrix suitable for the computational requirements of the image processing network. The combination of these two steps not only meets the requirement of constraining the number of bits occupied by the elements in the input matrix to reduce the computational load of the image processing network, but also allows for the targeted enhancement or compression of the signal in the desired region, avoiding accuracy loss caused by bit number transformation and ensuring image quality.

[0011] In one possible implementation, the nonlinear transformation is used to compress regions in the first image that are not perceptible to the human eye, and to enhance regions in the first image that are perceptible to the human eye.

[0012] The human eye is generally more sensitive to dark areas in an image than to bright areas. Therefore, even if there are significant changes in a bright area, the human eye may not easily perceive them. Based on this principle of human perception, the principle for selecting the parameter values ​​of the nonlinear transformation can include: compressing the signal of the areas in the image patch that the human eye is not sensitive to (bright areas) and enhancing the signal of the areas in the image patch that the human eye is more sensitive to (dark areas) through nonlinear transformation. That is, different regions on an image patch can be roughly divided into two categories: one is the area that the human eye is more sensitive to (dark areas), and the other is the area that the human eye is not sensitive to (bright areas). The nonlinear transformation corresponding to the selected parameter values ​​can enhance the signal of the area that the human eye is more sensitive to after the nonlinear transformation, and / or compress the signal of the area that the human eye is not sensitive to. Therefore, in this application, one (set) of parameter values ​​can be selected, and after its corresponding nonlinear transformation, the signal of the area (dark area) that is more sensitive to human eye perception in the image block is enhanced and / or the area (bright area) that is less sensitive to human eye perception is compressed. This application can also select multiple (sets) of parameter values, and the nonlinear transformation corresponding to each (set) of parameter values ​​is different for the area to be enhanced and / or compressed on the image block. The image block is subjected to multiple nonlinear transformations, and each nonlinear transformation yields a transformed image block. In this way, the image blocks after different nonlinear transformations can purposefully achieve signal changes in different areas.

[0013] In one possible implementation, the step of performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix includes: performing a first nonlinear transformation on the first pixel matrix to obtain a first transformation result matrix; performing a second nonlinear transformation on the first pixel matrix to obtain a second transformation result matrix; wherein the parameter values ​​corresponding to the first nonlinear transformation and the parameter values ​​corresponding to the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

[0014] Two nonlinear transformations are performed on the first pixel matrix. Since the parameter values ​​corresponding to each nonlinear transformation are different, the areas enhanced or compressed by each linear transformation will also differ. When the first nonlinear transformation cannot enhance all areas that are sensitive to human perception, the second nonlinear transformation can supplement the first nonlinear transformation to ensure that the areas that are sensitive to human perception are enhanced as comprehensively as possible. In one possible implementation, the first pixel matrix can also be subjected to three or more nonlinear transformations.

[0015] Accordingly, the step of quantizing the transformation result matrix to obtain a quantization matrix includes: performing the quantization process on the first transformation result matrix to obtain a first quantization matrix; and performing the quantization process on the second transformation result matrix to obtain a second quantization matrix.

[0016] In one possible implementation, the nonlinear transformation is a gamma transformation, and the parameter value corresponding to the gamma transformation is the gamma value.

[0017] Optionally, the range of the gamma value is (0, 10], for example (0, 4).

[0018] The nonlinear transformation used in this application can be a gamma transformation, with the parameter value being the gamma value. Two or more gamma transformations correspond to different gamma values; that is, when performing multiple gamma transformations on the first pixel matrix, a different gamma value is selected each time.

[0019] Optionally, this application can calculate the transformation result matrix according to the following formula (1):

[0020] X′=X 1 / gamma (1)

[0021] Where X represents the first pixel matrix, and X' represents the transformation result matrix corresponding to the value of gamma.

[0022] Based on the principle of human eye perception, the principles for selecting gamma values ​​may include: the gamma transformation corresponding to the selected gamma value can enhance the signal in areas where the human eye is more sensitive to perception after the gamma transformation, and / or compress the signal in areas where the human eye is less sensitive to perception.

[0023] In one possible implementation, the nonlinear transformation is an S-curve transformation, and the parameter values ​​corresponding to the S-curve transformation include the values ​​of x0, k, and L.

[0024] The nonlinear transformation used in this application can be an S-curve transformation, whose parameter values ​​include the values ​​of x0, k, and L. Two or more S-curve transformations correspond to different values ​​of x0, k, and / or L.

[0025] Optionally, this application can calculate the transformation result matrix according to the following formula (2):

[0026]

[0027] Where X represents the first pixel matrix, x0 represents the midpoint of the S-curve, k represents the growth rate of the S-curve, L represents the maximum value of the S-curve, and X' represents the transformation result matrix corresponding to the values ​​of x0, k, and L.

[0028] The S-curve parameters include three parameters: x0, k, and L. At least two sets of S-curve parameter values ​​must be different, meaning that some or all of the three parameters differ. For example, in each set of selected S-curve parameter values, only the values ​​of k, x0, or L change; or, in each set of selected S-curve parameter values, the values ​​of k and x0, k and L, or x0 and L change; or, in each set of selected S-curve parameter values, the values ​​of x0, k, and L all change. The principles for selecting S-curve parameter values ​​can include: the S-curve transformation corresponding to the selected values ​​of x0, k, and L can enhance the signal in areas where the human eye is more sensitive to perception, and / or compress the signal in areas where the human eye is less sensitive to perception, after the S-curve transformation.

[0029] Optionally, the quantization matrix can be calculated according to the following formula (3):

[0030]

[0031] Where X' represents the transformation result matrix, Y represents the quantization matrix, round() means rounding to the nearest integer, max() means taking the maximum value, and abs() means taking the absolute value.

[0032] Optionally, this application can combine nonlinear transformation and quantization into one step, that is, simultaneously perform nonlinear transformation and quantization on the first pixel matrix to obtain a quantization matrix. For example, the correspondence between the first pixel matrix and multiple quantization matrices can be obtained in advance using methods such as formulas (1) and (3), or formulas (2) and (3), and then the correspondence between the first pixel matrix and the quantization matrices can be stored in the form of a table. In practical applications, after obtaining the first pixel matrix, the corresponding quantization matrix is ​​obtained according to the lookup table.

[0033] In one possible implementation, before performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix, the method further includes: subtracting the first pixel matrix from a preset matrix to obtain a pixel difference matrix, wherein the elements in the preset matrix have equal values; correspondingly, performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix includes: performing the nonlinear transformation on the pixel difference matrix to obtain the transformation result matrix.

[0034] The values ​​of all elements in the preset matrix are equal, and their values ​​are related to the minimum element value in the first pixel matrix. Subtracting the preset matrix from the first pixel matrix means subtracting the value of the corresponding element in the preset matrix from the element in the first pixel matrix. By subtracting the preset matrix from the first pixel matrix, the area to be enhanced and / or compressed can be changed without changing the parameter values. For example, the original brightness range [0.0, 0.2] is changed to [0.0, 0.585] after gamma transformation corresponding to gamma1. After subtracting 0.1 from each element value, the brightness range [0.1, 0.3] is changed to [0.0, 0.585] after gamma transformation corresponding to gamma1.

[0035] In one possible implementation, after obtaining the first pixel matrix of the first image to be processed, the method further includes: obtaining a first precision compensation matrix based on the first pixel matrix, the first precision compensation matrix being used to compensate for the precision loss caused by quantization processing; performing the quantization processing on the first precision compensation matrix to obtain a first precision compensation quantization matrix, the values ​​of the elements in the first precision compensation quantization matrix being represented using the second format; and inputting the first precision compensation quantization matrix into the image processing network.

[0036] In one possible implementation, after performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix, the method further includes: obtaining a second precision compensation matrix based on the transformation result matrix, the second precision compensation matrix being used to compensate for the precision loss caused by quantization processing; performing the quantization processing on the second precision compensation matrix to obtain a second precision compensation quantization matrix, the values ​​of the elements in the second precision compensation quantization matrix being represented using the second format; and inputting the second precision compensation quantization matrix into the image processing network.

[0037] In one possible implementation, a first second precision compensation matrix is ​​obtained based on a first transformation result matrix; a second second precision compensation matrix is ​​obtained based on a second transformation result matrix; the first second precision compensation matrix is ​​subjected to the quantization process to obtain a first second precision compensation quantization matrix; the second second precision compensation matrix is ​​subjected to the quantization process to obtain a second second precision compensation quantization matrix; and the first second precision compensation quantization matrix and the second second precision compensation quantization matrix are input into the image processing network.

[0038] Since quantization transforms the number of bits occupied by the elements in the matrix from greater than 8 to less than or equal to 8, the range of values ​​is reduced, leading to a loss of precision. Therefore, precision compensation can be performed on the quantization matrix to ensure that image quality is not significantly affected. This application provides two methods to obtain a precision compensation matrix, where the values ​​of the elements are the precision compensation values ​​of the corresponding elements in the quantization matrix.

[0039] The first method is to obtain the first precision compensation matrix based on the first pixel matrix. The first precision compensation matrix is ​​obtained based on the first pixel matrix, which has not undergone nonlinear transformation, and the number of first precision compensation matrices is the same as the number of first pixel matrices.

[0040] Optionally, this application can calculate the first precision compensation matrix according to the following formula (4):

[0041]

[0042] Where X represents the first pixel matrix, Q1 represents the first precision compensation matrix, k represents the pre-set scaling factor, and floor() represents taking the integer value.

[0043] The second method is to obtain the second precision compensation matrix based on the transformation result matrix. The second precision compensation matrix is ​​obtained from the transformation result matrix, which is obtained by performing a nonlinear transformation on the first pixel matrix. The number of second precision compensation matrices is the same as the number of transformation result matrices.

[0044] Optionally, the second precision compensation matrix can be calculated according to the following formula (5):

[0045]

[0046] Where X' represents the transformation result matrix, Q2 represents the second precision compensation matrix, k represents the pre-set scaling factor, and floor() represents taking the integer value.

[0047] After obtaining the precision compensation matrix, in order to adapt to the data format supported by the image processing network, the precision compensation matrix also needs to be quantized to obtain the precision compensation quantization matrix. The values ​​of the elements in the precision compensation quantization matrix are all represented in the second format mentioned above. For example, the corresponding precision compensation quantization matrix can be calculated from the first precision compensation matrix or the second precision compensation matrix using formula (3). In this case, X in formula (3) represents the first precision compensation matrix or the second precision compensation matrix, and Y represents the first precision compensation quantization matrix corresponding to the first precision compensation matrix or the second precision compensation quantization matrix corresponding to the second precision compensation matrix.

[0048] In one possible implementation, before performing the quantization process on the first precision compensation matrix to obtain the first precision compensation quantization matrix, the method further includes: performing the nonlinear transformation on the first precision compensation matrix to obtain the first precision compensation transformation result matrix; correspondingly, performing the quantization process on the first precision compensation matrix to obtain the first precision compensation quantization matrix includes: performing the quantization process on the first precision compensation transformation result matrix to obtain the first precision compensation quantization matrix.

[0049] In one possible implementation, before performing the quantization process on the second precision compensation matrix to obtain the second precision compensation quantization matrix, the method further includes: performing the nonlinear transformation on the second precision compensation matrix to obtain the second precision compensation transformation result matrix; correspondingly, performing the quantization process on the second precision compensation matrix to obtain the second precision compensation quantization matrix includes: performing the quantization process on the second precision compensation transformation result matrix to obtain the second precision compensation quantization matrix.

[0050] This application also allows for the following steps: before quantizing the precision compensation matrix, a nonlinear transformation is performed on the precision compensation matrix to obtain a precision compensation transformation result matrix, and then the precision compensation transformation result matrix is ​​quantized to obtain a precision compensation quantization matrix. The parameter values ​​used for the nonlinear transformation of the precision compensation matrix can be the same as or different from the parameter values ​​used for the nonlinear transformation of the first pixel matrix. The precision compensation matrix can undergo only one nonlinear transformation or multiple nonlinear transformations.

[0051] Secondly, this application provides a post-processing method for the output of an image processing network, wherein the image processing network is a neural network with image processing capabilities. The method includes: obtaining a processing matrix output by the image processing network, wherein the values ​​of the elements in the processing matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8; performing inverse quantization processing on the processing matrix to obtain an inverse quantization matrix, wherein the values ​​of the elements in the inverse quantization matrix are all represented in a first format, and the number of bits occupied by the first format is greater than 8; performing a nonlinear inverse transform on the inverse quantization matrix to obtain an inverse transform result matrix; and obtaining a second pixel matrix of a second image block based on the inverse transform result matrix, wherein the second image block is a processed image block.

[0052] The processing matrix is ​​output by the image processing network, so the values ​​of its elements still use the data format supported by the image processing network. The processing matrix corresponds to the quantization matrix mentioned above. The quantization matrix is ​​the preprocessed matrix input to the image processing network, while the processing matrix is ​​the output of the image processing network before post-processing. In other words, one matrix is ​​the input to the image processing network, and the other is its output.

[0053] Dequantization refers to upsampling the values ​​of elements in a matrix, increasing the number of bits they occupy. For example, if the values ​​of elements in the processing matrix occupy 8 bits, the values ​​of elements in the dequantized matrix after dequantization will occupy 12 bits.

[0054] Optionally, this application can calculate the inverse quantization matrix according to the following formula (6):

[0055]

[0056] Where Y represents the processing matrix, X' represents the dequantization matrix, n represents the bit width of the elements in the dequantization matrix (e.g., 12 bits), round() represents rounding to the nearest integer, max() represents taking the maximum value, and abs() represents taking the absolute value.

[0057] The inverse nonlinear transform corresponds to the nonlinear transform, and the parameter values ​​used are also the same.

[0058] This application, based on the preprocessing of the matrix input image processing network, reverses the matrix output by the image processing network to obtain the second pixel matrix of the processed second image. This can both meet the requirement of constraining the number of bits occupied by the values ​​of the elements in the input matrix in order to reduce the computational load of the image processing network, and can purposefully enhance or compress the signal in the desired region, avoiding the accuracy loss caused by bit number transformation and ensuring image quality.

[0059] In one possible implementation, obtaining the second pixel matrix of the second image block based on the inverse transform result matrix includes: when there is only one inverse transform result matrix, using the inverse transform result matrix as the second pixel matrix; or, when there are at least two inverse transform result matrices, merging the at least two inverse transform result matrices to obtain the second pixel matrix.

[0060] In one possible implementation, merging at least two inverse transform result matrices to obtain the second pixel matrix includes: performing a weighted average of the at least two inverse transform result matrices; or, adding the at least two inverse transform result matrices; or, determining the corresponding element of the second pixel matrix based on a set threshold and the corresponding element in the at least two inverse transform result matrices.

[0061] Optionally, a weighted average can be taken from at least two inverse transformation result matrices, and the calculation formula (9) is shown below:

[0062]

[0063] Where X1, X2, ..., Xn represent one of at least two inverse transform result matrices, X' represents the second pixel matrix, and α1, α2, ..., αn represent the second pixel matrix. n The weights of X1, X2, ..., Xn are represented, and n represents the number of matrices in the inverse transformation result matrix.

[0064] Optionally, at least two inverse transformation result matrices can be added together, as shown in formula (10) below:

[0065] X′=X1+X2+…+Xn(10)

[0066] Where X1, X2, ..., Xn represent one of at least two inverse transformation result matrices, X' represents the second pixel matrix, and n represents the number of inverse transformation result matrices.

[0067] Optionally, the corresponding element of the second pixel matrix can be determined based on a set threshold and the corresponding elements in at least two inverse transform result matrices. For example, if two thresholds T1 and T2 are preset, the second pixel matrix X' can be obtained according to the following formula:

[0068]

[0069] In the above formula, T1 and T2 are two pre-set thresholds, X3(i,j) represents the element in the i-th row and j-th column of the inverse transformation result matrix X3, X4(i,j) represents the element in the i-th row and j-th column of the inverse transformation result matrix X4, and X'(i,j) represents the element in the i-th row and j-th column of the second pixel matrix X'.

[0070] In one possible implementation, the nonlinear inverse transform is a gamma inverse transform, and the parameter value corresponding to the gamma inverse transform is the gamma value.

[0071] Optionally, the range of the gamma value is (0, 10).

[0072] The nonlinear inverse transform can be the gamma inverse transform, using the same gamma value as the gamma transform used in the preprocessing stage. For example, if the gamma value of a single gamma transform is 'a', and the corresponding quantization matrix is ​​A, and the image processing network outputs a processing matrix B from the input quantization matrix A, then the gamma value used in the inverse gamma transform of processing matrix B is also 'a'.

[0073] Optionally, this application can calculate the inverse transformation result matrix according to the following formula (7):

[0074] X = X′ gamma (7)

[0075] Where X' represents the inverse quantization matrix, and X represents the transformation result matrix corresponding to the value of gamma.

[0076] In one possible implementation, the nonlinear inverse transform is an S-curve inverse transform, and the parameter values ​​corresponding to the S-curve inverse transform include the values ​​of x0, k, and L.

[0077] The nonlinear inverse transform can be an inverse S-curve transform, and the S-curve parameter values ​​used are the same as those used when performing the S-curve transform in the preprocessing stage.

[0078] Optionally, this application can calculate the inverse transformation result matrix according to the following formula (8):

[0079]

[0080] Where X' represents the inverse quantization matrix, x0 represents the midpoint of the S-curve, k represents the growth rate of the curve, L represents the maximum value of the curve, and X represents the inverse transformation result matrix corresponding to the values ​​of x0, k, and L.

[0081] Optionally, this application can combine inverse quantization and nonlinear inverse transformation into one step, that is, simultaneously perform inverse quantization and nonlinear inverse transformation on the processing matrix to obtain the inverse transformation result matrix. For example, the correspondence between the processing matrix and the inverse transformation result matrix can be obtained in advance using methods such as formulas (6) and (7), or formulas (6) and (8), and then the correspondence between the processing matrix and the inverse transformation result matrix can be stored in the form of a table. In actual application, after obtaining the processing matrix, the corresponding inverse transformation result matrix can be obtained according to the lookup table.

[0082] In one possible implementation, before obtaining the second pixel matrix of the second image block based on the inverse transform result matrix, the method further includes: obtaining the precision compensation processing matrix output by the image processing network, wherein the values ​​of the elements in the precision compensation processing matrix are all represented using the second format; and performing the inverse quantization processing on the precision compensation processing matrix to obtain a precision compensation inverse quantization matrix, wherein the values ​​of the elements in the precision compensation inverse quantization matrix are all represented using the first format.

[0083] Similarly, since quantization transforms the number of bits occupied by the elements in the matrix from greater than 8 to less than or equal to 8, the range of values ​​is reduced, leading to a loss of precision. Therefore, precision compensation can be considered for the output matrix of the image processing network to ensure that the image quality is not significantly affected. When the input matrix of the image processing network includes a precision-compensated quantization matrix, the output matrix of the image processing network also includes a precision-compensated quantization matrix, and the number of output precision-compensated quantization matrices is the same as the number of input precision-compensated quantization matrices.

[0084] In one possible implementation, after performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes: performing precision compensation on the inverse transform result matrix based on the precision compensation inverse quantization matrix to obtain the inverse transform compensation matrix; correspondingly, obtaining the second pixel matrix of the second image block based on the inverse transform result matrix includes: obtaining the second pixel matrix based on the inverse transform compensation matrix.

[0085] Optionally, when there is one precision compensation inverse quantization matrix, the one precision compensation inverse quantization matrix is ​​added to at least two inverse transform result matrices to obtain at least two inverse transform compensation matrices; or, when there are multiple precision compensation inverse quantization matrices, the multiple precision compensation inverse quantization matrices are added to the corresponding inverse transform result matrices to obtain at least two inverse transform compensation matrices, wherein the multiple precision compensation inverse quantization matrices correspond to the at least two inverse transform result matrices.

[0086] In one possible implementation, after performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes: performing precision compensation on the second pixel matrix based on the precision compensation inverse quantization matrix to obtain a second pixel compensation matrix; wherein the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0087] Optionally, when there is only one precision compensation inverse quantization matrix, the one precision compensation inverse quantization matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix; or, when there are multiple precision compensation inverse quantization matrices, the multiple precision compensation inverse quantization matrices are merged, and the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix.

[0088] This application obtains a precision-compensated inverse quantization matrix after inverse quantization processing, and precision compensation can be achieved using two methods:

[0089] One approach is to perform precision compensation on the inverse transform result matrix using one or more precision-compensated inverse quantization matrices to obtain an inverse transform compensation matrix. When there is only one precision-compensated inverse quantization matrix, it can be added to the inverse transform result matrix to obtain at least two inverse transform compensation matrices. When there are multiple precision-compensated inverse quantization matrices, the corresponding precision-compensated inverse quantization matrices and inverse transform result matrices can be added to obtain the inverse transform compensation matrix based on the correspondence between the multiple precision-compensated inverse quantization matrices and inverse transform result matrices. In this case, the merging operation involves at least two inverse transform compensation matrices.

[0090] Another method is to perform precision compensation on the second pixel matrix based on one or more precision compensation inverse quantization matrices to obtain the second pixel compensation matrix. When there is only one precision compensation inverse quantization matrix, it can be added to the second pixel matrix to obtain the second pixel compensation matrix. When there are multiple precision compensation inverse quantization matrices, any of the above-mentioned merging methods can be used to merge the multiple precision compensation inverse quantization matrices, and then the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix. In this case, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0091] In one possible implementation, after performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes performing the nonlinear inverse transformation on the precision compensation inverse quantization matrix to obtain the precision compensation inverse transformation result matrix.

[0092] In one possible implementation, the inverse transformation result matrix can be precision compensated based on the one or more precision compensation inverse transformation result matrices to obtain an inverse transformation compensation matrix; correspondingly, the second pixel matrix is ​​obtained based on the inverse transformation compensation matrix.

[0093] Optionally, when there is one precision compensation inverse transform result matrix, the one precision compensation inverse transform result matrix is ​​added to the inverse transform result matrix to obtain the at least two inverse transform compensation matrices; or, when there are multiple precision compensation inverse transform result matrices, the multiple precision compensation inverse transform result matrices are added to the corresponding inverse transform result matrices to obtain the at least two inverse transform compensation matrices, wherein the multiple precision compensation inverse transform result matrices correspond to the inverse transform result matrix.

[0094] In one possible implementation, the second pixel matrix can be precision compensated based on the one or more precision compensation inverse transform result matrices to obtain a second pixel compensation matrix; correspondingly, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0095] Optionally, when there is only one precision compensation inverse transformation result matrix, the one precision compensation inverse transformation result matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix; or, when there are multiple precision compensation inverse transformation result matrices, the multiple precision compensation inverse transformation result matrices are merged, and the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix.

[0096] This application can also perform nonlinear inverse transformation on one or more precision compensation inverse quantization matrices after obtaining one or more precision compensation inverse quantization matrices to obtain one or more precision compensation inverse transformation result matrices, and then perform precision compensation based on one or more precision compensation inverse transformation result matrices.

[0097] Similarly, precision compensation can also be achieved using the following two methods based on one or more precision compensation inverse transform result matrices:

[0098] One approach is to add the precision-compensated inverse transform result matrix to the inverse transform result matrix when there is only one such matrix. When there are multiple precision-compensated inverse transform result matrices, the corresponding precision-compensated inverse transform result matrices and inverse transform result matrices can be added to obtain the inverse transform compensation matrix, based on the correspondence between the multiple precision-compensated inverse transform result matrices and at least two inverse transform result matrices. In this case, the merging operation involves at least two inverse transform compensation matrices.

[0099] Another method is that when there is only one precision compensation inverse transform result matrix, it can be added to the second pixel matrix to obtain the second pixel compensation matrix. When there are multiple precision compensation inverse transform result matrices, any of the above merging methods can be used to merge the multiple precision compensation inverse transform result matrices, and then the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix. At this time, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0100] It should be noted that the processing of the precision compensation matrix in the post-processing stage corresponds to the processing of the precision compensation matrix in the pre-processing stage. That is, if the precision compensation matrix is ​​subjected to nonlinear transformation and quantization in the pre-processing stage, then the precision compensation matrix will be subjected to inverse quantization and inverse nonlinear transformation in the post-processing stage. The number of nonlinear transformations and the selected parameter values ​​are the same. If the precision compensation matrix is ​​only quantized in the pre-processing stage, then the precision compensation matrix will only be subjected to inverse quantization in the post-processing stage.

[0101] Thirdly, this application provides a preprocessing apparatus for an image processing network, wherein the image processing network is a neural network with image processing capabilities. The apparatus includes: an acquisition module for acquiring a first pixel matrix of a first image to be processed, wherein the values ​​of the elements in the first pixel matrix are all represented in a first format, and the first format occupies more than 8 bits; a transformation module for performing a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix; a quantization module for performing quantization processing on the transformation result matrix to obtain a quantization matrix, wherein the values ​​of the elements in the quantization matrix are all represented in a second format, and the second format occupies less than or equal to 8 bits; and inputting the quantization matrix into the image processing network.

[0102] In one possible implementation, the nonlinear transformation is a gamma transformation, and the parameter value corresponding to the gamma transformation is the gamma value.

[0103] In one possible implementation, the gamma value ranges from (0, 10).

[0104] In one possible implementation, the nonlinear transformation is an S-curve transformation, and the parameter values ​​corresponding to the S-curve transformation include the values ​​of x0, k, and L.

[0105] In one possible implementation, the transformation module is specifically used to perform a first nonlinear transformation on the first pixel matrix to obtain a first transformation result matrix; and to perform a second nonlinear transformation on the first pixel matrix to obtain a second transformation result matrix; wherein the parameter values ​​corresponding to the first nonlinear transformation and the parameter values ​​corresponding to the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

[0106] In one possible implementation, the quantization module is specifically used to perform the quantization process on the first transformation result matrix to obtain a first quantization matrix; and to perform the quantization process on the second transformation result matrix to obtain a second quantization matrix.

[0107] In one possible implementation, the nonlinear transformation is used to compress regions in the first image that are not perceptible to the human eye, and to enhance regions in the first image that are perceptible to the human eye.

[0108] In one possible implementation, the acquisition module is further configured to subtract the first pixel matrix from a preset matrix to obtain a pixel difference matrix, wherein the elements in the preset matrix have equal values; the transformation module is further configured to perform the nonlinear transformation on the pixel difference matrix to obtain the transformation result matrix.

[0109] In one possible implementation, the acquisition module is further configured to acquire a first precision compensation matrix based on the first pixel matrix, the first precision compensation matrix being used to compensate for the precision loss caused by quantization processing; the quantization module is further configured to perform the quantization processing on the first precision compensation matrix to obtain a first precision compensation quantization matrix, the values ​​of the elements in the first precision compensation quantization matrix being represented using the second format; and input the first precision compensation quantization matrix into the image processing network.

[0110] In one possible implementation, the transformation module is further configured to perform the nonlinear transformation on the first precision compensation matrix to obtain a first precision compensation transformation result matrix; the quantization module is further configured to perform the quantization process on the first precision compensation transformation result matrix to obtain a first precision compensation quantization matrix.

[0111] In one possible implementation, the acquisition module is further configured to acquire a second precision compensation matrix based on the transformation result matrix, the second precision compensation matrix being used to compensate for the precision loss caused by quantization processing; the quantization module is further configured to perform the quantization processing on the second precision compensation matrix to obtain a second precision compensation quantization matrix, the values ​​of the elements in the second precision compensation quantization matrix being represented using the second format; and input the second precision compensation quantization matrix into the image processing network.

[0112] In one possible implementation, the transformation module is further configured to perform the nonlinear transformation on the second precision compensation matrix to obtain a second precision compensation transformation result matrix; the quantization module is further configured to perform the quantization process on the second precision compensation transformation result matrix to obtain a second precision compensation quantization matrix.

[0113] Fourthly, this application provides a post-processing apparatus for an image processing network, wherein the image processing network is a neural network with image processing capabilities. The apparatus includes: an acquisition module for acquiring a processing matrix output by the image processing network, wherein the values ​​of the elements in the processing matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8; an inverse quantization module for performing inverse quantization processing on the processing matrix to obtain an inverse quantization matrix, wherein the values ​​of the elements in the inverse quantization matrix are all represented in a first format, and the number of bits occupied by the first format is greater than 8; an inverse transform module for performing a nonlinear inverse transform on the inverse quantization matrix to obtain an inverse transform result matrix; and a merging module for obtaining a second pixel matrix of a second image block based on the inverse transform result matrix, wherein the second image block is a processed image block.

[0114] In one possible implementation, the merging module is specifically used to use the inverse transformation result matrix as the second pixel matrix when there is only one inverse transformation result matrix; or, when there are at least two inverse transformation result matrices, to merge at least two inverse transformation result matrices to obtain the second pixel matrix.

[0115] In one possible implementation, the merging module is specifically used to perform a weighted average of the at least two inverse transform result matrices; or, to add the at least two inverse transform result matrices; or, to determine the element at the corresponding position of the second pixel matrix based on a set threshold and the element at the corresponding position in the at least two inverse transform result matrices.

[0116] In one possible implementation, the nonlinear inverse transform is a gamma inverse transform, and the parameter value corresponding to the gamma inverse transform is the gamma value.

[0117] In one possible implementation, the gamma value ranges from (0, 10).

[0118] In one possible implementation, the nonlinear inverse transform is an S-curve inverse transform, and the parameter values ​​corresponding to the S-curve inverse transform include the values ​​of x0, k, and L.

[0119] In one possible implementation, the acquisition module is further configured to acquire the precision compensation processing matrix output by the image processing network, wherein the values ​​of the elements in the precision compensation processing matrix are all represented in the second format; the inverse quantization module is further configured to perform the inverse quantization processing on the precision compensation processing matrix to obtain a precision compensation inverse quantization matrix, wherein the values ​​of the elements in the precision compensation inverse quantization matrix are all represented in the first format.

[0120] In one possible implementation, the inverse quantization module is further configured to perform precision compensation on the inverse transform result matrix according to the precision compensation inverse quantization matrix to obtain an inverse transform compensation matrix; the merging module is further configured to obtain the second pixel matrix according to the inverse transform compensation matrix.

[0121] In one possible implementation, the inverse quantization module is further configured to perform precision compensation on the second pixel matrix according to the precision compensation inverse quantization matrix to obtain a second pixel compensation matrix; wherein the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0122] In one possible implementation, the inverse transformation module is further configured to perform the nonlinear inverse transformation on the precision compensation inverse quantization matrix to obtain a precision compensation inverse transformation result matrix.

[0123] In one possible implementation, the inverse transform module is further configured to perform precision compensation on the inverse transform result matrix based on the precision compensation inverse transform result matrix to obtain an inverse transform compensation matrix; the merging module is further configured to obtain the second pixel matrix based on the inverse transform compensation matrix.

[0124] In one possible implementation, the inverse transform module is further configured to perform precision compensation on the second pixel matrix based on the precision compensation inverse transform result matrix to obtain a second pixel compensation matrix; wherein the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0125] Fifthly, this application provides an image processing apparatus, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the apparatus performs the method as described in any one of the first or second aspects above.

[0126] In a sixth aspect, this application provides an image processing apparatus, comprising: one or more processors and a transmission interface; the processors being configured to invoke one or more programs stored in a memory via the transmission interface; and when the one or more programs are executed by the one or more processors, the apparatus performs the method as described in any one of the first to second aspects above.

[0127] In a seventh aspect, this application provides a computer-readable storage medium storing program instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any one of the first to second aspects above.

[0128] Eighthly, this application provides a computer program product comprising program instructions that, when executed by a computer or processor, cause the computer or processor to implement the method described in any one of the first to second aspects above. Attached Figure Description

[0129] Figure 1 An exemplary schematic diagram of the image processing apparatus provided in this application;

[0130] Figure 2 An exemplary schematic diagram of the image processing apparatus provided in this application;

[0131] Figure 3 An exemplary flowchart of the input preprocessing method for the image processing network provided in this application;

[0132] Figure 4 An exemplary flowchart of the output post-processing method of the image processing network provided in this application;

[0133] Figure 5 An exemplary block diagram of an input preprocessing and output postprocessing method for an image processing network is shown;

[0134] Figure 6 An exemplary schematic diagram of the gamma transformation is shown;

[0135] Figure 7 An exemplary block diagram of an input preprocessing and output postprocessing method for an image processing network is shown;

[0136] Figure 8 An exemplary block diagram of an input preprocessing and output postprocessing method for an image processing network is shown;

[0137] Figure 9 An exemplary block diagram of an input preprocessing and output postprocessing method for an image processing network is shown;

[0138] Figure 10 This is a schematic diagram of the structure of a preprocessing device embodiment of the image processing network of this application;

[0139] Figure 11 This is a schematic diagram of the structure of a post-processing device embodiment of the image processing network of this application. Detailed Implementation

[0140] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0141] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0142] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0143] Figure 1 An exemplary schematic diagram of the image processing apparatus provided in this application. (e.g.) Figure 1 As shown, the image processing apparatus 100 is used to implement the embodiments disclosed in this application. In one embodiment, the image processing apparatus 100 may be a decoder, an encoder, or any device, chip, or component with image processing capabilities.

[0144] The image processing device 100 may include: an input port 110 and a receiver unit (Rx) 120 for receiving data; a processor (or logic unit, central processing unit, CPU) 130 for processing data. For example, the processor 130 may be a neural network processor; a transmitter unit (Tx) 140 and an output port 150 for transmitting data; and a memory 160 for storing data. It should be noted that the input port 110 and the receiver unit 120 may represent an antenna (or transceiver port) and a receiver unit with signal processing capabilities, respectively. The input port 110 and the receiver unit 120 may also be integrated modules with data transmission, reception, and processing capabilities.

[0145] The image processing apparatus 100 may further include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to an inlet port 110, a receiving unit 120, a transmitting unit 140, and an outlet port 150, for use as an outlet or inlet for optical or electrical signals.

[0146] Processor 130 can be implemented in hardware and software. Processor 130 can be implemented as one or more processor chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or digital signal processing (DSPs), etc. Processor 130 communicates with input port 110, receiving unit 120, transmitting unit 140, output port 150, and memory 160. Processor 130 includes an image processing module 170 (e.g., a neural network (NN) based image processing module 170). Image processing module 170 is used to implement the embodiments disclosed in this application. For example, image processing module 170 performs, processes, prepares, or provides various encoding, decoding, or image transformation operations. Therefore, image processing module 170 provides a substantial improvement to the functionality of image processing apparatus 100. Alternatively, image processing module 170 can be implemented with instructions stored in memory 160 and executed by processor 130.

[0147] Memory 160 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs when an executable program is selected, and to store instructions and data read during program execution. Memory 160 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0148] Figure 2 An exemplary schematic diagram of the image processing apparatus provided in this application. (e.g.) Figure 2 As shown, the image processing apparatus 200 is used to implement the embodiments disclosed in this application. In one embodiment, the image processing apparatus 200 may be a decoder, an encoder, or any device, chip, or component with image processing capabilities.

[0149] The processor 202 in the image processing apparatus 200 may be a central processing unit. Alternatively, the processor 202 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor, such as the processor 202 shown in the figure, using more than one processor is faster and more efficient.

[0150] The memory 204 in the image processing apparatus 200 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 204. Memory 204 may include code and data 206 accessed by the processor 202 via bus 212. Memory 204 may also include an operating system 208 and an application program 210, which includes at least one program that allows the processor 202 to execute the methods described in this application. For example, application program 210 may include applications 1 to N, and also includes an image processing application that executes the methods described in this application.

[0151] The image processing apparatus 200 may also include one or more output devices, such as a display 218, which may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 218 may be coupled to the processor 202 via a bus 212.

[0152] Although bus 212 in the image processing apparatus 200 is described herein as a single bus, bus 212 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components of the image processing apparatus 200 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, the image processing apparatus 200 can have a wide variety of configurations.

[0153] exist Figure 1 or Figure 2 Based on the illustrated embodiment, to suit the application scenarios of neural networks, the image processing device may further include a training engine. The training engine is used to train a neural network with image processing capabilities. Training data may include pixel matrices corresponding to various images or image patches, constraints, etc. The training data can be stored in a database. The training engine trains a neural network for image processing based on the training data. This image processing includes, for example, image transformation, super-resolution processing, encoding / decoding, filtering, etc. It should be noted that this application does not limit the source of the training data; for example, it may be obtained from the cloud or other sources. The neural network trained by the training engine can be applied in the image processing device.

[0154] It should be noted that the training engine can also be set up in the cloud, where the aforementioned neural network with image processing capabilities is trained. The image processing device downloads and uses this neural network from the cloud. For example, if the training engine trains a neural network with image transformation capabilities, the image processing device downloads this neural network from the cloud and then uses it to perform image transformation processing on the image or image patch to be processed, obtaining the transformed image or image patch. As another example, if the training engine trains a neural network with encoding capabilities, the image processing device downloads this neural network from the cloud and then uses it to encode the image or image patch to be processed, obtaining the bitstream of the image or image patch.

[0155] The following explains the terminology and related technologies of neural networks:

[0156] (1) Neural Network

[0157] A neural network (NN) is a machine learning model. A neural network is a network formed by connecting many individual neural units together; that is, the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from that local receptive field, which can be a region composed of several neural units.

[0158] (2) Deep Neural Networks

[0159] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." Based on their position, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. In deep neural networks, more hidden layers allow the network to better represent complex real-world scenarios.

[0160] (3) Convolutional Neural Network

[0161] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with a trainable filter and an input image or a convolutional feature map.

[0162] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of desired class numbers of output. Therefore, the neural network can include multiple hidden layers, the parameters of which can be pre-trained based on training data relevant to a specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0163] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0164] (4) Recurrent Neural Network

[0165] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains ineffective for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network remembers previous information and applies it to the calculation of the current output. That is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. Theoretically, RNNs can process sequential data of any length. RNNs aim to give machines the ability to remember, much like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0166] Neural networks typically involve very high computational demands. Therefore, the computational load can be reduced by limiting the number of bits in the data processed by the neural network. For example, neural networks typically process data using 8 bits. However, to ensure optimal image processing performance, image data is usually represented int16, float16, or float32 formats. The more bits a pixel value occupies, the more detailed information it provides. Therefore, before inputting image data into the neural network, quantization is performed to reduce the number of bits required, ensuring that the image data format meets the computational needs of the neural network.

[0167] This application provides an input preprocessing method and an output postprocessing method for an image processing network. The image data is preprocessed before being input into the neural network to reduce the number of bits occupied by the image data. After the neural network outputs the processed image data, the image data is postprocessed to restore the number of bits occupied by the image data.

[0168] It should be noted that the input preprocessing method and output postprocessing method of the image processing network provided in this application can be applied to the process of training a neural network using the aforementioned training engine. That is, the training data can be preprocessed using the methods provided in this application before being input into the training engine, and the data output from the training engine can be post-processed using the methods provided in this application. The input preprocessing method and output postprocessing method of the image processing network provided in this application can also be applied to any application scenario of the neural network. That is, the image data to be processed can be preprocessed using the methods provided in this application before being input into the trained neural network, and the processed image data output from the neural network can be post-processed using the methods provided in this application.

[0169] In one possible implementation, the image or image patch can be represented as a matrix, where the elements of the matrix correspond to the pixels in the image or image patch. This can also be described as the corresponding element in the matrix corresponding to the corresponding pixel in the image patch. For example, if the image patch is 64×64, it means the pixel distribution of the image patch is 64 rows and 64 columns, and x(i,j) represents the pixel in the i-th row and j-th column of the image patch. The element distribution of the matrix corresponding to this image patch also includes 64 rows and 64 columns, for a total of 64×64 elements, and A(i,j) represents the element in the i-th row and j-th column of the matrix. A(i,j) corresponds to x(i,j), and the value of A(i,j) can be a value representing the luminance, chrominance, or other relevant characteristics of the pixel, such as the brightness or chrominance value of x(i,j).

[0170] Figure 3 An exemplary flowchart of the input preprocessing method for the image processing network provided in this application. This process 300 can be performed by... Figure 1 or Figure 2 The image processing apparatus shown performs the operation. Process 300 is described as a series of steps or operations; it should be understood that process 300 can be performed in various orders and / or occur simultaneously, and is not limited to... Figure 3 The execution order is shown. Figure 3 As shown, the input preprocessing method of this image processing network includes:

[0171] Step 301: Obtain the first pixel matrix of the first image to be processed.

[0172] An image processing network refers to a trained neural network capable of image processing. Its technical principles and training methods are described above and will not be repeated here. This image processing network supports a second data format, which occupies less than or equal to 8 bits. Generally, image processing networks can process data in formats less than or equal to 8 bits.

[0173] The elements in the first pixel matrix correspond to the pixels in the first image block to be processed; that is, the first image block is represented by the first pixel matrix. The first image block can be a complete image frame or one of the image blocks after the image frame has been divided. The elements in the first pixel matrix are all represented using a first format, which occupies more than 8 bits. For example, the first format occupies 10 or 12 bits.

[0174] It should be noted that this application focuses on the number of bits occupied by the values ​​of each element in the first pixel matrix, and does not specifically limit the meaning of these element values. For example, the value of an element in the first pixel matrix can be the chromaticity value of a pixel at a corresponding position in the first image block, or it can be the luminance value of a pixel at a corresponding position in the first image block.

[0175] Step 302: Perform a nonlinear transformation on the first pixel matrix to obtain the transformation result matrix.

[0176] Optionally, this application may perform a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix.

[0177] Optionally, this application may also perform a first nonlinear transformation on the first pixel matrix to obtain a first transformation result matrix; and perform a second nonlinear transformation on the first pixel matrix to obtain a second transformation result matrix; wherein the parameter values ​​corresponding to the first nonlinear transformation and the parameter values ​​corresponding to the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

[0178] It can be seen that the number of transformation result matrices is the same as the number of nonlinear transformations performed on the first pixel matrix.

[0179] In one possible implementation, the nonlinear transformation is a gamma transformation. Two or more gamma transformations correspond to different gamma values; that is, when performing multiple gamma transformations on the first pixel matrix, a different gamma value is selected each time. Optionally, the range of the gamma value can be (0, 10], for example (0, 4). Generally, the human eye is more sensitive to dark areas in an image and less sensitive to bright areas. Therefore, even if there are significant changes in the image in bright areas, the human eye is not easily aware of them. Based on this principle of human eye perception, the principle for selecting the gamma value can include: compressing the signal of the areas in the image patch that the human eye is less sensitive to (bright areas) and enhancing the signal of the areas in the image patch that the human eye is more sensitive to (dark areas) through gamma transformation. In other words, different regions on the image patch can be roughly divided into two categories: areas that the human eye is more sensitive to and areas that the human eye is less sensitive to. The nonlinear transformation corresponding to the selected parameter value can enhance the signal of the areas that the human eye is more sensitive to after the nonlinear transformation, and / or compress the signal of the areas that the human eye is less sensitive to.

[0180] Optionally, this application can calculate the transformation result matrix according to the following formula (1):

[0181] X′=X 1 / gamma (1)

[0182] Where X represents the first pixel matrix, and X' represents the transformation result matrix corresponding to the value of gamma.

[0183] For example, select two gamma values, gamma1 and gamma2, and calculate them using formula (1) respectively:

[0184] X1′=X 1 / gamma1

[0185] X2′=X 1 / gamma2

[0186] X1' and X2' are two transformation result matrices obtained by performing two gamma transformations on the first pixel matrix X.

[0187] In one possible implementation, the nonlinear transformation is an S-curve transformation, where at least two S-curve transformations correspond to at least two different sets of S-curve parameter values.

[0188] Optionally, this application can calculate the transformation result matrix according to the following formula (2):

[0189]

[0190] Where X represents the first pixel matrix, x0 represents the midpoint of the S-curve, k represents the degree of growth of the curve, L represents the maximum value of the curve, and X' represents the transformation result matrix corresponding to the values ​​of x0, k, and L.

[0191] When performing multiple S-curve transformations on the first pixel matrix, different S-curve parameter values ​​are selected each time. The S-curve parameters include three parameters: x0, k, and L. At least two sets of different S-curve parameter values ​​can refer to differences in some or all of the aforementioned three parameters. For example, in each set of selected S-curve parameter values, only the values ​​of k, x0, or L change; or, in each set of selected S-curve parameter values, the values ​​of k and x0, k and L, or x0 and L change; or, in each set of selected S-curve parameter values, the values ​​of x0, k, and L all change. The principle for selecting S-curve parameter values ​​can include: the S-curve transformation corresponding to the selected values ​​of x0, k, and L can enhance the signal in areas that are more sensitive to human eye perception after the S-curve transformation, and / or compress the signal in areas that are less sensitive to human eye perception.

[0192] In one possible implementation, before performing a nonlinear transformation on the first pixel matrix, the first pixel matrix can be subtracted from a preset matrix to obtain a pixel difference matrix, and then the pixel difference matrix can be subjected to at least two nonlinear transformations to obtain at least two transformation result matrices.

[0193] The values ​​of all elements in the preset matrix are equal, and their values ​​are related to the minimum element value in the first pixel matrix. Subtracting the preset matrix from the first pixel matrix means subtracting the value of the corresponding element in the preset matrix from the element in the first pixel matrix. By subtracting the preset matrix from the first pixel matrix, the area to be enhanced and / or compressed can be changed without changing the parameter values. For example, the original brightness range [0.0, 0.2] is changed to [0.0, 0.585] after gamma transformation corresponding to gamma1. After subtracting 0.1 from each element value, the brightness range [0.1, 0.3] is changed to [0.0, 0.585] after gamma transformation corresponding to gamma1.

[0194] Step 303: Quantize the transformation result matrix to obtain the quantization matrix.

[0195] The values ​​of the elements in the quantization matrix are all represented in a second format, which occupies less than or equal to 8 bits. This second format is a data format supported by the image processing network. It can be seen that the purpose of quantization is to reduce the number of bits occupied by the values ​​of the elements in the matrix so as to meet the computational requirements of the image processing network.

[0196] After obtaining one or more transformation result matrices in step 302, quantization processing can be performed on each transformation result matrix. In this way, the number of quantization matrices can be obtained as many as there are transformation result matrices.

[0197] Optionally, the quantization matrix can be calculated according to the following formula (3):

[0198]

[0199] Where X' represents the transformation result matrix, Y represents the quantization matrix, round() means rounding to the nearest integer, max() means taking the maximum value, and abs() means taking the absolute value.

[0200] For example, the two transformation result matrices X1' and X2' obtained from the above calculations are quantized using formula (3) to obtain:

[0201]

[0202]

[0203] Y1 and Y2 are two quantized matrices obtained by quantizing the transformation result matrices X1' and X2' respectively.

[0204] It should be noted that, in addition to the above-mentioned quantification methods, this application may also use other methods to achieve quantification, and no specific limitations are made in this regard.

[0205] Optionally, this application can combine steps 302 and 303 into one step, that is, simultaneously perform nonlinear transformation and quantization processing on the first pixel matrix to obtain a quantization matrix. For example, the correspondence between the first pixel matrix and multiple quantization matrices can be obtained in advance using methods such as formulas (1) and (3), or formulas (2) and (3), and then the correspondence between the first pixel matrix and the quantization matrices can be stored in the form of a table. In practical applications, after obtaining the first pixel matrix, the corresponding quantization matrix is ​​obtained according to the lookup table.

[0206] Step 304: Input the quantization matrix into the image processing network.

[0207] The values ​​of the elements in the quantization matrix are all represented in the second format, which is a data format supported by the image processing network. The quantization matrix can be directly input into the image processing network.

[0208] In the above steps, the first pixel matrix is ​​first subjected to one or more nonlinear transformations to obtain one or more transformation result matrices, and then the transformation result matrices are quantized to obtain quantization matrices. As described in step 302, based on the parameter value selection principle, the elements in the transformation result matrices correspond to the pixels in the first image block. The transformation result matrix obtained by one nonlinear transformation compresses the signal in the bright area and enhances the signal in the dark area in the image block. The transformation result matrices obtained by multiple nonlinear transformations corresponding to different parameter values ​​focus on enhancing the characteristics of different regions in the first image block. Therefore, the quantization matrices obtained from the transformation result matrices can all be input into the image processing network.

[0209] This application first performs a nonlinear transformation on the pixel matrix corresponding to the image block to be processed to obtain a transformation result matrix. The nonlinear transformation can change the signal intensity of different regions in the original image block; therefore, suitable parameter values ​​can be selected to achieve at least two transformation result matrices, aiming to enhance or compress the signal in the desired region. Then, the transformation result matrix is ​​quantized to obtain a quantization matrix, which reduces the number of bits occupied by the elements in the matrix, making the quantization matrix suitable for the computational requirements of the image processing network. The combination of these two steps not only meets the requirement of constraining the number of bits occupied by the elements in the input matrix to reduce the computational load of the image processing network, but also aims to aim to enhance or compress the signal in the desired region, avoiding accuracy loss caused by bit number transformation and ensuring image quality.

[0210] Since quantization transforms the number of bits occupied by the elements in the matrix from greater than 8 to less than or equal to 8, the range of values ​​is reduced, resulting in a loss of precision. Therefore, precision compensation can be considered for the quantization matrix to ensure that the image quality is not significantly affected.

[0211] This application provides two methods for obtaining a precision compensation matrix, in which the values ​​of the elements are the precision compensation values ​​of the corresponding elements in the quantization matrix.

[0212] The first method involves obtaining the first precision compensation matrix based on the first pixel matrix.

[0213] The first precision compensation matrix is ​​obtained from the first pixel matrix, which has not undergone nonlinear transformation. The number of first precision compensation matrices is the same as the number of first pixel matrices.

[0214] Optionally, this application can calculate the first precision compensation matrix according to the following formula (4):

[0215]

[0216] Where X represents the first pixel matrix, Q1 represents the first precision compensation matrix, k represents the pre-set scaling factor, and floor() represents taking the integer value.

[0217] The second method involves obtaining the second-precision compensation matrix based on the transformation result matrix.

[0218] The second precision compensation matrix is ​​obtained from the transformation result matrix, which is obtained by performing a nonlinear transformation on the first pixel matrix. The number of second precision compensation matrices is the same as the number of transformation result matrices.

[0219] Optionally, the second precision compensation matrix can be calculated according to the following formula (5):

[0220]

[0221] Where X' represents the transformation result matrix, Q2 represents the second precision compensation matrix, k represents the pre-set scaling factor, and floor() represents taking the integer value.

[0222] After obtaining the precision compensation matrix, in order to adapt to the data format supported by the image processing network, the precision compensation matrix also needs to be quantized to obtain the precision compensation quantization matrix. The values ​​of the elements in the precision compensation quantization matrix are all represented in the second format mentioned above. For example, the corresponding precision compensation quantization matrix can be calculated from the first precision compensation matrix or the second precision compensation matrix using formula (3). In this case, X in formula (3) represents the first precision compensation matrix or the second precision compensation matrix, and Y represents the first precision compensation quantization matrix corresponding to the first precision compensation matrix or the second precision compensation quantization matrix corresponding to the second precision compensation matrix.

[0223] Optionally, before quantizing the precision compensation matrix, the aforementioned nonlinear transformation can be performed on the precision compensation matrix to obtain a precision compensation transformation result matrix, and then quantization can be performed on the precision compensation quantization matrix to obtain a precision compensation quantization matrix. The parameter values ​​used for one or more nonlinear transformations of the precision compensation matrix can be equal to or not exactly the same as the parameter values ​​used for the nonlinear transformation of the first pixel matrix. For example, the gamma value used for the gamma transformation can be one of at least two gamma values, indicating that only one gamma transformation is performed on the precision compensation matrix; it can also be both of the at least two gamma values, indicating that at least two gamma transformations are performed on the precision compensation matrix; or it can be one or more values ​​that are completely different from the at least two gamma values. This application does not specifically limit the size or number of parameter values ​​involved in this case. For example, the corresponding precision compensation transformation result matrix can be calculated by formula (1) or formula (2) for the first precision compensation matrix or the second precision compensation matrix. In this case, X in formula (1) or formula (2) represents the first precision compensation matrix or the second precision compensation matrix, and Y represents the first precision compensation transformation result matrix corresponding to the first precision compensation matrix or the second precision compensation transformation result matrix corresponding to the second precision compensation matrix.

[0224] It should be noted that, in addition to the two methods mentioned above, this application may also use other methods to obtain the accuracy compensation matrix, and no specific limitations are imposed on them.

[0225] Figure 4 An exemplary flowchart of the output post-processing method of the image processing network provided in this application. This process 400 can be performed by... Figure 1 or Figure 2 The image processing apparatus shown performs the operation. Process 400 is described as a series of steps or operations; it should be understood that process 400 can be performed in various orders and / or occur simultaneously, and is not limited to... Figure 4 The execution order is shown. Figure 4As shown, the post-processing method for the output of this image processing network includes:

[0226] Step 401: Obtain the processing matrix output by the image processing network.

[0227] The values ​​of all elements in this processing matrix are represented in a second format, which occupies 8 bits or less and is a data format supported by the image processing network. Since the processing matrix is ​​output by the image processing network, the values ​​of its elements still use the data format supported by the image processing network. The processing matrix and... Figure 3 The quantization matrix in the illustrated embodiment corresponds to the matrix that is input to the image processing network after preprocessing, while the processing matrix is ​​the matrix output by the image processing network before post-processing. In other words, the quantization matrix is ​​the input to the image processing network, and the processing matrix is ​​the output of the image processing network.

[0228] Step 402: Perform inverse quantization on the processing matrix to obtain the inverse quantization matrix.

[0229] The values ​​of the elements in the dequantization matrix are all represented using the first format, which occupies more than 8 bits. Dequantization refers to upsampling the values ​​of the elements in the matrix, increasing the number of bits required. For example, if the values ​​of the elements in the processing matrix occupy 8 bits, the values ​​of the elements in the dequantized matrix after dequantization will occupy 12 bits.

[0230] Optionally, this application can calculate the inverse quantization matrix according to the following formula (6):

[0231]

[0232] Where Y represents the processing matrix, X' represents the dequantization matrix, n represents the bit width of the elements in the dequantization matrix (e.g., 12 bits), round() represents rounding to the nearest integer, max() represents taking the maximum value, and abs() represents taking the absolute value.

[0233] For example, by performing inverse quantization on the two processing matrices Y3 and Y4 using formula (6), we obtain:

[0234]

[0235]

[0236] X3' and X4' are two dequantized matrices obtained by dequantizing the processing matrices Y3 and Y4 respectively.

[0237] It should be noted that, in addition to the above-mentioned dequantization processing method, this application may also use other methods to implement dequantization processing, and no specific limitations are made in this regard.

[0238] Step 403: Perform a nonlinear inverse transform on the inverse quantization matrix to obtain the inverse transform result matrix.

[0239] In one possible implementation, the nonlinear transformation can be an inverse gamma transformation, using the same gamma value as that used in the preprocessing stage. For example, if the gamma value of a single gamma transformation is 'a', and the corresponding quantization matrix is ​​A, and the image processing network outputs a processing matrix B from the input quantization matrix A, then the inverse gamma transformation of processing matrix B also uses the value 'a'.

[0240] Optionally, this application can calculate the inverse transformation result matrix according to the following formula (7):

[0241] X = X′ gamma (7)

[0242] Where X' represents one of at least two inverse quantization matrices, and X represents the inverse transformation result matrix corresponding to the value of gamma.

[0243] For example, X3' and X4' are two inverse quantization matrices, where the gamma value of the inverse gamma transform of X3' is gamma1, and the gamma value of the inverse gamma transform of X4' is gamma2. These two gamma values ​​are calculated using formula (7):

[0244] X3=X3′ gamma1

[0245] X4=X4′ gamma2

[0246] X3 and X4 are two inverse transformation result matrices obtained by performing gamma inverse transformation on the inverse quantization matrices X3' and X4', respectively.

[0247] In one possible implementation, the nonlinear transformation can be an inverse S-curve transformation, where the S-curve parameter values ​​used are the same as those used in at least two sets of S-curve parameter values ​​during the preprocessing stage.

[0248] Optionally, this application can calculate the inverse transformation result matrix according to the following formula (8):

[0249]

[0250] Where X' represents one of at least two inverse quantization matrices, x0 represents the midpoint of the S-curve, k represents the growth rate of the curve, L represents the maximum value of the curve, and X represents the inverse transformation result matrix corresponding to the values ​​of x0, k, and L.

[0251] For example, X3' and X4' are two inverse quantization matrices. The S-curve parameter values ​​for the inverse S-curve transformation of X3' include x01, k1, and L1, and the S-curve parameter values ​​for the inverse S-curve transformation of X4' include x02, k2, and L2. The S-curve parameter values ​​are calculated using formula (8) based on these two sets of S-curve parameter values.

[0252]

[0253]

[0254] X3 and X4 are two inverse transformation result matrices obtained by performing gamma inverse transformation on the inverse quantization matrices X3' and X4', respectively.

[0255] Optionally, this application can combine steps 402 and 403 into one step, that is, simultaneously perform inverse quantization and nonlinear inverse transform processing on the processing matrix to obtain the inverse transform result matrix. For example, the correspondence between the processing matrix and the inverse transform result matrix can be obtained in advance using methods such as formula (6) plus formula (7) or formula (6) plus formula (8), and then the correspondence between the processing matrix and the inverse transform result matrix can be stored in the form of a table. In actual application, after obtaining the processing matrix, the corresponding inverse transform result matrix can be obtained according to the lookup table.

[0256] Step 404: Obtain the second pixel matrix of the second image block based on the inverse transformation result matrix.

[0257] The second image block is the processed image block.

[0258] Optionally, a weighted average can be taken from at least two inverse transformation result matrices, and the calculation formula (9) is shown below:

[0259]

[0260] Where X1, X2, ..., Xn represent one of at least two inverse transform result matrices, X' represents the second pixel matrix, and α1, α2, ..., αn represent the second pixel matrix. n The weights of X1, X2, ..., Xn are represented, and n represents the number of matrices in the inverse transformation result matrix.

[0261] For example, two inverse transformation result matrices X3 and X4 are obtained, where the weight of X3 is α and the weight of X4 is β. The second pixel matrix X' can be obtained using formula (9):

[0262]

[0263] Optionally, at least two inverse transformation result matrices can be added together, as shown in formula (10) below:

[0264] X′=X1+X2+…+Xn(10)

[0265] Where X1, X2, ..., Xn represent one of at least two inverse transformation result matrices, X' represents the second pixel matrix, and n represents the number of inverse transformation result matrices.

[0266] For example, after obtaining two inverse transformation result matrices X3 and X4, the second pixel matrix X' can be obtained using formula (10):

[0267] X′=X3+X4

[0268] Optionally, the corresponding element of the second pixel matrix can be determined based on a set threshold and the corresponding elements in at least two inverse transform result matrices. For example, if two thresholds T1 and T2 are preset, the second pixel matrix X' can be obtained according to the following formula:

[0269]

[0270] In the above formula, T1 and T2 are two pre-set thresholds, X3(i,j) represents the element in the i-th row and j-th column of the inverse transformation result matrix X3, X4(i,j) represents the element in the i-th row and j-th column of the inverse transformation result matrix X4, and X'(i,j) represents the element in the i-th row and j-th column of the second pixel matrix X'.

[0271] It should be noted that, in addition to the methods mentioned above for merging at least two inverse transform result matrices, other methods can also be used to merge at least two inverse transform result matrices, and this application does not make any specific limitations on this.

[0272] This application, based on the preprocessing before the matrix input image processing network, reverses the matrix output by the image processing network to obtain the second pixel matrix corresponding to the processed second image. This can both meet the requirement of constraining the number of bits occupied by the values ​​of the elements in the input matrix in order to reduce the computational load of the image processing network, and can purposefully enhance or compress the signal in the desired region, avoiding the accuracy loss caused by bit number transformation and ensuring image quality.

[0273] Similarly, since quantization transforms the number of bits occupied by the elements in the matrix from greater than 8 to less than or equal to 8, the range of values ​​is reduced, resulting in a loss of precision. Therefore, precision compensation can be performed on the matrix of the output image processing network to ensure that the image quality is not significantly affected.

[0274] When the input matrix of the image processing network includes a precision compensation quantization matrix, the output matrix of the image processing network also includes a precision compensation processing matrix, and the number of output precision compensation processing matrices is the same as the number of input precision compensation quantization matrices.

[0275] In one possible implementation, the precision compensation processing matrix output by the image processing network is first obtained, and the values ​​of the elements in this matrix are all represented in a second format. Then, the precision compensation processing matrix is ​​dequantized to obtain a precision compensation dequantization matrix, and the values ​​of the elements in this dequantization matrix are all represented in a first format.

[0276] This application obtains the precision compensation inverse quantization matrix after inverse quantization processing using the above method, and precision compensation can be performed based on this precision compensation inverse quantization matrix. This application can implement precision compensation using the following two methods:

[0277] One approach is to perform precision compensation on the inverse transform result matrix using one or more precision-compensated inverse quantization matrices to obtain an inverse transform compensation matrix. When there is only one precision-compensated inverse quantization matrix, it can be added to the inverse transform result matrix to obtain at least two inverse transform compensation matrices. When there are multiple precision-compensated inverse quantization matrices, the corresponding precision-compensated inverse quantization matrices and inverse transform result matrices can be added to obtain the inverse transform compensation matrix based on the correspondence between the multiple precision-compensated inverse quantization matrices and inverse transform result matrices. In this case, the merging operation involves at least two inverse transform compensation matrices.

[0278] Another method is to perform precision compensation on the second pixel matrix based on one or more precision compensation inverse quantization matrices to obtain the second pixel compensation matrix. When there is only one precision compensation inverse quantization matrix, it can be added to the second pixel matrix to obtain the second pixel compensation matrix. When there are multiple precision compensation inverse quantization matrices, any of the above-mentioned merging methods can be used to merge the multiple precision compensation inverse quantization matrices, and then the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix. In this case, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0279] Optionally, after obtaining the precision compensation inverse quantization matrix, a nonlinear inverse transformation can be performed on the precision compensation inverse quantization matrix to obtain the precision compensation inverse transformation result matrix, and then precision compensation can be performed based on the precision compensation inverse transformation result matrix.

[0280] Similarly, based on the precision compensation inverse transform result matrix, precision compensation can also be achieved using the following two methods:

[0281] One approach is to add the precision-compensated inverse transform result matrix to the inverse transform result matrix when there is only one such matrix. When there are multiple precision-compensated inverse transform result matrices, the corresponding precision-compensated inverse transform result matrices and inverse transform result matrices can be added to obtain the inverse transform compensation matrix, based on the correspondence between the multiple precision-compensated inverse transform result matrices and at least two inverse transform result matrices. In this case, the merging operation involves at least two inverse transform compensation matrices.

[0282] Another method is that when there is only one precision compensation inverse transform result matrix, it can be added to the second pixel matrix to obtain the second pixel compensation matrix. When there are multiple precision compensation inverse transform result matrices, any of the above merging methods can be used to merge the multiple precision compensation inverse transform result matrices, and then the merged matrix is ​​added to the second pixel matrix to obtain the second pixel compensation matrix. At this time, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0283] It should be noted that the processing of the precision compensation matrix in the post-processing stage corresponds to the processing of the precision compensation matrix in the pre-processing stage. That is, if the precision compensation matrix is ​​subjected to nonlinear transformation and quantization in the pre-processing stage, then the precision compensation matrix will be subjected to inverse quantization and inverse nonlinear transformation in the post-processing stage. The number of nonlinear transformations and the selected parameter values ​​are the same. If the precision compensation matrix is ​​only quantized in the pre-processing stage, then the precision compensation matrix will only be subjected to inverse quantization in the post-processing stage.

[0284] The following uses several specific embodiments to illustrate... Figure 3 and Figure 4 The method shown will be described in further detail.

[0285] Example 1

[0286] Figure 5 An exemplary block diagram of an image processing network's input preprocessing and output postprocessing methods is shown, such as... Figure 5 As shown, the object of preprocessing is the first pixel matrix X of the first image block, and the elements in the first pixel matrix X correspond to the pixels in the first image block. The values ​​of the elements in the first pixel matrix X are represented in a first format, for example, the values ​​of the elements in the first pixel matrix occupy 10 bits or 12 bits.

[0287] Input preprocessing:

[0288] Two gamma transformations (gamma transformation 1 and gamma transformation 2) are performed on the first pixel matrix X. The gamma value used in gamma transformation 1 is gamma1, and the gamma value used in gamma transformation 2 is gamma2. The gamma transformation can be performed using formula (1) to obtain:

[0289] X1′=X 1 / gamma1

[0290] X2′=X 1 / gamma2

[0291] The gamma value typically ranges from (0, 10), and a commonly used gamma value in image processing is 2.2. Figure 6 An exemplary schematic diagram of the gamma transformation is shown, such as... Figure 6 As shown, gamma0 = 2.2, and the values ​​of gamma1 and gamma2 can be one greater than 2.2 and the other less than 2.2. For example, gamma1 = 3.0 and gamma2 = 1.6. From the gamma transformation curves corresponding to gamma0, gamma1, and gamma2 respectively, it can be seen that in the lower value range, the slope of the gamma transformation curve corresponding to gamma1 is greater than the slope of the gamma transformation curve corresponding to gamma0. This indicates that within this range, using the gamma transformation corresponding to gamma1 can widen the range of the original values. For example, if the original value range is [0.0, 0.2], the range after the gamma transformation corresponding to gamma1 is [0.0, 0.585]. The range of values ​​after the transformation is [0.0, 0.481]. In the range of higher values, the slope of the gamma transformation curve corresponding to gamma2 is larger than that of the gamma transformation curve corresponding to gamma0. This means that the gamma transformation corresponding to gamma2 can expand the range of the original values ​​in this range. For example, if the range of the original values ​​is [0.5, 1.0], the range of values ​​after the gamma transformation corresponding to gamma1 is [0.648, 1.0], and the range of values ​​after the gamma transformation corresponding to gamma0 is [0.73, 1.0].

[0292] The first pixel matrix X undergoes a first gamma transformation to obtain a transformation result matrix X1', and then undergoes a second gamma transformation to obtain a transformation result matrix X2'.

[0293] It should be noted that, based on the principles of human visual perception, in areas of an image patch where human perception is not obvious, the signal in these areas is compressed through gamma transformation; conversely, in areas of an image patch where human perception is obvious, the signal in these areas is enhanced through gamma transformation. That is, different regions on an image patch can be roughly divided into two categories: regions where human perception is obvious and regions where human perception is not obvious. One or more gamma values ​​can be selected that can enhance the signal in the regions where human perception is obvious after gamma transformation, and / or one or more gamma values ​​can be selected that can compress the signal in the regions where human perception is not obvious after gamma transformation. This application does not specifically limit the number of gamma transformations or the gamma values ​​selected for each transformation.

[0294] After obtaining two transformation result matrices X1' and X2', the transformation result matrix X1' is quantized to obtain quantization matrix Y1, and the transformation result matrix X2' is quantized to obtain quantization matrix Y2. The values ​​of each element in these two quantization matrices are represented in the second format, that is, the number of bits occupied by each element in the quantization matrix is ​​less than or equal to 8. Quantization can be performed using formula (3) to obtain:

[0295]

[0296]

[0297] The quantization matrices Y1 and Y2 are input into the image processing network.

[0298] Post-output processing:

[0299] The image processing network outputs two processing matrices, Y3 and Y4. Processing matrix Y3 corresponds to quantization matrix Y1, meaning the image processing network processes Y1 and outputs Y3. Processing matrix Y4 corresponds to quantization matrix Y2, meaning the image processing network processes Y2 and outputs Y4. Formula (6) can be used to inverse-quantize processing matrices Y3 and Y4:

[0300]

[0301]

[0302] The processing matrix Y3 is dequantized using formula (6) to obtain the dequantized matrix X3', and the processing matrix Y4 is dequantized using formula (6) to obtain the dequantized matrix X4'.

[0303] The inverse gamma transformation is performed using formula (7):

[0304] X3=X3′ gamma1

[0305] X4=X4′ gamma2

[0306] The inverse transformation matrix X3' is obtained by performing gamma inverse transformation using formula (7), and the inverse transformation matrix X4' is obtained by performing gamma inverse transformation using formula (7).

[0307] The inverse transformation result matrices X3 and X4 are merged using the merging method in step 404 to obtain the second pixel matrix X', which corresponds to the processed second image block.

[0308] Example 2

[0309] Figure 7 An exemplary block diagram of an image processing network's input preprocessing and output postprocessing methods is shown, such as... Figure 7 As shown, in addition to using Figure 5 In addition to the input preprocessing and output postprocessing performed by the method in the illustrated embodiment, this embodiment can also calculate the precision compensation matrix based on the first pixel matrix X using formula (4). Then, the precision compensation matrix Q1 is quantized using formula (3) to obtain the precision compensation quantization matrix.

[0310] The image processing network processes the precision compensation quantization matrix Q1' and outputs the precision compensation processing matrix Q2'. Formula (6) is then used to inverse quantize the precision compensation processing matrix Q2' to obtain the precision compensation inverse quantization matrix. The precision compensation inverse quantization matrix Q2 is added to the inverse transformation result matrices X3 and X4 respectively, and then the two added matrices are merged (using formula (10)) to obtain the second pixel matrix X′=(Q2+X3)+(Q2+X4).

[0311] Example 3

[0312] Figure 8 An exemplary block diagram of an image processing network's input preprocessing and output postprocessing methods is shown, such as... Figure 8 As shown, in addition to using Figure 5 In addition to the input preprocessing and output postprocessing performed by the method in the illustrated embodiment, this embodiment can also calculate the precision compensation matrix using formula (5) based on the transformation result matrix X1'. Then, the precision compensation matrix Q1 is quantized using formula (3) to obtain the precision compensation quantization matrix. Based on the transformation result matrix X2', the accuracy compensation matrix is ​​obtained by performing accuracy compensation calculation using formula (5). Then, the precision compensation matrix Q2 is quantized using formula (3) to obtain the precision compensation quantization matrix.

[0313] The image processing network processes the precision compensation quantization matrix Q1' and outputs the precision compensation processing matrix Q3'. Formula (6) is then used to inverse quantize the precision compensation processing matrix Q3' to obtain the precision compensation inverse quantization matrix. After processing the precision compensation quantization matrix Q2', the precision compensation processing matrix Q4' is output. Then, the precision compensation processing matrix Q4' is dequantized using formula (6) to obtain the precision compensation dequantization matrix. Add the precision compensation inverse quantization matrix Q3 to the inverse transformation result matrix X3, and add the precision compensation inverse quantization matrix Q4 to the inverse transformation result matrix X4. Then merge the two added matrices (using formula (10)) to obtain the second pixel matrix X′=((Q3+X3)+(Q4+X4)).

[0314] Example 4

[0315] Figure 9 An exemplary block diagram of an image processing network's input preprocessing and output postprocessing methods is shown, such as... Figure 9 As shown, before performing a nonlinear transformation on the first pixel matrix, the first pixel matrix X is subtracted from the preset matrix T to obtain the pixel difference matrix X” = XT, and then the transformation is performed using... Figure 5 The method in the illustrated embodiment performs input preprocessing and output postprocessing.

[0316] The values ​​of all elements in the preset matrix T are equal. By subtracting the preset matrix T from the first pixel matrix X, the area to be enhanced and / or compressed can be changed without changing the parameter values.

[0317] It should be noted that the above four embodiments are four examples, but they do not constitute a limitation on the processing procedures of the preprocessing and postprocessing stages of this application, and this application does not make any specific limitations in this regard.

[0318] Figure 10 This is a schematic diagram of the structure of the preprocessing device of the image processing network of this application, as shown below. Figure 10 As shown, the device in this embodiment can be Figure 1 or Figure 2 The image processing apparatus shown may include: an acquisition module 1001, a transformation module 1002, and a quantization module 1003.

[0319] The acquisition module 1001 is used to acquire the first pixel matrix of the first image to be processed, wherein the values ​​of the elements in the first pixel matrix are all represented in a first format, and the first format occupies more than 8 bits; the transformation module 1002 is used to perform a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix; the quantization module 1003 is used to perform quantization processing on the transformation result matrix to obtain a quantization matrix, wherein the values ​​of the elements in the quantization matrix are all represented in a second format, and the second format occupies less than or equal to 8 bits; and the quantization matrix is ​​input into the image processing network.

[0320] In one possible implementation, the nonlinear transformation is a gamma transformation, and the parameter value corresponding to the gamma transformation is the gamma value.

[0321] In one possible implementation, the gamma value ranges from (0, 10).

[0322] In one possible implementation, the nonlinear transformation is an S-curve transformation, and the parameter values ​​corresponding to the S-curve transformation include the values ​​of x0, k, and L.

[0323] In one possible implementation, the transformation module 1002 is specifically used to perform a first nonlinear transformation on the first pixel matrix to obtain a first transformation result matrix; and to perform a second nonlinear transformation on the first pixel matrix to obtain a second transformation result matrix; wherein the parameter values ​​corresponding to the first nonlinear transformation and the parameter values ​​corresponding to the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

[0324] In one possible implementation, the quantization module 1003 is specifically used to perform the quantization process on the first transformation result matrix to obtain a first quantization matrix; and to perform the quantization process on the second transformation result matrix to obtain a second quantization matrix.

[0325] In one possible implementation, the nonlinear transformation is used to compress regions in the first image that are not perceptible to the human eye, and to enhance regions in the first image that are perceptible to the human eye.

[0326] In one possible implementation, the acquisition module 1001 is further configured to subtract the first pixel matrix from the preset matrix to obtain a pixel difference matrix, wherein the values ​​of the elements in the preset matrix are equal; the transformation module 1002 is further configured to perform the nonlinear transformation on the pixel difference matrix to obtain the transformation result matrix.

[0327] In one possible implementation, the acquisition module 1001 is further configured to acquire a first precision compensation matrix based on the first pixel matrix, the first precision compensation matrix being used to compensate for the precision loss caused by quantization processing; the quantization module 1003 is further configured to perform the quantization processing on the first precision compensation matrix to obtain a first precision compensation quantization matrix, the values ​​of the elements in the first precision compensation quantization matrix being represented using the second format; and input the first precision compensation quantization matrix into the image processing network.

[0328] In one possible implementation, the transformation module 1002 is further configured to perform the nonlinear transformation on the first precision compensation matrix to obtain a first precision compensation transformation result matrix; the quantization module 1003 is further configured to perform the quantization process on the first precision compensation transformation result matrix to obtain a first precision compensation quantization matrix.

[0329] In one possible implementation, the acquisition module 1001 is further configured to acquire a second precision compensation matrix based on the transformation result matrix, the second precision compensation matrix being used to compensate for the precision loss caused by quantization processing; the quantization module 1003 is further configured to perform the quantization processing on the second precision compensation matrix to obtain a second precision compensation quantization matrix, the values ​​of the elements in the second precision compensation quantization matrix being represented using the second format; and input the second precision compensation quantization matrix into the image processing network.

[0330] In one possible implementation, the transformation module 1002 is further configured to perform the nonlinear transformation on the second precision compensation matrix to obtain a second precision compensation transformation result matrix; the quantization module 1003 is further configured to perform the quantization process on the second precision compensation transformation result matrix to obtain a second precision compensation quantization matrix.

[0331] The apparatus of this embodiment can be used to perform Figure 3 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0332] Figure 11 This is a schematic diagram of the structure of an embodiment of the post-processing device of the image processing network of this application, as shown below. Figure 11 As shown, the device in this embodiment can be Figure 1 or Figure 2 The image processing apparatus shown may include: an acquisition module 1101, an inverse quantization module 1102, an inverse transform module 1103, and a merging module 1104.

[0333] The acquisition module 1101 is used to acquire the processing matrix output by the image processing network, wherein the values ​​of the elements in the processing matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8; the inverse quantization module 1102 is used to perform inverse quantization processing on the processing matrix to obtain an inverse quantization matrix, wherein the values ​​of the elements in the inverse quantization matrix are all represented in a first format, and the number of bits occupied by the first format is greater than 8; the inverse transform module 1103 is used to perform a nonlinear inverse transform on the inverse quantization matrix to obtain an inverse transform result matrix; and the merging module 1104 is used to obtain the second pixel matrix of the second image block based on the inverse transform result matrix, wherein the second image block is the processed image block.

[0334] In one possible implementation, the merging module 1104 is specifically used to use the inverse transformation result matrix as the second pixel matrix when there is only one inverse transformation result matrix; or, when there are at least two inverse transformation result matrices, to merge at least two inverse transformation result matrices to obtain the second pixel matrix.

[0335] In one possible implementation, the merging module 1104 is specifically used to perform a weighted average of the at least two inverse transform result matrices; or, to add the at least two inverse transform result matrices; or, to determine the element at the corresponding position of the second pixel matrix based on a set threshold and the element at the corresponding position in the at least two inverse transform result matrices.

[0336] In one possible implementation, the nonlinear inverse transform is a gamma inverse transform, and the parameter value corresponding to the gamma inverse transform is the gamma value.

[0337] In one possible implementation, the gamma value ranges from (0, 10).

[0338] In one possible implementation, the nonlinear inverse transform is an S-curve inverse transform, and the parameter values ​​corresponding to the S-curve inverse transform include the values ​​of x0, k, and L.

[0339] In one possible implementation, the acquisition module 1101 is further configured to acquire the precision compensation processing matrix output by the image processing network, wherein the values ​​of the elements in the precision compensation processing matrix are all represented in the second format; the inverse quantization module 1102 is further configured to perform the inverse quantization processing on the precision compensation processing matrix to obtain a precision compensation inverse quantization matrix, wherein the values ​​of the elements in the precision compensation inverse quantization matrix are all represented in the first format.

[0340] In one possible implementation, the inverse quantization module 1102 is further configured to perform precision compensation on the inverse transform result matrix according to the precision compensation inverse quantization matrix to obtain an inverse transform compensation matrix; the merging module 1104 is further configured to obtain the second pixel matrix according to the inverse transform compensation matrix.

[0341] In one possible implementation, the inverse quantization module 1102 is further configured to perform precision compensation on the second pixel matrix according to the precision compensation inverse quantization matrix to obtain a second pixel compensation matrix; wherein the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0342] In one possible implementation, the inverse transformation module 1103 is further configured to perform the nonlinear inverse transformation on the precision compensation inverse quantization matrix to obtain a precision compensation inverse transformation result matrix.

[0343] In one possible implementation, the inverse transformation module 1103 is further configured to perform precision compensation on the inverse transformation result matrix based on the precision compensation inverse transformation result matrix to obtain an inverse transformation compensation matrix; the merging module 1104 is further configured to obtain the second pixel matrix based on the inverse transformation compensation matrix.

[0344] In one possible implementation, the inverse transformation module 1103 is further configured to perform precision compensation on the second pixel matrix based on the precision compensation inverse transformation result matrix to obtain a second pixel compensation matrix; wherein, the elements in the second pixel compensation matrix correspond to the pixels in the second image block.

[0345] The apparatus of this embodiment can be used to perform Figure 4 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0346] It should be noted that, Figure 10 The image processing network shown includes a preprocessing unit and... Figure 11 The post-processing device of the image processing network shown can be applied to the same device, that is, the device can both preprocess the image data input to the image processing network and postprocess the image data output to the image processing network; or it can be applied to different devices, that is, one device is used to preprocess the image data input to the image processing network and another device is used to postprocess the image data output to the image processing network.

[0347] The processor and memory mentioned in the above embodiments can be located on an integrated circuit or chip, and the processor has image processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), and the integrated circuit or chip can be an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0348] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0349] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0350] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0351] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0352] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0353] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0354] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0355] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An input preprocessing method for an image processing network, characterized in that, The image processing network is a neural network with image processing capabilities, and the method includes: Obtain the first pixel matrix of the first image to be processed. The values ​​of the elements in the first pixel matrix are all represented in a first format, and the first format occupies more than 8 bits. A transformation result matrix is ​​obtained by performing a nonlinear transformation on the first pixel matrix. The nonlinear transformation is used to compress areas in the first image that are not sensitive to human eye perception and to enhance areas in the first image that are sensitive to human eye perception. The transformation result matrix is ​​quantized to obtain a quantization matrix. The values ​​of the elements in the quantization matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8. The quantization matrix is ​​input into the image processing network; The step of performing a nonlinear transformation on the first pixel matrix to obtain the transformation result matrix includes: A first nonlinear transformation is performed on the first pixel matrix to obtain a first transformation result matrix; A second nonlinear transformation is performed on the first pixel matrix to obtain a second transformation result matrix; The parameter values ​​corresponding to the first nonlinear transformation and the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

2. The method according to claim 1, characterized in that, The nonlinear transformation is a gamma transformation, and the parameter value corresponding to the gamma transformation is the gamma value.

3. The method according to claim 2, characterized in that, The range of the gamma values ​​is (0, 10).

4. The method according to claim 1, characterized in that, The nonlinear transformation is an S-curve transformation, and the parameter values ​​corresponding to the S-curve transformation include the values ​​of x0, k, and L.

5. The method according to claim 1, characterized in that, The step of quantizing the transformation result matrix to obtain the quantization matrix includes: The first transformation result matrix is ​​subjected to the quantization process to obtain the first quantization matrix; The second transformation result matrix is ​​subjected to the quantization process described above to obtain the second quantization matrix.

6. The method according to any one of claims 1-5, characterized in that, Before performing a nonlinear transformation on the first pixel matrix to obtain the transformation result matrix, the method further includes: Subtracting the first pixel matrix from the preset matrix yields a pixel difference matrix, where the elements in the preset matrix have equal values. Accordingly, the step of performing a nonlinear transformation on the first pixel matrix to obtain the transformation result matrix includes: The nonlinear transformation is performed on the pixel difference matrix to obtain the transformation result matrix.

7. The method according to any one of claims 1-5, characterized in that, After obtaining the first pixel matrix of the first image to be processed, the process further includes: A first precision compensation matrix is ​​obtained based on the first pixel matrix. The first precision compensation matrix is ​​used to compensate for the precision loss caused by quantization processing. The first precision compensation quantization matrix is ​​obtained by performing the quantization process on the first precision compensation matrix, and the values ​​of the elements in the first precision compensation quantization matrix are all represented in the second format. The first precision compensation quantization matrix is ​​input into the image processing network.

8. The method according to claim 7, characterized in that, Before performing the quantization process on the first precision compensation matrix to obtain the first precision compensation quantization matrix, the method further includes: The first precision compensation matrix is ​​subjected to the nonlinear transformation to obtain the first precision compensation transformation result matrix; Accordingly, the step of performing the quantization process on the first precision compensation matrix to obtain the first precision compensation quantization matrix includes: The first precision compensation quantization matrix is ​​obtained by performing the quantization process on the first precision compensation transformation result matrix.

9. The method according to any one of claims 1-5 and 8, characterized in that, After performing a nonlinear transformation on the first pixel matrix to obtain the transformation result matrix, the method further includes: A second precision compensation matrix is ​​obtained based on the transformation result matrix. The second precision compensation matrix is ​​used to compensate for the precision loss caused by quantization processing. The second precision compensation quantization matrix is ​​obtained by performing the quantization process on the second precision compensation matrix, and the values ​​of the elements in the second precision compensation quantization matrix are all represented in the second format. The second precision compensation quantization matrix is ​​input into the image processing network.

10. The method according to claim 9, characterized in that, Before performing the quantization process on the second precision compensation matrix to obtain the second precision compensation quantization matrix, the method further includes: The second precision compensation matrix is ​​subjected to the nonlinear transformation to obtain the second precision compensation transformation result matrix; Accordingly, the step of performing the quantization process on the second precision compensation matrix to obtain the second precision compensation quantization matrix includes: The second precision compensation quantization matrix is ​​obtained by performing the quantization process on the second precision compensation transformation result matrix.

11. A post-processing method for the output of an image processing network, characterized in that, The image processing network is a neural network with image processing capabilities, and the method includes: Obtain the processing matrix output by the image processing network. The values ​​of the elements in the processing matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8. The processing matrix is ​​dequantized to obtain a dequantization matrix. The values ​​of the elements in the dequantization matrix are all represented in a first format, and the first format occupies more than 8 bits. Perform a nonlinear inverse transform on the inverse quantization matrix to obtain the inverse transform result matrix; The second pixel matrix of the second image block is obtained from the inverse transformation result matrix, and the second image block is the processed image block.

12. The method according to claim 11, characterized in that, The step of obtaining the second pixel matrix of the second image block based on the inverse transform result matrix includes: When the inverse transformation result matrix is ​​a single matrix, the inverse transformation result matrix is ​​used as the second pixel matrix; or, When the inverse transformation result matrix includes at least two matrices, the at least two inverse transformation result matrices are merged to obtain the second pixel matrix.

13. The method according to claim 12, characterized in that, The step of merging at least two of the inverse transform result matrices to obtain the second pixel matrix includes: Take a weighted average of the at least two inverse transformation result matrices; or, Add the at least two of the inverse transformation result matrices together; or, The element at the corresponding position of the second pixel matrix is ​​determined based on the set threshold and the element at the corresponding position in the at least two inverse transformation result matrices.

14. The method according to any one of claims 11-13, characterized in that, The nonlinear inverse transform is a gamma inverse transform, and the parameter value corresponding to the gamma inverse transform is the gamma value.

15. The method according to claim 14, characterized in that, The range of the gamma value is (0, 10).

16. The method according to any one of claims 11-13, characterized in that, The nonlinear inverse transform is an S-shaped curve inverse transform, and the parameter values ​​corresponding to the S-shaped curve inverse transform include the values ​​of x0, k, and L.

17. The method according to any one of claims 11-13 and 15, characterized in that, Before obtaining the second pixel matrix of the second image block based on the inverse transform result matrix, the method further includes: Obtain the precision compensation processing matrix output by the image processing network, wherein the values ​​of the elements in the precision compensation processing matrix are all represented in the second format; The precision compensation processing matrix is ​​subjected to the inverse quantization process to obtain the precision compensation inverse quantization matrix, and the values ​​of the elements in the precision compensation inverse quantization matrix are all represented in the first format.

18. The method according to claim 17, characterized in that, After performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes: The inverse transformation result matrix is ​​obtained by performing precision compensation on the precision compensation inverse quantization matrix; Accordingly, obtaining the second pixel matrix of the second image block based on the inverse transform result matrix includes: The second pixel matrix is ​​obtained based on the inverse transformation compensation matrix.

19. The method according to claim 17, characterized in that, After performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes: The second pixel compensation matrix is ​​obtained by performing precision compensation on the second pixel matrix based on the precision compensation inverse quantization matrix. The elements in the second pixel compensation matrix correspond to the pixels in the second image block.

20. The method according to claim 17, characterized in that, After performing the inverse quantization process on the precision compensation processing matrix to obtain the precision compensation inverse quantization matrix, the method further includes: The precision compensation inverse quantization matrix is ​​subjected to the nonlinear inverse transformation to obtain the precision compensation inverse transformation result matrix.

21. The method according to claim 20, characterized in that, After performing the nonlinear inverse transform on the precision compensation inverse quantization matrix to obtain the precision compensation inverse transform result matrix, the method further includes: Based on the precision compensation inverse transform result matrix, the inverse transform compensation matrix is ​​obtained by performing precision compensation on the inverse transform result matrix. The step of obtaining the second pixel matrix of the second image block based on the inverse transform result matrix includes: The second pixel matrix is ​​obtained based on the inverse transformation compensation matrix.

22. The method according to claim 20, characterized in that, After performing the nonlinear inverse transform on the precision compensation inverse quantization matrix to obtain the precision compensation inverse transform result matrix, the method further includes: The second pixel compensation matrix is ​​obtained by performing precision compensation on the second pixel matrix based on the precision compensation inverse transformation result matrix; The elements in the second pixel compensation matrix correspond to the pixels in the second image block.

23. A preprocessing apparatus for an image processing network, characterized in that, The image processing network is a neural network with image processing capabilities, and the device includes: The acquisition module is used to acquire the first pixel matrix of the first image to be processed. The values ​​of the elements in the first pixel matrix are all represented in a first format, and the first format occupies more than 8 bits. The transformation module is used to perform a nonlinear transformation on the first pixel matrix to obtain a transformation result matrix. The nonlinear transformation is used to compress areas in the first image that are not sensitive to human eye perception and to enhance areas in the first image that are sensitive to human eye perception. A quantization module is used to quantize the transformation result matrix to obtain a quantization matrix. The values ​​of the elements in the quantization matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8. The quantization matrix is ​​then input into the image processing network. The transformation module is specifically used to perform a first nonlinear transformation on the first pixel matrix to obtain a first transformation result matrix; and to perform a second nonlinear transformation on the first pixel matrix to obtain a second transformation result matrix; wherein the parameter values ​​corresponding to the first nonlinear transformation and the parameter values ​​corresponding to the second nonlinear transformation are different; the transformation result matrix includes the first transformation result matrix and the second transformation result matrix.

24. The apparatus according to claim 23, characterized in that, The nonlinear transformation is a gamma transformation, and the parameter value corresponding to the gamma transformation is the gamma value.

25. The apparatus according to claim 24, characterized in that, The range of the gamma value is (0, 10).

26. The apparatus according to claim 23, characterized in that, The nonlinear transformation is an S-curve transformation, and the parameter values ​​corresponding to the S-curve transformation include the values ​​of x0, k, and L.

27. The apparatus according to claim 23, characterized in that, The quantization module is specifically used to perform the quantization process on the first transformation result matrix to obtain a first quantization matrix; and to perform the quantization process on the second transformation result matrix to obtain a second quantization matrix.

28. The apparatus according to any one of claims 23-27, characterized in that, The acquisition module is further configured to subtract the first pixel matrix from the preset matrix to obtain a pixel difference matrix, wherein the values ​​of the elements in the preset matrix are equal. The transformation module is further configured to perform the nonlinear transformation on the pixel difference matrix to obtain the transformation result matrix.

29. The apparatus according to any one of claims 23-27, characterized in that, The acquisition module is further configured to acquire a first precision compensation matrix based on the first pixel matrix, wherein the first precision compensation matrix is ​​used to compensate for the precision loss caused by quantization processing. The quantization module is further configured to perform the quantization process on the first precision compensation matrix to obtain a first precision compensation quantization matrix, wherein the values ​​of the elements in the first precision compensation quantization matrix are all represented in the second format; The first precision compensation quantization matrix is ​​input into the image processing network.

30. The apparatus according to claim 29, characterized in that, The transformation module is further configured to perform the nonlinear transformation on the first precision compensation matrix to obtain the first precision compensation transformation result matrix; The quantization module is further configured to perform the quantization process on the first precision compensation transformation result matrix to obtain the first precision compensation quantization matrix.

31. The apparatus according to any one of claims 23-27 and 30, characterized in that, The acquisition module is further configured to acquire a second precision compensation matrix based on the transformation result matrix, the second precision compensation matrix being used to compensate for the precision loss caused by quantization processing; The quantization module is further configured to perform the quantization process on the second precision compensation matrix to obtain a second precision compensation quantization matrix, wherein the values ​​of the elements in the second precision compensation quantization matrix are all represented using the second format; The second precision compensation quantization matrix is ​​input into the image processing network.

32. The apparatus according to claim 31, characterized in that, The transformation module is further configured to perform the nonlinear transformation on the second precision compensation matrix to obtain the second precision compensation transformation result matrix; The quantization module is further configured to perform the quantization process on the second precision compensation transformation result matrix to obtain the second precision compensation quantization matrix.

33. A post-processing apparatus for an image processing network, characterized in that, The image processing network is a neural network with image processing capabilities, and the device includes: The acquisition module is used to acquire the processing matrix output by the image processing network. The values ​​of the elements in the processing matrix are all represented in a second format, and the number of bits occupied by the second format is less than or equal to 8. The dequantization module is used to perform dequantization processing on the processing matrix to obtain a dequantization matrix. The values ​​of the elements in the dequantization matrix are all represented in a first format, and the first format occupies more than 8 bits. The inverse transformation module is used to perform a nonlinear inverse transformation on the inverse quantization matrix to obtain the inverse transformation result matrix. The merging module is used to obtain the second pixel matrix of the second image block based on the inverse transformation result matrix, wherein the second image block is the processed image block.

34. The apparatus according to claim 33, characterized in that, The merging module is specifically used to use the inverse transformation result matrix as the second pixel matrix when there is only one inverse transformation result matrix; or, when there are at least two inverse transformation result matrices, to merge at least two inverse transformation result matrices to obtain the second pixel matrix.

35. The apparatus according to claim 34, characterized in that, The merging module is specifically used to perform a weighted average of the at least two inverse transform result matrices; or, to add the at least two inverse transform result matrices; or, to determine the element at the corresponding position of the second pixel matrix based on a set threshold and the element at the corresponding position in the at least two inverse transform result matrices.

36. The apparatus according to any one of claims 33-35, characterized in that, The nonlinear inverse transform is a gamma inverse transform, and the parameter value corresponding to the gamma inverse transform is the gamma value.

37. The apparatus according to claim 36, characterized in that, The range of the gamma value is (0, 10).

38. The apparatus according to any one of claims 33-35, characterized in that, The nonlinear inverse transform is an S-shaped curve inverse transform, and the parameter values ​​corresponding to the S-shaped curve inverse transform include the values ​​of x0, k, and L.

39. The apparatus according to any one of claims 33-35 and 37, characterized in that, The acquisition module is further configured to acquire the precision compensation processing matrix output by the image processing network, wherein the values ​​of the elements in the precision compensation processing matrix are all represented in the second format; The inverse quantization module is further configured to perform the inverse quantization process on the precision compensation processing matrix to obtain a precision compensation inverse quantization matrix, wherein the values ​​of the elements in the precision compensation inverse quantization matrix are all represented in the first format.

40. The apparatus according to claim 39, characterized in that, The inverse quantization module is further configured to perform precision compensation on the inverse transform result matrix based on the precision compensation inverse quantization matrix to obtain an inverse transform compensation matrix; The merging module is further configured to obtain the second pixel matrix based on the inverse transformation compensation matrix.

41. The apparatus according to claim 39, characterized in that, The inverse quantization module is further configured to perform precision compensation on the second pixel matrix based on the precision compensation inverse quantization matrix to obtain a second pixel compensation matrix; The elements in the second pixel compensation matrix correspond to the pixels in the second image block.

42. The apparatus according to claim 39, characterized in that, The inverse transformation module is also used to perform the nonlinear inverse transformation on the precision compensation inverse quantization matrix to obtain the precision compensation inverse transformation result matrix.

43. The apparatus according to claim 42, characterized in that, The inverse transformation module is further configured to perform precision compensation on the inverse transformation result matrix based on the precision compensation inverse transformation result matrix to obtain an inverse transformation compensation matrix; The merging module is further configured to obtain the second pixel matrix based on the inverse transformation compensation matrix.

44. The apparatus according to claim 42, characterized in that, The inverse transformation module is further configured to perform precision compensation on the second pixel matrix based on the precision compensation inverse transformation result matrix to obtain a second pixel compensation matrix; The elements in the second pixel compensation matrix correspond to the pixels in the second image block.

45. An image processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the apparatus implements the method as described in any one of claims 1-10 or 11-22.

46. ​​A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed on a computer or processor, cause the computer or processor to perform the method of any one of claims 1-10 or 11-22.

47. A computer program product, characterized in that, The computer program product includes program instructions that, when executed by a computer or processor, cause the computer or processor to perform the method of any one of claims 1-10 or 11-22.

Images