Image processing method, system, apparatus, electronic device and readable storage medium

Abstract

The present application discloses an image processing method, a system, an apparatus, an electronic device and a readable storage medium. The method comprises: on the basis of at least one padding pixel in an input image, a coefficient in a convolution kernel corresponding one-to-one with the at least one padding pixel and an output offset corresponding to the convolution kernel, calculating to obtain first data, the input image being an input image of a deconvolutional layer in a neural network model; on the basis of at least one non-padding pixel in the input image and a coefficient in the convolution kernel corresponding one-to-one with the at least one non-padding pixel, calculating to obtain second data; and on the basis of the first data and the second data, calculating to obtain a pixel value of an output pixel.

Description

Image processing method, system, device, electronic device and readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202311113036.6, filed on August 30, 2023, entitled “Image processing method, system, device, electronic device and readable storage medium,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application belongs to the technical field of electronic equipment, and specifically relates to an image processing method, system, device, electronic equipment and readable storage medium. Background Art

[0004] At present, for the operation of the deconvolution layer in the convolutional neural network model, the convolutional neural network model is usually first quantized using an asymmetric quantization method to convert the floating-point numbers into fixed-point numbers. This quantization process will change the pixel values ​​of the filled pixels in the input image from 0 to non-zero; then, each pixel point in the input image after quantization is zero-point offset to keep the pixel values ​​of the filled pixels at 0, thereby reducing the amount of computation required for subsequent convolution operations.

[0005] However, due to the high parallelism of operations in the convolutional neural network model and the large bandwidth of the input image, a large number of subtractors and registers need to be set up when performing the above-mentioned zero-point offset processing, resulting in a large hardware area and power consumption.

[0006] Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide an image processing method, system, device, electronic device and readable storage medium, which can solve the problems of large hardware footprint and power consumption.

[0008] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: calculating first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel corresponding one-to-one to at least one padded pixel, and an output bias corresponding to a convolution kernel, wherein the input image is an input image of a deconvolution layer in a neural network model; calculating second data based on at least one non-padded pixel in the input image, and a coefficient in a convolution kernel corresponding one-to-one to at least one non-padded pixel; and calculating a pixel value of an output pixel based on the first data and the second data.

[0009] In a second aspect, an embodiment of the present application provides an image processing system, which includes: a first chip and a second chip that are different from each other, the second chip being a chip dedicated to the calculation of a neural network model; wherein the first chip is used to calculate first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel that corresponds one-to-one to at least one padded pixel, and an output bias corresponding to a convolution kernel, and the input image is the input image of a deconvolution layer in the neural network model; the second chip is used to calculate second data based on at least one non-padded pixel in the input image, and a coefficient in a convolution kernel that corresponds one-to-one to at least one non-padded pixel; the second chip is also used to calculate the pixel value of an output pixel based on the first data and the second data.

[0010] In a third aspect, an embodiment of the present application provides an image processing device, which includes a computing module; the computing module is used to calculate first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel that corresponds one-to-one to at least one padded pixel, and an output bias corresponding to a convolution kernel, wherein the input image is an input image of a deconvolution layer in a neural network model; the computing module is also used to calculate second data based on at least one non-filled pixel in the input image, and a coefficient in a convolution kernel that corresponds one-to-one to at least one non-filled pixel; the computing module is also used to calculate a pixel value of an output pixel based on the first data and the second data.

[0011] In a fourth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0012] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a sixth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0014] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0015] In an embodiment of the present application, first data can be calculated based on at least one padded pixel in the input image, a coefficient in a convolution kernel that corresponds one-to-one to at least one padded pixel, and an output bias corresponding to the convolution kernel, where the input image is the input image of the deconvolution layer in the neural network model; and second data can be calculated based on at least one non-padded pixel in the input image and a coefficient in a convolution kernel that corresponds one-to-one to at least one non-padded pixel; and the pixel value of an output pixel can be calculated based on the first data and the second data. Through this solution, during the operation of the deconvolution layer, the pixel value of the output pixel can be calculated based on the pixels in the input image and the convolution kernel, and the entire operation process no longer requires zero-point offset processing for each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thereby reducing the hardware footprint and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a schematic diagram of the operation principle of the deconvolution layer in the related art;

[0017] FIG2 is a flowchart of an image processing method according to an embodiment of the present application;

[0018] FIG3 is a second flowchart of the image processing method provided in an embodiment of the present application;

[0019] FIG4 is a third flowchart of the image processing method provided in an embodiment of the present application;

[0020] FIG5 is a fourth flowchart of the image processing method provided in an embodiment of the present application;

[0021] FIG6 is a schematic diagram of an image processing method according to an embodiment of the present application;

[0022] FIG7 is a second schematic diagram of the image processing method provided in an embodiment of the present application;

[0023] FIG8 is a third schematic diagram of the image processing method provided in an embodiment of the present application;

[0024] FIG9 is a fourth schematic diagram of the image processing method provided in an embodiment of the present application;

[0025] FIG10 is a fifth schematic diagram of the image processing method provided in an embodiment of the present application;

[0026] FIG11 is a sixth schematic diagram of the image processing method provided in an embodiment of the present application;

[0027] FIG12 is a schematic diagram of an image processing system provided in an embodiment of the present application;

[0028] FIG13 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0029] FIG14 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0030] FIG15 is a hardware diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0032] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0033] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0034] The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one".

[0035] The image processing method, system, device, electronic device, and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0036] With the continuous development and advancement of artificial intelligence technology, neural network models are emerging in an endless stream, with broad application prospects in areas such as system identification, pattern recognition, and intelligent control. However, the implementation of these neural network models requires powerful computing power, and currently, these models are commonly implemented using custom Application Specific Integrated Circuit (ASIC) chips specifically designed for neural network model calculations.

[0037] Currently, convolutional neural network models, as a type of neural network model, typically include a computationally intensive hierarchy consisting of convolutional, pooling, and deconvolution layers. As shown in Figure 1, the deconvolution layer operates by first inserting a number of zero-filling pixels (i.e., padding) between every two adjacent pixels in the input image. Then, convolution is performed on each pixel in the input image after the padding has been inserted, increasing the size of the output image. In hardware implementation, to reduce the computational complexity of custom ASIC chips, the convolutional neural network model is typically quantized, converting floating-point numbers to fixed-point values. Asymmetric quantization is a common quantization method, but this quantization process can cause the values ​​of the padding pixels to change from 0 to non-zero. Each pixel in the quantized input image is then zero-shifted (i.e., subtracted from the zero-point). For the deconvolution layer, this zero-point is the non-zero value of the original input image pixels (i.e., non-padding pixels) after quantization. This causes the values ​​of the padding pixels subsequently inserted between pixels to become zero, thereby reducing the number of convolution operations.

[0038] However, according to the above method, although the zero-point offset processing in asymmetric quantization can greatly reduce the amount of computation required for subsequent convolution operations, due to the high parallelism of operations in the convolutional neural network model and the large bandwidth of the input image, a large number of subtractors and registers need to be set when performing the above-mentioned zero-point offset processing, which will result in a large area occupied and power consumption of the customized ASIC chip.

[0039] In order to solve the above problems, the embodiments of the present application provide an image processing method, system, device, electronic device and readable storage medium. The image processing method provided by the embodiments of the present application can be applied to the operation process of the deconvolution layer of the convolutional neural network model.

[0040] In the image processing method provided in the embodiment of the present application, first data can be calculated based on at least one padded pixel in the input image, a coefficient in a convolution kernel that corresponds one-to-one to at least one padded pixel, and an output bias corresponding to the convolution kernel, where the input image is the input image of the deconvolution layer in the neural network model; and second data can be calculated based on at least one non-padded pixel in the input image and a coefficient in a convolution kernel that corresponds one-to-one to at least one non-padded pixel; and the pixel value of an output pixel can be calculated based on the first data and the second data. Through this solution, during the operation of the deconvolution layer, the pixel value of the output pixel can be calculated based on the pixels in the input image and the convolution kernel, and the entire operation process no longer requires zero-point offset processing for each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thereby reducing the occupied area and power consumption of the custom ASIC chip.

[0041] It should be noted that the image processing method provided in the embodiments of the present application can be executed by an image processing device, an electronic device, or a functional module in an electronic device, etc. In some embodiments of the present application, the image processing method provided in the embodiments of the present application is described by taking an electronic device executing the image processing method as an example.

[0042] Figure 2 shows a flow chart of the image processing method provided by an embodiment of the present application. As shown in Figure 2, the image processing method provided by an embodiment of the present application may include the following steps 201 to 203.

[0043] Step 201: The electronic device calculates first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel corresponding one-to-one to the at least one padded pixel, and an output bias corresponding to the convolution kernel.

[0044] The above input image is the input image of the deconvolution layer in the neural network model.

[0045] In some embodiments of the present application, the above-mentioned neural network model may be a convolutional neural network model.

[0046] In some embodiments of the present application, the input image may be a multi-channel input image.

[0047] In some embodiments of the present application, the convolution kernel has the same number of channels as the input image.

[0048] In some embodiments of the present application, all coefficients in the above-mentioned convolution kernel are obtained in the training of the above-mentioned neural network model.

[0049] In some embodiments of the present application, one or more convolution kernels may be used in the entire operation process of the above-mentioned deconvolution layer.

[0050] It should be noted that the output bias is a learnable parameter in the convolutional neural network model. This output bias can be trained like a weight. The output bias is usually a single value, and there is only one output bias corresponding to each convolution kernel. This output bias can be used to eliminate the linearity of the convolutional layer output, making the convolutional neural network model more robust and improving the performance of the convolutional layer.

[0051] In some embodiments of the present application, a filling pixel is a pixel inserted between two original image pixels (ie, non-filling pixels described below) of the input image.

[0052] In some embodiments of the present application, the electronic device can calculate the above-mentioned first data through the first chip based on the above-mentioned at least one filling pixel, the coefficient in the above-mentioned convolution kernel corresponding one-to-one to the at least one filling pixel, and the output bias corresponding to the convolution kernel.

[0053] In some embodiments of the present application, the above-mentioned first chip can be a central processing unit (CPU) or a microcontroller unit (MCU) in an electronic device, etc., except for a chip dedicated to the calculation of a neural network model.

[0054] In some embodiments of the present application, in combination with FIG. 2 , as shown in FIG. 3 , the above step 201 may be specifically implemented through the following steps 201 a and 201 b.

[0055] Step 201a: The electronic device multiplies the pixel value of each padded pixel in the at least one padded pixel by the corresponding coefficient in a convolution kernel to obtain at least one third data having the same number as the at least one padded pixel.

[0056] In some embodiments of the present application, each of the above-mentioned filling pixels has a one-to-one corresponding coefficient in the above-mentioned convolution kernel, and the electronic device can multiply the pixel value of each filling pixel by a one-to-one corresponding coefficient to obtain the above-mentioned at least one third data.

[0057] For example, assuming that the at least one padded pixel includes pixel 1, pixel 2, pixel 3, and pixel 4, and the coefficient corresponding to pixel 1 in the convolution kernel is coefficient a, the coefficient corresponding to pixel 2 in the convolution kernel is coefficient b, the coefficient corresponding to pixel 3 in the convolution kernel is coefficient c, and the coefficient corresponding to pixel 4 in the convolution kernel is coefficient d; then the electronic device can multiply the pixel value of pixel 1 by coefficient a to obtain data A, multiply the pixel value of pixel 2 by coefficient b to obtain data B, multiply the pixel value of pixel 3 by coefficient c to obtain data C, and multiply the pixel value of pixel 4 by coefficient d to obtain data D. It can be seen that the electronic device obtains four data, namely, data A, data B, data C, and data D (i.e., the at least one third data), and the number of obtained data is the same as the number of the at least one padded pixel.

[0058] Step 201b: The electronic device adds the sum of at least one third data to the output bias to obtain the first data.

[0059] In some embodiments of the present application, the electronic device may first add the at least one third data to obtain the sum of the at least one third data, and then add the sum of the at least one third data to the output bias to obtain the first data.

[0060] It should be noted that the pixel value of each of the above-mentioned filling pixels and the coefficients in the above-mentioned convolution kernel are parameters related to the above-mentioned neural network model, so that the electronic device can be controlled by software and execute the above-mentioned steps 201a and 201b through the above-mentioned first chip.

[0061] In an embodiment of the present application, since the electronic device can obtain the above-mentioned first data by multiplying and adding the pixel value of each of the above-mentioned filling pixels with the corresponding coefficient in the above-mentioned convolution kernel, and then adding the above-mentioned output bias, the operation of the filling pixel part can be completed without performing zero-point offset processing, and thus there is no need to add subtractors and registers, which can reduce the design area of ​​the hardware.

[0062] Step 202: The electronic device calculates and obtains second data based on at least one non-filled pixel in the input image and a coefficient in a convolution kernel that corresponds one-to-one to the at least one non-filled pixel.

[0063] In some embodiments of the present application, the non-filled pixels are original image pixels in the input image.

[0064] In some embodiments of the present application, each non-filled pixel in the at least one non-filled pixel has a corresponding coefficient in the one convolution kernel.

[0065] In some embodiments of the present application, the electronic device can calculate the above-mentioned second data through a second chip based on the above-mentioned at least one non-filled pixel and the coefficients in the above-mentioned convolution kernel that correspond one-to-one to the at least one non-filled pixel.

[0066] In the embodiment of the present application, the second chip is different from the first chip.

[0067] In some embodiments of the present application, the second chip may be a chip dedicated to the calculation of the neural network model; for example, the second chip is a customized ASIC chip dedicated to the calculation of the neural network model.

[0068] In some embodiments of the present application, in combination with FIG. 2 , as shown in FIG. 4 , the above step 202 may be specifically implemented through the following steps 202 a and 202 b.

[0069] Step 202a: The electronic device multiplies the pixel value of each non-filled pixel in the at least one non-filled pixel by a corresponding coefficient in a convolution kernel to obtain at least one fourth data having the same number as the at least one non-filled pixel.

[0070] Step 202b: The electronic device adds at least one fourth data to obtain second data.

[0071] For the specific description of the embodiments of the present application, reference can be made to the relevant descriptions in the above-mentioned step 201a and step 201b. In order to avoid repetition, they will not be repeated here.

[0072] In an embodiment of the present application, since the electronic device can use the pixel value of each non-filled pixel as the multiplication and addition result of the corresponding coefficient in the convolution kernel as the second data, the operation of the non-filled pixel part can be completed without performing zero-point offset processing, and thus there is no need to add subtractors and registers, which can reduce the design area of ​​the hardware.

[0073] Step 203: The electronic device calculates a pixel value of an output pixel according to the first data and the second data.

[0074] In some embodiments of the present application, in combination with FIG. 2 , as shown in FIG. 5 , the above step 203 may be specifically implemented through the following step 203 a.

[0075] Step 203a: The electronic device adds the first data and the second data to obtain a pixel value of an output pixel.

[0076] In some embodiments of the present application, the electronic device may add the first data and the second data through the second chip to obtain the pixel value of the output pixel.

[0077] In some embodiments of the present application, after the electronic device obtains the above-mentioned first data through the above-mentioned first chip, it can configure the first data to the storage unit in the above-mentioned second chip through bus configuration; so that after the second chip obtains the first data and calculates the above-mentioned second data, it can add the first data and the second data to obtain the pixel value of the above-mentioned output pixel.

[0078] In an embodiment of the present application, since the electronic device can add the above-mentioned first data and the above-mentioned second data to obtain the pixel value of the above-mentioned output pixel, the entire operation of the deconvolution layer can be completed without performing zero-point offset processing, and thus there is no need to add subtractors and registers, which can reduce the design area and power consumption of the hardware.

[0079] The image processing method provided in the embodiments of the present application is exemplarily described below with reference to the accompanying drawings.

[0080] For example, assuming that the above-mentioned convolution kernel is a 2×2 convolution kernel and the deconvolution step size of the above-mentioned deconvolution layer is 2; then, as shown in FIG6 , the electronic device can first insert a padding element between every two adjacent pixels of the input image, and then perform a convolution operation on the padded input image with a convolution kernel of 2×2 and a convolution step size of 1, and finally add an output bias to each pixel after the convolution to obtain the final output result of the deconvolution layer.

[0081] Specifically, as shown in FIG7 , since the size of the above-mentioned convolution kernel is 2×2, each convolution operation requires the participation of a 2×2 image window, and every other input pixel Pin (i.e., the above-mentioned non-filled pixel) in the input image has a filling pixel pad, so there is an input pixel Pin and three filling pixels pad in this 2×2 image window. The pixel point (pad or Pin) at the corresponding position and the coefficient (C) in the convolution kernel are multiplied and added to obtain the convolution result Pcon, and then the convolution result Pcon is added to the output bias corresponding to the convolution kernel to obtain the pixel value Pout of an output pixel of the deconvolution layer. The Pout can be specifically calculated by the following formula (1): Pout = pad*(C 0_0 +C 0_1 +…+C 0_n +C 1_0 +C 1_1 +…+C 1_n +C 2_0 +C 2_1 +…+C 2_n ) +(P in_0 *C 3_0 +P in_1 *C 3_1 +…Pin_n *C 3_n )+bias; (1)

[0082] It can be seen that the first and third terms in the above formula (1) are completely composed of the parameters of the neural network model (filling pixels and convolution kernel coefficients). Therefore, in the image processing method provided in the embodiment of the present application, the first chip can be controlled by software to calculate in advance, and then the result of the addition of the two terms is written into the second chip as the new output bias bias_new.

[0083] In the convolution operation, since the image selection window of the convolution operation with a step size of 1 will slide on the padded input image, there are four ways to correspond image pixels and convolution kernel coefficients. Figure 8 shows the specific correspondence between the four corresponding ways; so for each convolution kernel, there are also four new output biases that can be calculated in advance by the first chip under software control, namely: bias_new0=pad*(C 0_0 +C 0_1 +…+C 0_n +C 1_0 +C 1_1 +…+C 1_n +C 2_0 +C 2_1 +… +C 2_n )+bias; bias_new1=pad*(C 0_0 +C 0_1 +…+C 0_n +C 1_0 +C 1_1 +…+C 1_n +C 3_0 +C 3_1 +… +C 3_n )+bias; bias_new2=pad*(C 0_0 +C 0_1 +…+C 0_n +C 2_0 +C 2_1 +…+C 2_n +C 3_0 +C 3_1 +… +C 3_n )+bias; bias_new3=pad*(C 1_0 +C 1_1 +…+C 1_n +C 2_0 +C 2_1 +…+C 2_n +C 3_0 +C 3_1 +… +C 3_n)+bias;

[0084] The specific calculation process is shown in Figure 9. Since the operations involving the padded pixels are completed by the first chip under software control, the second chip only needs to complete the multiplication and accumulation operations of the original input image pixels and the convolution kernel coefficients, and then add the new output bias bias_new. Figure 10 shows the specific calculation process for an output result Pout, where the deconvolution layer calculation formula is shown in the following formula (2): Pout = (Pin_0*C3_0+Pin_1*C3_1+…Pin_n*C3_n)+bias_new; (2)

[0085] Similarly, due to the sliding of the image selection window during the convolution operation, each input image pixel has four corresponding ways to the coefficient of the convolution kernel, and the pixel values ​​of the four image pixels can be output, namely: Pout0 = (Pin_0*C3_0+Pin_1*C3_1+…Pin_n*C3_n)+bias_new0; Pout1 = (Pin_0*C2_0+Pin_1*C2_1+…Pin_n*C2_n)+bias_new1; Pout2 = (Pin_0*C1_0+Pin_1*C1_1+…Pin_n*C1_n)+bias_new2; Pout3 = (Pin_0*C0_0+Pin_1*C0_1+…Pin_n*C0_n)+bias_new3;

[0086] Finally, the arrangement order of the pixels of the output image is shown in Figure 11. Each input pixel will obtain 4 output pixels after the deconvolution layer operation, so as to achieve the expansion of the size of the input image.

[0087] In this way, by transferring the calculation of the neural network model parameters in the deconvolution layer to the first chip under software control, the calculation of the second chip is simplified from 2×2 convolution to 1×1 dot product, reducing the computational effort to 1 / 4 of the original amount, thereby reducing the power consumption of the second chip. Furthermore, the addition of the first and second data can be completed by the existing second chip without adding new hardware logic. This also avoids the subtraction operations caused by zero-point offset processing in related technologies, eliminating the need for a large number of subtractors and registers, thus reducing the hardware area and power consumption of the second chip.

[0088] In the image processing method provided in the embodiment of the present application, during the operation of the deconvolution layer, the pixel value of the output pixel can be calculated based on the pixels in the input image and the convolution kernel, and the entire operation process no longer requires zero-point offset processing for each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thereby reducing the hardware footprint and power consumption.

[0089] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0090] Figure 12 shows a schematic diagram of an image processing system provided by an embodiment of the present application. As shown in Figure 12, the image processing system 120 provided by an embodiment of the present application may include: a first chip 121 and a second chip 122, which are different from each other, and the second chip 122 is a chip dedicated to the calculation of the neural network model.

[0091] The first chip 121 is configured to calculate first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel corresponding to the at least one padded pixel, and an output bias corresponding to the convolution kernel; the input image is the input image of the deconvolution layer in the neural network model. The second chip 122 is configured to calculate second data based on at least one unpadded pixel in the input image and a coefficient in the convolution kernel corresponding to the at least one unpadded pixel. The second chip 122 is further configured to calculate a pixel value of an output pixel based on the first data and the second data.

[0092] In one possible implementation, the first chip 121 can be specifically used to multiply the pixel value of each of the at least one filling pixel by the corresponding coefficient in the convolution kernel to obtain at least one third data having the same number as the at least one filling pixel; and add the sum of the at least one third data to the output bias to obtain the first data.

[0093] In one possible implementation, the second chip 122 can be specifically used to multiply the pixel value of each non-filled pixel in the at least one non-filled pixel by the corresponding coefficient in the convolution kernel to obtain at least one fourth data having the same number as the at least one non-filled pixel; and add the at least one fourth data to obtain the second data.

[0094] In a possible implementation, the second chip 122 may be configured to add the first data and the second data to obtain a pixel value of the output pixel.

[0095] In the image processing system provided by the embodiment of the present application, on the one hand, part of the operations in each convolution operation of the deconvolution layer can be handed over to the first chip in addition to the second chip dedicated to the calculation of the neural network model, thereby reducing the power consumption of the second chip; on the other hand, during the entire operation process of the deconvolution layer, there is no need to perform zero-point offset processing on each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thus reducing the hardware area of ​​the second chip. The image processing system provided by the embodiment of the present application can implement each process implemented by the above-mentioned method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0096] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0097] As shown in FIG13 , an embodiment of the present application provides an image processing device 130 , which may include a computing module 131 .

[0098] Calculation module 131 can be configured to calculate first data based on at least one padded pixel in an input image, a coefficient in a convolution kernel that corresponds one-to-one with the at least one padded pixel, and an output bias corresponding to the convolution kernel, where the input image is an input image of a deconvolution layer in a neural network model. Calculation module 131 can also be configured to calculate second data based on at least one non-padded pixel in the input image and a coefficient in the convolution kernel that corresponds one-to-one with the at least one non-padded pixel. Calculation module 131 can also be configured to calculate a pixel value of an output pixel based on the first data and the second data.

[0099] In one possible implementation, the calculation module 131 can be specifically used to multiply the pixel value of each filling pixel in the above-mentioned at least one filling pixel by the corresponding coefficient in the above-mentioned convolution kernel to obtain at least one third data with the same number as the at least one filling pixel; and add the sum of the at least one third data to the above-mentioned output bias to obtain the above-mentioned first data.

[0100] In one possible implementation, the calculation module 131 can be specifically used to multiply the pixel value of each non-filled pixel in the above-mentioned at least one non-filled pixel by the corresponding coefficient in the above-mentioned convolution kernel to obtain at least one fourth data with the same number as the at least one non-filled pixel; and add the at least one fourth data to obtain the above-mentioned second data.

[0101] In a possible implementation, the calculation module 131 may be specifically configured to add the first data and the second data to obtain a pixel value of the output pixel.

[0102] In the image processing device provided in the embodiment of the present application, during the operation process of the deconvolution layer, the pixel value of the output pixel can be calculated based on the pixels in the input image and the convolution kernel, and the entire operation process no longer requires zero-point offset processing for each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thereby reducing the occupied area and power consumption of the customized ASIC chip.

[0103] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0104] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0105] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0106] As shown in Figure 14, an embodiment of the present application also provides an electronic device 1400, including a processor 1401 and a memory 1402. The memory 1402 stores programs or instructions that can be run on the processor 1401. When the program or instructions are executed by the processor 1401, the various steps of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0107] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0108] FIG15 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0109] As shown in Figure 15, the electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0110] Those skilled in the art will appreciate that the electronic device 1000 may further include a power source (such as a battery) to power various components. The power source may be logically connected to the processor 1010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG15 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0111] In which, the processor 1010 can be used to calculate first data based on at least one padded pixel in the input image, a coefficient in a convolution kernel that corresponds one-to-one to the at least one padded pixel, and an output bias corresponding to the convolution kernel, where the input image is an input image of a deconvolution layer in a neural network model; and calculate second data based on at least one non-padded pixel in the input image and a coefficient in the convolution kernel that corresponds one-to-one to the at least one non-padded pixel; and calculate a pixel value of an output pixel based on the first data and the second data.

[0112] In one possible implementation, the processor 1010 can be specifically used to multiply the pixel value of each filling pixel in the above-mentioned at least one filling pixel by the corresponding coefficient in the above-mentioned convolution kernel to obtain at least one third data with the same number as the at least one filling pixel; and add the sum of the at least one third data to the above-mentioned output bias to obtain the above-mentioned first data.

[0113] In one possible implementation, the processor 1010 can be specifically used to multiply the pixel value of each non-filled pixel in the above-mentioned at least one non-filled pixel by the corresponding coefficient in the above-mentioned convolution kernel to obtain at least one fourth data with the same number as the at least one non-filled pixel; and add the at least one fourth data to obtain the above-mentioned second data.

[0114] In a possible implementation, the processor 1010 may be specifically configured to add the first data and the second data to obtain a pixel value of the output pixel.

[0115] In the electronic device provided in the embodiment of the present application, during the operation process of the deconvolution layer, the pixel value of the output pixel can be calculated based on the pixels in the input image and the convolution kernel, and the entire operation process no longer requires zero-point offset processing for each pixel point of the input image, thereby eliminating the need to set up a large number of subtractors and registers, thereby reducing the occupied area and power consumption of the customized ASIC chip.

[0116] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0117] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0118] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0119] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0120] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0122] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0123] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0124] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0126] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, the method comprising: Calculate and obtain first data according to at least one padding pixel in an input image, a coefficient in a convolution kernel corresponding one-to-one to the at least one padding pixel, and an output bias corresponding to the convolution kernel, wherein the input image is an input image of a deconvolution layer in a neural network model; Obtain second data by calculation according to at least one non-filled pixel in the input image and a coefficient in the convolution kernel that corresponds one-to-one to the at least one non-filled pixel; A pixel value of an output pixel is calculated based on the first data and the second data.

2. The method according to claim 1, wherein: The step of calculating the first data according to at least one padding pixel in the input image, a coefficient in a convolution kernel corresponding to the at least one padding pixel, and an output bias corresponding to the convolution kernel includes: Multiplying the pixel value of each padded pixel in the at least one padded pixel by the corresponding coefficient in the one convolution kernel to obtain at least one third data having the same number as the at least one padded pixel; The sum of the at least one third data is added to the output bias to obtain the first data.

3. The method according to claim 1, wherein: The step of calculating the second data according to at least one non-filled pixel in the input image and a coefficient in the convolution kernel corresponding to the at least one non-filled pixel in a one-to-one manner comprises: Multiplying the pixel value of each non-filled pixel in the at least one non-filled pixel by the corresponding coefficient in the one convolution kernel to obtain at least one fourth data having the same number as the at least one non-filled pixel; The at least one fourth data is added to obtain the second data.

4. The method according to any one of claims 1 to 3, wherein: The step of calculating a pixel value of an output pixel according to the first data and the second data comprises: The first data and the second data are added to obtain a pixel value of the one output pixel.

5. An image processing system, the system comprising: A first chip and a second chip that are different from each other, wherein the second chip is a chip dedicated to calculation of a neural network model; The first chip is used to calculate the first data according to at least one padding pixel in the input image, a coefficient in a convolution kernel corresponding to the at least one padding pixel, and an output bias corresponding to the convolution kernel, and the input image is an input image of the deconvolution layer in the neural network model; The second chip is used to calculate the second data according to at least one non-filled pixel in the input image and the coefficients in the one convolution kernel corresponding to the at least one non-filled pixel one by one; The second chip is further used to calculate a pixel value of an output pixel according to the first data and the second data.

6. The system according to claim 5, wherein: The first chip is specifically used to multiply the pixel value of each filling pixel in the at least one filling pixel by the corresponding coefficient in the convolution kernel to obtain at least one third data with the same number as the at least one filling pixel; and add the sum of the at least one third data to the output bias to obtain the first data.

7. The system according to claim 5, wherein: The second chip is specifically used to multiply the pixel value of each non-filled pixel in the at least one non-filled pixel by the corresponding coefficient in the convolution kernel to obtain at least one fourth data which is the same as the number of the at least one non-filled pixel; and add the at least one fourth data to obtain the second data.

8. A system according to any one of claims 5 to 7, wherein: The second chip is specifically used to add the first data and the second data to obtain the pixel value of the output pixel.

9. An image processing device, the device comprising a computing module; The calculation module is used to calculate the first data according to at least one padding pixel in the input image, a coefficient in a convolution kernel corresponding to the at least one padding pixel, and an output bias corresponding to the convolution kernel, wherein the input image is an input image of a deconvolution layer in the neural network model; The calculation module is further used to calculate the pixel value based on at least one non-filled pixel in the input image and the system in the one convolution kernel that corresponds to the at least one non-filled pixel. Number, calculate and obtain the second data; The calculation module is further used to calculate a pixel value of an output pixel according to the first data and the second data.

10. The device according to claim 9, wherein: The calculation module is specifically used to multiply the pixel value of each filling pixel in the at least one filling pixel by the corresponding coefficient in the convolution kernel to obtain at least one third data with the same number as the at least one filling pixel; and add the sum of the at least one third data to the output bias to obtain the first data.

11. The device according to claim 9, wherein: The calculation module is specifically used to multiply the pixel value of each non-filled pixel in the at least one non-filled pixel by the corresponding coefficient in the convolution kernel to obtain at least one fourth data which is the same as the number of the at least one non-filled pixel; and add the at least one fourth data to obtain the second data.

12. The device according to any one of claims 9 to 11, wherein: The calculation module is specifically used to add the first data and the second data to obtain the pixel value of the output pixel.

13. An electronic device, wherein: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 4 are implemented.

14. A readable storage medium, wherein: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 4 are implemented.

15. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method as described in any one of claims 1 to 4.

16. A computer program product, wherein the computer program product is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 4.

Images