Image signal processing method, system and related devices

Abstract

Embodiments of the present application provide an image signal processing method, a system and related devices. The method includes: obtaining an input data corresponding to a raw data; performing a smoothing transform on the plurality of target pixels of the input data to obtain a transformed data; performing an encoding operation on the transformed data to obtain an encoded data; performing a decoding operation on the encoded data to obtain a decoded data; performing an inverse smoothing transform on the plurality of transformed pixels of the decoded data to obtain a reconstructed data; performing an image signal processing operation on the reconstructed data. The embodiments of the present applications provide a nearly reversible transformation for an input data, which may decrease loss of information and increase compression rate.

Description

IMAGE SIGNAL PROCESSING METHOD, SYSTEM AND RELATEDDEVICESTECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of signal processing, more specifically related to an image signal processing method, system and related devices.BACKGROUND

[0002] Currently, both surveillance cameras for a professional application and mobile phone cameras in the consumer market use an image sensor (e.g., a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor (CIS)) to capture an optical signal in a scene and transform the optical signal into raw data of a digital signal (that is, raw data). The raw data from the image sensor contains light intensity values captured from a scene, but these data are not intrinsically recognizable to the human eye. In order to obtain a color image that is suitable for human eyes (generally in red green blue (RGB) format or luminance and chrominance (YUV) format), the raw data may be processed through a sequence of image processing operations using an image signal processor (ISP).SUMMARY

[0003] Embodiments of the present application provides an image signal processing method, a system and related devices. The embodiments of the present applications provide a nearly reversible transformation for an input data, which may decrease loss of information and increase compression rate.

[0004] According to a first aspect, an embodiment of the present application provides an image signal processing method. The method includes: obtaining input data corresponding to raw data, where the input data includes a plurality of target pixels and a plurality of first reservedpixels; performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, where the smooth transform is a reversible transform, the transformed data includes a plurality of first transformed pixels and the plurality of first reserved pixels, the plurality of first transformed pixels is in one-to-one correspondence with the plurality of target pixels, and a pixel value of each of the plurality of first transformed pixels is determined according to pixel values a corresponding target pixel and at least one neighboring pixel of the corresponding target pixel; performing an encoding operation on the transformed data to obtain encoded data.

[0005] According to the smoothing transform, noise of the input image data may be decreased, and a higher compression ratio may be obtained. In addition, the smooth transform is a reversible transform, so transform results obtained according the smoothing transform is reversible. Based on the reversible transform, a noise data in the input image data may be kept, and the noise data is important for an Al-based image signal processing. Further, in contrast to neural networks based solutions, the algorithm provided by the present application is simple and does not increase the latency of the raw data processing significantly.

[0006] In a possible design of the first aspect, the performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, includes: dividing the input data into F blocks, where the F blocks includes F i target blocks, F is a positive integer, F i is a positive integer, Fi is less than or equal to F; transforming target pixels among the Fi target blocks into first transformed pixels to obtain the transformed data, where the transformed data includes Fi transformed blocks, the Fi transformed blocks are in one-to-one correspondence with the Fi target blocks, each of the Fl transformed blocks are determined according to a corresponding target block, and a size of the target block is a same as a size of the transformed block.

[0007] In a possible design of the first aspect, the transforming the target pixels among the Fi target blocks into the first transformed pixels to obtain the transformed data, includes: transforming target pixels in a tthtarget block among the F i target blocks into first transformed pixels to obtain a tthtransformed block among the Fi transformed blocks, where the tthtransformed block includes first transformed pixels determined based on the target pixels in the tthtarget block, and the tthtransformed block further includes first reserved pixels in the tthtargetblock, t=l, Fi.

[0008] In a possible design of the first aspect, a pixel value of any one of the first transformed pixels in the tthtransformed block satisfies: V_NP=(woxV_GP+wi x V_GPni+w2xV_GPn2+w3xV_GPn3) / (wo+wi+w2+w3), where V_NP is a pixel value of a first transformed pixel NP in the tthtransformed block, V_GP is a pixel value of a target pixel GP corresponding to the first transformed pixel NP in the tthtarget block, the first transformed pixel NP is any one of the first transformed pixels in the tthtransformed block, V_GPniis a pixel value of a first neighboring pixel of the target pixel GP, V_GPn2 is a pixel value of a second neighboring pixel of the target pixel GP, V_GPn3 is a pixel value of a third neighboring pixel of the target pixel GP, wo is a weight of the target pixel GP, wi is a weight of the first neighboring pixel, W2 is a weight of the second neighboring pixel, and W3 is a weight of the third neighboring pixel.

[0009] In a possible design of the first aspect, before the performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, the method further includes: determining a variance of the plurality of target pixels meets a first preset condition; and / or determining a mean of the plurality of target pixels meets a second preset condition.

[0010] In a possible design of the first aspect, when the first reserved pixels in the tthtarget block includes pixels in a top row and a leftmost column in the Ithtarget block, the first neighboring pixel of the target pixel GP is a left neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a top-left neighboring pixel of the target pixel GP, and the third neighboring pixel of the target pixel GP is a top neighboring pixel of the target pixel GP.

[0011] In a possible design of the first aspect, the F i target blocks are arranged in M rows and N columns, the tthtarget block is in an i* row of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 0, where mod represents a modulo operation.

[0012] In a possible design of the first aspect, when the first reserved pixels in the tthtarget block includes pixels in a bottom row and a rightmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a right neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a bottom-right neighboring pixel of the targetpixel GP, the third neighboring pixel of the target pixel GP is a bottom neighboring pixel of the target pixel GP.

[0013] In a possible design of the first aspect, the Fi target blocks are arranged in M rows and N columns, the t* target block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 1, where mod represents a modulo operation.

[0014] In a possible design of the first aspect, when the first reserved pixels in the tthtarget block includes pixels in a bottom row and a leftmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a left neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a bottom-left neighboring pixel of the target pixel GP, the third neighboring pixel of the target pixel GP is a bottom neighboring pixel of the target pixel GP.

[0015] In a possible design of the first aspect, the Fi target blocks are arranged in M rows and N columns, the t* target block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1 ; j mod 2 = 0, where mod represents a modulo operation.

[0016] In a possible design of the first aspect, when the first reserved pixels in the tthtarget block includes pixels in a top row and a rightmost column in the tlhtarget block, the first neighboring pixel of the target pixel GP is a right neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a top-right neighboring pixel of the target pixel GP, the third neighboring pixel of the target pixel GP is a top neighboring pixel of the target pixel GP.

[0017] In a possible design of the first aspect, the Fi target blocks are arranged in M rows and N columns, the Ithtarget block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 1, where mod represents a modulo operation.

[0018] In a possible design of the first aspect, the Fi target blocks includes a plurality of first target block pairs, each of the plurality of first target block pairs includes a first target block and a second target block, the first target block is located to a left of the second target block, first reserved pixels in the first target block include pixels in a rightmost column in the firsttarget block, first reserved pixels in the second target block include pixels in a leftmost column in the second target block, and the pixels in the rightmost column in the first target block are the pixels in the leftmost column of the second target block.

[0019] In a possible design of the first aspect, the F i target blocks includes a plurality of second target block pairs, each of the plurality of second target block pairs includes a third target block and a fourth target block, the third target block is located above the fourth target block, first reserved pixels in the third block includes pixels in the bottom row in the third block; first reserved pixels in the fourth blocks includes pixels in the top row in the fourth block; the pixels in the bottom row in the third block are the pixels in the top row in the fourth block.

[0020] In a possible design of the first aspect, wo, wi, W2, and W3 are not identical.

[0021] According to the abovementioned technical solution, restrictions on the weights used for the smoothing transform and the inverse smoothing transform are not very strict. Therefore, the weights may be adapted according to the input data. Based on the adaptive weights, some important areas of the input data may have less distortion, and noise areas may have more distortion and higher compression rate.

[0022] In a possible design of the first aspect, wo satisfies:

[0024] where Kbase is a base K- value, kVar is a fixed scaling coefficient, maxi is a maximum intensity of target pixels corresponding to second transformed pixels in decoded data, mean is a mean intensity of the target pixels, VT is a constant, variance is a variance intensity of the target pixels, threshold is a fixed constant.

[0025] In a possible design of the first aspect, wi, w?, and W3 satisfy: wi = W2 = W3.

[0026] In a possible design of the first aspect, the encoded data is used to obtain decoded data by performing a decoding operation, where the decoded data includes a plurality of second transformed pixels and a plurality of second reserved pixels; the plurality of second transformed pixels of the decoded data are used to obtain reconstructed data by performing an inverse smoothing transform, where the reconstructed data includes a plurality of reconstructed pixels and the plurality of second reserved pixels, the plurality of reconstructed pixels is in one-to-one correspondence with the plurality of second transformed pixels, and a pixel value of each of theplurality of reconstructed pixels is determined according to pixel values of a corresponding second transformed pixel and at least one neighboring pixel of the reconstructed pixel; and the reconstructed data is used to obtain an image signal processing result by performing an image signal processing operation.

[0027] According to a second aspect, an embodiment of the present application provides an image signal processing method. The method includes: obtaining encoded data; performing a decoding operation on the encoded data to obtain a decoded data, where the decoded data includes a plurality of second transformed pixels and a plurality of second reserved pixels; performing an inverse smoothing transform on the plurality of second transformed pixels of the decoded data to obtain a reconstructed data, where the reconstructed data includes a plurality of reconstructed pixels and the plurality of second reserved pixels, the plurality of reconstructed pixels is in one-to-one correspondence with the plurality of second transformed pixels, and a pixel value of each of the plurality of reconstructed pixels is determined according to pixel values of a corresponding second transformed pixel and at least one neighboring pixel of the reconstructed pixel; performing an image signal processing operation on the reconstructed data.

[0028] According to the inverse smoothing transform, noise of the input image data may be gotten back, and the noise data is important for an Al -based image signal processing. Therefore, the Al-based image signal processing may obtain a better result according to image data with noise. Further, in contrast to neural networks based solutions, the algorithm provided by the present application is simple and does not increase the latency of the raw data processing significantly.

[0029] In a possible design of the second aspect, the performing an inverse smoothing transform on the plurality of second transformed pixels of . the decoded data to obtain a reconstructed data, includes: dividing the decoded data into F blocks, where the F blocks includes Fi transformed blocks, F is a positive integer, Fi is a positive integer, Fi is less than or equal to F; transforming second transformed pixels among the Fi transformed blocks into reconstructed pixels to obtain the reconstructed data, where the reconstructed data includes F i reconstructed blocks, the Fi reconstructed blocks are in one-to-one correspondence with the Fi transformed blocks, each of the Fl reconstructed blocks are determined according to a corresponding transformed block, and a size of the transformed block is a same as a size of thereconstructed block.

[0030] In a possible design of the second aspect, the transforming the second transformed pixels among the Fi transformed blocks into the reconstructed pixels to obtain the reconstructed data, includes: transforming second transformed pixels in a t* transformed block among the Fi transformed blocks into reconstructed pixels to obtain a tthreconstructed block among the F i reconstructed blocks, where the tthreconstructed block includes reconstructed pixels obtained according to the second transformed pixels in the 1thtransformed block, and the tthreconstructed block further includes second reserved pixels in the tthtransformed block, t=l , ..., Fi.

[0031] In a possible design of the second aspect, pixel value of any one of the reconstructed pixels in the tthreconstructed block satisfies: V_RP=((WO+WI+W2+W3)XV_NP - wixV_RPni- W2xV_RPn2 - W3xV_RPn3) / wo, where V_RP is a pixel value of a reconstructed pixel RP in the tthreconstructed block, V_NP is a pixel value of a second transformed pixel NP corresponding to the reconstructed pixel RP in the t* reconstructed block, the reconstructed pixel RP is any one of the reconstructed pixels in the tthreconstructed block, V_RPniis a pixel value of a first neighboring pixel of the reconstructed pixel RP, V_RPn2 is a pixel value of a second neighboring pixel of the reconstructed pixel RP, V_RPn3 is a pixel value of a third neighboring pixel of the reconstructed pixel RP, wo is a weight of the second transformed pixel FP, wi is a weight of the first neighboring pixel, W2 is a weight of the second neighboring pixel, and W3 is a weight of the third neighboring pixel.

[0032] In a possible design of the second aspect, when the second reserved pixels in the tthtransformed block includes pixels in a top row and a leftmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a left neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a top-left neighboring pixel of the reconstructed pixel RP, and the third neighboring pixel of the reconstructed pixel RP is a top neighboring pixel of the reconstructed pixel RP.

[0033] In a possible design of the second aspect, the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an 1throw of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 0, where mod represents a modulo operation.

[0034] In a possible design of the first aspect, when the second reserved pixels in the tthtransformed block includes pixels in a bottom row and a rightmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a right neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a bottom-right neighboring pixel of the reconstructed pixel RP, the third neighboring pixel of the reconstructed pixel RP is a bottom neighboring pixel of the reconstructed pixel RP.

[0035] In a possible design of the second aspect, the F i transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an i* row of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 1, where mod represents a modulo operation.

[0036] In a possible design of the second aspect, when the second reserved pixels in the tthtransformed block includes pixels in a bottom row and a leftmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a left neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a bottom- left neighboring pixel of the . reconstructed pixel RP, the third neighboring pixel of the reconstructed pixel RP is a bottom neighboring pixel of the reconstructed pixel FP.

[0037] In a possible design of the second aspect, the F i transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an 1throw of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 0, where mod represents a modulo operation.

[0038] In a possible design of the second aspect, when the second reserved pixels in the tthtransformed block includes pixels in a top row and a rightmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a right neighboring pixel of the reconstructed pixel FP, the second neighboring pixel of the reconstructed pixel RP is a top- right neighboring pixel of the reconstructed pixel FP, the third neighboring pixel of the reconstructed pixel RP is a top neighboring pixel of the reconstructed pixel FP.

[0039] In a possible design of the second aspect, the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 1, where mod represents a modulo operation.

[0040] In a possible design of the second aspect, the Fi transformed blocks includes aplurality of first transformed block pairs, each of the plurality of first transformed block pairs includes a first transformed block and a second transformed block, the first transformed block is located to a left of the second transformed block, second reserved pixels in the first transformed block include pixels in a rightmost column in the first transformed block, second reserved pixels in the second transformed block include pixels in a leftmost column in the second transformed block, and the pixels in the rightmost column in the first transformed block are the pixels in the leftmost column of the second transformed block.

[0041] In a possible design of the second aspect, the Fi transformed blocks includes a plurality of second transformed block pairs, each of the plurality of second transformed block pairs includes a third transformed block and a fourth transformed block, the third transformed block is located above the fourth transformed block, second reserved pixels in the third block includes pixels in the bottom row in the third block; second reserved pixels in the fourth blocks includes pixels in the top row in the fourth block; the pixels in the bottom row in the third block are the pixels in the top row in the fourth block.

[0042] In a possible design of the second aspect, wo, wi, W2, and W3 are not identical.

[0043] According to the abovementioned technical solution, restrictions on the weights used for the smoothing transform and the inverse smoothing transform are not very strict. Therefore, the weights may be adapted according to the input data. Based on the adaptive weights, some important areas of the input data may have less distortion, and noise areas may have more distortion and higher compression rate.

[0044] In a possible design of the second aspect, wo satisfies:

[0046] where Kbase is a base K- value, kVar is a fixed scaling coefficient, maxi is a maximum intensity of target pixels corresponding to the second transformed pixels in the decoded data, mean is a mean intensity of the target pixels, VT is a constant, variance is a variance intensity of the target pixels, threshold is a fixed constant.

[0047] In a possible design of the second aspect, wi, W2, and W3 satisfy: wi = W2 = W3.

[0048] In a possible design of the second aspect, the encoded data is obtained by performing an encoding operation on transformed data, and the transformed data is obtained by performinga smoothing transform on a plurality of target pixel of input data, where the input data includes the plurality of target pixels and a plurality of first reserved pixels, the transformed data includes a plurality of first transformed pixels and the plurality of first reserved pixels, the plurality of first transformed pixels is in one-to-one correspondence with the plurality of target pixels, and a pixel value of each of the plurality of first transformed pixels is determined according to pixel values of a corresponding target pixel and at least one neighboring pixel of the corresponding target pixel.

[0049] According to a third aspect, an embodiment of the present application provides an electronic device, and the electronic device has a function of implementing the method in the first aspect or any possible implementation of the first aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the function.

[0050] According to a fourth aspect, an embodiment of the present application provides an electronic device, and the electronic device has a function of implementing the method in the second aspect or any possible implementation of the second aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the function.

[0051] According to a fifth aspect, an embodiment of the present application provides a computer readable storage medium, including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0052] According to a sixth aspect, an embodiment of the present application provides a computer readable storage medium, including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0053] According to a seventh aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory,the processor is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0054] According to an eighth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0055] According to a ninth aspect, an embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method in the first aspect or any possible implementation of the first aspect.

[0056] According to a tenth aspect, an embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method in the second aspect or any possible implementation of the second aspect.

[0057] According to an eleventh aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0058] According to a twelfth aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0059] According to a thirteenth aspect, an embodiment of the present application provides an image processing system. The image processing system includes a first electronic device anda second electronic device. The first electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect, and the second electronic device is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0060] For example, the first electronic device may be the electronic device in the third aspect, and the second electronic device may be the electronic device in the fourth aspect. For another example, the first electronic device may be the electronic device in the seventh aspect, and the second electronic device may be the electronic device in the eighth aspect.

[0061] In addition, an embodiment of the present application provides an electronic device, and the electronic device has a function of implementing the method in the first aspect or any possible implementation of the first aspect and has a function of implementing the method in the second aspect or any possible implementation of the second aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more modules corresponding to the function.

[0062] An embodiment of the present application provides a computer readable storage medium, including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect and perform the method in the second aspect or any possible implementation of the second aspect.

[0063] An embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible implementation of the first aspect and perform the method in the second aspect or any possible implementation of the second aspect.

[0064] An embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposedperforms the method in the first aspect or any possible implementation of the first aspect and performs the method in the second aspect or any possible implementation of the second aspect.

[0065] According to an eleventh aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect and perform the method in the second aspect or any possible implementation of the second aspect.DESCRIPTION OF DRAWINGS

[0066] FIG. 1 is a schematic diagram of an image processing system.

[0067] FIG. 2 illustrates a schematic diagram of an electronic device.

[0068] FIG. 3 illustrates a diagram of an image signal processing procedure provided by some embodiments of the present application.

[0069] FIG. 4 illustrates an image processing method according to some embodiments of the present application.

[0070] FIG. 5 illustrates a dividing result of an input data according to some embodiments of the present application.

[0071] FIG. 6 illustrates a dividing result of another input data according to some embodiments of the present application.

[0072] FIG. 7 illustrates a target block transformed block obtained according to the target block.

[0073] FIG. 8 illustrates a target block transformed block obtained according to the target block.

[0074] FIG. 9 illustrates a target block transformed block obtained according to the target block.

[0075] FIG. 10 illustrates a target block transformed block obtained according to the target block.

[0076] FIG. 11 illustrates approaches for transforming a plurality of target blocks.

[0077] FIG. 12 illustrates an example of the first target block pair.

[0078] FIG. 13 illustrates an example of the second target block pair.

[0079] FIG. 14 illustrates another target block according to some embodiments provided by the present application.

[0080] The inverse transform procedure illustrated in FIG. 15 is an inverse operation of the transform procedures illustrated in FIG. 14.

[0081] FIG. 16 illustrates a schematic block diagram of an encoding device according to some embodiments of the present application.

[0082] FIG. 17 illustrates a schematic block diagram of a decoding device according to some embodiments of the present application.

[0083] FIG. 18 illustrates a schematic block diagram of an electronic device according to some embodiments of the present application.

[0084] FIG. 19 illustrates a schematic block diagram of an electronic device according to some embodiments of the present application.

[0085] FIG. 20 illustrates a schematic block diagram of an electronic device according to some embodiments of the present application.DESCRIPTION OF EMBODIMENTS

[0086] The following describes the technical solutions in the present application with reference to the accompanying drawings.

[0087] For a better understanding of embodiments provided by the present application, the following briefly describes some related basic concepts.

[0088] Raw data typically comes in a form of a color filter array (CFA). The CFA consists of an array of tiny color filters placed over individual photodiodes on the image sensor capture color information. These color filters are usually red, green, and blue (RGB), and they allow different wavelengths of light to pass through to the corresponding photodiodes. By capturing the intensity of light for each color channel at each pixel, the image sensor may obtain the raw data.

[0089] Usually, the raw data may have noises that comes from the image sensor. These noises are important for an artificial intelligence (Al)-based ISP, because a neural network forprocessing the raw data may be trained in presence of the noises. Without these noises, an output of the Al -based ISP may become out of original distribution and the neural network behavior may become unexpected.

[0090] FIG. l is a schematic diagram of an image processing system. As shown in FIG. 1 , an image processing system 100 includes a camera module 110 and an image processing module 120.

[0091] The camera module 110 includes a lens 111 and an image sensor 112. The image sensor 112 may be a CCD sensor or a CMOS image sensor. The lens 111 may be configured to obtain an optical signal corresponding to a photographed object in an image shooting scene. The image sensor 112 may be configured to convert the optical signal passing through the lens 111 into an electrical signal, perform analog-to-digital (A / D) conversion on the electrical signal, and output a corresponding digital signal. The digital signal output by the image sensor 112 is an original image shot by the camera module 110, where the original image is the raw data. In some embodiments, the raw data may also be referred to as a raw image or a digital negative.

[0092] The image processing module may be configured to may perform a series of processing on the received raw data, and finally obtain an image that may be displayed, for example, a JPEG image. The JPEG image may be transmitted to a display for display, and / or transmitted to a memory for storage.

[0093] A process in which the image processing module 120 processes the raw data to generate the JPEG image may include: performing image signal processing on the raw data; converting the image from a raw domain to a YUV domain, where an image in the YUV domain may be referred to as a YUV image; then, processing the YUV image by using a YUV domain post-processing algorithm; and finally, encoding, in a JPEG encoding manner, a YUV image obtained through YUV domain processing, to obtain the JPEG image.

[0094] For example, the image signal processing on the raw data may include bad pixel correction (DPC), raw domain noise reduction, black level correction (BLC), lens shading correction (LSC), automatic white balance (AWB), demosaic color interpolation, color correction (CCM), dynamic range compression (DRC), gamma correction, 3D look up table (LUT), YUV domain noise reduction, sharpening, detail enhancement, and the like. The YUV domain post-processing algorithm may include multi-frame registration, fusion, and noisereduction of a high-dynamic range (HDR) image, and a super resolution (SR) algorithm, a face beautification algorithm, a distortion correction algorithm, and a blurring algorithm that are used for improving definition.

[0095] In some embodiments, some image processing algorithms in the YUV domain are migrated to the raw domain, so that better image processing effects can be achieved. For example, image processing algorithms such as multi-frame registration, fusion, and noise reduction of the HDR in the YUV domain may be migrated to the raw domain. Advantages of performing image processing in the raw domain may include: A raw image includes higher bit information than a YUV image; and because the raw image does not undergo ISP, information such as colors and details is not damaged, and the like.

[0096] In some embodiments, the cameral module 110 and the image processing module 120 may belong to a single device. For example, an electronic device may include the cameral module 110 and the image processing module 120 (that is, the image processing system 100). The electronic device may be a terminal device. For example, the terminal device may be a mobile phone, a tablet personal computer, a notebook computer, a digital camera, a video camera, a security camera, a vlogging camera, or the like. The cameral module 110 may transmit the raw data to the image processing module 120 via an inner communication system (e.g., an interface and / or a bus). For theses embodiments, the image processing module may be referred to as a local image processing module or a local ISP.

[0097] In some other embodiments, the camera module 110 may upload the raw data to a cloud, and the image processing module 120 may run in the cloud. The image processing module 120 may make full use of big data resources and computing resources in the cloud to perform the raw domain image processing, a format conversion, and YUV domain processing on the raw data to obtain a final image (e.g., the JPEG image) and return the final image to an electronic device that may be used to store or display the final image. An electronic device including the cameral module 110 may further include a communication module. The cameral module 110 may transmit the raw data to the communication module via the inner communication system, and the communication module may transmit the raw data to the image processing module 120 in the cloud via the Internet. Further, in some embodiments, the communication module may be further configured to obtain the final image from the imageprocessing module 120. In some embodiments, the image processing module 120 may transmit the final image to another electronic device or store the final image in a storage module in the cloud. For these embodiments, the image processing module may be referred to as a cloud image processing module or a cloud ISP.

[0098] For convenience, in the following embodiments, a raw data processing method that uses the local ISP may be referred to as a local processing method, while a raw data processing method that uses the cloud ISP may be referred to as a cloud processing method.

[0099] FIG. 2 illustrates a schematic diagram of an electronic device. As shown in FIG. 2, an electronic device 200 may be the terminal device. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, a communication module 250, an audio module 260, a camera module 270, a camera lens 271, an image sensor 272, a display screen 280, and the like. The communication module 250 may include a wireless communication and / or a mobile communication module.

[0100] It may be understood that the schematic structure in this embodiment of this application constitutes no specific limitation on the electronic device 200. In some other embodiments of this application, the electronic device 120 may include more or fewer components than those shown in the figure, or some components may be combined, or some components may be split, or components are arranged in different manners. The components shown in the figure may be implemented by using hardware, software, or a combination of software and hardware.

[0101] The processor 210 may include one or more processing units. For example, the processor 210 may include a general purpose processor, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), and the like. Different processing units may be separate components, or may be integrated into one or more processors.

[0102] The controller may be a nerve center and a command center of the electronic device 200. The controller may generate an operation control signal based on an instruction operation code and a time sequence signal, to complete control of instruction reading and instructionexecution.

[0103] The memory may be further disposed in the processor 210, to store an instruction and data. In some embodiments, the memory in the processor 210 is a cache. The memory may store an instruction or data that is used or cyclically used by the processor 210. If the processor 210 needs to use the instruction or the data again, the processor 210 may directly invoke the instruction or the data from the memory, so as to avoid repeated access, and reduce a waiting time of the processor 210, thereby improving system efficiency.

[0104] In some embodiments, the processor 210 may include one or more interfaces. The interface may be an integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general- purpose input / output (GPIO) interface, a universal serial bus (USB) interface, and / or the like.

[0105] The I2C interface is a two-way synchronization serial bus, and includes one serial data line (serial data line, SDA) and one serial clock line (SCL). In some embodiments, the processor 210 may include a plurality of groups of I2C buses. The processor 210 may be separately coupled to a touch sensor, a charger, a flash, the camera lens 271 , and the like over different I2C bus interfaces. For example, the processor 210 may be coupled to the touch sensor over an I2C interface, so that the processor 210 communicates with the touch sensor over an I2C bus interface, to implement a touch function of the electronic device 200.

[0106] The I2S interface may be configured to perform audio communication. In some embodiments, the processor 210 may include a plurality of groups of I2S buses. The processor 210 may be coupled to the audio module 260 over an I2S bus, to implement communication between the processor 210 and the audio module 260. In some embodiments, the audio module 260 may transmit an audio signal to the communication module 250 over an I2S interface, to implement a function of answering a call over a Bluetooth headset.

[0107] The PCM interface may be also configured to perform audio communication, to perform sampling, quantization, and encoding on an analog signal. In some embodiments, the audio module 260 may be coupled to the communication module 250 over a PCM bus interface. In some embodiments, the audio module 260 may also transmit an audio signal to the communication module 250 over a PCM interface, to implement a function of answering a callover a Bluetooth headset. Both the I2S interface and the PCM interface may be configured to perform audio communication.

[0108] The UART interface is a universal serial data line, and is configured to perform asynchronous communication. The bus may be a two-way communications bus. The UART interface switches to-be-transmitted data between serial communication and parallel communication. In some embodiments, the UART interface is usually configured to connect the processor 210 to the communication module 250. For example, the processor 210 communicates with a Bluetooth module in the communication module 250 over the UART interface, to implement a Bluetooth function. In some embodiments, the audio module 260 may transmit an audio signal to the communication module 250 over the UART interface, to implement a function of playing music over a Bluetooth headset.

[0109] The MIPI interface may be configured to connect the processor 210 to a peripheral component such as the display screen 280 or the camera lens 271. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 210 communicates with the camera lens 271 over the CSI interface, to implement a photographing function of the electronic device 200. The processor 210 communicates with the display screen 280 over the DSI interface, to implement a display function of the electronic device 200.

[0110] The GPIO interface may be configured by using software. The GPIO interface may be configured as a control signal, or may be configured as a data signal. In some embodiments, the GPIO interface may be configured to connect the processor 210 to the camera lens 271, the display screen 280, the communication module 250, the audio module 260, the sensor module 180, and the like. The GPIO interface may be further configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, or the like.

[0111] The USB interface 230 is an interface that meets a USB standard specification, and may be specifically a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 230 may be configured to connect to the charger to charge the electronic device, or may be configured to transmit data between the electronic device 200 and a peripheral device, or may be configured to connect to a headset, to play audio over the headset. The interface may be further configured to connect to another electronic device such as an ARdevice.

[0112] It may be understood that a schematic interface connection relationship between the modules in this embodiment of this application is merely an example for description, and constitutes no limitation on the structure of the electronic device 200. In some other embodiments of this application, the electronic device 200 may alternatively use an interface connection manner different from that in the foregoing embodiment, or use a combination of a plurality of interface connection manners.

[0113] The charging management module 240 is configured to receive a charging input from the charger. The charger may be a wireless charger, or may be a wired charger. In some embodiments in which the charger is a wired charger, the charging management module 240 may receive a charging input from the wired charger over the USB interface 230. In some embodiments in which the charger is a wireless charger, the charging management module 240 may receive a charging input from the wireless charger over a wireless charging coil of the electronic device 200. When charging the battery 242, the charging management module 240 may further supply power to the electronic device over the power management module 241.

[0114] The power management module 241 is configured to connect to the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives an input of the battery 242 and / or the charging management module 240, to supply power to the processor 210, the internal memory 221, an external memory, the display screen 280, the camera lens 271, the communications module 250, and the like. The power management module 241 may be further configured to monitor parameters such as a battery capacity, a battery cycle count, and a battery state of health (electric leakage and impedance). In some other embodiments, the power management module 241 may be alternatively disposed in the processor 210. In some other embodiments, the power management module 241 and the charging management module 240 may be alternatively disposed in a same component.

[0115] A wireless communication function of the electronic device 200 may be implemented by using an antenna 1, an antenna 2, the communications module 250, the modem processor, a baseband processor, and the like.

[0116] The antenna 1 and the antenna 2 are configured to transmit and receive an electromagnetic wave signal. Each antenna of the electronic device 200 may be configured tocover one or more communication frequency bands. Different antennas may be multiplexed to improve utilization of the antennas. For example, the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna may be used in combination with a tuning switch.

[0117] As previously mentioned, in some embodiments, the communications module 250 may include the mobile communication module. The mobile communication module may provide a solution to wireless communication applied to the electronic device 200. The mobile communications module may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communications module may receive an electromagnetic wave over the antenna 1 , perform processing such as filtering and amplification on the received electromagnetic wave, and transmit a processed electromagnetic wave to the modem processor for demodulation. The mobile communications module may further amplify a signal modulated by the modem processor, and convert the signal into an electromagnetic wave for radiation over the antenna 1. In some embodiments, at least some function modules of the mobile communications module may be disposed in the processor 210. In some embodiments, at least some function modules of the mobile communications module and at least some modules of the processor 210 may be disposed in a same component.

[0118] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a to-be-sent low-frequency baseband signal into an intermediate-and- high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. Then, the demodulator transmits the low- frequency baseband signal obtained through demodulation to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal over an audio device (which is not limited to a loudspeaker, a telephone receiver, and the like), or displays an image or a video over the display screen 280. In some embodiments, the modem processor may be an independent component. In some other embodiments, the modem processor may be separate from the processor 210, and the modem processor and the mobile communications module or another function module may be disposed in a same component.

[0119] In some other embodiments, the communication module 250 may include thewireless communication module. The wireless communications module may provide a solution to wireless communication applied to the electronic device 100, for example, a wireless local area network (WLAN) (for example, a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (n NFC) technology, and an infrared (IR) technology. The wireless communications module may be one or more components into which at least one communication processing module is integrated. The wireless communications module receives an electromagnetic wave over the antenna 2, performs frequency modulation and filtering processing on an electromagnetic wave signal, and sends a processed signal to the processor 210. The wireless communications module may further receive a to-be-sent signal from the processor 210, perform frequency modulation and amplification on the signal, and convert the signal into an electromagnetic wave for radiation over the antenna 2.

[0120] In some embodiments, the antenna 1 and the mobile communications module of the electronic device 200 are coupled, and the antenna 2 and the wireless communications module of the electronic device 200 are coupled, so that the electronic device 200 may communicate with a network and another device by using a wireless communications technology. The wireless communications technology may include a global system for mobile communications (GSM), a general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), a fifth generation (5G ) system, a new radio system (NR), a future generation communication systems (e.g., a sixth generation (6G), etc), a BT, a GNSS, a WLAN, NFC, FM, an IR technology, and / or the like. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation system (SB AS).

[0121] The electronic device 200 implements a display function over the GPU, the display screen 280, the application processor, and the like. The GPU is a microprocessor for image processing, and connects to the display screen 280 and the application processor. The GPU is configured to perform mathematical and geometrical calculation, and is configured to perform graphics rendering. The processor 210 may include one or more GPUs, and execute a programinstruction to generate or change display information.

[0122] The display screen 280 is configured to display an image, a video, and the like. The display screen 280 includes a display panel. The display panel may use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLed, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), and the like. In some embodiments, the electronic device 200 may include one or more display screens 280.

[0123] The electronic device 200 may implement a photographing function over the camera module 270, the video codec, the GPU, the display screen 280, the application processor, and the like.

[0124] In some embodiments, the ISP in the electronic device 200 may be configured to process data fed back by the camera module 270. For example, during photo taking, a shutter is pressed, a ray of light is transmitted to a light-sensitive element of the camera lens 271 through a lens, an optical signal is converted into an electrical signal (that is the aforementioned raw data) by the image sensor 272, and the image sensor 272 transmits the raw data to the ISP for processing, and the ISP converts the raw data into a final image. In some embodiments, the ISP may further perform algorithm optimization on noise, luminance, and complexion of the final image. The ISP may further optimize parameters such as exposure and a color temperature of a photographing scenario. In some embodiments, the ISP may be disposed in the camera module 270.

[0125] In some other embodiments, the communication module 250 may obtain the raw data from the cameral module 270 and transmit the obtained raw data to the cloud ISP. The cloud ISP may process the raw data and transmit the final image to the electronic device 200. The communication module 250 may receive the final image from the cloud ISP and transmit the final image to a memory (e.g., the internal memory 221).

[0126] In some embodiments, the ISP in the electronic device 200 or the cloud ISP may be the Al-based IPS. The IPS or the cloud ISP may process the raw data by using a neural network, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or the like.

[0127] The camera module 270 is configured to capture a static image or a video. An optical image of a body is generated through the camera lens 271, and the optical image is projected tothe camera sensor 272 to obtain the raw data. The camera sensor 272 may be a CCD sensor or a CIS. The camera sensor 272 converts an optical signal into an electrical signal (that is, the raw data). As previously mentioned, in some embodiments, the camera sensor 272 may transmit the raw data to the ISP in the electronic device, so that the ISP converts the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV.

[0128] The digital signal processor is configured to process a digital signal, and in addition to a digital image signal, may further process another digital signal. For example, when the electronic device 200 performs frequency selection, the digital signal processor is configured to perform Fourier transform and the like on frequency energy.

[0129] The video codec is configured to compress or decompress a digital video. The electronic device 200 may support one or more video codecs. In this way, the electronic device 200 may play or record videos in a plurality of encoding formats, for example, moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0130] The NPU is a neural-network (NN) computing processor. The NPU may be used to implement an application such as intelligent cognition of the electronic device 200, for example, image recognition, facial recognition, voice recognition, and text understanding.

[0131] The external memory interface 220 may be configured to connect to an external storage card such as a micro SD card, to extend a storage capability of the electronic device 200. The external storage card communicates with the processor 210 over the external memory interface 220, to implement a data storage function, for example, to store files such as music and videos in the external storage card.

[0132] The internal memory 221 may be configured to store computer executable program code, and the executable program code includes an instruction. The processor 210 runs the instruction stored in the internal memory 221, to perform various function applications and data processing of the electronic device 200. The internal memory 221 may include a program storage region and a data storage region. The program storage region may store an operating system, an application required by at least one function (for example, a voice playing function or an image playing function), and the like. The data storage region may store data (for example,audio data and an address book) and the like created when the electronic device 200 is used. In addition, the internal memory 221 may include a high-speed random access memory, or may include a non-volatile memory such as at least one magnetic disk memory, a flash memory, or a universal flash storage (UFS).

[0133] The local ISP and the cloud ISP may be the Al-based ISP. The Al-based ISP may process the raw data by using the neural network (NN). Therefore, the Al-based ISP may be referred to as a NN-based ISP. The neural network may include neurons. A neuron may be an operation unit that uses xsand an intercept 1 as inputs, and an output of the operation unit may be as follows:

[0135] Herein, s=l, 2, ..., or n, n is a natural number greater than 1, Wsis a weight of xs, and b is bias of the neuron, f is an activation function of the neuron, and the activation function is used to introduce a non-linear feature into the neural network, and to convert an input signal in the neuron into an output signal. The output signal of the activation function may be used as an input of a next convolutional layer. The activation function may be a sigmoid function. The neural network is a network formed by connecting many single neurons together. To be specific, an output of a neuron may be an input of another neuron. An input of each neuron may be connected to a local receptive field of a previous layer to extract a feature of the local receptive field. The local receptive field may be a region including several neurons.

[0136] A deep neural network (DNN), also referred to as a multi-layer neural network, may be understood as a neural network having many hidden layers. The “many” herein does not have a special measurement standard. The DNN is divided based on locations of different layers, and a neural network in the DNN may be divided into three types: an input layer, a hidden layer, and an output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. Layers are fully connected. To be specific, any neuron at the ithlayer is certainly connected to any neuron at the (i+l)thlayer. Although the DNN looks to be complex, the DNN is actually not complex in terms of work at each layer, and is simply expressed as the following linear relationship:, where x is an input vector, y is an output vector, b is a bias vector, W is a weight matrix (also referred to as a coefficient),and a( ) is an activation function. At each layer, the output vector y is obtained by performing such a simple operation on the input vector x . Because there are many layers in the DNN, there are also many coefficients W and bias vectors b. Definitions of these parameters in the DNN are as follows: The coefficient W is used as an example. It is assumed that in a DNN having three layers, a linear coefficient from the fourth neuron at the second layer to the second neuron at the third layer is defined as w’4. The superscript 3 represents a layer at which the coefficient W is located, and the subscript 24 corresponds to an output third-layer index 2 and an input second-layer index 4. In conclusion, a coefficient from the kthneuron at the (L— l)thlayer to the jthneuron at the Lthlayer is defined as WJlk. It should be noted that there is no parameter W at the input layer. In the deep neural network, more hidden layers make the network more capable of describing a complex case in the real world.

[0137] Theoretically, a model with a larger quantity of parameters indicates higher complexity and a larger “capacity”, and indicates that the model can complete a more complex learning task. Training the deep neural network is a process of learning a weight matrix, and a final objective of the training is to obtain a weight matrix of all layers of the trained deep neural network (a weight matrix including vectors W at many layers).

[0138] A convolutional neural network (CNN) is a deep neural network having a convolutional structure. The convolutional neural network includes a feature extractor including a convolutional layer and a sub sampling layer. The feature extractor may be considered as a filter. A convolution process may be considered as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map). The convolutional layer is a neuron layer that is in the convolutional neural network and at which convolution processing is performed on an input signal. At the convolutional layer of the convolutional neural network, one neuron may be connected only to some adjacent-layer neurons. A convolutional layer usually includes a plurality of feature planes, and each feature plane may include some neurons arranged in a rectangular form. Neurons in a same feature plane share a weight. The shared weight herein is a convolution kernel. Weight sharing may be understood as that an image information extraction manner is irrelevant to a location. Aprinciple implied herein is that statistical information of a part of an image is the same as that of another part. This means that image information learned in a part can also be used in another part. Therefore, image information obtained through the same learning can be used for all locations in the image. At a same convolutional layer, a plurality of convolution kernels may be used to extract different image information. Usually, a larger quantity of convolution kernels indicate richer image information reflected by a convolution operation.

[0139] The convolution kernel may be initialized in a form of a random-size matrix. In a process of training the convolutional neural network, the convolution kernel may obtain an appropriate weight through learning. In addition, a direct benefit brought by weight sharing is that connections between layers of the convolutional neural network are reduced and an overfitting risk is lowered.

[0140] A recurrent neural network (RNN) is used to process sequence data. In a conventional neural network model, from an input layer to a hidden layer and then to an output layer, the layers are fully connected, and nodes at each layer are not connected. Such a common neural network resolves many difficult problems, but is still incapable of resolving many other problems. For example, if a word in a sentence is to be predicted, a previous word usually needs to be used, because adjacent words in the sentence are not independent. A reason why the RNN is referred to as the recurrent neural network is that a current output of a sequence is also related to a previous output of the sequence. A specific representation form is that the network memorizes previous information and applies the previous information to calculation of the current output. To be specific, nodes at the hidden layer are connected, and an input of the hidden layer not only includes an output of the input layer, but also includes an output of the hidden layer at a previous moment. Theoretically, the RNN can process sequence data of any length. Training for the RNN is the same as training for a conventional CNN or DNN. An error back propagation algorithm is also used, but there is a difference: if the RNN is expanded, a parameter such as W of the RNN is shared. This is different from the conventional neural network described in the foregoing example. In addition, during use of a gradient descent algorithm, an output in each step depends not only on a network in the current step, but also on a network status in several previous steps. The learning algorithm is referred to as a back propagation through time (BPTT) algorithm.

[0141] Now that there is a convolutional neural network, why is the recurrent neural network required? A reason is simple. In the convolutional neural network, it is assumed that elements are independent of each other, and an input and an output are also independent, such as a cat and a dog. However, in the real world, many elements are interconnected. For example, stocks change with time. For another example, a person says: I like traveling, and my favorite place is Yunnan. I will go if there is a chance. If there is blank filling, people should know that “Yunnan” will be filled in the blank. A reason is that the people can deduce the answer based on content of the context. However, how can a machine do this? The RNN emerges. The RNN is intended to make the machine capable of memorizing like a human. Therefore, an output of the RNN needs to depend on current input information and historical memorized information.

[0142] In a process of training the deep neural network, because it is expected that an output of the deep neural network is close to a predicted value that is actually expected as much as possible, a predicted value of a current network and a target value that is actually expected may be compared, and then a weight vector of each layer of the neural network is updated based on a difference between the predicted value and the target value (certainly, there is usually an initialization process before the first update, to be specific, parameters are preconfigured for all layers of the deep neural network). For example, if the predicted value of the network is large, the weight vector is adjusted to decrease the predicted value, and adjustment is continuously performed until the deep neural network can predict the target value that is actually expected or a value that is very close to the target value that is actually expected. Therefore, “how to obtain, through comparison, a difference between a predicted value and a target value” needs to be predefined. This is a loss function or an objective function. The loss function and the objective function are important formulas used to measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training of the deep neural network is a process of minimizing the loss as much as possible.

[0143] The convolutional neural network may correct a value of a parameter in an initial super-resolution model in a training process according to an error back propagation (BP) algorithm, so that an error loss of reconstructing the super-resolution model becomes smaller. Specifically, an input signal is transferred forward until an error loss occurs at an output, andthe parameter in the initial super-resolution model is updated based on back propagation error loss information, to make the error loss converge. The back propagation algorithm is an error- loss-centered back propagation motion intended to obtain a parameter, such as a weight matrix, of an optimal super-resolution model.

[0144] FIG. 3 illustrates a diagram of an image signal processing procedure provided by some embodiments of the present application.

[0145] Referring to FIG. 3, an image sensor 301 may be configured to obtain raw data and transmit the raw data to a preprocessing module 302. The preprocessing module 302 may be configured to process the obtained raw data to obtain transformed data. An encoding module 303 may be configured to perform an encoding operation on the transformed data to obtain encoded data. The encoded data may be transmitted to a decoding module 311 via a bus or the Internet. The decoding module 311 may be configured to perform a decoding operation on the encoded data to obtain decoded data. A postprocessing module 312 may be configured to obtain the decoded data and process the obtained decoded data to obtain reconstructed data. The ISP 313 may be configured to perform a series of processing on the reconstructed data and obtain an image that may be displayed, for example, a JPEG image. When the ISP 313 is a local ISP, the ISP 313 may transmit the JPEG image to a storage module of an electronic device including the image sensor 301 and the ISP 313 via the BUS. When the ISP 303 is a cloud ISP, the ISP 313 may transmit the JPEG image to an electronic device including the image sensor 301 via the Internet, and the electronic device including the image sensor 301 may store and display the JPEG image.The raw data may include a plurality of first pixels and a plurality of second pixels. The plurality of second pixels may be kept unchanged, and each of the first pixels may be transformed according to its neighboring pixels by the preprocessing module 302. The transformed data may include the plurality of second pixels and the transformed first pixels. For convenience, the transformed pixel may be referred to as a third pixel. The transform operation performed by the preprocessing module 302 may smooth noise of the raw data, which improves a compression ration of the encoding module 302 when encoding the transformed data. The postprocessing module 312 may perform an inverse transform operation corresponding to the transform operation performed by the preprocessing module 302. In other word, the transform operationperformed by the preprocessing module 302 is reversible . In other words, the original pixels in the raw data may be recovered from the transformed data. For convenience, a term “transform operation” may be referred to the transform operation performed by the preprocessing module 302. In other words, the transform operation is an operation that is used for obtaining the transformed data. A term “inverse transform operation” may be referred to the transform operation performed by the processed module 312. In other words, the inverse transform operation is an operation that is used to transform the transformed data back to the raw data. In order to distinguish between the raw data obtained by the image processor 301 and the raw data obtained by the inverse transform operation (that is, an output of the postprocessing module 312), the raw data obtained by the inverse transform operation may be referred to as reconstructed data.

[0146] The raw data obtained by the image sensor 301 may include noise data. The transform operation may reduce or remove the noise data. Since the raw data may be recovered by the inverse transform operation, the noise data in the raw data may be also recovered. When the ISP 313 is the Al-based ISP, the noise data in the raw data is important, since a neural network that is for processing the raw data by the Al -based ISP may be trained in presence of noise. Without the noise data, an output of the Al-based ISP may be out of an original distribution, and the neural network behavior may become unexpected.

[0147] For convenience, the image signal processing procedure illustrated in FIG. 3 may be divided into two procedures, a transform procedure and an inverse transform procedure. The transform procedure includes operations performed by the preprocessing module 302 and the encoding module 303, while the inverse transform procedure includes operations performed by the decoding module 311 and the postprocessing module 312. Referring to FIG. 2, the preprocessing module 302 and the encoding module 303 may be the general purpose processor. When the ISP 313 is the local ISP, the ISP 313 may be the ISP in the electronic device 200, the postprocessing module 312 and the decoding module 311 may be the general purpose processor. When the ISP 313 is the cloud ISP, the ISP 313 may be an ISP in the cloud, the postprocessing module 312 may be, and the decoding module 311 may be a general purpose processor.

[0148] The transform procedure is introduced blow through FIG. 4, and the inverse transform procedure is introduced below through FIG. 15. The inverse transform procedureillustrated in FIG. 15 is an inverse operation of the transform procedures illustrated in FIG. 4.

[0149] FIG. 4 illustrates an image processing method according to some embodiments of the present application. The image processing method illustrated in FIG. 4 may be performed by an encoding device, and the encoding device may be an electronic device. The electronic device may be a mobile phone, a tablet personal computer, a notebook computer, a digital camera, a video camera, a security camera, a vlogging camera, or the like. In some other embodiments, the image processing method illustrated in FIG. 4 may be performed by a component in the encoding device. For example, the image processing method may be performed by a processor, a system on chip (SOC), a processing circuit, or the like. For convenience, it is assumed that the method illustrated in FIG. 4 is performed by the encoding device.

[0150] 401, The encoding device obtains input data corresponding to raw data.

[0151] In some embodiments, the input data is the raw data. In other words, the encoding device may receive the raw data obtained by an image sensor of the encoding device.

[0152] In some other embodiments, the input data may be processed raw data. For example, after obtaining the raw data from the image sensor, the encoding device may perform an image signal processing on the raw data. The image signal processing may include one or more of the following: gamma correction, color correction, automatic white balance, or the like. The input data may be a processing result of the image signal processing.

[0153] 402, The encoding device may perform a transform operation on the input data to obtain a transformed data.

[0154] 403, The encoding device may perform an encoding operation on the transformed data to obtain an encoded data.

[0155] In some embodiments, before performing the encoding operation, the encoding device may perform the image signal processing on the transformed data. For example, the encoding device may perform the gamma correction on the raw data to obtain the input data and perform the color correction on the transformed data. Then, the encoding device performs the encoding operation.

[0156] 404, The encoding device may transmit the encoded data to a decoding device.

[0157] The decoding device may be an electronic device including an ISP. When the ISP isa local ISP, the decoding device is the encoding device.

[0158] The input data may include a plurality of first pixels and a plurality of second pixels. The transform operation is used to transform each of the first pixels, and the transformed first pixels may be referred to as third pixels. Meanwhile, the second pixels may be kept unchanged. Therefore, the transformed data may include the plurality of second pixels and a plurality of third pixels. The plurality of first pixels are in one-to-one correspondence with the plurality of third pixels. Each of the third pixels is a transform result of a corresponding first pixels.

[0159] In some embodiments, the encoding device may divide the input data into several blocks. For convenience, it is assumed that the encoding device may divided the input data into F blocks, where F is a positive integer greater than two.

[0160] In some embodiments, the encoding device may divide the input data by using a pattern having a fixed size. For example, a size of the pattern is PxQ (P -column by Q-row), where Q and P are positive integer greater than or equal to two. In some embodiments, the input data may be divided exactly by the pattern. In other words, a size of each of the F blocks is PxQ.

[0161] FIG. 5 illustrates a dividing result of an input data according to some embodiments of the present application. Referring to FIG. 5, the input data is divided into 25 blocks, and the 25 blocks are arranged in 5 rows and 5 columns. Each of the 25 blocks has PxQ pixels.

[0162] In some other embodiments, the input data may be not divisible by the pattern. In these embodiments, sizes of some blocks among the F blocks are PXQ, while the sizes of other blocks among the F blocks are smaller than PxQ.

[0163] FIG. 6 illustrates a dividing result of another input data according to some embodiments of the present application. Referring to FIG. 6, the input data is divided into 25 blocks, and the 25 blocks are arranged in 5 rows and 5 columns. Blocks in the first column to the fourth columns have the same size, which is PxQ. Sizes of blocks in the fifth columns is less than PxQ, which is (P / 2)xQ.

[0164] For convenience, a block with size PxQ may be referred to as a first block, whereas a block with size smaller than PxQ may be referred to as a second block. It is assumed that the F blocks include F i first blocks, where Fi is a positive integer, F i is less than or equal to F. When Fi is less than F, the F blocks may further include F2 second block(s), where F2 is a positive integer, and a sum of Fi and F2 is equal to F.

[0165] Referring to FIG. 5, the input data may be divided into 25 first blocks. Referring to FIG. 6, the input data may be divided into 20 first blocks and 5 second blocks.

[0166] In some other embodiments, the encoding device may divide the input data by using two or more patterns of varying sizes. Therefore, the F blocks may include two or more groups of blocks, and blocks belonging to the same group have the same size. FIG. 6 illustrates a dividing result of another input data according to some embodiments of the present application. The input data is divided into several blocks by using four patterns. Sizes of the four patterns are (2xP)x(2*Q), PxQ, (P / 2)xQ, and Px(Q / 2) respectively.

[0167] Regardless of whether the input data is divided by one or more patterns, the transform operation on each of the F blocks is similar. For convenience, the following embodiments assume that the input data is divided by one pattern with the size PxQ.

[0168] In some embodiments, the transform operation is performed on both of the first blocks and the second blocks. In some other embodiments, the transform operation is only performed on the first blocks. In other words, transform operation is not perform on the second blocks. For convenience, the blocks on which the transform operation is performed may be referred to as a target block, the following embodiments assume that the target blocks include the first blocks. In other word, the F blocks include Fi target blocks.

[0169] Each of the target blocks includes several first pixels and several second pixels. The purpose of the transform operation is transforming the first pixels into third pixels, and the transforming result is the transformed block. There are several approaches for transforming the first pixels into the third pixels.

[0170] Approach 1:

[0171] For the approach 1, pixels in a top row and a leftmost column of the target block are the second pixels, and other pixels are the first pixels. In other words, pixels in a first row and a first column of the target block are the second pixels, while other pixels in the target block, except for those in the first row and first column, are the first pixels. For convenience, this target block (that is, a target block in which pixels in the top row and the leftmost column are the second pixels) may be referred to as “target block Al”. The first row of the target block is the top row of the target block, the first column of the target block is the leftmost column of the target block, a last row of the target block is a bottom row of the target block, and a last columnof the target block is the rightmost column of the target block.

[0172] Each of the first pixels in the target block and its three neighboring pixels are used to determine a third pixel. For example, it is assumed that a size of the target block is P*Q (P- column by Q-row), then the target block may include (P-l)x(Q-l) first pixels, and P+Q-l second pixels. A transformed block determined according to the target block may include (P- 1)X(Q-1) third pixels, and P+Q-l second pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the target block includes P’xQ’ first pixels, and the transformed block determined according to the target block includes P’xQ’ third pixels. The P’xQ’ third pixels are in one-to-one correspondence with the P’xQ’ first pixels, and a location of each of the third pixels and a location of a corresponding first pixel are the same. Each of the third pixels is determined according to the corresponding first pixel and neighboring pixels of the corresponding first pixel. The neighboring pixels of the corresponding first pixel may include a left neighboring pixel of the corresponding first pixel, a top-left neighboring pixel of the corresponding first pixel, and a top neighboring pixel of the corresponding first pixel. Pixel values of the third pixel, the corresponding first pixel, and the neighboring pixels of the corresponding first pixel satisfy:

[0173] V_TP=(WOXV_FP+WI XV_FPL+W2XV_FPT_L+W3XV_FPT) / (WO+WI+W2+W3), (1.1)

[0174] wherein V_TP is a pixel value of the third pixel, V_FP is a pixel value of the corresponding first pixel, V_FPL is a pixel value of the left neighboring pixel of the corresponding first pixel, V_FPT_L is a pixel value of a top-left neighboring pixel of the corresponding first pixel, V_FPT is a pixel value of a top neighboring pixel of the corresponding first pixel, wo is a weight of the first pixel, wi is a weight of the left neighboring pixel, W2 is a weight of the top-left neighboring pixel, and W3 is a weight of the top neighboring pixel.

[0175] FIG. 7 (a) illustrates an example of a target block, and FIG. 7 (b) illustrates an example of a transformed block obtained according to the target block illustrated in FIG. 7(a).

[0176] Referring to FIG. 7(a), the target block includes 6><5 pixels. Pixels in a first row and a first column of the target block are the second pixels. Therefore, the 6x5 pixels includes 6+5- 1 second pixels, and 5 x4 first pixels. FP denotes the first pixel, and SP denotes the second pixel. FPx,y represents the first pixel located at a ythcolumn and a xthrow in the target block, SPa,p represents the second pixel located at a [3thcolumn and a athrow in the target block. Referringto FIG. 7 (a), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 2 to 6. a and P are positive integer, possible values of a are 1 to 5, and possible values of P are 1 to 6.

[0177] Referring to FIG. 7(b), the transformed block includes 6x5 pixels. Pixels in a first row and a first column of the transformed block are the second pixels. Therefore, the 6x5 pixels includes 6+5-1 second pixels, and 5x4 third pixels. TP denotes the third pixel, and SP denotes the second pixel. TPx>yrepresents the third pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the second pixel located at a pthcolumn and a athrow in the transformed block. Referring to FIG. 7 (b), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 2 to 6. a and P are positive integer, possible values of a are 1 to 5, and possible values of P are 1 to 6.

[0178] The transform operation is not performed on the second pixels. Therefore, the second pixels in the transformed block are the same as the second pixels in the target block. In other words, a value of a second pixel SPi,i in FIG. 7 (a) is the same as a value of a second pixel SPi.i in FIG. 7 (b), a value of a second pixel SPi,2 in FIG. 7 (a) is the same as a value of a second pixel SPi,2 in FIG. 7 (b), a value of a second pixel SPi,3 in FIG. 7 (a) is the same as a value of a second pixel SPi,3 in FIG. 7 (b), and so on.

[0179] The 5x4 first pixels in FIG. 7 (a) are in one-to-one correspondence with the 5x4 third pixels in FIG. 7 (b). As previously mentioned, the location of the third pixel is the same as the location of the corresponding first pixel. Therefore, a third pixel TP2,2 corresponds to a first pixel FP2,2, a third pixel TP2 corresponds to a first pixel FP2 , and so on. A value of each of the third pixels is determined according to values of the corresponding first pixel and neighboring pixels of the corresponding first pixel. For example, a value of the third pixel TP2,2 is determined according to a value of the first pixel FP2.2, a value of a left neighboring pixel of the first pixel FP2.2 (that is, a second pixel SP2,i), a value of a top-left neighboring pixel of the first pixel FP2,2 (that is, a second pixel SPi.i), and a value of a top neighboring pixel of the first pixel FP2.2 (that is, a second pixel SPi,2). Further, the value of the third pixel TP2.2, the value of the first pixel FP2,2, the value of the second pixel SP2,i, the value of the second pixel SPi.i, and the value of the second pixel SPi,2 satisfy the formula 1.1. For another example, a value of the pixel TP3 is determined according to a value of a first pixel FP3 , a value of a left neighboringpixel of the first pixel FP3 (that is, a first pixel FPa^), a value of a top-left neighboring pixel of the first pixel FP3 (that is, the first pixel FP2,2), and a value of a top neighboring pixel of the first pixel FP3 (that is, the first pixel FP2 ). The value of the third pixel TP3 , the value of the first pixel FP3 , the value of the first pixel FP3.2, the value of the first pixel FP2,2, and the value of the first pixel FP2 satisfy the formula 1.1.

[0180] Approach 2:

[0181] For the approach 2, pixels in a bottom row and a rightmost column of the target block are the second pixels, and the other pixels are the first pixels. In other words, pixels in a last row and a last column of the target block are the second pixels, while other pixels in the target block, except for those in the last row and the last column, are the first pixels. For convenience, this target block (that is, a target block in which pixels in the bottom row and the rightmost column are the second pixels) may be referred to as “target block A2”. The first row of the target block is the top row of the target block, the first column of the target block is the leftmost column of the target block, a last row of the target block is a bottom row of the target block, and a last column of the target block is the rightmost column of the target block.

[0182] Each of the first pixels in the target block and its three neighboring pixels are used to determine a third pixel. For example, it is assumed that a size of the target block is P*Q (P- column by Q-row), then the target block may include (P-I)x(Q-l) first pixels, and P+Q-l second pixels. A transformed block determined according to the target block may include (P- l)x(Q-l) third pixels, and P+Q-l second pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the target block includes P’xQ’ first pixels, and the transformed block determined according to the target block includes P’xQ’ third pixels. The P’xQ’ third pixels are in one-to-one correspondence with the P’xQ’ first pixels, and a location of each of the third pixels and a location of a corresponding first pixel are the same. Each of the third pixels is determined according to the corresponding first pixel and neighboring pixels of the corresponding first pixel. The neighboring pixels of the corresponding first pixel may include a right neighboring pixel of the corresponding first pixel, a bottom-right neighboring pixel of the corresponding first pixel, and a bottom neighboring pixel of the corresponding first pixel. Pixel values of the third pixel, the corresponding first pixel, and the neighboring pixels of the corresponding first pixel satisfy:

[0183] V_TP=(woxV_FP+wiXV_FPR+W2XV_FPB_R+W3XV_FPB) / (WO+WI+W2+W3), (1.2)

[0184] wherein V_TP is a pixel value of the third pixel, V_FP is a pixel value of the corresponding first pixel, V_FPR is a pixel value of the right neighboring pixel of the corresponding first pixel, V_FPB_R is a pixel value of a bottom-right neighboring pixel of the corresponding first pixel, V_FPB is a pixel value of a bottom neighboring pixel of the corresponding first pixel, wo is a weight of the first pixel, wi is a weight of the right neighboring pixel, W2 is a weight of the bottom-right neighboring pixel, and W3 is a weight of the bottom neighboring pixel.

[0185] FIG. 8 (a) illustrates an example of a target block, and FIG. 8 (b) illustrates an example of a transformed block obtained according to the target block illustrated in FIG. 8(a).

[0186] Referring to FIG. 8(a), the target block includes 6X5 pixels. Pixels in a last row and a last column of the target block are the second pixels. Therefore, the 6X5 pixels includes 6+5- 1 second pixels, and 5><4 first pixels. FP denotes the first pixel, and SP denotes the second pixel. FPx,yrepresents the first pixel located at a ythcolumn and a xthrow in the target block, SPa,p represents the second pixel located at a 0thcolumn and a athrow in the target block. Referring to FIG. 8 (a), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0187] Referring to FIG. 8(b), the transformed block includes 6><5 pixels. Pixels in a last row and a last column of the transformed block are the second pixels. Therefore, the 6x5 pixels includes 6+5-1 second pixels, and 5X4 third pixels. TP denotes the third pixel, and SP denotes the second pixel. TPx,yrepresents the third pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the second pixel located at a 0thcolumn and a athrow in the transformed block. Referring to FIG. 8 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0188] The transform operation is not performed on the second pixels. Therefore, the second pixels in the transformed block are the same as the second pixels in the target block. In other words, a value of a second pixel SPs.i in FIG. 8 (a) is the same as a value of a second pixel SPs.i in FIG. 8 (b), a value of a second pixel SPs,2 in FIG. 8 (a) is the same as a value of a secondpixel SPs.2 in FIG. 8 (b), a value of a second pixel SP53 in FIG. 8 (a) is the same as a value of a second pixel SP53 in FIG. 8 (b), and so on.

[0189] The 5x4 first pixels in FIG. 8 (a) are in one-to-one correspondence with the 5x4 third pixels in FIG. 8 (b). As previously mentioned, the location of the third pixel is the same as the location of the corresponding first pixel. Therefore, a third pixel TPia corresponds to a first pixel FP2.2, a third pixel TP23 corresponds to a first pixel FP23, and so on. A value of each of the third pixels is determined according to values of the corresponding first pixel and neighboring pixels of the corresponding first pixel. For example, a value of the third pixel TP?^ is determined according to a value of the first pixel FPi^, a value of a right neighboring pixel of the first pixel FP2.2 (that is, a first pixel FP23), a value of a bottom-right neighboring pixel of the first pixel F?2^ (that is, a first pixel FP33), and a value of a bottom neighboring pixel of the first pixel FP2.2 (that is, a first pixel FPa^). Further, the value of the third pixel TP2.2, the value of the first pixel FP2A the value of the first pixel FPy , the value of the first pixel FP33, and the value of the first pixel FPs^ satisfy the formula 1.2. For another example, a value of the pixel TP33 is determined according to a value of a first pixel FP33, a value of a right neighboring pixel of the first pixel FP33 (that is, a first pixel FPs,4), a value of a bottom-right neighboring pixel of the first pixel FP33 (that is, the first pixel FP4.4), and a value of a bottom neighboring pixel of the first pixel FP33 (that is, the first pixel FP43). The value of the third pixel TP33, the value of the first pixel FP33, the value of the first pixel FP3.4, the value of the first pixel FP4.4, and the value of the first pixel FP43 satisfy the formula 1.2.

[0190] Approach 3:

[0191] For the approach 3, pixels in a bottom row and a leftmost column of the target block are the second pixels, and other pixels are the first pixels. In other words, pixels in a last row and a first column of the target block are the second pixels, while other pixels in the target block, except for those in the last row and first column, are the first pixels. For convenience, this target block (that is, a target block in which pixels in the bottom row and the leftmost column are the second pixels) may be referred to as “target block A3”. The first row of the target block is the top row of the target block, the first column of the target block is the leftmost column of the target block, a last row of the target block is a bottom row of the target block, and a last column of the target block is the rightmost column of the target block.

[0192] Each of the first pixels in the target block and its three neighboring pixels are used to determine a third pixel. For example, it is assumed that a size of the target block is PxQ (P- column by Q-row), then the target block may include (P-l)x(Q-l) first pixels, and P+Q-l second pixels. A transformed block determined according to the target block may include (P- l)x(Q-l) third pixels, and P+Q-l second pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the target block includes P’xQ’ first pixels, and the transformed block determined according to the target block includes P’xQ’ third pixels. The P’xQ’ third pixels are in one-to-one correspondence with the P’xQ’ first pixels, and a location of each of the third pixels and a location of a corresponding first pixel are the same. Each of the third pixels is determined according to the corresponding first pixel and neighboring pixels of the corresponding first pixel. The neighboring pixels of the corresponding first pixel may include a left neighboring pixel of the corresponding first pixel, a bottom-left neighboring pixel of the corresponding first pixel, and a bottom neighboring pixel of the corresponding first pixel. Pixel values of the third pixel, the corresponding first pixel, and the neighboring pixels of the corresponding first pixel satisfy:

[0193] V_TP=(wox V_FP+wi x V_FPL+W2X V_FPB_L+W3XV_FPB) / (WO+WI+W2+W3), (1.3)

[0194] wherein V_TP is a pixel value of the third pixel, V_FP is a pixel value of the corresponding first pixel, V_FPL is a pixel value of the left neighboring pixel of the corresponding first pixel, V_FPB_L is a pixel value of a bottom -left neighboring pixel of the corresponding first pixel, V_FPB is a pixel value of a bottom neighboring pixel of the corresponding first pixel, wo is a weight of the first pixel, wi is a weight of the left neighboring pixel, W2 is a weight of the bottom -left neighboring pixel, and W3 is a weight of the bottom neighboring pixel.

[0195] FIG. 9 (a) illustrates an example of a target block, and FIG. 9 (b) illustrates an example of a transformed block obtained according to the target block illustrated in FIG. 9(a).

[0196] Referring to FIG. 9(a), the target block includes 6x5 pixels. Pixels in a last row and a first column of the target block are the second pixels. Therefore, the 6x5 pixels includes 6+5- 1 second pixels, and 5x4 first pixels. FP denotes the first pixel, and SP denotes the second pixel. FPx,y represents the first pixel located at a ythcolumn and a xthrow in the target block, SPa,p represents the second pixel located at a 0thcolumn and a athrow in the target block. Referringto FIG. 9 (a), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 2 to 6. a and are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0197] Referring to FIG. 9(b), the transformed block includes 6x5 pixels. Pixels in a last row and a first column of the transformed block are the second pixels. Therefore, the 6x5 pixels includes 6+5-1 second pixels, and 5x4 third pixels. TP denotes the third pixel, and SP denotes the second pixel. TPx,y represents the third pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the second pixel located at a 0thcolumn and a athrow in the transformed block. Referring to FIG. 9 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 2 to 6. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0198] The transform operation is not performed on the second pixels. Therefore, the second pixels in the transformed block are the same as the second pixels in the target block. In other words, a value of a second pixel SPi.i in FIG. 9 (a) is the same as a value of a second pixel SP i, i in FIG. 9 (b), a value of a second pixel SP2,i in FIG. 9 (a) is the same as a value of a second pixel SP2,i in FIG. 9 (b), a value of a second pixel SP3,i in FIG. 9 (a) is the same as a value of a second pixel SP3,i in FIG. 9 (b), and so on.

[0199] The 5x4 first pixels in FIG. 9 (a) are in one-to-one correspondence with the 5x4 third pixels in FIG. 9 (b). As previously mentioned, the location of the third pixel is the same as the location of the corresponding first pixel. Therefore, a third pixel TP2.2 corresponds to a first pixel FP2,2, a third pixel TP2 corresponds to a first pixel FP2 , and so on. A value of each of the third pixels is determined according to values of the corresponding first pixel and neighboring pixels of the corresponding first pixel. For example, a value of the third pixel TP2.2 is determined according to a value of the first pixel FP2,2, a value of a left neighboring pixel of the first pixel FP2,2 (that is, a second pixel SP2,i), a value of a bottom-left neighboring pixel of the first pixel FP2.2 (that is, a second pixel SPs.i), and a value of a bottom neighboring pixel of the first pixel FP2,2 (that is, a first pixel FP3,2). Further, the value of the third pixel TP2.2, the value of the first pixel FP2,2, the value of the second pixel SP2.1, the value of the second pixel SP3 , and the value of the first pixel FP3.2 satisfy the formula 1.3. For another example, a value of the pixel TP3 is determined according to a value of a first pixel FP3 , a value of a leftneighboring pixel of the first pixel FP3 (that is, a first pixel FP3.2), a value of a bottom-left neighboring pixel of the first pixel FP3 (that is, the first pixel FP4.2), and a value of a bottom neighboring pixel of the first pixel FP3 (that is, the first pixel FP4 ). The value of the third pixel TP3 , the value of the first pixel FP3 , the value of the first pixel FP3.2, the value of the first pixel FP4,2, and the value of the first pixel FP4 satisfy the formula 1.3.

[0200] Approach 4:

[0201] For the approach 4, pixels in a top row and a rightmost column of the target block are the second pixels, and the other pixels are the first pixels. In other words, pixels in a first row and a last column of the target block are the second pixels, while other pixels in the target block, except for those in the first row and the last column, are the first pixels. For convenience, this target block (that is, a target block in which pixels in the top row and the rightmost column are the second pixels) may be referred to as “target block A4”. The first row of the target block is the top row of the target block, the first column of the target block is the leftmost column of the target block, a last row of the target block is a bottom row of the target block, and a last column of the target block is the rightmost column of the target block.

[0202] Each of the first pixels in the target block and its three neighboring pixels are used to determine a third pixel. For example, it is assumed that a size of the target block is PxQ (P- column by Q-row), then the target block may include (P-l)x(Q-l) first pixels, and P+Q-l second pixels. A transformed block determined according to the target block may include (P- l)x(Q-l) third pixels, and P+Q-l second pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the target block includes P’xQ’ first pixels, and the transformed block determined according to the target block includes P’xQ’ third pixels. The P’xQ’ third pixels are in one-to-one correspondence with the P’xQ’ first pixels, and a location of each of the third pixels and a location of a corresponding first pixel are the same. Each of the third pixels is determined according to the corresponding first pixel and neighboring pixels of the corresponding first pixel. The neighboring pixels of the corresponding first pixel may include a right neighboring pixel of the corresponding first pixel, a top-right neighboring pixel of the corresponding first pixel, and a top neighboring pixel of the corresponding first pixel. Pixel values of the third pixel, the corresponding first pixel, and the neighboring pixels of the corresponding first pixel satisfy:

[0203] V_TP=(woxV_FP+wi x V_FPR+W2XV_FPT_R+W3XV_FPT) / (WO+WI+W2+W3), (1.4)

[0204] wherein V_TP is a pixel value of the third pixel, V_FP is a pixel value of the corresponding first pixel, V_FPR is a pixel value of the right neighboring pixel of the corresponding first pixel, V_FPT_R is a pixel value of a top-right neighboring pixel of the corresponding first pixel, V_FPT is a pixel value of a top neighboring pixel of the corresponding first pixel, wo is a weight of the first pixel, wi is a weight of the right neighboring pixel, W2 is a weight of the top-right neighboring pixel, and W3 is a weight of the top neighboring pixel.

[0205] FIG. 10 (a) illustrates an example of a target block, and FIG. 10 (b) illustrates an example of a transformed block obtained according to the target block illustrated in FIG. 10(a).

[0206] Referring to FIG. 10(a), the target block includes 6x5 pixels. Pixels in a first row and a last column of the target block are the second pixels. Therefore, the 6x5 pixels includes 6+5-1 second pixels, and 5x4 first pixels. FP denotes the first pixel, and SP denotes the second pixel. FPx,y represents the first pixel located at a ythcolumn and a xthrow in the target block, SPa,p represents the second pixel located at a [3thcolumn and a athrow in the target block. Referring to FIG. 10 (a), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 1 to 5. a and p are positive integer, possible values of a are 1 to 5, and possible values of are 1 to 6.

[0207] Referring to FIG. 10(b), the transformed block includes 6X5 pixels. Pixels in a first row and a last column of the transformed block are the second pixels. Therefore, the 6X5 pixels includes 6+5-1 second pixels, and 5X4 third pixels. TP denotes the third pixel, and SP denotes the second pixel. TPx,yrepresents the third pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the second pixel located at a pthcolumn and a athrow in the transformed block. Referring to FIG. 10 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and P are positive integer, possible values of a are 1 to 5, and possible values of p are 1 to 6.

[0208] The transform operation is not performed on the second pixels. Therefore, the second pixels in the transformed block are the same as the second pixels in the target block. In other words, a value of a second pixel SPi.i in FIG. 10 (a) is the same as a value of a second pixel SPi,i in FIG. 10 (b), a value of a second pixel SPi,2 in FIG. 10 (a) is the same as a value of a second pixel SPi,2 in FIG. 10 (b), a value of a second pixel SPi,3 in FIG. 10 (a) is the sameas a value of a second pixel SPi,3 in FIG. 10 (b), and so on.

[0209] The 5x4 first pixels in FIG. 10 (a) are in one-to-one correspondence with the 5x4 third pixels in FIG. 10 (b). As previously mentioned, the location of the third pixel is the same as the location of the corresponding first pixel. Therefore, a third pixel TP2,2 corresponds to a first pixel FP2.2, a third pixel TP23 corresponds to a first pixel FP23, and so on. A value of each of the third pixels is determined according to values of the corresponding first pixel and neighboring pixels of the corresponding first pixel. For example, a value of the third pixel TP2,2 is determined according to a value of the first pixel FP2.2, a value of a right neighboring pixel of the first pixel FP2,2 (that is, a first pixel FP23), a value of a top-right neighboring pixel of the first pixel FP2.2 (that is, a second pixel SP13), and a value of a top neighboring pixel of the first pixel FP2,2 (that is, a second pixel SPi,2). Further, the value of the third pixel TP2.2, the value of the first pixel FP2.2, the value of the first pixel FP2,3, the value of the second pixel SPi,3, and the value of the second pixel SPi,2 satisfy the formula 1.4. For another example, a value of the a pixel TP33 is determined according to a value of a first pixel FP33, a value of a right neighboring pixel of the first pixel FP33 (that is, a first pixel FP3.4), a value of a top-right neighboring pixel of the first pixel FP33 (that is, the first pixel FP2,4), and a value of a top neighboring pixel of the first pixel FP33 (that is, the first pixel FP23). The value of the third pixel TP33, the value of the first pixel FP33, the value of the first pixel FPs^, the value of the first pixel FP2,4, and the value of the first pixel FP23 satisfy the formula 1.4.

[0210] Blocks illustrates in FIG. 7 to FIG. 11 are an example of the target block. In some embodiments, the size of the target block (that is the number of pixels in the target block) may be a positive integer greater than or equal to 2. In some embodiments, the size of the target block (that is the number of pixels in the target block) may be power of 2 (e.g., 4, 8, 16, 32, 64, or the like). In general, the higher size of block, the higher compression and higher distortion may be achieved.

[0211] In some embodiments, only one of the approach 1 to the approach 4 is used for transforming the F 1 target blocks. In other words, any two of the F 1 target blocks are transformed by using the same approach.

[0212] In some other embodiments, two or more of the approach 1 to the approach 4 are used for transforming the Fi target blocks. For example, the Fi target blocks may be arrangedin M rows and N columns, M and N are positive integer greater than one. In some embodiments, target blocks at an odd row may be transformed to transformed blocks according to the approach 2, and target blocks at an even row may be transformed to transformed blocks according to the approach 1. In some other embodiments, target block at an odd column may be transformed to transformed blocks according to the approach 2, and target blocks at an even column may be transformed to transformed blocks according to the approach 3. In some other embodiments, a target block in an ilhrow of the M rows and a jlhcolumn of the N columns (hereinafter referred to as a target block TBij) and the approach for transforming the target block TBij may satisfy an arrangement illustrate in Table 1Table 1

[0213] “mod” represents to a modulo operation. For a target block located at the first row and the first column among the MxN target blocks, i =1, j =1, 1 mod 2 =1, 1 mod 2 =1, then, according to Table 1, the approach 2 may be used to transform the target block. For a target block located at the first row and the second column among the MxN target blocks, i =1, j =2, 1 mod 2 =1, 2 mod 2 =0, then, according to Table 1, the approach 3 may be used to transform the target block. For a target block located at the second row and the first column among the MxN target blocks, i =2, j =1, 2 mod 2 =0, 1 mod 2 =1, then, according to Table 1, the approach 4 may be used to transform the target block. For a target block located at the second row and the second column among the MxN target blocks, i =2, j =2, 2 mod 2 =0, 2 mod 2 =0, then, according to Table 1 , the approach 1 may be used to transform the target block.

[0214] FIG. 11 illustrates approaches for transforming a plurality of target blocks.

[0215] The plurality of target blocks includes 5x5 target blocks. Al represents the approach 1, A2 represents the approach 2, A3 represents the approach 3, and A4 represents the approach 4. Referring to FIG. 11, target blocks at an odd row and an odd column may be transformed totransformed block by using the approach 2, target blocks at the odd row and an even column may be transformed to transformed blocks by using the approach 3, target blocks at an even row and the odd column may be transformed to transformed blocks by using the approach 4, and target blocks at the even row and the even column may be transformed to transformed blocks by using the approach 1.

[0216] In some embodiments, two adjacent target blocks may share pixels in the same column. The two adjacent target blocks may be referred to as a first target block pair. The first target block pair may include a first target block and a second target block. The first target block is located to a left of the second target block. Second pixels in the first target block include pixels in a rightmost column in the first target block, and second pixels in the second target block include pixels in a leftmost column in the second target block, and the pixels in the rightmost column in the first target block are the pixels in the leftmost column of the second target block.

[0217] FIG. 12 illustrates an example of the first target block pair.

[0218] Referring to FIG. 12, the first target block pair includes 11 ><5 pixels. Both the first target block and the second target block include 6x5 pixels. More specifically, the first target block includes pixels in a first 6 columns among the 11 columns illustrated in FIG. 12, and the second target block includes pixels in a last 6 columns among the 11 columns. In other words, pixels in a sixth column belong to both the first target block and the second target blocks. The sixth column is a rightmost column of the first target block, and the sixth column is a leftmost column of the second target block. Second pixels of the first target block include pixels in the sixth column. For example, the first target block may be the target block A2 or the target block A5. Second pixels of the second target block include pixels in the sixth column. For example, the second target block may be the target block Al and the target block A3.

[0219] In some embodiments, two adjacent target blocks may share pixels in the same row. The two adjacent target blocks may be referred to as a second target block pair. The second target block pair may include a third target block and a fourth target block. The third target block is located above the fourth target block. Second pixels in the third target block include pixels in a bottom row in the third target block, and second pixels in the fourth target block include pixels in a top row in the fourth target block, and the pixels in the bottom row in thethird target block are the pixels in the top row of the fourth target block.

[0220] FIG. 13 illustrates an example of the second target block pair.

[0221] Referring to FIG. 13, the second target block pair includes 6x9 pixels. Both the third target block and the fourth target block include 6x5 pixels. More specifically, the third target block includes pixels in a first 5 rows among the 9 rows illustrated in FIG. 13, and the fourth target block includes pixels in a last 5 rows among the 9 rows. In other words, pixels in a fifth row belong to both the third target block and the fourth target blocks. The fifth row is a bottom row of the third target block, and the fifth row is a top row of the fourth target block. Second pixels of the third target block include pixels in the fifth row. For example, the third target block may be the target block A2 or the target block A3. Second pixels of the fourth target block include pixels in the fifth row. For example, the fourth target block may be the target block Al and the target block A4.

[0222] In general, if a size of a target block is PXQ, a size of the first target block pair may be (2xP-l)xQ, and pixels in a Pthcolumn among 2xP-l columns may belong to both the first target block and the second target block. Similarly, if a size of a target block is PXQ, a size of the second target block pair may be Px(2xQ-l), and pixels in a Qthrow among 2xQ-l rows may belong to both the third target block and the fourth target block.

[0223] Referring to the formulas 1.1 to 1.4, the value of the third pixel is determined according to values of a group of pixels, the group of pixels includes the corresponding first pixel and its neighboring pixels. Further, referring to the formulas 1.1 to 1.4, each pixel in the group of pixels has a weight. For convenience, “first neighboring pixel”, “second neighboring pixel” and “third neighboring pixel” are used to referred to as the neighboring pixels of the corresponding first pixels. For example, for the approach 1, the first neighboring pixel may be the left neighboring pixel, the second neighboring pixel may be the top-left neighboring pixel, and the third neighboring pixel may be the top neighboring pixel; for the approach 2, the first neighboring pixel may be the right neighboring pixel, the second neighboring pixel may be the bottom-right neighboring pixel, and the third neighboring pixel may be the bottom neighboring pixel; for the approach 3, the first neighboring pixel may be the left neighboring pixel, the second neighboring pixel may be the bottom-left neighboring pixel, and the third neighboring pixel may be the bottom neighboring pixel; for the approach 4, the first neighboring pixel maybe the right neighboring pixel, the second neighboring pixel may be the top-right neighboring pixel, and the third neighboring pixel may be the top neighboring pixel. Correspondingly, the formulas 1.1 to 1.4 may be rewritten as:

[0224] V_TP=(woxV_FP+wixV_FPni+w2xV_FPn2+w3xV_FPn3) / (wo+wi+w2+w3), (1.5)

[0225] wherein V_TP is a pixel value of a third pixel TP in the transformed block, V_FP is a pixel value of a first pixel FP corresponding to the third pixel TP in the target block, the third pixel TP is any one of the third pixels in the transformed block, V_FPniis a pixel value of a first neighboring pixel of the first pixel FP, V_FPn2 is a pixel value of a second neighboring pixel of the first pixel FP, V_FPn3 is a pixel value of a third neighboring pixel of the first pixel FP, wo is a weight of the first pixel FP, wi is a weight of the first neighboring pixel, W2 is a weight of the second neighboring pixel, and W3 is a weight of the third neighboring pixel.

[0226] The approach 1 to the approach 4 are some possible embodiments provided by the present application. Based on the disclosure of the present application, other approaches may be obtained. For example, FIG. 14 illustrates another target block according to some embodiments provided by the present application.

[0227] Referring to FIG. 14, second pixels of the target block include pixels in a first two columns and pixels in a first two rows, and other pixels are first pixels. Neighboring pixels of a first pixel include two pixels located to a left of the first pixel, two pixels located above the first pixel, and four pixels located to atop-left of the first pixel. For example, neighboring pixels of the first pixel FP3.3 include: a second pixel SPs.i, a second pixel SP3 , a second pixel SP2J, a second pixel SP2.2, a second pixel SPi.i, a second pixel SPi,2, a second pixel SP1 , and a second pixel SP2 .

[0228] FIG. 15 illustrates the inverse transform procedure according to some embodiments of the present application. The image processing method illustrated in FIG. 15 may be performed by a decoding device, and the decoding device may be an electronic device. The electronic device may be a mobile phone, a tablet personal computer, a notebook computer, a digital camera, a video camera, a security camera, a vlogging camera, a server, or the like. In some other embodiments, the image processing method illustrated in FIG. 15 may be performed by a component in the decoding device. For example, the image processing method may be performed by a processor, a system on chip (SOC), a processing circuit, or the like. In someembodiments, the image processing method illustrated in FIG. 4 (that is, the transform procedure) and the image processing method illustrated in FIG. 15 (that is, the inverse transform procedure) may be performed by one electronic device. For example, the image processing method illustrated in FIG. 4 and the image processing method illustrated in FIG. 15 may be performed by different components in the electronic device. In some other embodiments, the image processing method illustrated in FIG. 4 and the image processing method illustrated in FIG. 15 may be performed by different electronic devices or components belonging to different electronic devices. For convenience, it is assumed that the method illustrated in FIG. 15 is performed by the decoding device.

[0229] 1501, The decoding device obtains encoded data and performs a decoding operation on the encoded data to obtain decoded data.

[0230] For the transform procedure illustrated in FIG. 4, the encoding device derives the encoded data by performing the encoding operation on the transformed data. The encoded data mentioned in a step 1501 is the encoded data obtained by the encoding device in a step 403.

[0231] The decoded data may include a plurality of transformed pixels and a plurality of reserved pixels. As previously mentioned, the inverse transform procedure is the inverse operation of the transform procedure. Correspondingly, the step 1501 is an inverse step of the step 403. The step 403 is configured to obtain the encoded data based on the transformed data, and the step 1501 is configured to restore the encoded data back to the transformed data, and the decoded data is the restored transformed data.

[0232] Prior to obtaining the decoded data, both the transformed pixels and the reserved pixels obtained by the encoding device (that is, the transformed pixels and the reserved pixels in the transformed data) undergo encoding, transmission, and decoding processes. Each of these processes can potentially introduce issues, resulting in discrepancies between the transformed pixels in the decoded data and the transformed pixels obtained by the encoding device, as well as discrepancies between the reserved pixels in the decoded data and the reserved pixels obtained by the encoding device. For convenience, the transformed pixels in the transformed data may be referred to as first transformed pixels, the transformed pixels in the decoded data may be referred to second transformed pixels. The first transformed pixels are in one-to-one correspondence with the second transformed pixels, and a location of a first transformed pixelamong the transformed pixels is the same as a location of the corresponding second transform pixel. In some embodiments, the first transformed pixels may be the same as the second transformed pixels. In other words, a pixel value of each of the first transformed pixels is equal to a pixel value of the corresponding second transformed pixel. In some other embodiments, the first transformed pixels may be different from the second transformed pixels. In other words, a pixel value of at least one first transformed pixel is different from a pixel value of a corresponding second transformed pixel. Similarly, the reserved pixels in the transformed data may be referred to as first reserved pixels, and the reserved pixels in the decoded data may be referred to as second reserved pixels. The first reserved pixels are in one-to-one correspondence with the second reserved pixels, and a location of a first reserved pixel among the transformed pixels is the same as a location of the corresponding second reserved pixel. In some embodiments, the first reserved pixels may be the same as the second reserved pixels. In other words, a pixel value of each of the first reserved pixels is equal to a pixel value of the corresponding second reserved pixel. In some other embodiments, the first reserved pixels may be different from the second reserved pixels. In other words, a pixel value of at least one first reserved pixel is different from a pixel value of a corresponding second reserved pixel. However, for convenience, it is assumed that the encoding, transmission, and decoding processes are lossless processes in the embodiments. In other words, in the embodiments, the first transformed pixels are the same as the second transformed pixels, and the first reserved pixels are the same as the second reserved pixels.

[0233] 1502, The decoding device performs an inverse smoothing transform on the plurality of second transformed pixels of the decoded data to obtain a reconstructed data.

[0234] The reconstructed data include a plurality of reconstructed pixels and the plurality of reserved pixels. The plurality of reconstructed pixels are in one-to-one correspondence with the plurality of transformed pixels, and a pixel value of each of the plurality of reconstructed pixels is determined according to pixel values of a group of pixels. The group of pixels includes a corresponding transformed pixel and at least one neighboring pixel of the reconstructed pixel.

[0235] Similar to a relationship between the step 1501 and the step 403, a step 1502 is an inverse step of the step 402. As previously mentioned, the input data obtained by the encoding device in step 401 may include a plurality of first pixels and a plurality of second pixels, andthe transform operation performed by the encoding device is configured to transform the first pixels into the third pixels and keep the second pixels unchanged. Therefore, the first pixels may be referred to as target pixels, the second pixels may be referred to as reserved pixels, the third pixels may be referred to as the transformed pixels. The step 1502 is configured to restore the transformed pixels back to the target pixels, and the restored target pixels are the reconstructed pixels. Correspondingly, the reconstructed data is the restored input data.

[0236] 1503, The decoding device performs an image signal processing operation on the reconstructed.

[0237] The transform operation and the inverse transform operation are closely related. For example, in some embodiments, in order to perform the transform operation, the encoding device divides the input data into several blocks. Under this condition, the decoding device divides the decoded data into several blocks and performs the inverse smoothing transform based on the blocks. For example, when the input data is divided into F blocks, the decoded data may be also divided into F blocks, where F is a positive integer greater than two.

[0238] A dividing method for dividing the input data may be a same as a dividing method for dividing the decoded data. For example, when the input data is divided using a pattern of size PXQ, the decoded data may also be divided into blocks using the same pattern of PxQ. Details of the dividing method may be referred to the previous embodiments. For the sake of brevity, it will not be elaborated here.

[0239] For example, in some embodiments, the F blocks obtained according to the decoded data may include Fi transformed blocks. The decoding device may transform transformed pixels among the Fi transformed blocks into reconstructed pixels to obtain the reconstructed data. The reconstructed data includes F i reconstructed blocks, the F i reconstructed blocks are in one-to-one correspondence with the Fi transformed blocks, each of the Fl reconstructed blocks are determined according to a corresponding transformed block, and a size of the transformed block is a same as a size of the reconstructed block.

[0240] As previously mentioned, there are several approaches for transforming the first pixels (that is, the target pixels) into the third pixels (that is, the transformed pixels). Correspondingly, there are also several approach for transforming the transformed pixels into the reconstructed pixels. For convenience, a term “inverse approach” may be referred to anapproach for transforming the transformed pixels into the reconstructed pixels.

[0241] Inverse approach 1 :

[0242] The inverse approach 1 is an inverse approach corresponding to the approach 1. For the inverse approach 1 , pixels in a top row and a leftmost column of the transformed block are the reserved pixels, and other pixels are the transformed pixels. In other words, pixels in a first row and a first column of the transformed block are the reserved pixels, while other pixels in the transformed block, except for those in the first row and first column, are the transformed pixels. For convenience, this transformed block (that is, a transformed block in which pixels in the top row and the leftmost column are the reserved pixels) may be referred to as “transformed block Al”. The first row of the transformed block is the top row of the transformed block, the first column of the transformed block is the leftmost column of the transformed block, a last row of the transformed block is a bottom row of the transformed block, and a last column of the transformed block is the rightmost column of the transformed block.

[0243] Each of the transformed pixels in the transformed block and its three neighboring pixels are used to determine a reconstructed pixel. For example, it is assumed that a size of the transformed block is PxQ (P-column by Q-row), then the transformed block may include (P- l)x(Q-l) transformed pixels, and P+Q-l reserved pixels. A reconstructed block determined according to the transformed block may include (P-l)x(Q-l) reconstructed pixels, and P+Q-l reserved pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the transformed block includes P’xQ’ transformed pixels, and the reconstructed block determined according to the transformed block includes P’xQ’ reconstructed pixels. The P’xQ’ reconstructed pixels are in one-to-one correspondence with the P’xQ’ transformed pixels, and a location of each of the reconstructed pixels and a location of a corresponding transformed pixel are the same. Each of the reconstructed pixels is determined according to the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. The neighboring pixels of the corresponding transformed pixel may include a left neighboring pixel of the corresponding transformed pixel, a top-left neighboring pixel of the corresponding transformed pixel, and a top neighboring pixel of the corresponding transformed pixel. Pixel values of the reconstructed pixel, the corresponding first pixel, and the neighboring pixels of the corresponding transformed pixel satisfy:

[0244] V_RP=((WO+WI+W2+W3)XV_NP - WIXV_RPL - W2XV_RPT_L - W3XV_RPT) / WO, (2.1)

[0245] wherein V_RP is a pixel value of the reconstructed pixel, V_NP is a pixel value of the corresponding transformed pixel, V_RPL is a pixel value of the left neighboring pixel of the corresponding transformed pixel, V_RPT_L is a pixel value of a top-left neighboring pixel of the corresponding transformed pixel, V_RPT is a pixel value of a top neighboring pixel of the corresponding transformed pixel, wo is a weight of the transformed pixel, wi is a weight of the left neighboring pixel, W2 is a weight of the top-left neighboring pixel, and w3is a weight of the top neighboring pixel.

[0246] As previously mentioned, the inverse approach 1 is the inverse approach corresponding to the approach 1. Therefore, FIG. 7 (a) and FIG. 7 (b) may be used to depict the inverse approach 1. For the inverse approach 1 , FIG. 7 (b) illustrates the transformed block, and FIG. 7 (a) illustrates the reconstructed block.

[0247] Referring to FIG. 7(b), the transformed block includes 6><5 pixels. Pixels in a first row and a first column of the transformed block are the reserved pixels. Therefore, the 6X5 pixels includes 6+5-1 reserved pixels, and 5x4 third pixels. TP denotes the transformed pixel, and SP denotes the reserved pixel. TPx,y represents the transformed pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the reserved pixel located at a [3thcolumn and a athrow in the transformed block. Referring to FIG. 7 (b), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 2 to 6. a and P are positive integer, possible values of a are 1 to 5, and possible values of are 1 to 6.

[0248] Referring to FIG. 7(a), the reconstructed block includes 6X5 pixels. Pixels in a first row and a first column of the reconstructed block are the reserved pixels. Therefore, the 6X5 pixels includes 6+5-1 reserved pixels, and 5X4 reconstructed pixels. FP denotes the reconstructed pixel, and SP denotes the reserved pixel. FPx,yrepresents the reconstructed pixel located at a ythcolumn and a Xthrow in the reconstructed block, SPa,p represents the reserved pixel located at a pthcolumn and a athrow in the reconstructed block. Referring to FIG. 7 (a), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 2 to 6. a and P are positive integer, possible values of a are 1 to 5, and possible values of p are 1 to 6.

[0249] The inverse transform operation is not performed on the reserved pixels. Therefore, the reserved pixels in the transformed block are the same as the reserved pixels in thereconstructed block. In other words, a value of a reserved pixel SPi,i in FIG. 7 (b) is the same as a value of a reserved pixel SPi.i in FIG. 7 (a), a value of a reserved pixel SPi,2 in FIG. 7 (b) is the same as a value of a reserved pixel SPi,2 in FIG. 7 (a), a value of a reserved pixel SPi,3 in FIG. 7 (b) is the same as a value of a reserved pixel SPi,3 in FIG. 7 (a), and so on.

[0250] The 5x4 transformed pixels in FIG. 7 (b) are in one-to-one correspondence with the 5x reconstructed pixels in FIG. 7 (a). As previously mentioned, the location of the reconstructed pixel is the same as the location of the corresponding transformed pixel. Therefore, a transformed pixel TP2.2 corresponds to a reconstructed pixel FP2.2, a transformed pixel TP2 corresponds to a reconstructed pixel FP2.3, and so on. A value of each of the reconstructed pixels is determined according to values of the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. For example, a value of the reconstructed pixel FP2.2 is determined according to a value of the transformed pixel TP2.2, a value of a left neighboring pixel of the transformed pixel T?2,2 (that is, a reserved pixel SP2,i), a value of a top-left neighboring pixel of the transformed pixel TP2.2 (that is, a reserved pixel SP1,1), and a value of a top neighboring pixel of the transformed pixel TP2.2 (that is, a reserved pixel SP1,2). Further, the value of the reconstructed pixel FP2.2, the value of the transformed pixel TP2.2, the value of the reserved pixel SP2,i, the value of the reserved pixel SPi.i, and the value of the reserved pixel SPi,2 satisfy the formula 2.1. For another example, a value of the reconstructed pixel FP3 is determined according to a value of a transformed pixel TP3 , a value of a left neighboring pixel of the transformed pixel TP3 (that is, a transformed pixel TP3.2), a value of a top-left neighboring pixel of the transformed pixel TP3 (that is, the transformed pixel TP2.2), and a value of a top neighboring pixel of the transformed pixel TP3 (that is, the transformed pixel TP2 ). The value of the reconstructed pixel FP3 , the value of the transformed pixel TP3 , the value of the transformed pixel TPs,2, the value of the transformed pixel TP2.2, and the value of the transformed pixel TP2 satisfy the formula 2.1.

[0251] Inverse approach 2:

[0252] The inverse approach 2 is an inverse approach corresponding to the approach 2. For the approach 2, pixels in a bottom row and a rightmost column of the transformed block are the reserved pixels, and the other pixels are the transformed pixels. In other words, pixels in a last row and a last column of the transformed block are the reserved pixels, while other pixels inthe transformed block, except for those in the last row and the last column, are the transformed pixels. For convenience, this transformed block (that is, a transformed block in which pixels in the bottom row and the rightmost column are the reserved pixels) may be referred to as “transformed block A2”. The first row of the transformed block is the top row of the transformed block, the first column of the transformed block is the leftmost column of the transformed block, a last row of the transformed block is a bottom row of the transformed block, and a last column of the transformed block is the rightmost column of the transformed block.

[0253] Each of the transformed pixels in the transformed block and its three neighboring pixels are used to determine a reconstructed pixel. For example, it is assumed that a size of the transformed block is P*Q (P-column by Q-row), then the transformed block may include (P- 1)X(Q-1) transformed pixels, and P+Q-l reserved pixels. A reconstructed block determined according to the transformed block may include (P-l)x(Q-l) reconstructed pixels, and P+Q-l reserved pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the transformed block includes P’xQ’ transformed pixels, and the reconstructed block determined according to the transformed block includes P’xQ’ reconstructed pixels. The P’xQ’ reconstructed pixels are in one-to-one correspondence with the P’xQ’ transformed pixels, and a location of each of the reconstructed pixels and a location of a corresponding transformed pixel are the same. Each of the reconstructed pixels is determined according to the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. The neighboring pixels of the corresponding transformed pixel may include a right neighboring pixel of the corresponding transformed pixel, a bottom-right neighboring pixel of the corresponding transformed pixel, and a bottom neighboring pixel of the corresponding transformed pixel. Pixel values of the reconstructed pixel, the corresponding transformed pixel, and the neighboring pixels of the corresponding transformed pixel satisfy:

[0254] V_RP=((WO+WI+W2+W3)XV_NP - WIXV_RPR - W2XV_RPB_R - W3XV_RPB) / WO, (2.2)

[0255] wherein V_RP is a pixel value of the reconstructed pixel, V_NP is a pixel value of the corresponding transformed pixel, V_RPR is a pixel value of the right neighboring pixel of the corresponding transformed pixel, V_RPB_R is a pixel value of a bottom-right neighboring pixel of the corresponding transformed pixel, V_RPB is a pixel value of a bottom neighboring pixel of the corresponding transformed pixel, wo is a weight of the transformed pixel, wi is aweight of the right neighboring pixel, W2 is a weight of the bottom-right neighboring pixel, and W3 is a weight of the bottom neighboring pixel.

[0256] As previously mentioned, the inverse approach 2 is the inverse approach corresponding to the approach 2. Therefore, FIG. 8 (a) and FIG. 8 (b) may be used to depict the inverse approach 2. For the inverse approach 2, FIG. 8 (b) illustrates the transformed block, and FIG. 8 (a) illustrates the reconstructed block.

[0257] Referring to FIG. 8(b), the transformed block includes 6><5 pixels. Pixels in a last row and a last column of the transformed block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 transformed pixels. TP denotes the transformed pixel, and SP denotes the reserved pixel. TPx yrepresents the transformed pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the reserved pixel located at a 0thcolumn and a athrow in the transformed block. Referring to FIG. 8 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0258] Referring to FIG. 8(a), the reconstructed block includes 6x5 pixels. Pixels in a last row and a last column of the target block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 reconstructed pixels. FP denotes the reconstructed pixel, and SP denotes the reserved pixel. FPx,y represents the reconstructed pixel located at a ythcolumn and a xthrow in the reconstructed block, SPa,p represents the reserved pixel located at a 0thcolumn and a athrow in the reconstructed block. Referring to FIG. 8 (a), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0259] The inverse transform operation is not performed on the reserved pixels. Therefore, the reserved pixels in the transformed block are the same as the reserved pixels in the reconstructed block. In other words, a value of a reserved pixel SPs.i in FIG. 8 (b) is the same as a value of a reserved pixel SPs,i in FIG. 8 (a), a value of a reserved pixel SPs,2 in FIG. 8 (b) is the same as a value of a reserved pixel SPs,2 in FIG. 8 (a), a value of a reserved pixel SP5 in FIG. 8 (b) is the same as a value of a reserved pixel SP5 in FIG. 8 (a), and so on.

[0260] The 5x4 transformed pixels in FIG. 8 (b) are in one-to-one correspondence with the 5x4 reconstructed pixels in FIG. 8 (a). As previously mentioned, the location of thereconstructed pixel is the same as the location of the corresponding transformed pixel. Therefore, a reconstructed pixel FP23 corresponds to a transformed pixel TP23, a reconstructed pixel FPy corresponds to a transformed pixel TPy, and so on. A value of each of the reconstructed pixels is determined according to values of the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. For example, a value of the reconstructed pixel FP23 is determined according to a value of the transformed pixel TP23, a value of a right neighboring pixel of the transformed pixel TP23 (that is, a transformed pixel TP23), a value of a bottom-right neighboring pixel of the transformed pixel TP23 (that is, a transformed pixel TP33), and a value of a bottom neighboring pixel of the transformed pixel TP23 (that is, a transformed pixel TP33). Further, the value of the reconstructed pixel FP24, the value of the transformed pixel TP23, the value of the transformed pixel TP23, the value of the transformed pixel TP33, and the value of the transformed pixel TP33 satisfy the formula 2.2. For another example, a value of the reconstructed pixel FP33 is determined according to a value of a transformed pixel TP33, a value of a right neighboring pixel of the transformed pixel TP33 (that is, a transformed pixel IP3.4), a value of a bottom-right neighboring pixel of the transformed pixel TP33 (that is, the transformed pixel TP43), and a value of a bottom neighboring pixel of the transformed pixel TP33 (that is, the transformed pixel TP43). The value of the reconstructed pixel FP33, the value of the transformed pixel TP33, the value of the transformed pixel TP3.4, the value of the transformed pixel TP4.4, and the value of the transformed pixel TP43 satisfy the formula 2.2.[02611 Inverse approach 3:

[0262] The inverse approach 3 is an inverse approach corresponding to the approach 3. For the inverse approach 2, pixels in a bottom row and a leftmost column of the transformed block are the reserved pixels, and other pixels are the transformed pixels. In other words, pixels in a last row and a first column of the transformed block are the reserved pixels, while other pixels in the transformed block, except for those in the last row and first column, are the transformed pixels. For convenience, this transformed block (that is, a transformed block in which pixels in the bottom row and the leftmost column are the reserved pixels) may be referred to as “transformed block A3”. The first row of the transformed block is the top row of the transformed block, the first column of the transformed block is the leftmost column of the transformed block,a last row of the transformed block is a bottom row of the transformed block, and a last column of the transformed block is the rightmost column of the transformed block.

[0263] Each of the transformed pixels in the transformed block and its three neighboring pixels are used to determine a reconstructed pixel. For example, it is assumed that a size of the transformed block is PxQ (P-column by Q-row), then the transformed block may include (P- l)x(Q-l) transformed pixels, and P+Q-l reserved pixels. A reconstructed block determined according to the transformed block may include (P-l)x(Q-l) reconstructed pixels, and P+Q-l reserved pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the transformed block includes P’xQ’ transformed pixels, and the reconstructed block determined according to the transformed block includes P’xQ’ reconstructed pixels. The P’xQ’ reconstructed pixels are in one-to-one correspondence with the P’xQ’ transformed pixels, and a location of each of the reconstructed pixels and a location of a corresponding transformed pixel are the same. Each of the reconstructed pixels is determined according to the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. The neighboring pixels of the corresponding transformed pixel may include a left neighboring pixel of the corresponding transformed pixel, a bottom-left neighboring pixel of the corresponding transformed pixel, and a bottom neighboring pixel of the corresponding transformed pixel. Pixel values of the reconstructed pixel, the corresponding transformed pixel, and the neighboring pixels of the corresponding transformed pixel satisfy:

[0264] V_RP=((WO+WI+W2+W3)XV_NP - WIXV_RPL - W2XV_RPB_L - W3XV_RPB) / WO, (2.3)

[0265] wherein V_RP is a pixel value of the reconstructed pixel, V_NP is a pixel value of the corresponding transformed pixel, V_RPL is a pixel value of the left neighboring pixel of the corresponding transformed pixel, V_RPB_L is a pixel value of a bottom -left neighboring pixel of the corresponding transformed pixel, V_RPB is a pixel value of a bottom neighboring pixel of the corresponding transformed pixel, wo is a weight of the transformed pixel, wi is a weight of the left neighboring pixel, W2 is a weight of the bottom-left neighboring pixel, and W3 is a weight of the bottom neighboring pixel.

[0266] As previously mentioned, the inverse approach 3 is the inverse approach corresponding to the approach 3. Therefore, FIG. 9 (a) and FIG. 9 (b) may be used to depict the inverse approach 3. For the inverse approach 3, FIG. 9 (b) illustrates the transformed block, andFIG. 9 (a) illustrates the reconstructed block.

[0267] Referring to FIG. 9(b), the transformed block includes 6x5 pixels. Pixels in a last row and a first column of the transformed block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 transformed pixels. TP denotes the transformed pixel, and SP denotes the reserved pixel. TPx,yrepresents the transformed pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the reserved pixel located at a 0thcolumn and a athrow in the transformed block. Referring to FIG. 9 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 2 to 6. a and are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0268] Referring to FIG. 9(a), the reconstructed block includes 6x5 pixels. Pixels in a last row and a first column of the reconstructed block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 reconstructed pixels. FP denotes the reconstructed pixel, and SP denotes the reserved pixel. FPx>yrepresents the reconstructed pixel located at a ythcolumn and a xthrow in the reconstructed block, SPa,p represents the reserved pixel located at a 0thcolumn and a athrow in the reconstructed block. Referring to FIG. 9 (a), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 2 to 6. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0269] The inverse transform operation is not performed on the reserved pixels. Therefore, the reserved pixels in the reconstructed block are the same as the reserved pixels in the transformed block. In other words, a value of a reserved pixel SPi,i in FIG. 9 (b) is the same as a value of a reserved pixel SPi,i in FIG. 9 (a), a value of a reserved pixel SP2,i in FIG. 9 (b) is the same as a value of a reserved pixel SP2,i in FIG. 9 (a), a value of a reserved pixel SPa.i in FIG. 9 (b) is the same as a value of a reserved pixel SPa.i in FIG. 9 (a), and so on.

[0270] The 5x4 first pixels in FIG. 9 (b) are in one-to-one correspondence with the 5x4 third pixels in FIG. 9 (a). As previously mentioned, the location of the reconstructed pixel is the same as the location of the corresponding transformed pixel. Therefore, a reconstructed pixel FP2,2 corresponds to a transformed pixel TP2,2, a reconstructed pixel FP2 corresponds to a transformed pixel TP2.3, and so on. A value of each of the reconstructed pixels is determined according to values of the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. For example, a value of the reconstructed pixel FP2.2 isdetermined according to a value of the transformed pixel TP2.2, a value of a left neighboring pixel of the transformed pixel TP2.2 (that is, a reserved pixel SP2,i), a value of a bottom-left neighboring pixel of the transformed pixel TP2,2 (that is, a reserved pixel SP3,i), and a value of a bottom neighboring pixel of the transformed pixel TP2,2 (that is, a transformed pixel TP3,2). Further, the value of the reconstructed pixel FP2.2, the value of the transformed pixel FP2.2, the value of the reserved pixel SP2,i, the value of the reserved pixel SP .i, and the value of the transformed pixel TP3,2 satisfy the formula 2.3. For another example, a value of the reconstructed pixel FP3 is determined according to a value of a transformed pixel TP3 , a value of a left neighboring pixel of the transformed pixel TP3 (that is, a transformed pixel TPs.2), a value of a bottom-left neighboring pixel of the transformed pixel TP3 (that is, the transformed pixel TP4.2), and a value of a bottom neighboring pixel of the transformed pixel TP3 (that is, the transformed pixel TP4 ). The value of the reconstructed pixel FP3 , the value of the transformed pixel TP3 , the value of the transformed pixel TPs.2, the value of the transformed pixel TP4.2, and the value of the transformed pixel TP4 satisfy the formula 2.3.

[0271] Inverse approach 4:

[0272] The inverse approach 4 is an inverse approach corresponding to the approach 4. For the inverse approach 4, pixels in a top row and a rightmost column of the transformed block are the reserved pixels, and the other pixels are the transformed pixels. In other words, pixels in a first row and a last column of the transformed block are the reserved pixels, while other pixels in the transformed block, except for those in the first row and the last column, are the transformed pixels. For convenience, this transformed block (that is, a transformed block in which pixels in the top row and the rightmost column are the reserved pixels) may be referred to as “transformed block A4”. The first row of the transformed block is the top row of the transformed block, the first column of the transformed block is the leftmost column of the transformed block, a last row of the transformed block is a bottom row of the transformed block, and a last column of the transformed block is the rightmost column of the transformed block.

[0273] Each of the transformed pixels in the transformed block and its three neighboring pixels are used to determine a reconstructed pixel. For example, it is assumed that a size of the transformed block is PxQ (P-column by Q-row), then the transformed block may include (P- l)x(Q-l) transformed pixels, and P+Q-l reserved pixels. A reconstructed block determinedaccording to the transformed block may include (P-l)x(Q-l) reconstructed pixels, and P+Q-l reserved pixels. For convenience, P’ denotes P-1, and Q’ denotes Q-l. In other words, the transformed block includes P’xQ’ transformed pixels, and the reconstructed block determined according to the transformed block includes P’xQ’ reconstructed pixels. The P’xQ’ reconstructed pixels are in one-to-one correspondence with the P’xQ’ transformed pixels, and a location of each of the reconstructed pixels and a location of a corresponding transformed pixel are the same. Each of the reconstructed pixels is determined according to the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. The neighboring pixels of the corresponding transformed pixel may include a right neighboring pixel of the corresponding transformed pixel, a top-right neighboring pixel of the corresponding transformed pixel, and a top neighboring pixel of the corresponding transformed pixel. Pixel values of the reconstructed pixel, the corresponding transformed pixel, and the neighboring pixels of the corresponding transformed pixel satisfy:

[0274] V_RP=((WO+WI+W2+W3)XV_NP - WIXVJRPR - W2XV_RPT_R - W3XV_RPT) / WO, (2.4)

[0275] wherein V RP is a pixel value of the reconstructed pixel, V_NP is a pixel value of the corresponding transformed pixel, V_RPR is a pixel value of the right neighboring pixel of the corresponding transformed pixel, V_RPT_R is a pixel value of a top-right neighboring pixel of the corresponding transformed pixel, V_RPT is a pixel value of a top neighboring pixel of the corresponding transformed pixel, wo is a weight of the transformed pixel, wi is a weight of the right neighboring pixel, W2 is a weight of the top-right neighboring pixel, and W3 is a weight of the top neighboring pixel.

[0276] As previously mentioned, the inverse approach 4 is the inverse approach corresponding to the approach 4. Therefore, FIG. 10 (a) and FIG. 10 (b) may be used to depict the inverse approach 4. For the inverse approach 4, FIG. 10 (b) illustrates the transformed block, and FIG. 10 (a) illustrates the reconstructed block.

[0277] Referring to FIG. 10(b), the transformed block includes 6x5 pixels. Pixels in a first row and a last column of the transformed block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 third pixels. TP denotes the transformed pixel, and SP denotes the reserved pixel. TPx,yrepresents the transformed pixel located at a ythcolumn and a xthrow in the transformed block, SPa,p represents the reserved pixel located at a 0thcolumnand a athrow in the transformed block. Referring to FIG. 10 (b), x and y are positive integer, possible values of x are 1 to 4, and possible values of y are 1 to 5. a and are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0278] Referring to FIG. 10(a), the reconstructed block includes 6x5 pixels. Pixels in a first row and a last column of the reconstructed block are the reserved pixels. Therefore, the 6x5 pixels includes 6+5-1 reserved pixels, and 5x4 reconstructed pixels. FP denotes the reconstructed pixel, and SP denotes the reserved pixel. FPx,y represents the reconstructed pixel located at a ythcolumn and a xthrow in the reconstructed block, SPa,p represents the reserved pixel located at a 0thcolumn and a athrow in the reconstructed block. Referring to FIG. 10 (a), x and y are positive integer, possible values of x are 2 to 5, and possible values of y are 1 to 5. a and 0 are positive integer, possible values of a are 1 to 5, and possible values of 0 are 1 to 6.

[0279] The inverse transform operation is not performed on the reserved pixels. Therefore, the reserved pixels in the reconstructed block are the same as the reserved pixels in the transformed block. In other words, a value of a reserved pixel SPi.i in FIG. 10 (b) is the same as a value of a reserved pixel SPi.i in FIG. 10 (a), a value of a reserved pixel SPi,2 in FIG. 10 (b) is the same as a value of a reserved pixel SPi,2 in FIG. 10 (a), a value of a reserved pixel SPi,3 in FIG. 10 (b) is the same as a value of a reserved pixel SPi,3 in FIG. 10 (a), and so on.

[0280] The 5x4 transformed pixels in FIG. 10 (b) are in one-to-one correspondence with the 5x4 reconstructed pixels in FIG. 10 (a). As previously mentioned, the location of the reconstructed pixel is the same as the location of the corresponding transformed pixel. Therefore, a reconstructed pixel FP2.2 corresponds to a transformed pixel TP2.2, a reconstructed pixel FP2.3 corresponds to a transformed pixel TP2.3, and so on. A value of each of the reconstructed pixels is determined according to values of the corresponding transformed pixel and neighboring pixels of the corresponding transformed pixel. For example, a value of the reconstructed pixel FP2.2 is determined according to a value of the transformed pixel TP2.2, a value of a right neighboring pixel of the transformed pixel TP2,2 (that is, a transformed pixel TP2 ), a value of a top-right neighboring pixel of the transformed pixel TP2.2 (that is, a reserved pixel SP1 ), and a value of a top neighboring pixel of the transformed pixel TP2.2 (that is, a reserved pixel SPi,2). Further, the value of the reconstructed pixel FP2.2, the value of the transformed pixel TP2.2, the value of the transformed pixel TP2 , the value of the reserved pixelSPi,3, and the value of the reserved pixel SPi,2 satisfy the formula 2.4. For another example, a value of the a reconstructed pixel FP33 is determined according to a value of a transformed pixel TP3.3, a value of a right neighboring pixel of the transformed pixel TP33 (that is, a transformed pixel TP3.4), a value of a top-right neighboring pixel of the transformed pixel TP33 (that is, the transformed pixel TP2 / 1), and a value of a top neighboring pixel of the transformed pixel TP33 (that is, the transformed pixel TP23). The value of the reconstructed pixel FP33, the value of the transformed pixel TP33, the value of the transformed pixel TP3.4, the value of the transformed pixel TP2.4, and the value of the transformed pixel TP23 satisfy the formula 2.4.

[0281] As previously mentioned, the transform operation and the inverse transform operation are closely related. For the approaches and the inverse approaches, when a target block is transformed by using one of the four approaches, a transformed block corresponding to the target block may be transformed by using a corresponding inverse approach. For example, the approach 1 is used to transform a target block TBXinto a transformed block NBX. Then, the inverse approach is used to transform the transformed block NBXinto a reconstructed block RBX. Similarly, in some embodiments, a transformed block in an ithrow of the M rows and a jthcolumn of the N columns (hereinafter referred to as a transformed block TBij) and the inverse approach for transforming the transformed block TBij may satisfy an arrangement illustrate in Table 2Table 2

[0282] “mod” represents to a modulo operation. For a transformed block located at the first row and the first column among the MxN transformed blocks, i =1, j =1, 1 mod 2 =1, 1 mod 2 =1 , then, according to Table 3, the inverse approach 2 may be used to transform the transformed block. For a transformed block located at the first row and the second column among the MxN transformed blocks, i =1, j =2, 1 mod 2 =1, 2 mod 2 =0, then, according to Table 3, the inverseapproach 3 may be used to transform the transformed block. For a transformed block located at the second row and the first column among the MxN transformed blocks, i =2, j =1, 2 mod 2 =0, 1 mod 2 =1, then, according to Table 3, the inverse approach 4 may be used to transform the transformed block. For a transformed block located at the second row and the second column among the MxN transformed blocks, i =2, j =2, 2 mod 2 =0, 2 mod 2 =0, then, according to Table 3, the inverse approach 3 may be used to transform the transformed block.

[0283] Similarly, FIG. 11 may be also used to depict the inverse approaches. In this case, blocks illustrated in FIG. 11 are the transformed blocks. FIG. 11 illustrates 5x5 transformed blocks. Al represents the inverse approach 1, A2 represents the inverse approach 2, A3 represents the inverse approach 3, and A4 represents the inverse approach 4. Referring to FIG. 11 , transformed blocks at an odd row and an odd column may be transformed to reconstructed block by using the inverse approach 2, transformed blocks at the odd row and an even column may be transformed to reconstructed blocks by using the inverse approach 3, transformed blocks at an even row and the odd column may be transformed to reconstructed blocks by using the inverse approach 4, and transformed blocks at the even row and the even column may be transformed to reconstructed blocks by using the inverse approach 1.

[0284] In some embodiments, two adjacent transformed blocks may share pixels in the same column. The two adjacent transformed blocks may be referred to as a first transformed block pair. The first transformed block pair may include a first transformed block and a second transformed block. The first transformed block is located to a left of the second transformed block. Reserved pixels in the first transformed block include pixels in a rightmost column in the first transformed block, and reserved pixels in the second transformed block include pixels in a leftmost column in the second transformed block, and the pixels in the rightmost column in the first transformed block are the pixels in the leftmost column of the second transformed block.

[0285] FIG. 12 may be also used to depict the first transformed block pair.

[0286] Referring to FIG. 12, the first transformed block pair includes 11x5 pixels. Both the first transformed block and the second transformed block include 6x5 pixels. More specifically, the first transformed block includes pixels in a first 6 columns among the 11 columns illustrated in FIG. 12, and the second transformed block includes pixels in a last 6 columns among the 11columns. In other words, pixels in a sixth column belong to both the first transformed block and the second transformed blocks. The sixth column is a rightmost column of the first transformed block, and the sixth column is a leftmost column of the second transformed block. Reserved pixels of the first transformed block include pixels in the sixth column. For example, the first transformed block may be the transformed block A2 or the transformed block A5. Reserved pixels of the second transformed block include pixels in the sixth column. For example, the second transformed block may be the transformed block Al and the transformed block A3.

[0287] In some embodiments, two adjacent transformed blocks may share pixels in the same row. The two adjacent transformed blocks may be referred to as a second transformed block pair. The second transformed block pair may include a third transformed block and a fourth transformed block. The third transformed block is located above the fourth transformed block. Reserved pixels in the third transformed block include pixels in a bottom row in the third transformed block, and reserved pixels in the fourth transformed block include pixels in a top row in the fourth transformed block, and the pixels in the bottom row in the third transformed block are the pixels in the top row of the fourth transformed block.

[0288] FIG. 13 may be also used to depict the second transformed block pair.

[0289] Referring to FIG. 13, the second transformed block pair includes 6x9 pixels. Both the third transformed block and the fourth transformed block include 6x5 pixels. More specifically, the third transformed block includes pixels in a first 5 rows among the 9 rows illustrated in FIG. 13, and the fourth transformed block includes pixels in a last 5 rows among the 9 rows. In other words, pixels in a fifth row belong to both the third transformed block and the fourth transformed blocks. The fifth row is a bottom row of the third transformed block, and the fifth row is a top row of the fourth transformed block. Reserved pixels of the third transformed block include pixels in the fifth row. For example, the third transformed block may be the transformed block A2 or the transformed block A3. Reserved pixels of the fourth transformed block include pixels in the fifth row. For example, the fourth transformed block may be the transformed block Al and the transformed block A4.

[0290] In general, if a size of a target block is PXQ, a size of the first transformed block pair may be (2xP-l)xQ, and pixels in a Pthcolumn among 2xP-l columns may belong to both the first transformed block and the second transformed block. Similarly, if a size of a transformedblock is PxQ, a size of the second transformed block pair may be P*(2xQ-l), and pixels in a Qthrow among 2xQ-l rows may belong to both the third transformed block and the fourth transformed block.

[0291] Referring to the formulas 2.1 to 2.4, the value of the reconstructed pixel is determined according to values of a group of pixels, the group of pixels includes the corresponding transformed pixel and its neighboring pixels. Further, referring to the formulas 2.1 to 2.4, each pixel in the group of pixels has a weight. For convenience, “first neighboring pixel”, “second neighboring pixel” and “third neighboring pixel” are used to referred to as the neighboring pixels of the corresponding transformed pixels. For example, for the inverse approach 1, the first neighboring pixel may be the left neighboring pixel, the second neighboring pixel may be the top-left neighboring pixel, and the third neighboring pixel may be the top neighboring pixel; for the inverse approach 2, the first neighboring pixel may be the right neighboring pixel, the second neighboring pixel may be the bottom-right neighboring pixel, and the third neighboring pixel may be the bottom neighboring pixel; for the inverse approach 3, the first neighboring pixel may be the left neighboring pixel, the second neighboring pixel may be the bottom-left neighboring pixel, and the third neighboring pixel may be the bottom neighboring pixel; for inverse the approach 4, the first neighboring pixel may be the right neighboring pixel, the second neighboring pixel may be the top-right neighboring pixel, and the third neighboring pixel may be the top neighboring pixel. Correspondingly, the formulas 2.1 to 2.4 may be rewritten as:

[0292] V_RP=((WO+WI+W2+W3)X V_NP - wi x V_RPm - w2*V_RPn2 - w3xV_RPn3) / wo, (2.5)

[0293] where V_RP is a pixel value of a reconstructed pixel in the tthreconstructed block, V_NP is a pixel value of a transformed pixel corresponding to the reconstructed pixel in the tthreconstructed block, the reconstructed pixel is any one of the reconstructed pixels in the tthreconstructed block, V_RPniis a pixel value of a first neighboring pixel of the reconstructed pixel, V_RPn2is a pixel value of a second neighboring pixel of the reconstructed pixel, V_RPn3 is a pixel value of a third neighboring pixel of the reconstructed pixel, wo is a weight of the transformed pixel FP, wi is a weight of the first neighboring pixel, w2is a weight of the second neighboring pixel, and w3is a weight of the third neighboring pixel.

[0294] In some embodiments, values of wo, wi, w2, and w3are the same.

[0295] In some other embodiments, values of at least two of wo, wi, W2, and W3 are different.

[0296] In some embodiments, wo may be a preset value. In some other embodiments, wo may be obtained according to the input data (that is, the raw data). For convenience, a value obtained according to the input data may be referred to as a calculated value. Similarly, wi may be a preset value or a calculated value, W2 may be a preset value or a calculated value, W3 may be a preset value or a calculated value. In other words, in some embodiments, wo, wi, W2, and W3 are preset values; in some other embodiments, wo, wi, W2, and W3 are calculated values; in some other embodiments, one or more of wo, wi, W2, and W3 are calculated values while others are preset values.

[0297] In some embodiments, the preset value may be an empirical value. In some other embodiments, the preset value may be a simulated value obtained by simulation. In some other embodiments, the preset value may be trained by a neural network.

[0298] The calculated value may be calculated according to statistics of pixels in the input data. In some embodiments, the statistics of pixels in the input data may include statistics of all pixels in the input data. For example, the statistics of all pixels in the input data may include at least one of the followings: a maximum intensity of all pixels in the input data, a mean intensity of all pixels in the input data, a variance intensity of all pixels in the input data, or the like.

[0299] In some embodiments, the statistics of pixels in the input data may include statistics of the first pixels in the input data. For example, the statistics of the first pixels in the input data may include at least one of the followings: a maximum intensity of the first pixels, a mean intensity of the first pixels, a variance intensity of the first pixels, or the like.

[0300] In some embodiments, the statistics of pixels in the input data may include statistics of the second pixels in the input data. For example, the statistics of the second pixels in the input data may include at least one of the followings: a maximum intensity of the second pixels, a mean intensity of the second pixels, a variance intensity of the second pixels, or the like.

[0301] In some embodiments, the statistics of pixels in the input data may include both the statistics of the first pixels in the input data and the statistics of the second pixels in the input data.

[0302] For example, in some embodiments, the weight of the first pixel wo satisfies:

[0303] where wo is the weight of the first pixel, Kbase is a base K-value, and kVar is a fixed scaling coefficient, maxi is a maximum intensity of the first pixels, mean is a mean intensity of the first pixels, VT is a constant, variance is a variance intensity of the first pixels, threshold is a fixed constant, sigmoid ( ) represents a sigmoid function. The formula 2.1 allows less smoothing for dark and high contrast areas, which are more important for ISP. Distortion in low-intensity areas are more important for the image signal processing. High intensity pixels could be distorted more for human eye.

[0304] All target blocks may be processed with a K-value, and the K-value may be not lower than base K-value. The base K-value is a preset value. The higher the base K-value, the lower compression and the lower distortion may be achieved. For example, possible values of the base K-value may be 5 to 40. For example, the base K-value may be equal to 16, 18, 20, or the like. The base K-value may be an integer or a non-integer value.

[0305] kVAR determines range of K-value depending on average intensity and variance. In general, Kbase < K-value < Kbase + kVAR.

[0306] In some embodiments, possible values of the VT are 2 to 10. For example, VT may be equal to 2, 5, 8, or the like.

[0307] In some embodiments, possible values of the threshold are 20 to 200. For example, the threshold may be equal to 20, 50, 100, or the like.

[0308] In some other embodiments, the weight of the first pixel wo satisfies:

[0310] In some other embodiments, the weight of the first pixel wo satisfies:

[0312] In some other embodiments, the weight of the first pixel wo satisfies:

[0314] Meaning and possible values of function notations and function symbols in formulas 2.2 to 2.4 are the same as those in the formula 2.1. For the sake of brevity, it will not beelaborated here.

[0315] In some embodiments, the weight of the first neighboring pixel wi, the weight of the second neighboring pixel wi, and the weight of the third neighboring pixel W3 may satisfy: wi = W2 = W3. For example, in some embodiments, wi = W2 = W3 = 1.

[0316] In some embodiments, wo may be equal to the K-value, and wi = W2 - W3 = 1.

[0317] In some embodiments, wo, wi, W2, and W3 may be determined according to K-value.Table 3 illustrates a relationship between the K-value and wo, wi, W2, and W3.Table 3

[0318] There is some tradeoff between compression ratio and reconstruction quality (e.g., peak signal-to-noise ratio (PSNR). This tradeoff is parametrized by K-value. If the K-value is higher, the PSNR is higher and the compression ratio is lower. If the K-value is lower, the PSNR is lower and the compression ratio is higher. As a result, to achieve a higher compression ratio, a lower K-value should be selected; to achieve a higher PSNR, a higher K-value should be selected. For given set of RAW files, different K- values may be tried (single value is defined for whole set), a selected K-value is a K-value that results in highest compression ratio withoutsignificant loss of quality. The quality may be evaluated by objective metrics (e.g. PSNR) or subjective properties (e.g. no visual artifacts in the RGB images obtained after ISP).

[0319] FIG. 16 illustrates a schematic block diagram of an encoding device according to some embodiments of the present application. Referring to FIG. 16, an encoding device 1600 includes an obtaining module 1601, a preprocessing module 1602 and an encoding module 1603.

[0320] The obtaining module 1601 is configured to obtain input data corresponding to raw data.

[0321] The raw data is obtained by an image sensor, and the input data includes a plurality of target pixels and a plurality of first reserved pixels.

[0322] The preprocessing module 1602 is configured to perform a smoothing transform on the plurality of target pixels of the input data to obtain transformed data.

[0323] The transformed data includes a plurality of first transformed pixels and the plurality of first reserved pixels. The plurality of first transformed pixels is in one-to-one correspondence with the plurality of target pixels, and a pixel value of each of the plurality of first transformed pixels is determined according to pixel values of a corresponding target pixels and at least one neighboring pixels of the corresponding target pixel.

[0324] The encoding module 1603 is configured to perform an encoding operation on the transformed data to obtain encoded data.

[0325] The encoding device 1600 may be the aforementioned encoding device. The preprocessing module 1602 and the encoding module 1603 may be a processor (e.g., a general purpose processor). In some embodiments, the obtaining module 1601 may be a camera module including lens and a camera sensor. In some other embodiments, the obtaining module 1601 may be a receiving circuit that may receive the input data from a camera sensor.

[0326] Details on how to process the input data may refer to the above-mentioned embodiments and will not be described here.

[0327] FIG. 17 illustrates a schematic block diagram of a decoding device according to some embodiments of the present application. Referring to FIG. 17, a decoding device 1700 includes a decoding module 1701 and a postprocessing module 1702.

[0328] The decoding module 1701 is configured to perform a decoding operation onencoded data to obtain decoded data.

[0329] The encoded data is determined by the abovementioned encoding device 1600. The decoded data includes a plurality of second transformed pixels and a plurality of second reserved pixels.

[0330] The postprocessing module is configured to perform an inverse smoothing transform on the plurality of second transformed pixels of the decoded data to obtain reconstructed data.

[0331] The reconstructed data comprises a plurality of reconstructed pixels and the plurality of second reserved pixels. The plurality of reconstructed pixels is in one-to-one correspondence with the plurality of second transformed pixels. A pixel value of each of the plurality of reconstructed pixels is determined according to pixel values of a corresponding second transformed pixel and at least one neighboring pixel of the reconstructed pixel.

[0332] In some embodiments, the decoding device 1700 may further includes an ISP module. The ISP module may be configured to obtain reconstructed data from the processing module 1702 and perform an image signal processing operation on the reconstructed data.

[0333] In some other embodiments, the decoding device 1700 may further includes a transmitting module. The transmitting module may be configured to transmit the reconstructed to an electronic device including an ISP module (hereinafter referred to a “ISP device”). The ISP device may receive the reconstructed and perform an image signal processing operation on the reconstructed data.

[0334] The decoding device 1700 may be the aforementioned decoding device. The postprocessing module 1702 and the decoding module 1701 may be a processor (e.g., a general purpose processor).

[0335] Details on how to process the input data may refer to the above-mentioned embodiments and will not be described here.

[0336] The present application further provides a signal processing system. The signal processing system may include the encoding device 1600 and the decoding device 1700.

[0337] In some embodiments, an electronic device may include the signal processing system. In other words, the electronic device may include the encoding device 1600 and the decoding device 1700. In this situation, the encoding device 1600 and the decoding device 1700may be regarded as two modules of the electronic device. The preprocessing module 1602, the encoding module 1603, the decoding module 1701 and the postprocessing module 1702 may be performed by one or more processors of the electronic device.

[0338] In some embodiments, the encoding device 1600 may transmit the encoded data to the decoding device 1700 through a communication bus. In some other embodiments, the encoding device 1600 may transmit the encoded data to the decoding device 1700 via a memory module. For example, the encoding device 1600 may save the encoded data in the memory module, and the decoding module 1700 may read the memory module to obtain the encoded data. The memory module may be a volatile memory and a nonvolatile memory. For example, the memory may be a read-only memory (ROM), a random access memory (RAM), or the like.

[0339] For example, as shown in FIG. 18, an electronic device 1800 may include a camera lens 1801, an image sensor 1802, a processor 1803, and a memory 1804. The memory 1804 may be configured to store code, instructions, and the like executed by the processor 1803. The memory 1804 may be further configured to store data obtained during the signal processing procedure. For example, the memory 1804 may be further configured to store the input data, the encoded data, the decoded data, the reconstructed data, or the like.

[0340] The image sensor 1802 may be configured to obtain an input data by using the camera lens 1801. The input data includes a plurality of target pixels and a plurality of first reserved pixels.

[0341] The processor 1803 reads information and instructions in the memory 1804 and completes the steps of the foregoing methods. For example, the processor 1804 may be perform steps performed by the preprocessing module 1602, the encoding module 1603, the decoding module 1702, the postprocessing module 1701. For example, the processor 1804 may be configured to perform a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, perform an encoding operation on the transformed data to obtain an encoded data, perform a decoding operation on the encoded data to obtain a decoded data, and perform an inverse smoothing transform on a plurality of transformed pixels of the decoded data to obtain a reconstructed data.

[0342] In some embodiments, the processor 1803 may be further configured to perform an image signal processing operation on the reconstructed data.

[0343] In some embodiments, the electronic device 1800 may be the terminal device 200 illustrated in FIG. 2. The terminal device may be a mobile phone, a tablet computer, a notebook computer, an augmented reality (AR) device, a virtual reality (VR) device, a digital camera, a digital video camcorder, an action camera, a security camera or the like.

[0344] In some other embodiments, the encoding device 1600 may be an electronic device and the decoding device may be another electronic device. In other words, the system may include two electronic device, one of the two electronic devices is the encoding device 1600, while another electronic device is the decoding device 1700. In this situation, the encoding device 1600 may transmit the encoded data to the decoding device 1700 via wired or wireless communication.

[0345] For example, as shown in FIG. 19, an electronic device 1900 may include a camera lens 1901, an image sensor 1902, a processor 1903, a memory 1904, and a transmitter 1905. The memory 1904 may be configured to store code, instructions, and the like executed by the processor 1903. The memory 1904 may be further configured to store data obtained during the signal processing procedure. For example, the memory 1904 may be further configured to store the input data, the encoded data, or the like.

[0346] The image sensor 1902 may be configured to obtain input data by using the camera lens 1901. The input data includes a plurality of target pixels and a plurality of first reserved pixels.

[0347] The processor 1903 reads information and instructions in the memory 1904 and completes the steps of the foregoing methods. For example, the processor 1904 may be perform steps performed by the preprocessing module 1602 and the encoding module 1603. For example, the processor 1904 may be configured to perform a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, perform an encoding operation on the transformed data to obtain encoded data.

[0348] The transmitter 1905 may be configured to transmit the encoded data to another electronic device, e.g., an electronic device 2000 illustrated in FIG. 20.

[0349] Referring to FIG. 20, the electronic device 2000 includes a receiver 2001, a processor 2002, and a memory 2003. The memory 2003may be configured to store code, instructions, and the like executed by the processor 2002. The memory 2003 may be furtherconfigured to store data obtained during the signal processing procedure. For example, the memory 2003may be further configured to store the reconstructed data, the decoded data, or the like.

[0350] The receiver 2001 may be configured to receive the encoded data from the electronic device 1900.

[0351] The processor 2002 reads information and instructions in the memory 2003 and completes the steps of the foregoing methods. For example, the processor 2002 may be perform steps performed by the postprocessing module 1701 and the decoding module 1702. For example, the processor 2002 may be configured to perform a decoding operation on the encoded data to obtain decoded data, and perform an inverse smoothing transform on a plurality of second transformed pixels of the decoded data to obtain reconstructed data.

[0352] In some embodiments, the electronic device 1900 may be the terminal device 200 illustrated in FIG. 2. The terminal device may be mobile phone terminal, a tablet computer, a notebook computer, an AR device, a VR device, a digital camera, a digital video camcorder, an action camera, a security camera or the like.

[0353] In some embodiments, the processor 2002 may be further configured to perform an image signal processing operation on the reconstructed data.

[0354] In some other embodiments, the electronic device 2000 may further include a transmitter. The transmitter is configured to transmit the reconstructed data to the ISP device that is configured to perform the image signal processing operation.

[0355] In some embodiments, the electronic device 2000 may be the terminal device 200 illustrated in FIG. 2. The terminal device may be mobile phone terminal, a tablet computer, a notebook computer, an AR device, a VR device, a digital camera, a digital video camcorder, an action camera, a security camera or the like. In some other embodiments, the electronic device 2000 may be a computing device or a computing system. For example, the electronic device 2000 may be a personal computer, a server, or a cloud service system

[0356] The processor described this specification may be an integrated circuit chip and has a signal processing capability. In an implementation process, steps of the foregoing method embodiments may be completed by using a hardware integrated logic circuit in the processor, or by using instructions in a form of software. The processor may be a general purpose processor,a central processing unit (CPU), a system on chip (SoC) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor may implement or perform the methods, the steps, and the logical block diagrams that are disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the methods disclosed with reference to the embodiments of the present application may be directly performed and completed by the processor, or may be performed and completed by using a combination of hardware in the processor and a software module. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor reads information in the memory and completes the steps of the foregoing methods in combination with hardware in the processor.

[0357] It may be understood that the memory described this specification may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a programmable read- only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM) and is used as an external cache. By way of example rather than limitation, many forms of RAMs may be used, and are, for example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DR RAM).

[0358] The present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the aforementioned method.

[0359] The present application provides a chip system. The chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processoris configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the aforementioned method.

[0360] The present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to perform the aforementioned method.

[0361] The present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the aforementioned method performed by the encoding device.

[0362] The present application provides a chip system. The chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the aforementioned method performed by the encoding device.

[0363] The present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to perform the aforementioned method performed by the encoding device.

[0364] The present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the aforementioned method performed by the decoding device.

[0365] The present application provides a chip system. The chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the aforementioned method performed by the decoding device.

[0366] The present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to perform the aforementioned method performed by the decoding device.

[0367] The terms such as "first" and "second" mentioned in the specification are merely for a descriptive purpose, and cannot be understood as indicating or implying relative importance,or implicitly indicating a quantity of indicated technical features. Therefore, the features defined by "first" and "second" can explicitly or implicitly include one or more features.

[0368] In the embodiments of the present application, “at least one” means one or more, and “a plurality of’ means two or more. The term “and / or” describes an association relationship between associated objects and represents that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character “I” generally indicates an “or” relationship between the associated objects. “At least one of the following” and a similar expression thereof refer to any combination of these items, including any combination of one item or a plurality of items. For example, at least one of a, b, and c may indicate: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c may be singular or plural.

[0369] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0370] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0371] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may beimplemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0372] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0373] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0374] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disc.The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. An image signal processing method, wherein comprising: obtaining input image data corresponding to raw image data, wherein the input data comprises a plurality of target pixels and a plurality of first reserved pixels; performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, wherein the smooth transform is a reversible transform, the transformed data comprises a plurality of first transformed pixels and the plurality of first reserved pixels, the plurality of first transformed pixels is in one-to-one correspondence with the plurality of target pixels, and a pixel value of each of the plurality of first transformed pixels is determined according to pixel values of a corresponding target pixel and at least one neighboring pixel of the corresponding target pixel; performing an encoding operation on the transformed data to obtain encoded data.

2. The method according to claim 1 , wherein the performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, comprises: dividing the input data into F blocks, wherein the F blocks comprises F i target blocks, F is a positive integer, F i is a positive integer, F i is less than or equal to F ; transforming target pixels among the F i target blocks into first transformed pixels to obtain the transformed data, wherein the transformed data comprises Fi transformed blocks, the Fi transformed blocks are in one-to-one correspondence with the Fi target blocks, each of the Fl transformed blocks are determined according to a corresponding target block, and a size of the target block is a same as a size of the transformed block.

3. The method according to claim 2, wherein the transforming the target pixels among the Fi target blocks into the first transformed pixels to obtain the transformed data, comprises: transforming target pixels in a tthtarget block among the Fi target blocks into first transformed pixels to obtain a tthtransformed block among the F i transformed blocks, wherein the tthtransformed block comprises first transformed pixels determined based on the target pixels in the tthtarget block, and the tthtransformed block further comprises first reserved pixelsin the tthtarget block, t=l , ..., Fi.

4. The method according to claim 3, wherein a pixel value of any one of the first transformed pixels in the 1thtransformed block satisfies:V_NP=(W0xV_GP+WlxV_GPnl+W2xV_GPn2+ 3xV_GPn3) / (W0+Wl+W2+W3), wherein V_NP is a pixel value of a first transformed pixel NP in the tthtransformed block, V_GP is a pixel value of a target pixel GP corresponding to the first transformed pixel NP in the tthtarget block, the first transformed pixel NP is any one of the first transformed pixels in the Ithtransformed block, V_GPniis a pixel value of a first neighboring pixel of the target pixel GP, V_GPn2 is a pixel value of a second neighboring pixel of the target pixel GP, V_GPn3 is a pixel value of a third neighboring pixel of the target pixel GP, wo is a weight of the target pixel GP, wi is a weight of the first neighboring pixel, W2 is a weight of the second neighboring pixel, and W3 is a weight of the third neighboring pixel.

5. The method according to claim 4, wherein before the performing a smoothing transform on the plurality of target pixels of the input data to obtain transformed data, the method further comprises: determining a variance of the plurality of target pixels meets a first preset condition; and / or determining a mean of the plurality of target pixels meets a second preset condition.

6. The method according to claim 4 or 5, wherein when the first reserved pixels in the tthtarget block comprises pixels in a top row and a leftmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a left neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a top-left neighboring pixel of the target pixel GP, and the third neighboring pixel of the target pixel GP is a top neighboring pixel of the target pixel GP.

7. The method according to claim 6, wherein the Fi target blocks are arranged in M rows and N columns, the Ithtarget block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 0,wherein mod represents a modulo operation.

8. The method according to claim 4 or 5, wherein when the first reserved pixels in the t* target block comprises pixels in a bottom row and a rightmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a right neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a bottom-right neighboring pixel of the target pixel GP, the third neighboring pixel of the target pixel GP is a bottom neighboring pixel of the target pixel GP.

9. The method according to claim 8, wherein the Fi target blocks are arranged in M rows and N columns, the t* target block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 1, wherein mod represents a modulo operation.

10. The method according to claim 4 or 5, wherein when the first reserved pixels in the tthtarget block comprises pixels in a bottom row and a leftmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a left neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a bottom-left neighboring pixel of the target pixel GP, the third neighboring pixel of the target pixel GP is a bottom neighboring pixel of the target pixel GP.

11. The method according to claim 10, wherein the F i target blocks are arranged in M rows and N columns, the t* target block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 0, wherein mod represents a modulo operation.

12. The method according to claim 4 or 5, wherein when the first reserved pixels in the tthtarget block comprises pixels in a top row and a rightmost column in the tthtarget block, the first neighboring pixel of the target pixel GP is a right neighboring pixel of the target pixel GP, the second neighboring pixel of the target pixel GP is a top-right neighboring pixel of the target pixel GP, the third neighboring pixel of the target pixel GP is a top neighboring pixel of the target pixel GP.

13. The method according to claim 12, wherein the Fi target blocks are arranged in M rows and N columns, the tthtarget block is in an 1throw of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 1, wherein mod represents a modulo operation.

14. The method according to any one of claims 3 to 13, wherein the Fi target blocks comprises a plurality of first target block pairs, each of the plurality of first target block pairs comprises a first target block and a second target block, the first target block is located to a left of the second target block, first reserved pixels in the first target block comprise pixels in a rightmost column in the first target block, first reserved pixels in the second target block comprise pixels in a leftmost column in the second target block, and the pixels in the rightmost column in the first target block are the pixels in the leftmost column of the second target block.

15. The method according to any one of claims 3 to 14, wherein the Fi target blocks comprises a plurality of second target block pairs, each of the plurality of second target block pairs comprises a third target block and a fourth target block, the third target block is located above the fourth target block, first reserved pixels in the third block comprises pixels in the bottom row in the third block; first reserved pixels in the fourth blocks comprises pixels in the top row in the fourth block; the pixels in the bottom row in the third block are the pixels in the top row in the fourthblock.

16. The method according to any one of 4 to 13 wherein wo, wi, W2, and W3 are not identical.

17. The method according to any one of claims 4 to 13, and 16, wherein wo satisfies:where Kbase is a base K-value, kVar is a fixed scaling coefficient, maxi is a maximum intensity of target pixels corresponding to second transformed pixels in decoded data, mean is a mean intensity of the target pixels, VT is a constant, variance is a variance intensity of the target pixels, threshold is a fixed constant.

18. The method according to any one of claims 4 to 13, 16 and 17, wherein wi, W2, and W3 satisfy:W| = W2 = W3.

19. The method according to any one of claims 1 to 18, wherein the encoded data is used to obtain decoded data by performing a decoding operation, wherein the decoded data comprises a plurality of second transformed pixels and a plurality of second reserved pixels; the plurality of second transformed pixels of the decoded data are used to obtain reconstructed data by performing an inverse smoothing transform, wherein the reconstructed data comprises a plurality of reconstructed pixels and the plurality of second reserved pixels, the plurality of reconstructed pixels is in one-to-one correspondence with the plurality of second transformed pixels, and a pixel value of each of the plurality of reconstructed pixels is determined according to pixel values of a corresponding second transformed pixel and at least one neighboring pixel of the reconstructed pixel; and the reconstructed data is used to obtain an image signal processing result by performing an image signal processing operation.

20. An image signal processing method, wherein comprising: obtaining encoded data; performing a decoding operation on the encoded data to obtain decoded data, wherein the decoded data comprises a plurality of second transformed pixels and a plurality of second reserved pixels; performing an inverse smoothing transform on the plurality of second transformed pixelsof the decoded data to obtain reconstructed data, wherein the reconstructed data comprises a plurality of reconstructed pixels and the plurality of second reserved pixels, the plurality of reconstructed pixels is in one-to-one correspondence with the plurality of second transformed pixels, and a pixel value of each of the plurality of reconstructed pixels is determined according to pixel values of a corresponding second transformed pixel and at least one neighboring pixel of the reconstructed pixel; performing an image signal processing operation on the reconstructed data.

21. The method according to claim 20, wherein the performing an inverse smoothing transform on the plurality of second transformed pixels of the decoded data to obtain reconstructed data, comprises: dividing the decoded data into F blocks, wherein the F blocks comprises F i transformed blocks, F is a positive integer, Fi is a positive integer, Fi is less than or equal to F; transforming second transformed pixels among the Fi transformed blocks into reconstructed pixels to obtain the reconstructed data, wherein the reconstructed data comprises Fi reconstructed blocks, the Fi reconstructed blocks are in one-to-one correspondence with the Fi transformed blocks, each of the Fl reconstructed blocks are determined according to a corresponding transformed block, and a size of the transformed block is a same as a size of the reconstructed block.

22. The method according to claim 21, wherein the transforming the second transformed pixels among the F i transformed blocks into the reconstructed pixels to obtain the reconstructed data, comprises: transforming second transformed pixels in a tthtransformed block among the Fi transformed blocks into reconstructed pixels to obtain a tthreconstructed block among the F i reconstructed blocks, wherein the tthreconstructed block comprises reconstructed pixels obtained according to the second transformed pixels in the tthtransformed block, and the tthreconstructed block further comprises second reserved pixels in the tthtransformed block, t=l, ..., Fi.

23. The method according to claim 22, wherein pixel value of any one of the reconstructed pixels in the tthreconstructed block satisfies:V_RP=((WO+W1+W2+W3)XV_NP - Wl ><V_RPnl - W2xV_RPn2 - W3XV_RPn3) / wo,wherein V_RP is a pixel value of a reconstructed pixel RP in the tthreconstructed block, V_NP is a pixel value of a second transformed pixel NP corresponding to the reconstructed pixel RP in the tthreconstructed block, the reconstructed pixel RP is any one of the reconstructed pixels in the tthreconstructed block, V_RPniis a pixel value of a first neighboring pixel of the reconstructed pixel RP, V_RPn2 is a pixel value of a second neighboring pixel of the reconstructed pixel RP, V_RPn3 is a pixel value of a third neighboring pixel of the reconstructed pixel RP, wo is a weight of the second transformed pixel FP, wi is a weight of the first neighboring pixel, W2 is a weight of the second neighboring pixel, and W3 is a weight of the third neighboring pixel.

24. The method according to claim 23, wherein when the second reserved pixels in the tthtransformed block comprises pixels in a top row and a leftmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a left neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a top-left neighboring pixel of the reconstructed pixel RP, and the third neighboring pixel of the reconstructed pixel RP is a top neighboring pixel of the reconstructed pixel RP.

25. The method according to claim 24, wherein the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an i* row of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 0, wherein mod represents a modulo operation.

26. The method according to claim 23, wherein when the second reserved pixels in the tthtransformed block comprises pixels in a bottom row and a rightmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a right neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a bottom-right neighboringpixel of the reconstructed pixel RP, the third neighboring pixel of the reconstructed pixel RP is a bottom neighboring pixel of the reconstructed pixel RP.

27. The method according to claim 26, wherein the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 1, wherein mod represents a modulo operation.

28. The method according to claim 23, wherein when the second reserved pixels in the tthtransformed block comprises pixels in a bottom row and a leftmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a left neighboring pixel of the reconstructed pixel RP, the second neighboring pixel of the reconstructed pixel RP is a bottom-left neighboring pixel of the reconstructed pixel RP, the third neighboring pixel of the reconstructed pixel RP is a bottom neighboring pixel of the reconstructed pixel FP.

29. The method according to claim 28, wherein the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 1; j mod 2 = 0, wherein mod represents a modulo operation.

30. The method according to claim 23, wherein when the second reserved pixels in the tthtransformed block comprises pixels in a top row and a rightmost column in the tthtransformed block, the first neighboring pixel of the reconstructed pixel RP is a right neighboring pixel of the reconstructed pixel FP, the second neighboring pixel of the reconstructed pixel RP is a top-right neighboring pixelof the reconstructed pixel FP, the third neighboring pixel of the reconstructed pixel RP is a top neighboring pixel of the reconstructed pixel FP.3 1. The method according to claim 30, wherein the Fi transformed blocks are arranged in M rows and N columns, the tthtransformed block is in an ithrow of the M rows and a jthcolumn of the N columns, the i and the j satisfy: i mod 2 = 0; j mod 2 = 1, wherein mod represents a modulo operation.

32. The method according to any one of claims 22 to 31 , wherein the F i transformed blocks comprises a plurality of first transformed block pairs, each of the plurality of first transformed block pairs comprises a first transformed block and a second transformed block, the first transformed block is located to a left of the second transformed block, second reserved pixels in the first transformed block comprise pixels in a rightmost column in the first transformed block, second reserved pixels in the second transformed block comprise pixels in a leftmost column in the second transformed block, and the pixels in the rightmost column in the first transformed block are the pixels in the leftmost column of the second transformed block.

33. The method according to any one of claims 22 to 26, wherein the Fi transformed blocks comprises a plurality of second transformed block pairs, each of the plurality of second transformed block pairs comprises a third transformed block and a fourth transformed block, the third transformed block is located above the fourth transformed block, second reserved pixels in the third block comprises pixels in the bottom row in the third block; second reserved pixels in the fourth blocks comprises pixels in the top row in the fourth block; the pixels in the bottom row in the third block are the pixels in the top row in the fourth block.

34. The method according to any one of 23 to 31, wherein wo, wi, W2, and wa are not identical.

35. The method according to any one of claims 23 to 31, and 34, wherein wo satisfies:where Kbase is a base K-value, kVar is a fixed scaling coefficient, maxi is a maximum intensity of target pixels corresponding to the second transformed pixels in the decoded data, mean is a mean intensity of the target pixels, VT is a constant, variance is a variance intensity of the target pixels, threshold is a fixed constant.

36. The method according to any one of claims 23 to 31, 34 and 35, wherein wi, W2, andW3 satisfy:W1 = W2 = W3.

37. The method according to any one of claims 20 to 36, wherein the encoded data is obtained by performing an encoding operation on transformed data, and the transformed data is obtained by performing a smoothing transform on a plurality of target pixel of input data, wherein the input data comprises the plurality of target pixels and a plurality of first reserved pixels, the transformed data comprises a plurality of first transformed pixels and the plurality of first reserved pixels, the plurality of first transformed pixels is in one-to-one correspondence with the plurality of target pixels, and a pixel value of each of the plurality of first transformed pixels is determined according to pixel values of a corresponding target pixel and at least one neighboring pixel of the corresponding target pixel.

38. An image processing system, wherein the image processing system comprises modules for implementing the method according to any one of claims 1 to 19, and / or, the method according to any one of claims 20 to 37.

39. computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 19, and / or, the method according to any one of claims 20 to 37.

40. An electronic device, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that the electronic device performs the method according to any one of claims 1 to 19, and / or, the method according to any one ofclaims 20 to 37.

41. A chip system, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method according to any one of claims 1 to 19, and / or, the method according to any one of claims 20 to 37.

42. A computer program product, wherein when the computer program product runs on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 19, and / or, the method according to any one of claims 20 to 37.

Images