Image processing method and image processing device

Abstract

This invention provides an image processing method and an image processing apparatus. In the method, an input image is first encoded to output a first noisy image. The first noisy image is then denoised to output a first denoised image. The input image is then denoised based on the first denoised image to output a first image. This improves image quality.

Description

Technical Field

[0001] This invention relates to an image processing technique, and more particularly, to an image processing method and an image processing apparatus. Background Technology

[0002] With advancements in display and network technologies, the demand for high-definition images is gradually increasing. Image quality may degrade during capture, conversion, and / or transmission. Nowadays, image quality can be restored or even improved through post-processing. Summary of the Invention

[0003] This invention relates to an image processing method and an image processing apparatus that processes images based on thumbnail images to improve image quality.

[0004] According to embodiments of the present invention, the image processing method includes (but is not limited to) the following steps: performing a first encoding on an input image to output a first noisy image; denoising the first noisy image to output a first denoised image; and denoising the input image based on the first denoised image to output a first image.

[0005] According to an embodiment of the present invention, an image processing apparatus includes (but is not limited to) a memory and a processor. The memory is used to store program code. The processor is coupled to the memory. The processor is configured to load and execute the program code to perform a first encoding on an input image to output a first noisy image, to denoise the first noisy image to output a first denoised image, and to denoise the input image based on the first denoised image to output a first image.

[0006] According to embodiments of the present invention, the image processing method includes (but is not limited to) the following steps: inputting a standard dynamic range image, converting the image data format through spatial depth calculation, and outputting a first image data. Inputting the first image data into a first encoding calculation to output second image data. Inputting the second image data into a global calculation to output third image data. Inputting the second image data and the third image data into a first decoding calculation to output fourth image data. Inputting the first image data and the fourth image data into a second decoding calculation to output a high dynamic range image.

[0007] Based on the above, the image processing method and image processing apparatus according to embodiments of the present invention denoise the input image by referring to the encoded denoised image. This effectively improves image quality. Attached Figure Description

[0008] The accompanying drawings are included to further illustrate the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0009] Figure 1This is a block diagram of components of an image processing apparatus according to an embodiment of the present invention;

[0010] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0011] Figure 3 This is a schematic flowchart of a noise reduction process according to an embodiment of the present invention;

[0012] Figure 4 This is a schematic flowchart of a noise reduction process according to an embodiment of the present invention;

[0013] Figure 5 This is a flowchart of a noise reduction block according to an embodiment of the present invention;

[0014] Figure 6 This is a schematic diagram of the first pixel of an original image according to an embodiment of the present invention;

[0015] Figure 7A This is a schematic diagram of the second pixel of an input image according to an embodiment of the present invention;

[0016] Figure 7B This is a schematic diagram of average pooling according to an embodiment of the present invention;

[0017] Figure 8 This is a schematic diagram of pixel unpacking according to an embodiment of the present invention;

[0018] Figure 9A This is a flowchart of a two-stage optimization network according to an embodiment of the present invention;

[0019] Figure 9B This is a flowchart of a two-stage optimization network according to another embodiment of the present invention;

[0020] Figure 9C This is a flowchart of an image processing method according to an embodiment of the present invention;

[0021] Figure 10 This is a flowchart of high dynamic range (HDR) processing according to an embodiment of the present invention;

[0022] Figure 11 This is a flowchart of high dynamic range processing according to an embodiment of the present invention;

[0023] Figure 12 This is a schematic diagram of space-to-depth conversion according to an embodiment of the present invention;

[0024] Figure 13 This is a flowchart of the second encoding according to an embodiment of the present invention;

[0025] Figure 14A This is a flowchart of a convolution operation according to an embodiment of the present invention;

[0026] Figure 14B This is a flowchart of a convolution operation according to an embodiment of the present invention;

[0027] Figure 15 This is a flowchart of a high dynamic range block according to an embodiment of the present invention;

[0028] Figure 16 This is a schematic diagram of averaging calculation according to an embodiment of the present invention;

[0029] Figure 17 This is a schematic diagram of a multiplication operation according to an embodiment of the present invention;

[0030] Figure 18 This is a flowchart of a high dynamic range processing block according to an embodiment of the present invention;

[0031] Figure 19A This is a flowchart of a convolution operation according to an embodiment of the present invention;

[0032] Figure 19B This is a flowchart of a convolution operation according to an embodiment of the present invention;

[0033] Figure 20 This is a schematic diagram of depth-to-space conversion according to an embodiment of the present invention.

[0034] Explanation of icon numbers

[0035] 100: Image processing device;

[0036] 110: Memory;

[0037] 130: Processor;

[0038] S210~S230, S301~S303, S401~S407, S501~S503, S910~S920, S101~105, S1011, S1021, S111~S119, S122~S124, S1221~S1227, S1031~S1039, S131~134, S1321~S1327: Steps;

[0039] IM11: Input image;

[0040] DSM11: First noisy image;

[0041] DNM11, DNM12, DNM15, DNM23: First denoised images;

[0042] DNM21, DNM22: Second denoised images;

[0043] OM11, OM12, OM1: First image;

[0044] DSM12: Second noisy image;

[0045] CM1: First combined image;

[0046] Y0000, Y0001, Y0010, Y0011, Y1000, Y1001, Y1010, Y1011, Y0100, Y0101, Y0110, Y0111, Y1100, Y1101, Y1110, Y1111, Y00, Y01, Y10, Y11: Brightness values;

[0047] U00, U10, U01, U11, U: chromaticity values;

[0048] V00, V10, V01, V11, V: Concentration values;

[0049] DSM21: First Standard Dynamic Range Image;

[0050] DSM22: Second Standard Dynamic Range Image;

[0051] HDR11, HDR21: First high dynamic range images;

[0052] OM21, OM22: Second images;

[0053] DSM: Third Standard Dynamic Range Image;

[0054] DSM24: Fourth Standard Dynamic Range Image;

[0055] HDR22, HDR12: Second highest dynamic range images;

[0056] HDR23: The third highest dynamic range image;

[0057] HDR24: The fourth highest dynamic range image;

[0058] CM2: Second combined image;

[0059] CIM1, CIM2: Combined images. Detailed Implementation

[0060] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component symbols are used in the drawings and description to denote the same or similar parts.

[0061] Figure 1This is a block diagram of components of an image processing apparatus 100 according to an embodiment of the present invention. Please refer to... Figure 1 The image processing device 100 may be a smartphone, tablet computer, server, wearable device, smart home appliance, or other type of electronic device. The image processing device 100 includes (but is not limited to) a memory 110 and a processor 130.

[0062] The memory 110 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar component. In one embodiment, the memory 110 is used to record program code, software modules, configuration settings, data, or files (e.g., raw images, input images, processed images, combined images, and output images).

[0063] Processor 130 is coupled to memory 110. Processor 130 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), neural network accelerator, or other similar components or combinations thereof. In one embodiment, processor 130 is used to perform all or part of the operations of image processing device 100, and may load and execute program code, software modules, files, and data recorded in memory 110.

[0064] The methods described in the embodiments of the present invention will be described below in conjunction with the various devices, components, and modules in the image processing apparatus 100. The various processes of this method may be adjusted according to the implementation situation, and are not limited thereto.

[0065] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present invention. Please refer to... Figure 2The processor 130 performs a first encoding on the input image to output a first noisy image (step S210). Specifically, the input image can be a medical image, a surveillance image, a product image, or a person image, but this embodiment of the invention does not limit its type. In addition, the input image can be in red-green-blue (RGB) format or in luminance-chrominance (YUV) format, and this invention does not impose any limitations on this.

[0066] In one embodiment, the first encoding includes a first downsampling. The first downsampling is a process used to reduce the width and / or height of a feature map (e.g., an input image). In one embodiment, the first downsampling is average pooling. Pooling performs a specific operation through a filter (or kernel). For example, average pooling performs an averaging operation, and max pooling selects the largest pixel. Notably, the pixels covered by the sliding filter do not overlap with previously covered pixels. In another embodiment, the first downsampling is max pooling. In other embodiments, the first downsampling may be other processes used to reduce the feature map. Therefore, the width and / or height of the first noisy image is smaller than that of the input image.

[0067] Processor 130 denoises the first noisy image to output a first denoised image (step S220). Specifically, denoising (or noise reduction) is used to remove unwanted noise from an image and may improve image clarity and cleanliness. In one embodiment, processor 130 can identify noise information in the image using a noise detection model based on machine learning algorithms (e.g., Deep Neural Network (DNN), Multi-Layer Perceptron (MLP), Support Vector Machine (SVM), or other machine learning models). Noise information may be, for example, the probability that each pixel is noise or a decision on whether it is noise. The noise detection model is a machine learning model trained based on training samples (and labeling results). For example, the noise detection model includes one or more convolution operations, each using a trained filter. Processor 130 can remove or reduce noise in a feature map (e.g., the input image) based on the noise information.

[0068] In another embodiment, processor 130 can directly remove or reduce noise in the feature map using a denoising model based on machine learning algorithms. In other embodiments, denoising may also be smoothing or frequency domain filtering.

[0069] Processor 130 denoises the input image based on the first denoised image to output a first image (step S230). Specifically, the denoising technique can be found in the description of step S220 and will not be repeated here. It is worth noting that in step S230, processor 130 also refers to the first denoised image for denoising. That is, processor 130 refers to a denoised image that is smaller in width and / or height than the input image and denoises the input image.

[0070] In one embodiment, the processor 130 may combine the first denoised image and the input image to output a first combined image. For example, the processor 130 may assign the first denoised image and the input image to different channels and use the set of these channels as the first combined image. Taking the red-green-blue (RGB) format as an example, the intensity values ​​of the red, green, and blue of a pixel in the first denoised image are in the first to third channels, and the intensity values ​​of the corresponding pixel in the input image are in the fourth to sixth channels. However, depending on the format or number of channels, other allocation methods may also be possible.

[0071] Next, the processor 130 may perform one or more convolution operations on the first combined image to output a first image. In one embodiment, the first combined image is subjected to those convolution operations to output noise information (e.g., through a noise detection model), and the processor 130 may denoise the input image based on the noise information. In another embodiment, the first combined image is subjected to one or more convolution operations to directly output the first image (e.g., through a denoising model).

[0072] For example, Figure 3 This is a schematic flowchart of a noise reduction process according to an embodiment of the present invention. Please refer to... Figure 3 The processor 130 performs a first encoding (e.g., mean pooling) on ​​the input image IM11 (configured as W / 2 × H / 2 × 6 channels, where W is the width of the original image and H is the height of the original image) (step S301) and outputs a first noisy image DSM11 (configured as W / 4 × H / 4 × 6 channels). The processor 130 directly denoises the first noisy image DSM11 (step S302) and outputs a first denoised image DNM11 (configured as W / 4 × H / 4 × 6 channels). The processor 130 unpacks the first denoised image DNM11 (step S303) and outputs a first denoised image DNM12 (configured as W / 2 × H / 2 × 3 channels). Processor 130 denoises the input image IM11 based on the first denoised image DNM12 (step S304) and outputs the first image OM11 (configured as W / 2×H / 2×6 for width×height×channel). In this embodiment, the unpacking procedure described above is, for example, a format conversion.

[0073] In one embodiment, the processor 130 may perform a first encoding on the i-th noise image to output the (i+1)-th noise image. i is a positive integer greater than zero from 1 to M-1, where M is the order number. For example, if i is 2, the processor 130 may perform a first encoding on the second noise image to output the third noise image. The rest follow the same principle and will not be elaborated further here.

[0074] Next, the processor 130 can denoise the (i+1)th noisy image to output the (i+1)th denoised image. Similarly, referencing the denoised image with a smaller width and / or height, the processor 130 can denoise the image that has not yet been denoised. For example, if i is 3 and M is 5, the processor 130 can denoise the fourth noisy image to output the fourth denoised image.

[0075] For example, Figure 4 This is a schematic flowchart of a noise reduction process according to an embodiment of the present invention. Please refer to... Figure 4 Processor 130 performs a first encoding (e.g., mean pooling) on ​​the input image IM11 (configured as W / 2×H / 2×6, where W is the width of the original image and H is the height of the original image) (step S401) and outputs a first noisy image DSM11 (configured as W / 4×H / 4×6). Processor 130 performs a first encoding (e.g., mean pooling) on ​​the first noisy image DSM11 (step S402) and outputs a second noisy image DSM12 (configured as W / 8×H / 8×6). Processor 130 directly denoises the second noisy image DSM12 (step S403) and outputs a second denoised image DNM21 (configured as W / 8×H / 8×6). Processor 130 decrypts the second denoised image DNM21 (step S404) and outputs the second denoised image DNM22 (configured width × height × channel as W / 4 × H / 4 × 3). Processor 130 denoises the first noisy image DSM11 based on the second denoised image DNM22 (step S405) and outputs the first denoised image DNM15 (configured width × height × channel as W / 4 × H / 4 × 6). Processor 130 decrypts the first denoised image DNM15 (step S406) and outputs the first denoised image DNM23 (configured width × height × channel as W / 2 × H / 2 × 3). Processor 130 denoises the input image IM11 based on the first denoised image DNM23 (step S407) and outputs the first image OM12 (configured width × height × channel as W / 2 × H / 2 × 6).

[0076] It should be noted that the embodiments of the present invention are not limited to those described above. Figure 3 The second-order optimization shown or Figure 4 The 3rd-order optimization shown can also be achieved at 4th-order (i.e., M is 4), 5th-order (i.e., M is 5) or higher orders.

[0077] As described above, in this embodiment, the image format of the input image and the first image is, for example, luminance-chrominance density (YUV) format, but is not limited thereto. In other preferred embodiments, the image format of the input image and the first image may also be red-green-blue (RGB) format, and the present invention does not impose any limitations thereon.

[0078] In another preferred embodiment, when performing such Figure 3 Or Figure 4 Before the disclosed denoising process, the input image is, for example, pre-processed through a third encoding conversion to transform the original image (configured width × height × channels of W × H × 3) into the input image format (configured width × height × channels of W / 2 × H / 2 × 6). In other words, the input image is, for example, the original image after the third encoding. For instance, both the original and input images can be in luminance / chrominance density formats, where the original image format can be YUV444 and the input image format can be YUV420. The YUV444 image format can be converted to the YUV420 image format after, for example, a format conversion third encoding. Of course, in other preferred embodiments, the original image format can also be RGB format, and the input image format can be YUV420. In this embodiment, the above format conversion is, for example, a packing procedure.

[0079] It is worth mentioning that, in the embodiment where the input image has undergone the third encoding conversion process described above, after outputting the first image (configured as width × height × channels of W / 2 × H / 2 × 6), the processor 130 can further decrypt the first image and output the first image (configured as width × height × channels of W × H × 3). In other embodiments where the input image has undergone the third encoding conversion process described above, after outputting the first image (configured as width × height × channels of W / 2 × H / 2 × 6), the decryption process of the processor 130 can be omitted, and adjustments can be made in subsequent image processing programs to make the final output image configured as width × height × channels of W × H × 3.

[0080] As can be seen from the above description, whether or not the input image has undergone the conversion process of the third encoding in advance is within the spirit and scope of this invention, and no limitations are made herein.

[0081] Figure 5 This is a flowchart of a noise reduction block according to an embodiment of the present invention. Please refer to... Figure 5 ,by Figure 3 Taking step S304 as an example (or possibly...) Figure 4In step S405 or S407, processor 130 can combine the input image IM11 and the first denoised image DNM12 (step S501) and output the first combined image CM1. Processor 130 can perform multiple convolution operations on the first combined image CM1 (step S502) and output noise information. Then, processor 130 can denoise the input image IM11 according to the noise information (step S503) and output the first image OM11.

[0082] In one embodiment, the original image may also be pre-processed by the processor 130 through a pixel allocation operation. Taking the luminance-chrominance-density (YUV) format as an example, the processor 130 can allocate the luminance values ​​of two adjacent 2×2 first pixels in the original image to four of the six channels of a second pixel in the input image, allocate the chrominance values ​​of two adjacent 2×2 first pixels in the original image to one of the six channels of a second pixel in the input image, and allocate the density values ​​of two adjacent 2×2 first pixels in the original image to one of the six channels of a second pixel in the input image. This achieves the redistribution of pixels in the original image. Specifically, the processor 130 records the luminance, chrominance, and density values ​​of any first pixel in the original image in three channels. If dimensionality reduction is desired, the processor 130 records the luminance values ​​of two adjacent 2×2 first pixels together in the four channels of a second pixel in the input image, and records the chrominance and density values ​​of these first pixels (e.g., average, maximum, or other statistical values) in the two channels of the corresponding second pixel in the input image.

[0083] For example, Figure 6 This is a schematic diagram of the first pixel of the original image according to an embodiment of the present invention, and Figure 7A This is a schematic diagram of the second pixel of an input image according to an embodiment of the present invention. Please refer to... Figure 6 and Figure 7A , Figure 6 The top left 2×2 first pixels are assigned to the second pixel numbered "00" in the input image (e.g. Figure 7A The diagram shows the brightness values ​​Y0000, Y0001, Y0010, Y0011 of 2×2 first pixels, and the representative chromaticity values ​​U00 and saturation values ​​V00 of these first pixels. Figure 6 The top right 2×2 first pixels are assigned to the second pixel numbered "10" in the input image (e.g. Figure 7A The diagram shows the brightness values ​​Y1000, Y1001, Y1010, Y1011 of 2×2 first pixels, and the representative chromaticity values ​​U10 and saturation values ​​V10 of these first pixels. Figure 6 The first 2×2 pixels at the bottom left are assigned to the second pixel numbered "01" in the input image (e.g., Figure 7AThe diagram shows the brightness values ​​Y0100, Y0101, Y0110, Y0111 of 2×2 first pixels, the representative chromaticity value U01 and the density value V01 of these first pixels; Figure 6 The first 2×2 pixels at the bottom right are assigned to the second pixel numbered "11" in the input image (e.g. Figure 7A The diagram shows the luminance values ​​Y1100, Y1101, Y1110, and Y1111 of 2×2 first pixels, as well as the representative chromaticity value U11 and saturation value V11 of these first pixels.

[0084] It should be noted that the width and height of the original image can be redistributed not limited to 2×2 first pixels; sizes such as 4×4, 3×3, or other sizes can also be achieved.

[0085] In one embodiment, the processor 130 may perform pooling on two adjacent 2×2 second pixels in the input image. Pooling may perform specific operations (e.g., taking the maximum value or the average value) on the second pixels within a specific region (a 2×2 region in this embodiment).

[0086] For example, Figure 7B This is a schematic diagram of mean pooling according to an embodiment of the present invention. Please refer to... Figure 7A and Figure 7B The first channel of a pixel in the first noisy image is the average brightness value Y00 obtained by averaging the four brightness values ​​Y0000, Y0001, Y0010, and Y0011 of the second pixel numbered "00"; the second channel is the average brightness value Y01 obtained by averaging the four brightness values ​​Y0100, Y0101, Y0110, and Y0111 of the second pixel numbered "01"; and the third channel is the average brightness value Y1 obtained by averaging the four brightness values ​​Y1000, Y1001, Y1010, and Y1011 of the second pixel numbered "10". 0; The fourth channel is the average brightness value Y11 of the four brightness values ​​Y1100, Y1101, Y1110, and Y1111 of the second pixel numbered "11"; The fifth channel is the average chromaticity value U of the four chromaticity values ​​U00, U01, U10, and U11 of the four second pixels numbered "00", "01", "10", and "11"; The sixth channel is the average chromaticity value V of the four density values ​​V00, V01, V10, and V11 of the four second pixels numbered "00", "01", "10", and "11".

[0087] It should be noted that the pooling process is not limited to 2×2 first pixels; it can also be implemented in areas such as 4×4, 3×3, or other sizes.

[0088] In one embodiment, the processor 130 may reallocate the third pixel of the i-th denoised image to increase dimensionality (e.g., Figure 3 Step S303 Figure 4 Steps S404 and S406). Taking YUV format as an example, the processor 130 can allocate the four channels corresponding to the brightness values ​​of the six channels of the third pixel in the i-th denoised image to the undecoded image (e.g., Figure 3 The first denoised image DNM12, Figure 4 The second denoised image DNM22, or Figure 4 In the first denoised image DNM23), the four fourth pixels are respectively assigned one of the six channels of the third pixel in the i-th denoised image corresponding to the chromaticity value to the four fourth pixels in the undeconstructed image, and the six channels of the third pixel in the i-th denoised image corresponding to the concentration value to the four fourth pixels in the undeconstructed image.

[0089] Specifically, the processor 130 records the luminance, chroma, and saturation values ​​of any third pixel in the i-th denoised image in six channels (e.g., ...). Figure 7A (As shown). If a dimension is to be added, the processor 130 records the brightness values ​​of the four channels of a single third pixel in one channel of the four fourth pixels of the deconstructed image, and records the chroma value and density value of the third pixel in two channels of the four fourth pixels of the input image (that is, the chroma value and density value of the four fourth pixels are the same).

[0090] For example, Figure 8 This is a schematic diagram of pixel disassembly according to an embodiment of the present invention. Please refer to... Figure 8 , Figure 7B The luminance values ​​Y00, Y01, Y10, and Y11 of the four channels of the third pixel are assigned to one channel of the four fourth pixels, the chrominance value U of one channel of the third pixel is assigned to one channel of the four fourth pixels, and the density value V of one channel of the third pixel is assigned to one channel of the four fourth pixels.

[0091] It should be noted that the above embodiments are illustrated using the luminance chromaticity (YUV) format, but the red-green-blue (RGB) format may also be used in other embodiments.

[0092] In one embodiment, processor 130 may sharpen the i-th noisy image to output an i-th sharpened image. Sharpening is used to enhance the edge contours of an image and may improve image clarity and cleanliness. In one embodiment, processor 130 may sharpen the i-th noisy image using a sharpening model based on a machine learning algorithm. For example, the sharpening model includes one or more convolution operations, each using a trained filter. In another embodiment, processor 130 may use a high-pass filter, unsharp masking, or other edge sharpening techniques.

[0093] Next, the processor 130 may sharpen the input image based on the i-th sharpened image to output a first image. Similarly, a sharpened image with a smaller width and / or height is referenced, and the unsharpened image is sharpened. In one embodiment, the processor 130 may combine the i-th sharpened image and the input image, and perform one or more convolution operations on the combined image to output the first image.

[0094] It should be noted that sharpening and noise reduction can be performed simultaneously or separately.

[0095] Besides sharpening and noise reduction, high dynamic range (HDR) processing is also widely used in image processing. High dynamic range helps to enhance the visual experience.

[0096] Figure 9A This is a flowchart of a two-stage optimization network according to an embodiment of the present invention. Please refer to... Figure 9A The processor 130 can then perform a Local Optimizing Network (LON) (step S11). Next, the processor 130 can perform a Global Optimizing Network (GON) on the output of the Local Optimizing Network (LON) (step S12). The Local Optimizing Network of step S11 can be used for sharpening and denoising as described above, and can be implemented using the aforementioned sharpening and denoising related embodiments. For example, Figure 2 The steps shown Figure 3 or Figure 4 The architecture shown is illustrated. The global optimization network in step S12 can be used for high dynamic range processing, and its detailed embodiments will be described in subsequent embodiments.

[0097] However, two-stage networks are not limited to Figure 9A The order shown. Figure 9B This is a flowchart of a two-stage optimization network according to another embodiment of the present invention. Please refer to... Figure 9B The processor 130 first performs a global optimization network (step S13). Next, the processor 130 performs a local optimization network on the output of the global optimization network (step S14). The local optimization network of step S13 can be used for sharpening and denoising as described above, and can be implemented using the aforementioned sharpening and denoising related embodiments. For example, Figure 2 The steps shown Figure 3 or Figure 4 The architecture shown is illustrated. The global optimization network in step S14 can be used for high dynamic range processing, and its detailed embodiments will be described in subsequent embodiments.

[0098] For high dynamic range processing, Figure 9C This is a flowchart of an image processing method according to an embodiment of the present invention. Please refer to... Figure 9C The processor 130 can process the first image (such as...) Figure 9A The output of step S11 in the process) or a raw image ( Figure 9B The processor 130 performs high dynamic range processing on the input of step S13 to output a first high dynamic range image (step S910). Specifically, high dynamic range processing is used to enhance the dynamic range of exposure and may correctly represent the range of brightness from direct light to the darkest shadow in the real world. In one embodiment, the processor 130 can identify enhancement information in the image through an intensity mapping model based on a machine learning algorithm (e.g., deep neural network (DNN), multilayer perceptron (MLP), support vector machine (SVM), or other machine learning model). Enhancement information is, for example, the intensity value (e.g., brightness value, contrast value, or hue) corresponding to each pixel or a decision to adjust the intensity. The intensity mapping model is a machine learning model trained based on training samples (and labeled results). For example, the intensity mapping model includes one or more convolution operations, and each convolution operation uses a trained filter. The processor 130 can perform high dynamic range processing based on the enhancement information. That is, adjust the intensity values ​​(e.g., brightness value, contrast value, or hue) of the feature map (e.g., the first image or the original image).

[0099] In another embodiment, processor 130 can directly perform high dynamic range processing on the feature map using a high dynamic range model based on a machine learning algorithm. In other embodiments, processor 130 may utilize other high dynamic range algorithms.

[0100] The first image is, for example, Figure 3 The first image OM11 or Figure 3 The first image is OM12, but it could also be the output of other locally optimized networks. The original image can be a medical image, a surveillance image, a product image, or a person image, but this embodiment of the invention does not limit its type.

[0101] Processor 130 performs high dynamic range processing on the first image based on the first high dynamic range image to output the second image (step S920). Specifically, the high dynamic range technology can be found in the description of step S910, which will not be repeated here. It is worth noting that in step S920, processor 130 also refers to the first high dynamic range image when performing high dynamic range processing. That is, processor 130 refers to a high dynamic range image with a smaller width and / or height compared to the first image and performs high dynamic range processing on the first input image.

[0102] In one embodiment, when the processor 130 performs high dynamic range processing on the first image, it performs high dynamic range processing on the first image based on the first high dynamic range image to output the second image. Figure 9AFor example, the output of step S11 is the first image, and in step S12, the first image can be processed with high dynamic range (HMR) based on the first HMR image to output the second image. In another embodiment, when the processor 130 performs HMR processing on the original image, it performs HMR processing on the original image based on the first HMR image to generate the input image, and then processes the input image further, for example... Figure 3 or Figure 4 The first coding job is shown. Figure 9B For example, the input of step S13 is the original image, and in step S13, the original image can be processed with high dynamic range based on the first high dynamic range image to output the input image.

[0103] Here, let's first address Figure 9A The high dynamic range (HVR) processing of the global optimization network is illustrated below. In one embodiment, the processor 130 can combine the first HVR image and the first image to output a second combined image. For example, the processor 130 can assign the first HVR image and the first image to different channels and use the set of these channels as the second combined image. Taking the red-green-blue (RGB) format as an example, the intensity values ​​of the red, green, and blue of a pixel in the first HVR image are in the first to third channels, and the intensity values ​​of the corresponding pixel in the first image are in the fourth to sixth channels. However, depending on the format or number of channels, other allocation methods may be possible.

[0104] Next, processor 130 may perform one or more convolution operations on the second combined image to output a second image. In one embodiment, the second combined image is processed by those convolution operations to output enhancement information (e.g., through an intensity mapping model), and processor 130 may adjust the intensity value of the first input image based on the enhancement information. In another embodiment, the second combined image is processed by those convolution operations to directly output a second image (e.g., through a high dynamic range model).

[0105] In one embodiment, the high dynamic range processing is preceded by a depth-to-space transformation. The depth-to-space transformation is used to allocate multiple channels to a single channel. That is, pixels from multiple channels are merged into the same channel, or pixels that were originally in different channels are transformed to be located in the same channel. For example, the first, third, and fifth pixels from the first to third channels are assigned to a new first channel; the second, fourth, and sixth pixels from the fourth to sixth channels are assigned to a new second channel.

[0106] For example, Figure 10 This is a flowchart of high dynamic range processing according to an embodiment of the present invention. Please refer to [the following text is also included:] [Further details about the flowchart are needed for a complete translation.] Figure 9A and Figure 10After the processor 130 performs Local Optimizing Network (LON) and outputs the first image, it performs a spatial-to-depth transformation on the first image OM1 (configured as width × height × channels W × H × C, where W is the width of the first image OM1, H is the height of the first image OM1, and C is the number of channels of the first image OM1) (step S101), and outputs the first standard dynamic range image DSM21 (configured as width × height × channels W / 2 × H / 2 × 4 × C). The processor 130 performs high dynamic range processing on the first standard dynamic range image DSM21 (step S102), and outputs the first high dynamic range image HDR11 (configured as width × height × channels W / 2 × H / 2 × 4 × C). The processor 130 performs high dynamic range processing on the first standard dynamic range image DSM21 based on the first high dynamic range image HDR11 (step S105), and outputs the second image OM21 (configured as W×H×C with width×height×channel).

[0107] In one embodiment, processor 130 may define the first image as a first Standard Dynamic Range (SDR) image. Processor 130 may perform a second encoding on the j-th SDR image to output a (j+1)-th SDR image. j is a positive integer from 1 to N-1, and N is the number of levels. For example, if j is 2, processor 130 may perform a second encoding on the second SDR image to output a third SDR image. The rest follow the same principle and will not be elaborated further. The second encoding includes pooling processing, thereby reducing the image size.

[0108] Processor 130 can perform high dynamic range processing on the (j+1)th standard dynamic range image to output the (j+1)th high dynamic range image. Next, processor 130 can perform high dynamic range processing on the (j+1)th standard dynamic range image based on the (j+1)th high dynamic range image to output the jth high dynamic range image. Similarly, high dynamic range images with smaller widths and / or heights are referenced, and high dynamic range processing is performed on images that have not yet undergone high dynamic range processing. N is the number of levels and is a positive integer greater than two. For example, if j is 3 and N is 4, then processor 130 performs high dynamic range processing on the fourth standard dynamic range image based on the fourth high dynamic range image to output the third high dynamic range image.

[0109] Furthermore, when j=1, the processor 130 can perform high dynamic range processing on the second (j+1) standard dynamic range image based on the second (j+1) high dynamic range image to output the first high dynamic range image, and then let the first high dynamic range image perform high dynamic range processing on the first image to output the second image.

[0110] For example, Figure 11 This is a flowchart of high dynamic range processing according to an embodiment of the present invention. Please refer to... Figure 11 The processor 130 performs a spatial-to-depth transformation on the first image OM1 (configured as width × height × channels W × H × C, where W is the width of the first image OM1, H is the height of the first image OM1, and C is the number of channels of the first image OM1) (step S111), and outputs a first standard dynamic range image DSM21 (configured as width × height × channels W / 2 × H / 2 × 4 × C). The processor 130 performs a second encoding on the first standard dynamic range image DSM21 (step S112), and outputs a second standard dynamic range image DSM22 (configured as width × height × channels W / 4 × H / 4 × 4 × C). The processor 130 performs a second encoding on the second standard dynamic range image DSM22 (step S113), and outputs a third standard dynamic range image DSM23 (configured as width × height × channels W / 8 × H / 8 × 4 × C). Processor 130 performs a second encoding on the third standard dynamic range image DSM23 (step S114) and outputs a fourth standard dynamic range image DSM24 (configured as W / 16×H / 16×4*C in width×height×channel). Processor 130 directly performs high dynamic range processing on the fourth standard dynamic range image DSM24 (step S115) and outputs a fourth high dynamic range image HDR24 (configured as W / 16×H / 16×4C in width×height×channel). Processor 130 performs high dynamic range processing on the fourth standard dynamic range image DSM24 based on the fourth high dynamic range image HDR24 (step S116) and outputs a third high dynamic range image HDR23 (configured as W / 8×H / 8×C in width×height×channel). Processor 130 performs high dynamic range processing on the third standard dynamic range image DSM23 based on the third high dynamic range image HDR23 (step S117), and outputs a second high dynamic range image HDR22 (configured as W / 4×H / 4×C for width × height × channels). Processor 130 performs high dynamic range processing on the second standard dynamic range image DSM22 based on the second high dynamic range image HDR22 (step S118), and outputs a first high dynamic range image HDR21 (configured as W / 2×H / 2×C for width × height × channels). Processor 130 performs high dynamic range processing on the first standard dynamic range image DSM21 based on the first high dynamic range image HDR21 (step S118), and outputs a second image OM22 (configured as W×H×C for width × height × channels).

[0111] It should be noted that the embodiments of the present invention are not limited to those described above. Figure 10 The second-order optimization shown or Figure 11The fourth-order optimization shown can also be implemented at third-order (i.e., N is 3), fifth-order (i.e., N is 5), or even higher orders. Furthermore, the second encoding in steps S112, S113, and S114 can also be called the first encoding calculation, the high dynamic range processing in step S115 can also be called the global calculation, the high dynamic range processing in step S116 can also be called the first decoding calculation, and the high dynamic range processing in steps S117, S118, and S119 can also be called the second decoding calculation.

[0112] The above text is first aimed at Figure 9A The following is a detailed description of the implementation method where a Local Optimizing Network (LON) is first applied to output the first image, and then a Global Optimizing Network (GON) is applied to that first image. Next, please also refer to... Figure 9B and Figure 10 The processor 130 can also first target the original image ( Figure 9B After performing high dynamic range processing on the input (in step S13) to output the input image, the input image is then processed by a local optimization network to output the first image. Figure 9A The implementation method shown is the same, in Figure 9B In the illustrated embodiment, processor 130 also performs spatial-to-depth transformation on the original image (not shown) (step S101) and outputs a first standard dynamic range image (DSM21). Processor 130 also performs high dynamic range processing on the first standard dynamic range image (DSM21) (step S102) and outputs a first high dynamic range image (HDR11). Similarly, processor 130 also performs high dynamic range processing on the first standard dynamic range image (DSM21) based on the first high dynamic range image (HDR11) to generate the input image. In this way, the input image can subsequently undergo, for example, a local optimization network operation (as in step S14) to output the first image.

[0113] and Figure 9A The implementation method shown is the same, in Figure 9B In the embodiment shown, the processor 130 can also perform the following: Figure 11The disclosed high dynamic range (HVR) processing. For example, the original image can be defined as a first processed image, and the second encoding is performed on the first processed image to output a second processed image. Then, the HVR processing is performed on the second processed image to output a second HVR image. Afterwards, the HVR processing is performed on the second processed image based on the second HVR image to output a first HVR image, and then the first HVR image is used to perform HVR processing on the original image to output the input image. The aforementioned second encoding is, for example, pooling. Regarding... Figure 9B The implementation shown is similar to Figure 9A The implementation methods shown are the same, and will not be repeated here.

[0114] In one embodiment, processor 130 may perform a second encoding on a first, second, or other standard dynamic range image. In one embodiment, the second encoding includes a second downsampling. Second downsampling is a process for reducing the width and / or height of a feature map (e.g., an input image). In one embodiment, the second downsampling is an inverse desubpixel transformation. Inverse desubpixel transformation distributes multiple pixels on a single channel to single pixels in multiple channels. In another embodiment, the second downsampling is a pooling process. In a preferred embodiment, the pooling process is, for example, mean pooling. That is, taking the average of multiple pixels in any channel. In some embodiments, the second downsampling includes both inverse desubpixel and channel mean pooling. In other embodiments, the second downsampling may be other processes for reducing the feature map. Therefore, the width and / or height of the second-encoded image is smaller than the input first image or the original image.

[0115] In one embodiment, the second encoding includes at least one convolution operation.

[0116] In one embodiment, the processor 130 further performs a space-to-depth transformation on the first image or the original image. The space-to-depth transformation is used to distribute the single-channel first image or the original image across multiple channels for subsequent high dynamic range processing (e.g., ...). Figure 10 Step S102, or Figure 11 (Steps S112 to S119). That is, pixels in the same channel are split into multiple channels, or pixels that were originally in the same channel may be converted to be located in different channels. For example, the first, third, and fifth pixels in the first channel are assigned to a new first channel; the second, fourth, and sixth pixels in the first channel are assigned to a new second channel. In addition, the processor 130 can perform high dynamic range processing on the first image or the original image that has been assigned to multiple channels based on the first high dynamic range image to output a second image.

[0117] Figure 12 This is a schematic diagram of space-to-depth conversion according to an embodiment of the present invention. Please refer to... Figure 12 ,by Figure 11 Taking step S111 as an example (or possibly...) Figure 10 Step S101). Assume that the first image OM1 is configured with a width × height × channel of 4 × 4 × 2. In the spatial-to-depth conversion (step S111), the processor 130 may assign pixels A1, A3, A9, and A11 of channel C1 to the new channel C1, assign pixels A2, A4, A10, and A12 of channel C1 to the new channel C2, assign pixels A5, A7, A13, and A15 of channel C1 to the new channel C3, assign pixels A6, A8, A14, and A16 of channel C1 to the new channel C4, assign pixels B1, B3, B9, and B11 of channel C2 to the new channel C5, assign pixels B2, B4, B10, and B12 of channel C2 to the new channel C6, assign pixels B5, B7, B13, and B15 of channel C2 to the new channel C7, and assign pixels B6, B8, B14, and B16 of channel C2 to the new channel C8. Ultimately, a first-coded sample standard dynamic range image with a width × height × channel size of 2 × 2 × 8 is formed.

[0118] Figure 13 This is a flowchart of the second encoding according to an embodiment of the present invention. Please refer to... Figure 13 ,by Figure 10 Taking step S102 as an example (or possibly...) Figure 11 Steps S112, S113, and S114). Processor 130 may perform one or more convolution operations on the first image OM1 or the first standard dynamic range image DSM21 to output a first channel adjusted image (step S122). Processor 130 may combine the first channel adjusted image and the image input to one or more convolution operations (e.g., the first standard dynamic range image DSM21) to output a combined image CIM1 (step S123). Next, processor 130 may perform a second downsampling (e.g., mean pooling) on ​​the combined image CM2 to output a second standard dynamic range image DSM23 (step S124). Mean pooling can be referred to... Figure 7A and Figure 7B The explanation will not be repeated here.

[0119] Figure 14A This is a flowchart of a convolution operation according to an embodiment of the present invention. Please refer to... Figure 14A , Figure 13 Step S122 may include, for example, two convolution operations (steps S1221 and S1222).

[0120] Figure 14B This is a flowchart of a convolution operation according to an embodiment of the present invention. Please refer to... Figure 14B , Figure 13 Step S122 includes two convolution operations (steps S1223 and S1226) and multiple depthwise convolution operations (step S1224). The processor 130 can perform depthwise convolution operations on the output of step S1223 (step S1224). The processor 130 can combine the outputs of the depthwise convolution operations (step S1225) and perform convolution operations on the combined image (step S1226). Then, the processor 130 can add / superimpose the outputs of steps S1226 and S1224 (step S1227).

[0121] It should be noted that the convolution operation in step S122 may have other variations.

[0122] Figure 15 This is a flowchart of a high dynamic range block according to an embodiment of the present invention. Please refer to... Figure 15 ,by Figure 10 Taking step S102 as an example (or possibly...) Figure 11 Step S115). This high dynamic range block is used for high dynamic range processing without referencing other images. Processor 130 can perform a first convolution operation to output a second channel adjusted image (step S1031), and perform an averaging operation on the second channel adjusted image to output an average image (step S1032). Processor 130 can perform a second encoding on the average image (e.g., reduce image size) to output an encoded average image (step S1033). Processor 130 can multiply the second channel adjusted image and the encoded average image to output an encoded weighted image (step S1034). Then, processor 130 can sequentially perform a second convolution operation (step S1035) and one or more second encodings on the encoded weighted image (steps S1036-S1039). The second encoding can refer to Figure 13 , Figure 14A and Figure 14B The explanation will not be repeated here.

[0123] Figure 16 This is a schematic diagram of averaging calculation according to an embodiment of the present invention. Please refer to... Figure 16 Assume the second channel adjusted image has a configuration width × height × channel size of W × H × C. In the averaging operation (step S1032), the average value of pixels in each channel (distinguished by different background textures) is taken. Therefore, the average image has the same number of channels as the second channel adjusted image, but each channel has only one pixel (as shown on the right side of the figure).

[0124] Figure 17 This is a schematic diagram of multiplication operations according to an embodiment of the present invention. Please refer to... Figure 17Assume the second channel adjusted image has a configured width × height × channel size of W × H × C. In the multiplication operation (step S1034), the pixels in each channel (distinguished by different background patterns) are multiplied by a single pixel in the same channel of the encoded average image. Therefore, the width × height × channel size of the encoded weighted image is the same as that of the second channel adjusted image.

[0125] Figure 18 This is a flowchart of a high dynamic range processing block according to an embodiment of the present invention. Please refer to... Figure 18 ,by Figure 10 Taking step S105 as an example (or possibly...) Figure 11 (Steps S119, S118, S117, or S116). Processor 130 can combine the first standard dynamic range image DSM21 and the first high dynamic range image HDR11 after the second encoding of the first image OM1 (step S131) ​​and output the second combined image CM2. Processor 130 can perform one or more convolution operations on the second combined image CM2 (step S132) and output enhancement information. Then, processor 130 can perform high dynamic range processing on the first standard dynamic range image DSM21 according to the enhancement information (step S133) and output the combined image CIM2. For example, processor 130 can superimpose the enhancement information onto the first standard dynamic range image DSM21. Processor 130 can perform depth-to-space transformation on the combined image CIM2 (step S134) and output the second image OM21.

[0126] In one embodiment, the processor 130 may combine the (j+1)th high dynamic range image and the (j+1)th standard dynamic range image to output a third combined image. For example, Figure 11 The image shown combines a fourth high dynamic range image (HDR24) and a fourth standard dynamic range image (DSM24). Next, the processor 130 can perform a convolution operation on the third combined image to output a j-th high dynamic range image. For example, the third combined image, after convolution, outputs enhancement information, and the enhancement information, superimposed on the fourth standard dynamic range image, outputs a third high dynamic range image (HDR23). In one embodiment, the image resulting from the superposition of enhancement information and the standard dynamic range image can undergo a depth-to-space transformation to generate another high dynamic range image.

[0127] Figure 19A This is a flowchart of a convolution operation according to an embodiment of the present invention. Please refer to... Figure 19A , Figure 18 Step S132 includes two convolution operations (steps S1321 and S1322).

[0128] Figure 19B This is a flowchart of a convolution operation according to an embodiment of the present invention. Please refer to... Figure 19B , Figure 18Step S132 includes two convolution operations (steps S1323 and S1326) and multiple depthwise convolution operations (step S1324). The processor 130 can perform depthwise convolution operations on the output of step S1323 (step S1324). The processor 130 can combine the outputs of the depthwise convolution operations (step S1325) and perform convolution operations on the combined image (step S1326). Then, the processor 130 can add / superimpose the outputs of steps S1326 and S1324 (step S1327) to form enhanced information.

[0129] It should be noted that the convolution operation in step S132 may have other variations.

[0130] Figure 20 This is a schematic diagram of depth-to-space conversion according to an embodiment of the present invention. Please refer to... Figure 20 ,by Figure 18 Taking step S134 as an example, other high dynamic range processing blocks can be processed similarly. Assume the combined image CIM2 is configured with a width × height × channel size of 2 × 2 × 8. In the depth-to-space conversion (step S134), the processor 130 can allocate pixels A1 to A16 from channels C1 to C4 to the new channel C1, and allocate pixels B1 to B16 from channels C5 to C8 to the new channel C2. Finally, a second image OM21 with a width × height × channel size of 4 × 4 × 2 is formed.

[0131] In summary, in the image processing method and image processing apparatus of this invention, denoising / sharpening / high dynamic range processing is performed on an image that has not yet undergone denoising / sharpening / high dynamic range processing, based on a denoised / sharpened / high dynamic range image with a smaller width and height. This effectively improves image quality.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized by, The method comprises: performing a first encoding on an input image to output a first noise image; performing a first denoising operation on the first noise image to output a first denoised image; performing a format conversion on the first denoised image; and performing a second denoising operation on the input image according to the format-converted first denoised image to output a first image, further comprising: performing the first encoding on the input image to output a second noise image; denoising the second noise image to output a second denoised image; denoising the first noise image according to the second denoised image to output the first denoised image, and denoising the input image according to the first denoised image; combining the first denoised image and the input image to output a first combined image; and performing at least one convolution operation on the first combined image to output the first image; wherein the first encoding comprises a pooling process.

2. The method of claim 1, wherein the first combined image outputs noise information through the at least one convolution operation, and the step of denoising the input image according to the first denoised image comprises: denoising the input image according to the noise information.

3. The method of claim 1, further comprising: sharpening the first noise image to output a first sharpened image; and sharpening the input image according to the first sharpened image to output the first image.

4. The method of claim 1, further comprising: performing a high dynamic range (HDR) process on the first image or an original image to output a first HDR image; and performing the HDR process on the first image or the original image according to the first HDR image to generate a second image or the input image; wherein when the HDR process is performed on the first image, the HDR process is performed on the first image according to the first HDR image to output the second image; wherein when the HDR process is performed on the original image, the HDR process is performed on the original image according to the first HDR image to generate the input image, and the first encoding is further performed on the input image.

5. The method of claim 4, wherein the step of performing the HDR process on the first image or the original image according to the first HDR image comprises: combining the first HDR image and the first image or the original image to output a second combined image; and performing at least one convolution operation on the second combined image to output the second image or the input image; wherein when the first HDR image and the first image are combined to output the second combined image, and at least one convolution operation is performed on the second combined image, the second image is output. ​ ​ ​ ​ wherein the first high dynamic range image and the original image are combined to output a second combined image, and at least one convolution operation is performed on the second combined image to output the input image, and the first encoding is performed on the input image.

6. The image processing method of claim 5, wherein the second combined image outputs enhancement information through the at least one convolution operation, and the step of performing the high dynamic range processing on the first image or the original image according to the first high dynamic range image comprises: performing the high dynamic range processing on the first image or the original image according to the enhancement information.

7. The image processing method of claim 4, wherein the step of performing the high dynamic range processing on the input image according to the first high dynamic range image comprises: defining the first image as a first standard dynamic range image; performing a second encoding on the first standard dynamic range image to output a second standard dynamic range image; performing the high dynamic range processing on the second standard dynamic range image to output a second high dynamic range image; and performing the high dynamic range processing on the second standard dynamic range image according to the second high dynamic range image to output a first high dynamic range image, and then performing the high dynamic range processing on the first image according to the first high dynamic range image to output the second image; wherein the second encoding comprises a pooling process.

8. The image processing method of claim 7, wherein the second encoding further comprises: performing at least one convolution operation on the first image to output a first channel adjusted image; and combining the first channel adjusted image and the image input to the at least one convolution operation.

9. The image processing method of claim 4, further comprising a spatial-to-depth conversion for allocating the single-channel first image to a multi-channel to perform the high dynamic range processing to output the first high dynamic range image, and then performing the high dynamic range processing on the first image allocated to the multi-channel according to the first high dynamic range image to output the second image.

10. The image processing method of claim 7, wherein the high dynamic range processing comprises: performing a first convolution operation to output a second channel adjusted image; performing an averaging operation on the second channel adjusted image to output an averaged image; performing the second encoding on the averaged image to output an encoded averaged image; multiplying the second channel adjusted image and the encoded averaged image to output an encoded weighted image; and performing a second convolution operation and at least one of the second encoding on the encoded weighted image.

11. The image processing method of claim 7, wherein the high dynamic range processing comprises: combining the second high dynamic range image and the second standard dynamic range image to output a third combined image; and performing at least one convolution operation on the third combined image to output the first high dynamic range image. ​ ​ ​ 12. The image processing method of claim 4, wherein prior to performing the high dynamic range processing further comprises: performing a depth-to-space conversion, wherein the depth-to-space conversion is used to allocate multi-channels to a single channel.

13. The image processing method of claim 4, wherein performing the high dynamic range processing on the original image according to the first high dynamic range image comprises: defining the original image as a first processed image; performing a second encoding on the first processed image to output a second processed image; performing the high dynamic range processing on the second processed image to output a second high dynamic range image; and performing the high dynamic range processing on the second processed image according to the second high dynamic range image to output a first high dynamic range image, which in turn performs the high dynamic range processing on the original image to output the input image; wherein the second encoding comprises a pooling process.

14. The image processing method of claim 4, wherein the original image is encoded by a third encoding to form the input image.

15. An image processing apparatus comprising: a memory to store program code; and a processor coupled to the memory and configured to load and execute the program code to: perform a first encoding on an input image to output a first noise image; denoise the first noise image to output a first denoised image; denoise the input image according to the first denoised image to output a first image; perform the first encoding on the first noise image to output a second noise image; denoise the second noise image to output a second denoised image; denoise the first noise image according to the second denoised image to output the first denoised image and denoise the input image according to the first denoised image; combine the first denoised image and the input image to output a first combined image; and perform at least a convolution operation on the first combined image to output the first image, wherein the first encoding comprises a pooling process.

16. The image processing apparatus of claim 15, wherein the first combined image outputs a noise information after the at least a convolution operation, and the processor is further configured to: denoise the input image according to the noise information.

17. The image processing apparatus of claim 15, wherein the processor is further configured to: sharpen the first noise image to output a first sharpened image; and sharpen the input image according to the first sharpened image to output the first image.

18. The image processing apparatus of claim 15, wherein the processor is further configured to: perform a high dynamic range processing on the first image or an original image to output a first high dynamic range image; and perform the high dynamic range processing on the first image or the original image according to the first high dynamic range image to generate a second image or the input image. ​ ​ wherein when the high dynamic range processing is performed on the first image, performing the high dynamic range processing on the first image according to the first high dynamic range image to output the second image; wherein when the high dynamic range processing is performed on the original image, performing the high dynamic range processing on the original image according to the first high dynamic range image to generate the input image, and performing the first encoding on the input image again.

19. The image processing device of claim 18, wherein the processor is further configured to: combine the first high dynamic range image and the first image to output a second combined image; and perform at least a convolution operation on the second combined image to output the second image.

20. The image processing device of claim 19, wherein the second combined image outputs enhancement information through the at least a convolution operation, and the processor is further configured to: perform the high dynamic range processing on the first image according to the enhancement information.

21. The image processing device of claim 18, wherein the processor is further configured to: define the first image as a first standard dynamic range image; perform a second encoding on the first standard dynamic range image to output a second standard dynamic range image; and perform the high dynamic range processing on the second standard dynamic range image to output a second high dynamic range image; and perform the high dynamic range processing on the second standard dynamic range image according to the second high dynamic range image to output a first high dynamic range image, and further cause the first high dynamic range image to perform the high dynamic range processing on the first image to output the second image; wherein the second encoding comprises a pooling process.

22. The image processing device of claim 21, wherein the processor is further configured to: perform at least a convolution operation on the first image to output a first channel adjusted image; and combine the first channel adjusted image and an image input to the at least a convolution operation.

23. The image processing device of claim 18, wherein the processor is further configured to: perform a spatial-to-depth conversion, wherein the spatial-to-depth conversion is used to distribute the single channel of the first image to multiple channels to perform the high dynamic range processing to output the first high dynamic range image, and further cause the first high dynamic range image to perform the high dynamic range processing on the first image distributed to the multiple channels to output the second image.

24. The image processing device of claim 21, wherein the processor is further configured to: perform a first convolution operation to output a second channel adjusted image; perform an averaging operation on the second channel adjusted image to output an averaged image; perform the second encoding on the averaged image to output an encoded averaged image; multiply the second channel adjusted image and the encoded averaged image to output an encoded weighted image; and perform a second convolution operation and at least the second encoding on the encoded weighted image. ​ 25. The image processing device of claim 21, wherein the processor is further configured to: combine the second high dynamic range image and the second standard dynamic range image to output a third combined image; and perform at least one convolution operation on the third combined image to output the first high dynamic range image.

26. The image processing device of claim 18, wherein the processor is further configured to: perform a depth-to-space conversion, wherein the depth-to-space conversion is used to allocate multiple channels to a single channel.

27. The image processing device of claim 18, wherein the processor is further configured to: define the original image as a first processed image; perform a second encoding on the first processed image to output a second processed image; perform the high dynamic range processing on the second processed image to output a second high dynamic range image; and perform the high dynamic range processing on the second processed image from the second high dynamic range image to output a first high dynamic range image, which in turn performs the high dynamic range processing on the original image to output the input image; wherein the second encoding includes a pooling process.

28. The image processing device of claim 18, wherein the original image is formed into the input image after a third encoding.

29. An image processing method, comprising: inputting a standard dynamic range image into a spatial depth operation to convert an image data format, outputting a first image data; inputting the first image data into a first encoding operation to output a second image data; inputting the second image data into a global operation to output a third image data; inputting the second image data and the third image data into a first decoding operation to output a fourth image data; and inputting the first image data and the fourth image data into a second decoding operation to output a high dynamic range image.

30. The image processing method of claim 29, wherein the first encoding operation includes: downsizing an image size of the first image data.

31. The image processing method of claim 29, wherein the global operation includes: performing a first convolution operation to output a channel adjustment image; performing an averaging operation on the channel adjustment image to output an average image; downsizing an image size of the average image to output an encoded average image; multiplying the channel adjustment image and the encoded average image to output an encoded weighted image; and performing a second convolution operation on the encoded weighted image.

32. The image processing method of claim 29, wherein the first decoding operation includes: combining the second image data and the third image data to output a combined image; performing at least one convolution operation on the combined image to output an enhancement information; performing a dynamic range processing on the second image data from the enhancement information to output a combined image; and performing a depth-to-space conversion on the combined image to output the fourth image data. ​ ​ 33. The image processing method of claim 29, wherein the second coding operation comprises: combining the first image data and the fourth image data to output a combined image; performing at least a convolution operation on the combined image to output enhancement information; performing dynamic range processing on the first image data according to the enhancement information to output a combined image; and performing depth-to-space conversion on the combined image to output the high dynamic range image.

34. The image processing method of claim 29, wherein the first coding operation comprises: performing a first dynamic range processing on the first image data to output a first processed image; performing a first depth-to-space conversion on the first processed image to output a first converted image; performing a first spatial-to-depth conversion on the second image data to output a second converted image; performing a first spatial-to-depth conversion on the third image data to output a third converted image; performing a first spatial-to-depth conversion on the fourth image data to output a fourth converted image; performing a first spatial-to-depth conversion on the fifth image data to output a fifth converted image; perform

Images