Image processing system and related image processing method for image enhancement based on region control and multi-branch processing architecture

Abstract

An image processing system includes: a first image processing device for performing a first image enhancement process on a source image to generate a first enhanced image; one or more second image processing devices, each of which is used to perform a second image enhancement process on a reduced image generated based on the source image, thereby outputting one or more second enhanced images of the same size as the source image; and an output controller for analyzing the regional frequency characteristics of the source image to generate an analysis result, determining one or more regional weights based on the analysis result, and synthesizing the first enhanced image with the one or more second enhanced images based on the one or more regional weights to generate an output image.

Description

Technical Field

[0001] The present invention relates to image processing, and in particular to an image processing device and a related image processing method for image enhancement based on region control and a multi-branch processing architecture. Background Art

[0002] In deep learning networks for image enhancement, performing multiple enhancement processes simultaneously, such as super-resolution, noise reduction, deblurring, and sharpening, often makes it difficult to achieve balanced and natural results. Therefore, interpolation or a multi-block architecture is often used to blend the different enhancement effects. In the interpolation approach, multiple deep learning networks are trained for different types of enhancement, outputting multiple enhanced images. These images are then combined by weight manipulation, or the parameters of the deep learning networks are interpolated to synthesize different enhancement results. In contrast, a multi-block architecture uses multiple processing blocks within a single deep learning network for different types of enhancement. These blocks are trained in stages, and the outputs of each block are then weighted to combine the various enhancement effects. However, in the interpolation approach, multiple deep learning network architectures require multiple inferences to reach the final result, while parameter interpolation architectures cannot precisely control weights based on regional characteristics. On the other hand, in a multi-block architecture, since the processing blocks are attached to the main network, the individual enhancement processes are interdependent and cannot be performed simultaneously. Therefore, the above methods all have room for further improvement. Summary of the Invention

[0003] Therefore, the present invention proposes an image processing system and related methods based on a multi-branch processing architecture, which allows different types of image enhancement processing to be performed in parallel and simultaneously in different branches, thereby reducing computing time. In addition, in an embodiment of the present invention, some branches reduce the size of the source image before performing image processing, and then restore the image size after processing, thereby effectively reducing the computing burden of the system. In addition, the embodiment of the present invention also uses the technique of regional weight control, which determines the intensity of different types of image enhancement effects based on the regional frequency characteristics of the source image, thereby increasing the adjustability of the enhancement effect and allowing different enhancement effects to be better and more naturally integrated.

[0004] An embodiment of the present invention provides an image processing system. The image processing system includes: a first image processing device, one or more second image processing devices, and an output controller. The first image processing device includes a first deep learning network for performing a first image enhancement process on a source image, thereby generating a first enhanced image. Each of the one or more second image processing devices includes a second deep learning network for performing a second image enhancement process on a reduced image generated based on the source image, and the one or more second image processing devices output one or more second enhanced images of the same size as the source image. The output controller is coupled to the first image processing device and the one or more second image processing devices and is configured to analyze the regional frequency characteristics of the source image to generate an analysis result, determine one or more regional weights corresponding to the one or more second enhanced images based on the analysis result, and synthesize the first enhanced image with the one or more second enhanced images based on the one or more regional weights to generate an output image.

[0005] An embodiment of the present invention provides an image processing method. The image processing system method comprises: utilizing a first image processing device including a first deep learning network to perform a first image enhancement process on a source image, thereby generating a first enhanced image; utilizing one or more second image processing devices including a second deep learning network to perform a second image enhancement process on a reduced image generated based on the source image, and outputting one or more second enhanced images having the same size as the source image; analyzing the regional frequency characteristics of the source image to generate an analysis result, and determining one or more regional weights corresponding to the one or more second enhanced images based on the analysis result; and synthesizing the first enhanced image with the one or more second enhanced images based on the one or more regional weights to generate an output image. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A diagram showing the architecture of an image processing system according to a first embodiment of the present invention.

[0007] Figure 2 A diagram showing the architecture of an image processing system according to a second embodiment of the present invention.

[0008] Figure 3 A diagram showing the architecture of an image processing system according to a third embodiment of the present invention.

[0009] Figure 4 The method of generating regional weights in an embodiment of the present invention is shown.

[0010] Figure 5 The flowchart of the image processing method according to the embodiment of the present invention is shown.

[0011] Figure 6 It shows how to use hardware devices to implement the image processing method in the embodiment of the present invention. DETAILED DESCRIPTION

[0012] Numerous specific details are described below to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention without one or more of these specific details, or using other methods, components, or materials. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the core concepts of the present invention.

[0013] References in this specification to "one embodiment" mean that the particular features, structures, or characteristics described in that embodiment may be included in at least one embodiment of the present invention. Therefore, references to "in one embodiment" throughout this specification do not necessarily refer to the same embodiment. Furthermore, the aforementioned particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0014] Figure 1 The following is an architectural diagram of an image processing system according to an embodiment of the present invention. As shown, the image processing system 100 of the present invention includes a first image processing device 110, second image processing devices 120_1-120_2, and an output controller 130. Note that while only two second image processing devices 120_1-120_2 are shown in the illustrated embodiment, those skilled in the art will be able to understand in the subsequent description how to generalize and apply the core concepts disclosed in this embodiment to implement a complete image processing system with more or fewer second image processing devices, and such variations remain within the scope of the present invention.

[0015] The first image processing device 110 is primarily used to perform image enhancement processing on the source image IMG_S, while the second image processing devices 120_1-120_2 are used to perform image enhancement processing on images that are smaller than the source image IMG_S. After the first and second image processing devices 110-120_2 complete the image enhancement processing, the output controller 130 controls the output of the enhanced images generated by the first and second image processing devices 110-120_2. The output controller 130 assigns weights to the enhanced images generated by the different second image processing devices 120_1-120_N and combines them with the enhanced image generated by the first image processing device 110 based on the weights.

[0016] In this embodiment, the first image processing device 110 includes a deep learning network 119 composed of a convolutional layer 111, a residual block 112, a residual block 113, and a convolutional layer 114. The second image processing devices 120_1 to 120_2 respectively include deep learning networks 129_1 to 129_2 composed of convolutional layers 121_1 to 121_2, residual blocks 122_1 to 122_2, residual blocks 123_1 to 123_2, and convolutional layers 124_1 to 124_2. However, it should be noted that in some embodiments of the present invention, the first image processing device 110 and the second image processing devices 120_1 to 120_2 may include other types of deep learning network structures. Figure 1 In some embodiments of the present invention, the architectures of the deep learning networks included in the first image processing device 110 and the second image processing devices 120_1 to 120_2 may be different from each other.

[0017] Furthermore, the first image processing device 110 and the second image processing devices 120_1-120_2 may each perform different or the same image enhancement processing. In one embodiment, the image enhancement processing that the first image processing device 110 may perform includes (but is not limited to): super-resolution, noise reduction, deblurring, and sharpening. Furthermore, the image enhancement processing that the second image processing devices 120_1-120_2 may perform includes (but is not limited to): super-resolution, noise reduction, deblurring, and sharpening.

[0018] The deep learning network 119 in the first image processing device 110 is used to perform image enhancement processing on the source image IMG_S, thereby outputting an enhanced image IMG_ENH0. Furthermore, the second image processing devices 120_1 - 120_2 each include a downsampling unit 125_1 - 125_2, which downsamples the source image IMG_S to different magnifications to generate downscaled images IMG_DOWN1 and IMG_DOWN2. The downscaled images IMG_DOWN1 and IMG_DOWN2 are then enhanced by the deep learning networks 129_1 - 129_2. After processing, the deep learning networks 129_1 - 129_2 output enhanced images IMG_OUT1 and IMG_OUT2. In addition, the second image processing devices 120_1-120_2 further include upsampling units 126_1-126_2, respectively, for upsampling the enhanced images IMG_OUT1 and IMG_OUT2 at different rates, thereby outputting enhanced and resized images IMG_ENH1 and IMG_ENH2. The upsampling rates of the upsampling units 126_1-126_2 correspond to the downsampling rates of the downsampling units 125_1-125_2. For example, Figure 1As shown, the downsampling unit 125_1 may downsample the source image IMG_S (original image size: HxW) by 0.5 times to reduce the image size (reduced image size: (H / 2)x(H / 2)), and the upsampling unit 126_1 may upsample the enhanced image IMG_OUT1 by 2 times to restore the image size (restore to the image size: HxW); the downsampling unit 125_2 may downsample the source image IMG_S by 0.25 times to reduce the image size (reduced image size: (H / 4)x(H / 4)), and the upsampling unit 126_2 may upsample the enhanced image IMG_OUT2 by 4 times to restore the image size. However, this is not a limitation of the present invention. In embodiments of the present invention, the downsampling units 125_1-125_2 may employ downsampling ratios different from those of the present embodiment, thereby obtaining downscaled images IMG_DOWN1 and IMG_DOWN2 of different sizes. As long as the upsampling units 126_1-126_2 restore images IMG_OUT1 and IMG_OUT2 to the same size as the source image IMG_S, they can achieve the same effect as the present embodiment. In one embodiment, the image downsampling ratio may be related to the type of image enhancement processing to be performed. For example, if the deep learning network 129_1 is configured to perform noise reduction or super-resolution image enhancement processing, since these types of enhancement processing consume more computing resources and transmission bandwidth in the entire image processing system 100, the downsampling unit 125_1 may be configured to downsample at a higher ratio to obtain a smaller image size, thereby reducing the system's computing burden and transmission bandwidth requirements.

[0019] In addition, in addition to Figure 1 In addition to the architecture shown in FIG, some embodiments of the present invention also provide different architectures that can achieve the effects of the invention. Figure 2 As shown, in some embodiments of the present invention, the downsampling units 125_1 to 125_2 in the second image processing device 120_1 to 120_2 can be replaced by the pooling layer in the deep learning network 129_1 to 129_2. As is well known, the pooling layer can be used to compress data in the deep learning network. Therefore, the pooling layer 1225_1 to 1225_2 can also achieve the size reduction of the source image IMG_S in the present invention. The upsampling units 126_1 to 126_2 can be replaced by the pixel reorganization layer (Pixel Shuffle Layer) in the deep learning network, and the pixel reorganization layer 1226_1 to 1226_2 can perform super-resolution processing on the images IMG_OUT1 and IMG_OUT2, thereby generating an image of the same size as the source image IMG_S. In addition, in Figure 3In the illustrated embodiment, the downsampling units 125_1-125_2 in the second image processing devices 120_1-120_2 can be replaced by convolutional layers in a deep learning network. Convolutional layers 1235_1-1235_2 have a stride of n, thereby reducing the size of the source image IMG_S. Furthermore, the upsampling units 1236_1-1236_2 can be replaced by deconvolutional layers in a deep learning network. Deconvolutional layers 126_1-126_2 can restore the sizes of images IMG_OUT1 and IMG_OUT2 to match the size of the source image IMG_S.

[0020] The output controller 130 is used to combine the enhanced image IMG_ENH0 output by the first image processing device 110 and the enhanced images IMG_ENH1 to IMG_ENH2 output by the second image processing devices 120_1 to 120_2. The output controller 130 includes a regional analysis unit 132, a weight generation unit 134, an amplification unit 136_1 to 136_2, and a synthesis unit 138_1 to 138_2. The function of the output controller 130 is to perceive the regional frequency characteristics of the source image IMG_S, and determine the synthesis strength of the enhanced image according to the regional frequency characteristics, so as to achieve a fine image enhancement effect. The regional analysis unit 132 is used to perform partition analysis on the source image IMG_S. The regional analysis unit 132 may include a Sobel filter, a discrete cosine transform unit, or a convolutional neural network, which is used to convert the source image IMG_S into the frequency domain, so as to obtain the frequency distribution of the source image IMG_S (that is, the location of the high / low frequency parts in the image). Figure 4 In the illustrated embodiment, the region analysis unit 132 may divide the source image IMG_S into 6x4 regions R0-R23 and determine the regional frequency characteristics based on the frequency distribution to generate an analysis result. The weight generation unit 134 generates a regional weight A corresponding to the enhanced image IMG_ENH1 and a regional weight B corresponding to the enhanced image IMG_ENH2 based on the analysis result. The regional weights A and B are as follows: Figure 4As shown, the 6x4 weight coefficients A0-A23 and B0-B23, respectively, indicate the composite strength of the enhanced images IMG_ENH1 and IMG_ENH2 relative to the enhanced image IMG_ENH0. For example, weight coefficient A11 in region weight A indicates the strength to be used when compositeing the region of enhanced image IMG_ENH1 corresponding to region R11 of the source image IMG_S into the region of enhanced image IMG_ENH0 corresponding to region R11 of the source image IMG_S. Weight coefficient B15 in region weight B indicates the strength to be used when compositeing the region of enhanced image IMG_ENH2 corresponding to region R15 of the source image IMG_S into the region of enhanced image IMG_ENH0 corresponding to region R15 of the source image IMG_S. It should be noted that the number of partitions and the number of weight coefficients in the region weights are not limitations of the present invention; other possibilities exist in other embodiments of the present invention. On the other hand, when determining regional weights A and B, the image enhancement processing performed by the second image processing devices 120_1 and 120_2, as well as the regional frequency characteristics of the source image IMG_S, may be considered simultaneously. For example, in high-frequency regions of the source image IMG_S, a higher weighting coefficient may be used to enhance the sharpening processing performed by one of the second image processing devices 120_1 or 120_2 to highlight image edges and details. In low-frequency regions of the source image IMG_S, a higher weighting coefficient may be used to enhance the super-resolution processing performed by the other of the second image processing devices 120_1 or 120_2 to restore lost image details in the low-frequency regions. Once the weight generation unit 134 determines the region weights A and B, the amplification units 136_1 - 136_2 and the synthesis units 138_1 - 138_2 may combine the enhanced images IMG_ENH1 and IMG_ENH2 with the enhanced image IMG_ENH0 based on the weight coefficients A0 - A23 and B0 - B23, thereby generating an output image IMG_OUT with various enhancement effects.

[0021] Figure 5 The flowchart of the image processing method according to the embodiment of the present invention is shown. As shown in the figure, the image processing method according to the present invention includes the following steps:

[0022] S310: Using a first image processing device including a first deep learning network, performing a first image enhancement process on the source image to generate a first enhanced image;

[0023] S320: Using one or more second image processing devices including a second deep learning network, perform a second image enhancement process on the reduced image generated based on the source image, and output one or more second enhanced images having the same size as the source image.

[0024] S330: Analyze regional frequency characteristics of the source image to generate an analysis result, and determine one or more regional weights corresponding to the one or more second enhanced images according to the analysis result;

[0025] S340: Synthesize the first enhanced image and the one or more second enhanced images according to the one or more region weights to generate an output image.

[0026] Since the principles and specific details of the above steps have been described in detail in the previous embodiment, they will not be repeated here. It should be noted that the above process can better achieve image enhancement processing and further improve its image enhancement effect by adding other additional steps or making appropriate changes and adjustments. In addition, all the operations in the above embodiments of the present invention can be performed by Figure 6 The present invention is implemented by the apparatus 400 shown in FIG. A storage unit 410 (e.g., a non-volatile memory) in the apparatus 400 may be used to store source code, instructions, variables, or data. A hardware processing unit 420 (e.g., a general-purpose processor) in the apparatus 400 may execute the source code and instructions stored in the storage unit 410 and perform all operations in the aforementioned embodiments by referring to the variables or data therein.

[0027] In summary, the multi-branch image processing architecture provided by the present invention allows different types of image enhancement processing to be performed in parallel and simultaneously, thereby reducing computing time. Moreover, in the architecture of the present invention, since some image processing devices first reduce the size of the source image and then perform image enhancement, and restore the image size after the processing is completed, the computing burden of the system can be effectively reduced. In addition, the architecture of the present invention also uses the technique of regional weight control, and determines the intensity of different types of image enhancement effects based on regional frequency characteristics, thereby increasing the adjustability of the enhancement effect and allowing different enhancement effects to be better and more naturally integrated.

[0028] The embodiments of the present invention may be implemented using hardware, software, firmware, and combinations thereof. By using an appropriate instruction execution system, the embodiments of the present invention may be implemented using software or firmware stored in a memory and a corresponding instruction execution processor. As for hardware, any of the following technologies or combinations thereof may be used to implement the embodiments: individual arithmetic logic having logic gates that can perform logic functions based on data signals, application specific integrated circuits (ASICs) having appropriate combinational logic gates, programmable gate arrays (PGAs), or field programmable gate arrays (FPGAs), etc.

[0029] The processes and blocks in the flowcharts within the specification illustrate the architecture, functions, and operations that can be implemented by the systems, methods, and computer software products based on various embodiments of the present invention. In this regard, each block in the flowchart or functional block diagram can represent a module, segment, or portion of a source code that includes one or more executable instructions for implementing a specified logical function. In addition, each block in the functional block diagram and / or flowchart, as well as a combination of blocks, can essentially be implemented by a dedicated hardware system that performs a specified function or action, or by a combination of dedicated hardware and computer program instructions. These computer program instructions can also be stored in a computer-readable medium that can cause a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable medium implement the functions / actions specified by the blocks in the flowchart and / or functional block diagram.

[0030] The above description is only a preferred embodiment of the present invention. Equivalent changes and modifications made according to the scope of the patent application of the present invention are within the scope of the present invention.

[0031] Reference numerals

[0032] 100 Image Processing System

[0033] 110 first image processing device

[0034] 111, 114, 121_1 to 121_2, 124_1 to 124_2 convolutional layers

[0035] 112, 113, 122_1~122_2, 123_1~123_2 residual blocks

[0036] 119, 129_1~129_2 Deep Learning Network

[0037] 120_1 to 120_N Second image processing device

[0038] 125_1~125_2 Downsampling Unit

[0039] 126_1~126_2 upsampling unit

[0040] 1225_1~1225_2 Pooling layer

[0041] 1226_1~1226_2 pixel reconstruction layer

[0042] 1235_1~1235_2 convolutional layers

[0043] 1236_1~1236_2 Deconvolution layer

[0044] 130 Output Controller

[0045] 132 Regional Analysis Unit

[0046] 134 Weight Generation Unit

[0047] 136_1~136_2 Amplification unit

[0048] 138_1~138_2 synthesis unit

[0049] IMG_S Source Image

[0050] IMG_OUT output image

Claims

1. An image processing system comprising: A first image processing device, comprising a first deep learning network, configured to perform a first image enhancement process on a source image, thereby generating a first enhanced image; one or more second image processing devices, each second image processing device comprising a second deep learning network, the second deep learning network being configured to perform a second image enhancement process on a reduced image generated based on the source image, whereby the one or more second image processing devices output one or more second enhanced images having the same size as the source image, wherein a reduction ratio of the reduced image relative to the source image is determined based on a type of the second image enhancement process; wherein the first image enhancement process is different from the second image enhancement process; as well as An output controller, coupled to the first image processing device and the one or more second image processing devices, is used to analyze the regional frequency characteristics of the source image to generate an analysis result, and determine one or more regional weights corresponding to the one or more second enhanced images respectively according to the analysis result and the type of the second image enhancement processing, and synthesize the first enhanced image with the one or more second enhanced images according to the one or more regional weights to generate an output image.

2. The image processing system according to claim 1, wherein the output controller comprises: a region analysis unit, configured to divide the source image into NxM regions, and respectively determine NxM region frequency characteristics of the NxM regions, thereby obtaining the analysis result; as well as A weight generating unit is coupled to the region analyzing unit, and is used to determine a plurality of weight coefficients corresponding to the NxM regions respectively according to the analysis result, wherein the one or more region weights are respectively composed of the plurality of weight coefficients.

3. The image processing system according to claim 1 , wherein each second image processing device comprises: a downsampling unit, configured to reduce the size of the source image to generate the reduced image; as well as An upsampling unit, coupled to the second deep learning network, is configured to upscale the processed image output by the second deep learning network to generate the second enhanced image.

4. The image processing system according to claim 1 , wherein each second image processing device comprises: A pooling layer, configured to reduce the size of the source image to generate the reduced image; as well as A pixel reorganization layer is used to enlarge the size of the processed image output by the second deep learning network to generate the second enhanced image.

5. The image processing system according to claim 1 , wherein each second image processing device comprises: A convolutional layer having a stride of n, configured to reduce the size of the source image to generate the reduced image; as well as The deconvolution layer is used to enlarge the size of the processed image output by the second deep learning network to generate the second enhanced image.

6. The image processing system according to claim 1, wherein the first and second image enhancement processes respectively include at least one of super-resolution, noise reduction, deblurring, and sharpening.

7. The image processing system as claimed in claim 1, wherein the image processing system comprises a plurality of second image processing devices, and the plurality of second image processing devices respectively reduce the source image at different magnifications before performing image enhancement processing, and respectively perform different types of image enhancement processing on the images reduced at different magnifications.

8. An image processing method, comprising: Performing a first image enhancement process on the source image using a first image processing device including a first deep learning network, thereby generating a first enhanced image; performing a second image enhancement process on the reduced image generated based on the source image using one or more second image processing devices including a second deep learning network, and outputting one or more second enhanced images having the same size as the source image, wherein a reduction ratio of the reduced image relative to the source image is determined based on a type of the second image enhancement process; wherein the first image enhancement process is different from the second image enhancement process; Analyzing the regional frequency characteristics of the source image to generate an analysis result, and determining one or more regional weights corresponding to the one or more second enhanced images respectively according to the analysis result and the type of the second image enhancement process; as well as The first enhanced image is synthesized with the one or more second enhanced images according to the one or more region weights to generate an output image.

9. The image processing method according to claim 8, wherein the step of analyzing the regional frequency characteristics of the source image comprises: Dividing the source image into NxM regions, and determining NxM regional frequency characteristics of the NxM regions respectively, thereby obtaining the analysis result; as well as The step of determining the one or more region weights comprises: According to the analysis result, a plurality of weight coefficients corresponding to the NxM regions are determined, wherein the one or more region weights are respectively composed of the plurality of weight coefficients.

10. The image processing method according to claim 8, wherein the step of performing the second image enhancement process comprises: Using a downsampling unit to reduce the size of the source image, thereby generating the reduced image; as well as The processed image output by the second deep learning network is upscaled using an upsampling unit to generate the second enhanced image.

Images