Method and device for image magnification and enhancement

Abstract

The present disclosure relates to a method and apparatus for image magnification and enhancement. The present invention provides a system for image magnification and enhancement. The system comprises: an enlargement processing unit, a deep learning residue network unit and a synthesis processing unit. The enlargement processing unit is used to perform an enlargement process on a low-resolution image to output a high-resolution image. The deep learning residue network unit is used to output a high-resolution residue image corresponding to the low-resolution image based on a deep learning result. The synthesis processing unit is used to adjust the high-resolution residue image according to a weighting value to add the adjusted high-resolution residue image back to the high-resolution image, thereby generating an enhanced image, wherein the weighting value is different from the reference weighting value used for training the deep learning residue network unit.

Description

Technical Field

[0001] The present invention relates to the field of super-resolution algorithms, and more particularly to a method and device for image magnification and enhancement based on a deep learning architecture. Background Art

[0002] In recent years, single-frame image super-resolution algorithms have been widely studied, which can reconstruct low-resolution images into high-resolution images. Among them, the algorithm based on the deep-learning convolutional neural network (Deep-Learning Convolutional Neural Network) has achieved good results. However, the deep learning super-resolution algorithm has the defect of poor adjustability in practical applications. Generally speaking, after the neural network completes deep learning, the sharpness of the image cannot be adjusted unless it is trained again. On the other hand, in the existing deep learning super-resolution algorithm, an image is processed using common parameters, so it is impossible to fine-tune the local area of ​​the image, such as processing the sky area in the image to be smoother, processing the grass area to be sharper, or additionally enhancing the image with a large number of detailed textures. Therefore, the field needs a highly adjustable super-resolution algorithm. Summary of the invention

[0003] In view of this, the present invention proposes an image enhancement method and device. In the image enhancement method and device of the present invention, a variety of image magnification technologies are used for super-resolution, including a deep learning residue network based on deep learning technology. In addition, by weighting the output results of the deep learning residue network, the adjustability of the image enhancement is improved. In the present invention, the weight control unit can be adjusted according to the overall characteristics or local characteristics (such as edges or colors) of the input image. In this way, the present invention improves the adjustability of the deep learning residue network, thereby providing an image magnification and increase system that has good adaptability to various scenarios.

[0004] One embodiment of the present invention provides a system for image magnification and enhancement, the system comprising: an enlargement processing unit, a deep learning residue network unit and a synthesis processing unit. The enlargement processing unit is used to perform an enlargement process on a low-resolution image to output a high-resolution image. The deep learning residue network unit is used to output a high-resolution residue image corresponding to the low-resolution image based on a deep learning result. The synthesis processing unit is used to adjust the high-resolution residue image according to a weight value to add the adjusted high-resolution residue image back to the high-resolution image to generate an enhanced image, wherein the weight value is different from the reference weight value used when training the deep learning residue network unit.

[0005] An embodiment of the present invention provides a method for image magnification and enhancement, the method comprising: performing a magnification process on a low-resolution image to output a high-resolution image; based on a deep learning result, outputting a high-resolution residue image corresponding to the low-resolution image according to the low-resolution image; and adjusting the high-resolution residue image according to a weighting value to add the adjusted high-resolution residue image back to the high-resolution image to generate an enhanced image; wherein the weighting value is different from a reference weighting value used when obtaining the deep learning result. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of the structure of an image magnification and enhancement system according to an embodiment of the present invention is shown.

[0007] Figure 2 A schematic diagram of the training architecture of the deep learning residual network unit of the present invention is shown.

[0008] Figure 3 A flow chart of an image magnification and enhancement method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0009] In the following text, many specific details are described to provide readers with a thorough understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention without one or more specific details, or using other methods, components, materials, etc. In other cases, well-known structures, materials, or operations will not be shown or described in detail to avoid obscuring the core concepts of the present invention.

[0010] The reference to "an embodiment" in the specification means that the specific features, structures or characteristics described in the embodiment may be included in at least one embodiment of the present invention. Therefore, the "in an embodiment" that appears in various places in this specification does not necessarily mean the same embodiment. In addition, the aforementioned specific features, structures or characteristics may be combined in one or more embodiments in any suitable form.

[0011] Figure 1 Schematic diagram of the architecture of the image magnification and enhancement system of an embodiment of the present invention. As shown in the figure, the image magnification and enhancement system 100 includes: an enlargement processing unit 110, a deep learning residual network unit 120 and a synthesis processing unit 130. The enlargement processing unit 110 is used to perform an image enlargement process on a low-resolution image IMG_LR to output a high-resolution image IMG_HR. Among them, the enlargement processing unit 110 can be an interpolation filter, which realizes the image enlargement process by performing an interpolation operation, such as a bicubic filter (Bicubic Filter), or an edge preserving amplifier (EdgePreserving Scaler). The operation of the deep learning residual network unit 120 is based on a deep learning process, and mainly outputs a high-resolution residual image IMG_HR_RS corresponding to the low-resolution image IMG_LR based on the low-resolution image IMG_LR. The synthesis processing unit 130 is used to adjust the high-resolution residue image IMG_HR_RS according to a weight value WF, so as to add the adjusted high-resolution residue image IMG_HR_RS' back to the high-resolution image IMG_HR, and then generate an enhanced image IMG_EH. In one embodiment, the synthesis processing unit 130 may include an addition unit 132 and a multiplication unit 134. The multiplication unit 134 will amplify the high-resolution residue image IMG_HR_RS according to the weight value WF, so as to obtain the adjusted high-resolution residue image IMG_HR_RS'. The addition unit 132 adds the adjusted high-resolution residue image IMG_HR_RS' back to the high-resolution image IMG_HR, and then obtains the enhanced image IMG_EH. The feature of the present invention is that the weight value WF is different from the reference weight value WF_O used when training the deep learning residue network unit 120.

[0012] For the deep learning process of the deep learning residual network unit 120, please refer to Figure 2 The architecture diagram shown. Among them, the deep learning residue network (Deep-Learning Residue Network) applied by the deep learning residue network unit 120 can have any number of layers or structures, for example, it can be composed of multiple convolutional neural networks (Convolutional Neural Network). The deep learning residue network applied by the deep learning residue network unit 120 can be trained based on the input low-resolution image IMG_LRO and its corresponding best (Golden) high-resolution image: image IMG_GD as the target. As a result of the training, it is ultimately hoped that the deep learning residue network unit 120 can output the best residue image IMG_GD_RS. And it has the following relationship with the above-mentioned numerical values:

[0013] WF_O*IMG_GD_RS+IMG_HRO=IMG_GD

[0014] Wherein, IMG_HRO is the result of the input low-resolution image IMG_LRO through the upscaling processor 110. In the deep learning process performed by the deep learning residual network unit 120 of the present invention, the reference weighted value WF_O is preset to 1, so:

[0015] IMG_GD_RS=IMG_GD-IMG_HR

[0016] After the training of the deep learning process is completed, the deep learning residual network unit 120 can generate a corresponding high-resolution residual image based on the low-resolution image. The feature of at least one embodiment of the present invention is that the weighting value WF used by the synthesis processing unit 130 is different from the reference weighting value WF_O=1 used in the above-mentioned deep learning process. Such a feature allows the enhanced image IMG_EH output by the image magnification and enhancement system 100 to be different from the best high-resolution image IMG_GD in the deep learning process, thereby bringing adjustability to the image magnification and enhancement system 100. For example, when the weighting value WF is greater than the reference weighting value WF_O in the training process, a sharper enhanced image IMG_EH can be obtained, and when the weighting value WF is less than the reference weighting value WF_O, a softer enhanced image IMG_EH can be obtained. Please note that the low-resolution image IMG_LR or IMG_LRO, the high-resolution image IMG_HR or IMG_HRO, the enhanced image IMG_EH, the high-resolution residue image IMG_HR_RS or IMG_HR_RS', the best residue image IMG_GD_RS, and the best high-resolution image IMG_GD mentioned above may represent the pixel values ​​of one or more pixels in an image.

[0017] In an embodiment of the present invention, the weighted value WF adopted by the synthesis processing unit 130 may be generated based on different methods. In one embodiment, the image magnification and enhancement system 100 further includes a weighted value determination unit 140 for determining the weighted value WF adopted by the synthesis processing unit 130. The weighted value determination unit 140 can detect image characteristics and perform global adjustments, such as giving different weighted values ​​to dynamic images and static images. Alternatively, the weighted value determination unit 140 can also determine the weighted value WF based on a specific fixed value specified by the user. In different embodiments, the weighted value determination unit 140 can refer to the input low-resolution image IMG_LR, or the high-resolution image IMG_HR output by the magnification processor 110, so as to determine the content characteristics of the image, which include the global characteristics or local characteristics of the image.

[0018] Furthermore, the weighted value determination unit 140 can also determine the weighted value WF for each pixel in the image, thereby realizing the adjustment capability of the pixel level. In one embodiment, the weighted value determination unit 140 can determine the weighted value WF according to the high-frequency characteristics or boundary strength of the pixel values ​​in the neighboring area of ​​a specific pixel. For example, through Sobel Filters of different sizes (such as 3x3 / 5x5 / 7x7), the local area of ​​the low-resolution image IMG_LR or the high-resolution image IMG_HR can be analyzed to determine whether the high-frequency components are more or less, thereby determining whether the area is a flat area or a detail area. Alternatively, it can be analyzed that a specific area contains horizontal edges or vertical edges, thereby determining that the area is an edge area. After analyzing the regional characteristics of the image, the weighted value determination unit 140 can give different weighted values ​​WF according to its characteristics.

[0019] In another embodiment, the weighted value determination unit 140 may determine the weighted value WF according to the color characteristics of the pixels in the neighboring area of ​​a specific pixel. For example, a lower weighted value WF is set for the skin color area or the blue area in the image, so as to obtain a skin color area or a sky area with soft details in the enhanced image IMG_EH. Alternatively, a higher weighted value WF is set for the green area in the image, so as to obtain a grass area with clear details in the enhanced image IMG_EH.

[0020] In one embodiment, the weighted value WF may have a value less than 1 or less than the reference weighted value WF_O. This is because the deep learning residual network unit 120 may not have a good enhancement effect when facing an image / image area of ​​a specific pattern, and may even generate noise that does not exist originally, thereby bringing negative effects. Therefore, by setting the weighted value WF of the corresponding image area to be less than 1 or less than the reference weighted value WF_O, this negative effect can be effectively alleviated.

[0021] Figure 3 A flow chart of the method for image magnification and enhancement according to an embodiment of the present invention is shown. As shown in the figure, the flow chart of the method according to the present invention generally includes the following steps:

[0022] Step 210: Perform an enlargement process on a low-resolution image to output a high-resolution image

[0023] Step 220: Based on a deep learning result, output a high-resolution residual image corresponding to the low-resolution image according to the low-resolution image

[0024] Step 230: Adjust the high-resolution residual image according to a weighted value to add the adjusted high-resolution residual image back to the high-resolution image to generate an enhanced image; wherein the weighted value is different from a reference weighted value used when obtaining the deep learning result

[0025] The specific principles and details of the above steps have been explained in detail in the relevant description of the image magnification and enhancement system 100, and will not be further elaborated here for the sake of brevity.

[0026] In summary, under the framework of the present invention, through the technique of weight control, the deep learning residual network unit is allowed to adjust the sharpness according to the characteristics of the image without retraining. In addition, the present invention can also perform pixel-level weight control according to the local characteristics of the image (such as edges or colors), thereby highlighting or weakening local details. In this way, the present invention improves the adjustability of the deep learning residual network unit, thereby providing an image magnification and increase system that has good adaptability to various scenarios.

[0027] The embodiments of the present invention may be implemented using hardware, software, firmware, and combinations thereof. By using an appropriate instruction execution system, the embodiments of the present invention may be implemented using software or firmware stored in a memory. As for hardware, the embodiments of the present invention may be implemented using any of the following technologies or combinations thereof: an individual operation logic having logic gates that can perform logic functions according to data signals, an application specific integrated circuit (ASIC) having suitable combinational logic gates, a programmable gate array (PGA), or a field programmable gate array (FPGA), etc.

[0028] The processes and blocks in the flowcharts within the specification illustrate the architecture, functions and operations that can be implemented by the systems, methods and computer software products based on various embodiments of the present invention. In this regard, each block in the flowchart or functional block diagram can represent a module, segment or portion of a program code, which includes one or more executable instructions for implementing a specified logical function. In addition, each block in the functional block diagram and / or flowchart, as well as the combination of blocks, can be basically implemented by a dedicated hardware system that performs a specified function or action, or a combination of dedicated hardware and computer program instructions. These computer program instructions can also be stored in a computer-readable medium that can enable a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable medium implement the functions / actions specified by the blocks in the flowchart and / or functional block diagram.

[0029] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

[0030]

Explanation of symbols

[0031] 100 Image magnification and enhancement system

[0032] 110 Amplification Processing Unit

[0033] 120 Deep Learning Residual Network Units

[0034] 130 Synthesis Processing Unit

[0035] 132 Addition unit

[0036] 134 Multiplication Unit

[0037] 140 Weighted value determination unit

[0038] IMG_LR low resolution image

[0039] IMG_HR High resolution image

[0040] WF Weighted Value

[0041] IMG_HR_RS High resolution residual image

[0042] IMG_HR_RS' High resolution residual image

[0043] IMG_EH enhanced image

[0044] IMG_LRO Low-resolution image

[0045] IMG_GD_RS Best Residue Image

Claims

1. A system for image magnification and enhancement, comprising: An enlargement processing unit, used for performing an enlargement process on a low-resolution image to output a high-resolution image; a deep learning residual network unit, which outputs a high-resolution residual image corresponding to the low-resolution image based on the low-resolution image based on a deep learning result; a synthesis processing unit, coupled to the enlargement processing unit and the deep learning residual network unit, for adjusting the high-resolution residual image according to a weight value to add the adjusted high-resolution residual image back to the high-resolution image to generate an enhanced image, wherein the weight value is different from a reference weight value used when training the deep learning residual network unit; and A weighting value determination unit is coupled to the synthesis processing unit and is used to determine the weighting value according to the high-frequency characteristics, boundary intensity or color characteristics of pixels in a neighborhood area of ​​a specific pixel in one of the low-resolution image and the high-resolution image.

2. The system according to claim 1, wherein the weighting value is a fixed value specified by a user, or is determined according to content characteristics of the low-resolution image. 3 . The system according to claim 1 , wherein the weight value determination unit is used to determine a weight value applicable to each pixel according to one of the low-resolution image and the high-resolution image.

4. A method for image magnification and enhancement, comprising: Performing an enlargement process on a low-resolution image to output a high-resolution image; Based on a deep learning result, outputting a high-resolution residue image corresponding to the low-resolution image according to the low-resolution image; Determining a weight value applicable to each pixel according to high-frequency characteristics, boundary strength or color characteristics of pixels in a neighborhood of a specific pixel of one of the low-resolution image and the high-resolution image; as well as adjusting the high-resolution residual image according to a weight value to add the adjusted high-resolution residual image back to the high-resolution image, thereby generating an enhanced image; The weighted value is different from a reference weighted value used when obtaining the deep learning result. 5 . The method according to claim 4 , wherein the weighting value is a fixed value specified by a user, or is determined according to content characteristics of the low-resolution image.

Images