Image magnification device and method with super-resolution magnification mechanism

Abstract

The present application relates to an image magnification device and method with super-resolution magnification mechanism. The method comprises: receiving an input image by a magnification module of a neural network system, performing image magnification, and generating a magnified image; receiving the input image by a front-end convolution path included in a neural network module of the neural network system to perform convolution operation and generate a front-end operation output result; receiving the front-end operation output result by a plurality of branch convolution paths included in the neural network module respectively to perform convolution operation and generate a plurality of sets of output image residuals; weighting and mixing the output image residuals by a mixing module included in the neural network module according to weight settings related to a plurality of image regions of the input image to generate a set of final output image residuals; and enhancing the magnified image according to the set of final output image residuals by an enhancement module to generate an output magnified image.

Description

TECHNICAL FIELD

[0001] The present application relates to image upscaling technology, and in particular, to an image upscaling device and method with super-resolution upscaling mechanism. BACKGROUND

[0002] The conventional image upscaling technology cannot improve the resolution of the upscaling image. Therefore, in the upscaling image, the obvious blur, unclear edge and noise are easily observed. In recent years, the image super-resolution technology has been widely used in daily life. The main purpose of super-resolution is to obtain a high-resolution (HR) image from a low-resolution (LR) image and maintain the details as much as possible.

[0003] With the improvement of the resolution of the current digital display, from full HD to ultra HD and even higher resolution, the image upscaling technology with super-resolution upscaling mechanism becomes more important. How to make the upscaling image not only can be adjusted in the overall, but also can be enhanced for the regional characteristics is a problem to be solved. SUMMARY

[0004] In view of the problems of the prior art, one object of the present application is to provide an image upscaling device and method with super-resolution upscaling mechanism to improve the prior art.

[0005] The present application comprises an image upscaling device with super-resolution upscaling mechanism, the device comprising: a storage circuit configured to store a plurality of computer executable instructions; and a processing circuit electrically coupled to the storage circuit and configured to acquire and execute the computer executable instructions to operate a neural network system comprising an upscaling module, a neural network module and an enhancement module, and perform an image upscaling method, the image upscaling method comprising: receiving an input image by the upscaling module, performing image upscaling, generating an upscaling image; receiving the input image by a front-end convolution path comprised in the neural network module to perform convolution operation, generating a front-end operation output result; receiving the front-end operation output result by a plurality of branch convolution paths comprised in the neural network module respectively to perform convolution operation, generating a plurality of sets of output image residuals; weighting and mixing the output image residuals according to a weight setting related to a plurality of image regions of the input image by a mixing module comprised in the neural network module to generate a set of final output image residuals; and enhancing the upscaling image according to the set of final output image residuals by the enhancement module to generate an output upscaling image.

[0006] The application further includes an image magnification method with a super-resolution magnification mechanism, which comprises: receiving an input image by a magnification module of a neural network system, performing image magnification, and generating a magnified image; receiving the input image by a front-end convolution path included in a neural network module of the neural network system to perform convolution operation and generate front-end operation output results; receiving the front-end operation output results by a plurality of branch convolution paths included in the neural network module respectively to perform convolution operation and generate a plurality of sets of output image residuals; weighting and mixing the output image residuals according to weight settings related to a plurality of image regions of the input image by a mixing module included in the neural network module to generate a set of final output image residuals; and enhancing the magnified image according to the set of final output image residuals by an enhancement module of the neural network system to generate an output magnified image.

[0007] The application further includes an image magnification device with a super-resolution magnification mechanism, which comprises: a magnification circuit, a neural network circuit, and an enhancement circuit. The magnification circuit is configured to receive an input image, perform image magnification, and generate a magnified image. The neural network circuit includes a front-end convolution path, a plurality of branch convolution paths, and a mixing circuit. The front-end convolution path is configured to receive the input image, perform convolution operation, and generate front-end operation output results. The branch convolution paths are respectively configured to receive the front-end operation output results to perform convolution operation and generate a plurality of sets of output image residuals. The mixing circuit is configured to weight and mix the plurality of sets of output image residuals according to weight settings related to a plurality of image regions of the input image to generate a set of final output image residuals. The enhancement circuit is configured to enhance the magnified image according to the set of final output image residuals to generate an output magnified image.

[0008] The features, operations, and effects of the application are described in detail below in conjunction with the preferred embodiments and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A block diagram of an image magnification device with a super-resolution magnification mechanism in an embodiment of the application is shown;

[0010] Figure 2 A flowchart of an image magnification method with a super-resolution magnification mechanism in an embodiment of the application is shown; and

[0011] Figure 3 A block diagram of a neural network system realized according to the operation of an image magnification device in an embodiment of the application is shown. DETAILED DESCRIPTION

[0012] An object of the present application is to provide an image magnification device and method with super resolution magnification mechanism, which performs deep learning on input images according to different image characteristics to generate final output image residuals corresponding to the image characteristics to enhance the magnified input images, thereby achieving super resolution image magnification effect.

[0013] Please refer to Figure 1 . Figure 1 A block diagram of an image magnification device 100 with super resolution magnification mechanism in an embodiment of the present application is shown. The image magnification device 100 comprises a storage circuit 110 and a processing circuit 120.

[0014] In an embodiment, the storage circuit 110 can be, for example but not limited to, an optical disk, a random access memory (RAM), a read only memory (ROM), a floppy disk, a hard disk, or an optical magnetic disk. The storage circuit 110 is configured to store a plurality of computer executable instructions 115.

[0015] The processing circuit 120 is electrically coupled to the storage circuit 110. In an embodiment, the processing circuit 120 is configured to acquire and execute the computer executable instructions 115, and accordingly perform the functions of the image magnification device 100. In more detail, the processing circuit 120 performs super resolution image magnification on a low resolution input image LR via a deep learning mechanism to generate an output magnified image HR. When the size of the input image LR is WxH and the magnification factor is n times, the size of the output magnified image HR is nWxnH.

[0016] The operation of the image magnification device 100 will be described in detail in the following paragraphs, with reference to Figure 2 and Figure 3 .

[0017] Figure 2 A flowchart of an image magnification method 200 with super resolution magnification mechanism in an embodiment of the present application is shown. The image magnification method 200 can be applied to, for example Figure 1 the image magnification device 100 as shown, or operated by other hardware components such as a database, a general processor, a calculator, a server, or other unique hardware devices with specific logic circuits or devices with specific functions, such as integrating program codes and processors / chips into unique hardware.

[0018] In more detail, the image enlargement method 200 can be implemented using a computer program to control the components of the image enlargement device 100. The computer program can be stored in a non-transitory computer-readable recording medium, such as a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a USB flash drive, a magnetic tape, a database accessible via a network, or a computer-readable recording medium having the same function as those mentioned above that can be easily conceived by those skilled in the art.

[0019] Figure 3 A block diagram of the neural network system 300 implemented according to the operation of the image enlargement device 100 in an embodiment of the present application is shown. In more detail, when the computer-executable instructions 115 are executed by the processing circuit 120, the neural network system 300 will operate to perform the image enlargement method 200. That is, Figure 3 The modules in the neural network system 300 can be implemented via software operating on the processing circuit 120, but the present application does not exclude embodiments in which one or more of the modules are replaced or configured with firmware or hardware (such as a microprocessor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or the like, or a combination circuit thereof, but not limited thereto), for example, integrating the program code and the processor / chip into a unique hardware as mentioned above.

[0020] In an embodiment, the neural network system 300 includes an enlargement module 310, a neural network module 320, and an enhancement module 330. The neural network module 320 includes a front-end convolution path 340, a branch convolution path 350A, a branch convolution path 350B, a mixing module 360, and a weight generation module 370.

[0021] The image enlargement method 200 includes the following steps (it should be understood that, in the present embodiment, the order of the steps mentioned can be adjusted according to actual needs, even if they can be executed simultaneously or partially simultaneously, except for the order specified).

[0022] In step S210, the input image LR is received by the enlargement module 310, image enlargement is performed, and an enlarged image ER is generated.

[0023] In an embodiment, the enlargement module 310 can achieve the purpose of enlargement via various suitable operation mechanisms, such as but not limited to interpolation operation according to the pixels included in the input image LR. In an embodiment, when the size of the input image LR is WxH, and the enlargement factor is n times, the size of the enlarged image ER is nWxnH.

[0024] In step S220, the input image LR is received by the front-end convolution path 340 included in the neural network module 320 to perform convolution operation, and a front-end operation output result FO is generated.

[0025] In an embodiment, the front-end convolution path 340 includes a plurality of front-end convolution units CNN0-CNN2 connected in series. The front-end convolution unit CNN0 corresponds to a head layer, including a single convolution layer. The front-end convolution units CNN1, CNN2 each correspond to a residual block, and can include one or more convolution layers. The front-end convolution units CNN0-CNN2 sequentially perform convolution operations on the input image LR to generate a front-end operation output result FO.

[0026] It should be noted that, Figure 3 The number of front-end convolution units illustrated in FIG. 3 is merely an example. In different embodiments, the front-end convolution path 340 can include any number of one or more front-end convolution units.

[0027] At step S230, the plurality of branch convolution paths included in the neural network module 320 respectively receive the front-end operation output result FO to perform convolution operations, generating a plurality of sets of output image residuals.

[0028] In the present embodiment, the neural network module 320 includes two branch convolution paths 350A and 350B. The branch convolution paths 350A and 350B each include a plurality of branch convolution units connected in series and a pixel reorganization unit.

[0029] The branch convolution path 350A includes branch convolution units CNA0-CNAM and a pixel reorganization unit PSA.

[0030] The branch convolution units CNA0-CNAM-1 each correspond to a residual block, and can include one or more convolution layers. The branch convolution unit CNAM corresponds to a tail layer, including a single convolution layer. The branch convolution units CNA0-CNAM sequentially perform convolution operations on the front-end operation output result FO to generate a branch operation output result BOA.

[0031] In an embodiment, the branch operation output result BOA includes n x n sets of data with a size of W x H. The pixel reorganization unit PSA further performs pixel reorganization on the branch operation output result BOA to generate a set of data with a size of nW x nH as a set of output image residuals RVA.

[0032] The branch convolution path 350B includes branch convolution units CNB0-CNBM and a pixel reorganization unit PSB.

[0033] The branch convolution unit CNB0 corresponds to one residual block and can include one or more convolution layers. The branch convolution unit CNB1 corresponds to a tail convolution layer and includes a single convolution layer. The branch convolution units CNB0-CNBl sequentially perform convolution operations on the front-end operation output result FO to generate a branch operation output result BOB.

[0034] In an embodiment, the branch operation output result BOB includes n x n groups of data with a size of W x H. The pixel reorganization unit PSB further performs pixel reorganization on the branch operation output result BOB to generate a group of data with a size of nW x nH as a group of output image residuals RVB.

[0035] In an embodiment, the convolution units included in the branch convolution path 350A and the branch convolution path 350B perform convolution operations according to a plurality of groups of convolution operation parameters, respectively, and each group of convolution operation parameters corresponds to one of a plurality of image characteristics of the input image LR. The image characteristics can be, for example but not limited to, an edge, a texture, or a combination thereof.

[0036] For example, the branch convolution path 350A can be configured to be trained for object edge effect to strengthen the output image residual RVA for object edges to achieve the effect of clear edges with less noise and smoothness. In contrast, the branch convolution path 350B can be configured to be trained for object texture to strengthen the output image residual RVB for object texture to achieve the effect of highlighting texture.

[0037] It should be noted that the above-described correspondence between the branch convolution path and the image characteristic is only an example. In other embodiments, the branch convolution path can correspond to other types of image characteristics to achieve the effect of strengthening in a deep learning manner.

[0038] Further, Figure 3 The number and structure of the branch convolution paths illustrated in FIG. 3 are only examples. In different embodiments, the number of branch convolution paths included in the neural network module 320 can be any value of two or more, and deep learning can be performed for different image characteristics. Also, in different embodiments, the number of convolution units included in each branch convolution path can be any value of one or more.

[0039] In step S240, the mixing module 360 included in the neural network module 320 weights and mixes the output image residual RVA and the output image residual RVB according to the weight setting WS related to a plurality of image regions of the input image to generate a group of final output image residuals RVF.

[0040] In one embodiment, the weight setting WS is generated by a weight generation module 370 included in the neural network module 320. In more detail, the weight generation module 370 is configured to receive the input image LR to determine the image region characteristics possessed by each of the image regions included in the input image LR.

[0041] For example, the weight generation module 370 can include a high-pass filter, a Sobel filter for edge detection, an object edge direction determination unit, or a combination thereof, to distinguish between object edges and texture regions in the input image LR. In another example, the weight generation module 370 can also include a color determination unit, a segmentation unit, or a combination thereof, to distinguish between different objects such as sky, grass, etc.

[0042] Further, the weight generation module 370 generates a plurality of weight values corresponding to the output image residual RVA and the output image residual RVB as the weight setting WS according to the image region characteristics.

[0043] In one example, the weight generation module 370 can assign a larger weight value to the output image residual RVA corresponding to the regions in the input image LR that belong to object edges. The weight generation module 370 can assign a larger weight value to the output image residual RVB corresponding to the regions in the input image LR that belong to object textures.

[0044] In another example, the weight generation module 370 can distinguish between colors and objects in the input image LR and determine the corresponding image characteristics according to the distinguished objects. For example, the weight generation module 370 can distinguish between grass and tree regions and other regions in the input image LR and enhance the edges of the grass and tree regions. In such a case, the weight generation module 370 will assign a larger weight value to the output image residual RVA corresponding to the grass and tree regions in the input image LR. The weight generation module 370 will assign a larger weight value to the output image residual RVB corresponding to the other regions in the input image LR.

[0045] Therefore, the mixing module 360 can weight the output image residual RVA and the output image residual RVB by the weight setting WS according to the image region characteristics of each image region, and then mix them by operations such as, but not limited to, addition and / or multiplication to generate a set of final output image residuals RVF. The final output image residuals include a set of data with a size of nW x nH.

[0046] At step S250, the enhanced zoomed image ER is enhanced by the enhancement module 330 according to the final output image residual RVF to generate an output zoomed image HR. In an embodiment, the enhancement module 330 is configured to perform, for example but not limited to, an operation of superposition and / or multiplication between the final output image residual RVF and the pixels of the corresponding zoomed image ER to generate the output zoomed image HR. Wherein, the size of the output zoomed image HR is nW x nH.

[0047] It should be noted that the above-mentioned embodiments are only examples. In other embodiments, those of ordinary skill in the art can make changes without departing from the spirit of the present application. It should be appreciated that the steps mentioned in the above-mentioned embodiments can be adjusted in actual needs in terms of their order, and even can be executed simultaneously or partially simultaneously, except for the order specified.

[0048] In summary, the image zooming device and method with super-resolution zooming mechanism in the present application can learn deeply the input image LR according to different image characteristics to generate the final output image residual corresponding to these image characteristics, so as to strengthen the zoomed input image and achieve the effect of super-resolution image zooming.

[0049] Although the embodiments of the present application are described above, these embodiments are not intended to limit the present application. Those of ordinary skill in the art can make changes to the technical features of the present application according to the explicit or implicit content of the present application. Any such changes can be within the scope of the patent protection sought by the present application. In other words, the patent protection scope of the present application shall be subject to the patent protection scope defined by the application.

[0050]

Symbol Description

[0051] 100: image zooming device

[0052] 110: storage circuit

[0053] 115: computer executable instructions

[0054] 120: processing circuit

[0055] 200: image zooming method

[0056] S210-S250: steps

[0057] 300: neural network system

[0058] 310: zooming module

[0059] 320: neural network module

[0060] 330: enhancement module

[0061] 340: front-end convolution path

[0062] 350A, 350B: branch convolution path

[0063] 360: mixing module

[0064] 370: weight generation module

[0065] BOA, BOB: branch operation output result

[0066] CNA1-CNAM, CNB1-CNB2: branch convolution unit

[0067] PSA, PSB: pixel reorganization unit

[0068] CNN0-CNN2: front-end convolution unit

[0069] ER: enlarged image

[0070] FO: front-end operation output result

[0071] LR: input image

[0072] HR: output enlarged image

[0073] RVA, RVB: output image residual

[0074] RVF: final output image residual

[0075] WS: weight setting

Claims

1. An image magnification device with a super-resolution magnification mechanism, comprising: a storage circuit configured to store a plurality of computer-executable instructions; as well as a processing circuit electrically coupled to the storage circuit and configured to retrieve and execute the plurality of computer-executable instructions to operate as a neural network system comprising an enlargement module, a neural network module, and an enhancement module, and to perform an image enlargement method, the image enlargement method comprising: The amplification module receives an input image, performs image amplification, and generates an amplified image; A front-end convolution path included in the neural network module receives the input image to perform a convolution operation and generate a front-end operation output result; The plurality of branch convolution paths included in the neural network module respectively receive the front-end operation output results to perform convolution operations to generate multiple sets of output image residuals; A weight generation module receives the input image, determines an image region characteristic of each of a plurality of image regions corresponding to a plurality of image characteristics, and generates a plurality of weight values ​​corresponding to the plurality of sets of output image residuals as a plurality of weight settings based on the image region characteristics; a blending module included in the neural network module, weighting and blending the multiple sets of output image residuals according to the multiple weight settings associated with multiple image regions of the input image to generate a set of final output image residuals; and The enhancement module enhances the enlarged image according to the set of final output image residuals to generate an output enlarged image.

2. The image magnification device according to claim 1, wherein the image magnification method further comprises: The front-end convolution path includes a plurality of front-end convolution units connected in series, sequentially performing convolution operations on the input image to generate the front-end operation output result; The plurality of branch convolution units connected in series, each of the plurality of branch convolution paths, sequentially perform convolution operations on the front-end operation output result to generate a branch operation output result; and A pixel reorganization unit included in each of the plurality of branch convolution paths performs pixel reorganization on the branch operation output results to generate a set of output image residuals among the plurality of sets of output image residuals.

3. The image magnification device according to claim 2, wherein when the size of the input image is W×H and the magnification is n, the size of the magnified image is nW×nH, the branch operation output results include n×n groups of data of size W×H, the multiple groups of output image residuals each include a group of data of size nW×nH, the group of final output image residuals includes a group of data of size nW×nH, and the size of the output magnified image generated by the enhancement module is nW×nH.

4. The image magnification device according to claim 1, wherein the plurality of branched convolution paths respectively perform convolution operations according to a plurality of sets of convolution operation parameters, and the plurality of sets of convolution operation parameters each correspond to one of a plurality of image characteristics of the input image.

5. The image magnification device according to claim 4, wherein the image magnification method further comprises: The mixing module weights the output image residual using the multiple weight settings according to the image region characteristics of each image region in the multiple image regions, and then superimposes the output image residual with the enlarged image to generate the output enlarged image.

6. A method for image magnification with a super-resolution magnification mechanism, comprising: An enlargement module of a neural network system receives an input image and performs image enlargement to generate an enlarged image; A front-end convolution path included in a neural network module of the neural network system receives the input image to perform a convolution operation and generate a front-end operation output result; The plurality of branch convolution paths included in the neural network module respectively receive the front-end operation output results to perform convolution operations to generate multiple sets of output image residuals; A weight generation module included in the neural network module receives the input image, determines an image region characteristic of each of a plurality of image regions corresponding to a plurality of image characteristics, and generates a plurality of weight values ​​corresponding to the plurality of sets of output image residuals as a plurality of weight settings based on the image region characteristics; A mixing module included in the neural network module weights and mixes the multiple sets of output image residuals according to the multiple weight settings of the multiple image regions of the input image to generate a set of final output image residuals; as well as The upscaled image is enhanced by an enhancement module of the neural network system according to the set of final output image residuals to generate an output upscaled image.

7. The image magnification method according to claim 6, further comprising: The front-end convolution path includes a plurality of front-end convolution units connected in series, sequentially performing convolution operations on the input image to generate the front-end operation output result; The plurality of branch convolution units connected in series, each of the plurality of branch convolution paths, sequentially perform convolution operations on the front-end operation output result to generate a branch operation output result; and A pixel reorganization unit included in each of the plurality of branch convolution paths performs pixel reorganization on the branch operation output results to generate a set of output image residuals among the plurality of sets of output image residuals.

8. The image enlargement method according to claim 7, wherein when the size of the input image is W×H and a magnification is n, the size of the enlarged image is nW×nH, the branch operation output result includes n×n groups of data of size W×H, the multiple groups of output image residuals each include a group of data of size nW×nH, the group of final output image residuals includes a group of data of size nW×nH, and the size of the output enlarged image generated by the enhancement module is nW×nH.

9. The image magnification method according to claim 6, wherein the plurality of branched convolution paths respectively perform convolution operations according to a plurality of sets of convolution operation parameters, and wherein the plurality of sets of convolution operation parameters each correspond to one of a plurality of image characteristics of the input image, and the image magnification method further comprises: The mixing module weights the output image residual using the multiple weight settings according to the image region characteristics of each of the multiple image regions, and then superimposes the output image residual with the enlarged image to generate the output enlarged image.

10. An image magnification device with a super-resolution magnification mechanism, comprising: an amplification circuit configured to receive an input image, amplify the image, and generate an amplified image; A neural network circuit comprising: a front-end convolution path configured to receive the input image, perform a convolution operation, and generate a front-end operation output result; A plurality of branch convolution paths are respectively configured to receive the front-end operation output results to perform convolution operations to generate multiple sets of output image residuals; a weight generation circuit configured to receive the input image, determine an image region characteristic of each of a plurality of image regions corresponding to a plurality of image characteristics, and generate a plurality of weight values ​​corresponding to the plurality of sets of output image residuals as a plurality of weight settings based on the image region characteristics; a mixing circuit configured to weight and mix the multiple sets of output image residuals according to a weight setting of multiple image regions of the input image to generate a set of final output image residuals; as well as An enhancement circuit is configured to enhance the upscaled image according to the set of final output image residuals to generate an output upscaled image.

Images