Method and method for changing resolution of image

Abstract

A device for changing the resolution of an image, according to an embodiment of the present disclosure, comprises: at least one memory for storing instructions; and at least one processor, wherein the at least one processor executes the instructions so as to perform the steps of: downscaling an input image by a first scaling factor so as to generate a reduced image; upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor, so as to generate a pseudo input image; applying a Laplacian operation and variance to the input image so as to generate a first value; applying the Laplacian operation and variance to the pseudo input image so as to generate a second value; determining a second scaling factor on the basis of a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and using the second scaling factor so as to change the resolution of the image.

Description

Method and device for converting the resolution of an image

[0001] The present disclosure relates to a method and device for converting the resolution of an image.

[0002] The material described in this section merely provides background information for the present invention and does not constitute prior art.

[0003] Typically, a general interpolation method is used to convert low-resolution original images (for example, old images produced during the analog broadcasting era are expressed in SD-level 720x480 resolution) to high resolutions such as Full HD or 4K and then store or transmit them. However, this method has the problem that the image appears blurry compared to the number of pixels that can be expressed in high resolution, or the image itself does not show a high level of detailed expression, so the image appears blurry.

[0004] The present disclosure provides a method and apparatus for expanding a receptive field through a preprocessing step of downscaling an input image to an optimal resolution without changing a CNN-based image enhancement model.

[0005] The present disclosure provides a method and device for reducing the amount of computation by processing an input image at a lower resolution than the input image.

[0006] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] According to one aspect of the present disclosure, a device for converting the resolution of an image comprises: at least one memory for storing commands; and at least one processor, wherein the at least one processor is configured to perform the steps of: downscaling an input image by a first scaling factor to generate a reduced image; upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor to generate a pseudo input image; applying a Laplacian operation and a variance to the input image to generate a first value; applying the Laplacian operation and a variance to the pseudo input image to generate a second value; determining a second scaling factor based on a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and converting the resolution of the image using the second scaling factor.

[0008] According to another aspect of the present disclosure, a computer-implemented method using a device for converting the resolution of an image is provided, the method comprising: a step of downscaling an input image by a first scaling factor to generate a reduced image; a step of upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor to generate a pseudo-input image; a step of applying a Laplacian operation and a variance to the input image to generate a first value; a step of applying the Laplacian operation and a variance to the pseudo-input image to generate a second value; a step of determining a second scaling factor based on a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and a step of converting the resolution of the image using the second scaling factor.

[0009] The present disclosure can improve performance by expanding the receptive field through a preprocessing step of downscaling an input image to an optimal resolution without changing the existing CNN-based image enhancement model.

[0010] The present disclosure can reduce the amount of computation by processing an input image at a lower resolution than the input image.

[0011] The present disclosure can exhibit good performance in old animation work where line expression is important and texture details are relatively small.

[0012] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0013] Figure 1 is a diagram comparing the difference in image range accommodated by one receptive field according to the resolution of the input image.

[0014] FIG. 2 is a block diagram of a preprocessing device for expanding a receptive field of a CNN-based image processing model according to an embodiment of the present disclosure.

[0015] FIG. 3 is a flowchart illustrating a preprocessing operation for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0016] FIG. 4 is a flowchart illustrating a method for predicting optimal resolution in a preprocessing device for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0017] FIG. 5 is a diagram comparing the preprocessing effect for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0018] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0019] In addition, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by these terms. Throughout the specification, when a part is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be further included, unless specifically stated otherwise. In addition, terms such as "part" and "module" described in the specification mean a unit that processes at least one function or operation, and this may be implemented by hardware, software, or a combination of hardware and software.

[0020] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0021] Figure 1 is a diagram comparing the difference in image range accommodated by one receptive field according to the resolution of the input image.

[0022] With recent advancements in deep learning technology, convolutional neural networks (CNNs) are increasingly being utilized for image enhancement. CNN-based models are also being used to improve blurred images (e.g., image deblurring). The CNN-based model according to the embodiments of the present disclosure is referred to as a "CNN-based image processing model."

[0023] CNN generally secures a receptive field by deeply stacking multiple convolutional layers with 3x3 sized kernels, and by analyzing the relationship between components within the receptive field secured in this way, processing is possible by understanding the content and context of the image. As a result, as shown in reference number 110 of FIG. 1, the lower the resolution of the input image, the larger the receptive field in proportion, and as shown in reference number 120 of FIG. 1, the higher the resolution of the input image, the smaller the receptive field (120) in proportion. As the receptive field becomes smaller, the performance of understanding the input image may decrease. However, if the resolution of the input image is lowered without any constraints, high-frequency components may be lost, which may significantly reduce the detail of the image. Therefore, it is important to process downscaling to reduce only the redundant pixels.

[0024] When the original image is blurry and expressed with a resolution higher than necessary, the correlation between adjacent pixels is high and the texture depicted in detail is very small, so processing a CNN-based image processing model with the original resolution has the disadvantage of poor performance and increased computational load due to a narrow receptive field. In order to improve this problem, the embodiment of the present disclosure proposes a preprocessing method and device that can improve performance and reduce computational load by widening the receptive field of a CNN-based image processing model by downscaling the input image by predicting the optimal resolution of the input image (the resolution before interpolation for reprocessing is performed or the most optimal resolution for expressing the input image).

[0025] FIG. 2 is a block diagram of a preprocessing device for expanding a receptive field of a CNN-based image processing model according to an embodiment of the present disclosure.

[0026] A preprocessing device for expanding a receptive field of a CNN-based image processing model according to an embodiment of the present disclosure includes an optimal resolution prediction unit (210), a preprocessing downscaler (220), a CNN-based image processing unit (230), a postprocessing upscaler (240), etc.

[0027] The optimal resolution prediction unit (210) predicts the optimal resolution of a blurred input image. The embodiment of the present disclosure assumes a special situation in reality where an image can be reprocessed by interpolation (e.g., a case where a low-resolution original image is simply interpolated). Under this assumption, the optimal resolution prediction unit (210) predicts the actual optimal resolution of a blurred input image that has been interpolated at a magnification that is unknown in reality. Here, the optimal resolution means the resolution before the interpolation by reprocessing is performed, and means the optimal resolution that can most economically express the input image without using redundant pixels filled by the interpolation. The optimal resolution can be calculated or predicted by analyzing the high-frequency signal characteristics from a blurred input image of size WxH and using an objective function that minimizes redundant pixels. There may be various methods for predicting the optimal resolution based on this concept, and in the embodiment of the present disclosure, the optimal resolution is predicted through a high-frequency signal analysis using a Laplacian operation and a search technique based on an experimentally determined threshold (or a predetermined threshold).

[0028] The preprocessing downscaler (220) preprocesses the input image by downscaling it to the predicted optimal resolution.

[0029] The CNN-based image processing unit (230) processes the preprocessed input image through a CNN-based image processing model. The CNN-based image processing unit (230) receives and processes the downscaled image from the preprocessing downscaler (220) using an image processing model such as denoising, deblurring, and super-resolution based on a deep multi-layer neural network structure using convolutional layers.

[0030] The post-processing upscaler (240) upscales the output image of the CNN-based image processing model to its original resolution for post-processing. More specifically, the post-processing upscaler (240) restores the output image of the CNN-based image processing unit (230) to its original state by upscaling it to the same size as the input image resolution. At this time, it is not necessary to restore the image to the same size as the input image resolution, and the resolution can be adjusted to a different size as needed.

[0031] FIG. 3 is a flowchart illustrating a preprocessing method for expanding a receptive field of a CNN-based image processing model according to an embodiment of the present disclosure.

[0032] The preprocessing unit for receptive field expansion of a CNN-based image processing model predicts the optimal resolution of the input image in step 301. The specific method for predicting the optimal resolution is described with reference to Fig. 4.

[0033] The preprocessing unit for receptive field expansion of the CNN-based image processing model downscales the input image using the predicted optimal resolution in step 302.

[0034] The preprocessing unit for receptive field expansion of the CNN-based image processing model processes the downscaled image in step 303 using the CNN-based image processing model.

[0035] The preprocessing unit for expanding the receptive field of the CNN-based image processing model upscales the output image of the CNN-based image processing model to restore it to its original resolution in step 304.

[0036] FIG. 4 is a flowchart illustrating a method for predicting optimal resolution in a preprocessing device for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0037] In other words, Fig. 4 shows the detailed operation process performed in step 301 of Fig. 3.

[0038] The preprocessing unit for receptive field expansion of the CNN-based image processing model sets the parameters of the input image of size WxH in step 401.

[0039] The parameters of the input image include, for example, S (scaling factor) set to 1.0, D indicating the unit for updating the S value, T indicating the threshold for the iteration termination condition, I indicating the input image, W indicating the width of the input image, and H indicating the height of the input image. In other words, T means the threshold for terminating the iteration.

[0040] The preprocessing unit for receptive field expansion of the CNN-based image processing model sets the value obtained by subtracting D from S to S' in step 402. S is called the initial scaling factor, and S' is called the first scaling factor.

[0041] The preprocessing unit for receptive field expansion of the CNN-based image processing model downscales the input image (I) to S' in step 403 to generate a reduced image (ID).

[0042] The preprocessing unit for the receptive field expansion of the CNN-based image processing model upscales the reduced image (ID) to 1 / S' in step 404 to create a pseudo-input image (I p ) is created.

[0043] The preprocessing unit for receptive field expansion of the CNN-based image processing model calculates the Var(L(I)) value in step 405.

[0044] The preprocessing unit for the receptive field expansion of the CNN-based image processing model is Var(L(I) in step 406. p )) Calculate the value.

[0045] Steps 405 and 406 are the input image (I) and the pseudo-input image (I p ) is an operation that calculates the variance of the Laplacian operation result after calculating the Laplacian operator for each. Var(L(I)) value and Var(L(I p )) value, we can estimate whether the details of each input image and pseudo-input image remain. That is, the Var(L(I)) value and Var(L(I p )) The ratio of values ​​is used to estimate the magnitude of high frequency components (or the degree of loss of high frequency components) of the input image and the pseudo-input image and whether detail information of the input image and the pseudo-input image remains.

[0046] An embodiment of the present disclosure predicts optimal resolution through high-frequency signal analysis using a Laplacian operator and a search technique based on an experimentally determined threshold.

[0047] The Laplacian operator (L) satisfies the following mathematical equation when the second-order differential operator is applied. The result of the Laplacian operation (L(I)) for the input image (I) and the pseudo-input image (I) p ) results of the Laplacian operation (L(I p )) satisfies mathematical expression 1, and is expressed as a value obtained by performing second differentiation for each of the x-axis and y-axis, and combining the results of second differentiation for each direction of the x-axis and y-axis.

[0048]

[0049] The magnitude of high-frequency components in an image can be quantified by calculating the variance of the result of the Laplacian operation on the image, which indicates that the variance of the result of the Laplacian operation and the magnitude of the high-frequency components are proportional. The pseudo-input image (I p: Downscaling-Upscaling The closer the ratio of the high frequency component size of the processed image) to the high frequency component size of the input image (I) is to 1, the less high frequency component loss there is even when the input image is downscaled with the corresponding scaling factor. On the other hand, the closer the ratio is to 0, the greater the loss of high frequency components and the less detail in the image there is. As described later in step 407, this ratio can be compared with a threshold value (T) empirically set in advance to determine the scaling factor with the least high frequency component loss, and the final optimal resolution can be determined based on the determined scaling factor, and the determined resolution can be used as a predicted value.

[0050] The preprocessing unit for the receptive field expansion of the CNN-based image processing model is Var(L(I) in step 407. p )) value divided by the Var(L(I)) value is determined to exceed the T value.

[0051] Var(L(I p )) value divided by the Var(L(I)) value exceeds the T value, the preprocessing unit for receptive field expansion of the CNN-based image processing model proceeds to step 402.

[0052] However, Var(L(I p )) value divided by the Var(L(I)) value is less than or equal to the T value, the preprocessing unit for receptive field expansion of the CNN-based image processing model proceeds to step 408 to determine the scaling factor with the least high-frequency component loss, and calculates the optimal width using the determined scaling factor with the least high-frequency component loss.

[0053] In other words, Var(L(I p )) value divided by the Var(L(I)) value exceeds the T value, the preprocessing unit for receptive field expansion of the CNN-based image processing model repeatedly performs the operation by proceeding to step 402 until the desired resolution is found.

[0054] However, Var(L(I p )) value divided by the Var(L(I)) value is less than or equal to the T value, the preprocessing unit for receptive field expansion of the CNN-based image processing model is judged to exceed the desired resolution.

[0055] After step 408, the preprocessing unit for receptive field expansion of the CNN-based image processing model calculates the optimal height of the image using the scaling factor with the least high-frequency component loss determined in step 409.

[0056] The optimal width and optimal height of the image are used to predict the optimal resolution.

[0057] Although FIG. 4 describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure pertains may modify and apply various modifications and variations, such as changing the order described in FIG. 4 and executing one or more processes in parallel, without departing from the essential characteristics of one embodiment of the present disclosure, and thus is not limited to the chronological order of FIG. 4.

[0058] The optimal resolution prediction part of Fig. 4 can predict the optimal resolution using various measurement and prediction methods in addition to the method using the Laplacian operator.

[0059] FIG. 5 is a diagram comparing the preprocessing effect for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0060] Figure 5 (a) is an example of the original input image.

[0061] Figure 5 (b) is an example diagram showing the result of processing the original input image using the basic method.

[0062] FIG. 5 (c) is an exemplary diagram showing the preprocessing effect for receptive field expansion of a CNN-based image processing model according to an embodiment of the present disclosure.

[0063] It can be confirmed that method (c) using the original input image (a) by preprocessing it with a method according to an embodiment of the present disclosure can achieve clearer improvement results and faster processing time than the existing method (b) using the original input image (a) without preprocessing. For example, the processing time of the existing method (b) is, for example, 355 msec, whereas the processing time of method (c) using the preprocessing with a method according to an embodiment of the present disclosure is, for example, 231 msec.

[0064] In particular, the method according to the embodiment of the present disclosure can exhibit good performance in traditional animation work where line expression is important and texture details are relatively small.

[0065] An embodiment of the present disclosure can improve performance by expanding a receptive field through a preprocessing step of downscaling an input image to an optimal resolution without changing the structure of an existing CNN-based image enhancement model, while reducing the amount of computation by processing the input image at a lower resolution than the input image.

[0066] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0067] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0068] Various implementations of the systems and techniques described herein can be implemented by a programmable computer. The computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems, or a combination thereof), and at least one communication interface. For example, the programmable computer can be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant (PDA), a cloud computing system, or a mobile device.

[0069] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0070] CROSS-REFERENCE TO RELATED APPLICATION

[0071] This patent application claims priority to Korean Patent Application No. 10-2023-0141736, filed on October 23, 2023, which is incorporated herein by reference in its entirety.

Claims

1. In a device that converts the resolution of an image, At least one memory for storing instructions; and at least one processor, At least one processor executes the instructions, A process of creating a reduced image by downscaling an input image using a first scaling factor; A process of generating a pseudo input image by upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor; A process of generating a first value by applying a Laplacian operation and variance to the above input image; A process of generating a second value by applying the Laplacian operation and variance to the above-mentioned pseudo-input image; A process of determining a second scaling factor based on a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and A device configured to perform a process of converting the resolution of the image using the second scaling factor.

2. In paragraph 1, The process of determining the above second scaling factor is: A process for determining whether the above ratio value exceeds the predetermined threshold value; and A device comprising a process of generating an image of a converted resolution by multiplying the width and height of the input image by the second scaling factor having the least high-frequency component loss when the ratio value is less than or equal to the predetermined threshold value.

3. In paragraph 2, A process of updating the first value and the second value when the above ratio value exceeds the predetermined threshold value; and A device further comprising a process of re-determining the second scaling factor based on a comparison result of the ratio value of the updated first value and the second value and the predetermined threshold value.

4. In paragraph 1, The above ratio value is used to estimate the degree of loss of high frequency components of the input image and the pseudo input image and whether detail information of the input image and the pseudo input image remains.

5. In paragraph 1, A device in which the first scaling factor represents a value obtained by subtracting the update unit of the scaling factor from the initial scaling factor.

6. In paragraph 1, The above first value is generated as a variance value of the Laplacian operation for the input image, A device wherein the second value is generated as a variance value of a Laplacian operation on the pseudo input image.

7. In a computer implementation method using a device for converting the resolution of an image, A process of creating a reduced image by downscaling an input image using a first scaling factor; A process of generating a pseudo input image by upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor; A process of generating a first value by applying a Laplacian operation and variance to the above input image; A process of generating a second value by applying the Laplacian operation and variance to the above-mentioned pseudo-input image; A process of determining a second scaling factor based on a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and A computer-implemented method comprising a process of converting the resolution of the image using the second scaling factor.

8. In paragraph 7, The process of determining the above second scaling factor is: A process for determining whether the above ratio value exceeds the predetermined threshold value; and A computer-implemented method comprising the step of generating an image of a converted resolution by multiplying the width and height of the input image by the second scaling factor having the least high-frequency component loss when the ratio value is less than or equal to the predetermined threshold value.

9. In paragraph 8, A process of updating the first value and the second value when the above ratio value exceeds the predetermined threshold value; and A computer-implemented method further comprising a process of re-determining the second scaling factor based on a comparison result of the ratio value of the updated first value and the second value and the predetermined threshold value.

10. In paragraph 7, A computer-implemented method wherein the above ratio value is used to estimate the degree of loss of high-frequency components of the input image and the pseudo-input image and whether detail information of the input image and the pseudo-input image remains.

11. In paragraph 7, A computer-implemented method, wherein the first scaling factor represents a value obtained by subtracting the update unit of the scaling factor from the initial scaling factor.

12. In paragraph 7, The above first value is generated as a variance value of the Laplacian operation for the input image, A computer-implemented method, wherein the second value is generated as a variance value of a Laplacian operation on the pseudo-input image.

13. A computer-readable, non-transitory recording medium having stored thereon a command, which, when executed by the computer, causes the computer to: A process of creating a reduced image by downscaling an input image using a first scaling factor; A process of generating a pseudo input image by upscaling the reduced image by a value obtained by dividing 1 by the first scaling factor; A process of generating a first value by applying a Laplacian operation and variance to the above input image; A process of generating a second value by applying the Laplacian operation and variance to the above-mentioned pseudo-input image; A process of determining a second scaling factor based on a comparison result of a ratio value of the first value and the second value and a predetermined threshold value; and A computer-readable, non-transitory recording medium that causes a process of converting the resolution of the image using the second scaling factor.

14. In paragraph 13, The process of determining the above second scaling factor is: A process for determining whether the above ratio value exceeds the predetermined threshold value; and A computer-readable non-transitory recording medium, comprising a process of generating an image of a converted resolution by multiplying the width and height of the input image by the second scaling factor having the least high-frequency component loss when the ratio value is less than or equal to the predetermined threshold value.

15. In paragraph 14, A process of updating the first value and the second value when the above ratio value exceeds the predetermined threshold value; and A computer-readable non-transitory recording medium further comprising a process of re-determining the second scaling factor based on a comparison result of the ratio value of the updated first value and the second value and the predetermined threshold value.

16. In paragraph 13, A computer-readable, non-transitory recording medium, wherein the above ratio value is used to estimate the degree of loss of high-frequency components of the input image and the pseudo-input image and whether detail information of the input image and the pseudo-input image remains.

17. In paragraph 13, A computer-readable, non-transitory recording medium, wherein the first scaling factor represents a value obtained by subtracting the update unit of the scaling factor from the initial scaling factor.

18. In paragraph 13, The above first value is generated as a variance value of the Laplacian operation for the input image, A computer-readable, non-transitory recording medium, wherein the second value is generated as a variance value of a Laplacian operation on the pseudo-input image.

Images