Method for super-resolution and apparatus for super-resolution image processing

Abstract

Provided are a method for super-resolution and an apparatus for super-resolution image processing, the method including: determining a direction type of an input image based on a gradient of the input image; acquiring a first intermediate image based on a residual lookup table (LUT) corresponding to the kernel set mapped to the determined direction type, the first intermediate image being a super-resolution image corresponding to the input image; determining an LUT intensity graph of the input image based on the gradient of the input image and a preset tuning parameter; and generating an output image based on the LUT intensity map and the first intermediate image, the output image being a super-resolution image of the input image.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0088014 filed on Jul. 6, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following examples relate to methods and apparatus with super-resolution. Background Art

[0003] Image super-resolution technology can be a technology for obtaining a high-resolution (HR) image from a single low-resolution (LR) image. Image super-resolution technology may include not only improving the quality of LR images to HR, but also scaling technology for large scaling of small objects in the image. Super-resolution technology (e.g., nearest interpolation, bilinear interpolation, and bicubic interpolation) can use interpolation and various deep learning-based super-resolution technologies, and technologies for making deep learning models lightweight may include quantization, pruning, low-rank approximation, knowledge distillation, and compact network design to process super-resolution of images in real time in mobile environments (e.g., smart phones). Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one or more overall embodiments, a processor-implemented method with super-resolution includes: determining a direction type of an input image based on a gradient of the input image; acquiring a first intermediate image based on a residual lookup table (LUT) corresponding to a kernel set mapped to the determined direction type, the first intermediate image being a super-resolution image corresponding to the input image; determining a LUT intensity map of the input image based on the gradient of the input image and preset tuning parameters; and generating an output image based on the LUT intensity map and the first intermediate image, the output image being a super-resolution image of the input image.

[0006] The step of acquiring the first intermediate image may include: determining a receptive field corresponding to a reference pixel in the input image based on a kernel included in the kernel set; updating the values ​​of pixels in the receptive field using values ​​obtained by subtracting values ​​of reference pixels from values ​​of pixels in the receptive field; and obtaining values ​​stored in a residual LUT corresponding to the kernel for combinations of updated values ​​of pixels in the receptive field other than the reference pixels to obtain a first intermediate image.

[0007] The step of generating the output image may include: acquiring a second intermediate image based on the LUT intensity map and the first intermediate image; and generating the output image based on the second intermediate image and a baseline image acquired by applying interpolation for super-resolution to the input image.

[0008] The LUT intensity map for the input image may include a LUT intensity value for each pixel in the input image determined based on a product of a gradient of each pixel in the input image and a tuning parameter.

[0009] The LUT intensity value of a first pixel included in an input image may be determined as a predetermined maximum value in response to a product of a magnitude of a gradient of the first pixel and a tuning parameter being greater than a predetermined maximum value, may be determined as a predetermined minimum value in response to a product of a magnitude of a gradient of the first pixel and a tuning parameter being less than a predetermined minimum value, and may be determined as the product of a magnitude of a gradient of the first pixel and a tuning parameter in response to a product of a magnitude of a gradient of the first pixel and a tuning parameter being less than or equal to a maximum value and greater than or equal to a minimum value.

[0010] The kernel set may include multiple kernels, and the step of acquiring the first intermediate image may include acquiring the first intermediate image by weighted summing super-resolution operation results corresponding to the multiple kernels respectively based on weights determined for the multiple kernels respectively.

[0011] The step of acquiring a first intermediate image may include: updating the first intermediate image using an image acquired by overlapping blocks corresponding to different areas of the first intermediate image, each block of the first intermediate image may be a group of pixels corresponding to a super-resolution result of each pixel in the input image, and the pixel value of the area where the blocks are overlapped in the updated first intermediate image may be determined as the average value of the pixel values ​​of the overlapping blocks.

[0012] The step of determining the LUT intensity map of the input image may include: determining the LUT intensity value corresponding to the pixel detected as a corner in the input image based on the tuning parameters and the gradient of the pixel detected as a corner in the input image, and the step of generating the output image may include: in response to the first pixel in the first intermediate image corresponding to the pixel detected as a corner in the input image, generating the output image based on the first intermediate image and the LUT intensity value corresponding to the pixel detected as a corner in the input image.

[0013] The method may include setting the output image as the input image, and repeating the steps of determining the direction type of the input image, acquiring the first intermediate image, determining the LUT intensity map, and generating the output image.

[0014] In one or more general aspects, a processor-implemented method with super-resolution includes: determining a direction type of an input image based on a gradient of the input image; obtaining super-resolution operation results corresponding to the kernels respectively based on residual lookup tables corresponding to the kernels included in a kernel set, the kernel set being mapped to the determined direction type of the input image; and generating an output image based on weighted summation of the super-resolution operation results corresponding to the kernels respectively by weights determined for the kernels respectively, the output image being a super-resolution image of the input image, wherein the residual LUT includes super-resolution operation results of a combination of pixel values, the number of which is less than "1" than the size of the corresponding kernel.

[0015] The step of generating an output image may include: obtaining a first intermediate image by weighted summing up super-resolution operation results corresponding to the kernels respectively based on weights determined for the kernels respectively, the first intermediate image being a super-resolution image corresponding to the input image; and generating an output image based on the first intermediate image and a baseline image obtained by applying interpolation for super-resolution to the input image.

[0016] The step of obtaining a super-resolution operation result may include: determining a receptive field corresponding to a reference pixel in an input image based on a kernel included in a kernel set; updating a value of a pixel in the receptive field using a value obtained by subtracting a value of the reference pixel from a value of the pixel in the receptive field; and obtaining a value stored in a residual lookup table corresponding to the kernel for a combination of updated values ​​of pixels other than the reference pixel in the receptive field to obtain a super-resolution operation result.

[0017] The first weight determined for a first kernel among the kernels may be determined based on a relative positional relationship between the reference pixel and the further pixels in the receptive field indicated by the first kernel.

[0018] The step of generating an output image may include: updating the super-resolution operation result corresponding to the first kernel by overlapping blocks corresponding to different areas of the super-resolution operation result corresponding to the first kernel among the kernels, each block of the super-resolution operation result corresponding to the first kernel may be a set of super-resolution results of each pixel in the input image stored in the residual LUT corresponding to the first kernel, and the pixel value of the area where the blocks are overlapped in the updated super-resolution operation result corresponding to the first kernel may be determined as the average value of the pixel values ​​of the overlapped blocks.

[0019] The step of generating an output image may include: determining a weighted sum of super-resolution operation results corresponding to the kernels respectively for a first pixel in the input image that is detected as a corner; and determining a simple sum of super-resolution operation results corresponding to the kernels respectively for a second pixel in the input image that is not detected as a corner.

[0020] The method may include setting an output image as an input image, and repeating the steps of determining a direction type of the input image, obtaining super-resolution operation results respectively corresponding to the kernels, and generating an output image.

[0021] In one or more general aspects, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, configure the processor to perform any one, any combination, or all of the operations and / or methods described herein.

[0022] In one or more general aspects, a device for super-resolution image processing includes: one or more processors configured to determine a direction type of an input image based on a gradient of the input image; obtain a first intermediate image based on a residual lookup table (LUT) corresponding to a kernel set mapped to the determined direction type, the first intermediate image being a super-resolution image corresponding to the input image; determine a LUT intensity map of the input image based on the gradient of the input image and a preset tuning parameter; and generate an output image based on the LUT intensity map and the first intermediate image, the output image being a super-resolution image of the input image.

[0023] The kernel set may include multiple kernels, and in order to obtain a first intermediate image, the one or more processors may be configured to obtain the first intermediate image by weighted summing up the super-resolution operation results corresponding to the multiple kernels respectively based on weights determined for the multiple kernels respectively.

[0024] In one or more general aspects, a device for super-resolution image processing includes: one or more processors configured to: determine a direction type of an input image based on a gradient of the input image; obtain super-resolution operation results corresponding to the kernels respectively based on residual lookup tables (LUTs) respectively corresponding to kernels included in a kernel set, the kernel set being mapped to the determined direction type of the input image; and generate an output image based on weighted summation of super-resolution operation results corresponding to the kernels respectively by weights determined for the kernels respectively, the output image being a super-resolution image of the input image, wherein the residual LUT includes super-resolution operation results of a combination of pixel values, the number of which is less than "1" than the size of the corresponding kernel.

[0025] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example of a method for super-resolution is shown.

[0027] Figure 2 An example of a set of kernels mapped to direction types is shown.

[0028] Figure 3 An example of residual lookup table (LUT) processing is shown.

[0029] Figure 4 An example of a framework showing a method for super-resolution.

[0030] Figure 5 An example of a framework showing a method for super-resolution.

[0031] Figure 6 An example of a method for super-resolution is shown.

[0032] Figure 7 An example of determining the weight is shown.

[0033] Figure 8 An example of a framework showing a method for super-resolution.

[0034] Fig. 9 An example of a framework showing a method for super-resolution.

[0035] Fig.10 An example of a method for super-resolution based on an average value of overlapping regions is shown.

[0036] Fig.11 An example of a framework of a method for super-resolution including an overlap region average calculation process is shown.

[0037] Fig.12 An example of a framework of a method for super-resolution including an overlap region average calculation process is shown.

[0038] Fig.13 An example of a method for super-resolution that is repeatedly performed is shown.

[0039] FIG. 14A to FIG. 14C An example of a method for super-resolution for controlling artifacts is shown.

[0040] FIG. 15A to FIG. 15C An example of a method for super-resolution for corner case handling is shown.

[0041] Fig.16 An example of the configuration of a device is shown.

[0042] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0043] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for the operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of known features after understanding the disclosure of the present application can be omitted.

[0044] The phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. are intended to have a separate meaning, and unless the corresponding description and embodiments require such enumeration (for example, "at least one of A, B, and C") to be interpreted as having a connecting meaning, these phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. also include examples in which one or more of each of A, B and / or C (for example, any combination of one or more of each of A, B, and C) may exist.

[0045] Although terms such as "first", "second" and "third", or A, B, (a), (b), etc. may be used herein to describe various members, components, regions, layers or parts, these members, components, regions, layers or parts should not be limited by these terms. Each of these terms is not used to define, for example, the nature, order or sequence of the corresponding member, component, region, layer or part, but is only used to distinguish the corresponding member, component, region, layer or part from other members, components, regions, layers or parts. Therefore, without departing from the teachings of the examples, the first member, first component, first region, first layer or first part referred to in the examples described herein may also be referred to as the second member, second component, second region, second layer or second part.

[0046] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, the component or element may be directly "connected to," "coupled to," or "engaged to" the other component or element (e.g., directly in contact with the other component or element), or one or more other components or elements may reasonably be present in between. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to" another component or element, there may not be other components or elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0047] The terms used herein are only used to describe various examples and will not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As non-limiting examples, the terms "comprise", "include" and "have" illustrate the existence of stated features, quantities, operations, components, elements and / or their combinations, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or their combinations.

[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs and based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the disclosure of the relevant field and the present application, and should not be interpreted in an idealized or overly formal manner. The use of the term "may" with respect to an example or embodiment herein (for example, with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0049] Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, the same reference numerals denote the same elements, and repeated descriptions thereof will be omitted.

[0050] Figure 1 An example of a method for super-resolution is shown. Figure 1 Operations 110 to 140 of the embodiment may be performed in the order and manner shown. However, without departing from the spirit and scope of the illustrated examples, the order of one or more of the operations may be changed and / or one or more of the operations may be performed in parallel or simultaneously.

[0051] Reference Figure 1 , a method for super-resolution according to an example may include operation 110 : determining a direction type of an input image based on a gradient of the input image.

[0052] The input image according to the example may be a low resolution (LR) image and may correspond to an image that is an object of super resolution. For example, the input image may include various images such as raw (RAW) data obtained from a complementary metal oxide semiconductor (CMOS) image sensor (CIS) or an image pre-processed by an image signal processor (ISP) (e.g., processed using white balance (WB), demosaicing, etc.).

[0053] According to an example, the input image may include an image corresponding to a single frame or a video including a plurality of frames. As an example, when the input image is a video including a plurality of frames, the super-resolution method including operations 110 to 140 may be performed on each frame. Hereinafter, an image corresponding to a single frame will be described as an example.

[0054] According to one example, the gradient of an image may be data indicating a change in pixel values ​​between adjacent pixels in the image. For example, when the brightness difference between a background and an object included in the image is significant, the pixel value may have a large change at the boundary of the object, and therefore, the gradient at the boundary of the object in the image may be a value greater than the value of the gradient of other parts in the image.

[0055] According to one example, various edge detection methods (such as Canny, Sobel, and Prewitt) may be used to obtain the gradient of the image. As an example, a vector of the derivative of the image with respect to the x-axis and the derivative with respect to the y-axis may be obtained as the gradient of the image.

[0056] Operation 110 according to the example may include an operation of obtaining a derivative component for each axis of the input image by applying a mask (or mask) for the derivative about each axis to the input image. For example, the mask for the derivative may include at least one of a Sobel mask, a Scharr mask, a Prewitt mask, and a Roberts mask. For example, the axis of the image may include an x-axis as the horizontal axis of the two-dimensional image and a y-axis as the vertical axis of the two-dimensional image. The derivative component for each axis may be obtained by applying the mask for the derivative to each of the x-axis and the y-axis of the input image to perform a derivative operation on each axis.

[0057] For example, as represented by the following equation 1, the image can be obtained by performing an operation on the image using a Sobel mask. A The derivative component of about the x-axis G x and the derivative component about the y-axis G y .

[0058] Equation 1: In equation 1, for the image A The operation using Sobel mask can be image A Convolution operation between the pixel value and the Sobel mask.

[0059] According to one example, the direction type of the input image may be determined based on the gradient of the input image. The direction type of the input image may be determined for each pixel in the input image. The direction type of the image may be a type classified according to the direction along which the pixel value of the image changes, and may include a predetermined number of direction types. For example, the direction type may include a horizontal type, a vertical type, a slash type, and / or a backslash type.

[0060] Operation 110 according to an example may include an operation of determining a direction type of the input image as one of predetermined direction types based on an angle of a gradient of the input image obtained from a derivative component with respect to each axis.

[0061] For example, as shown in Equation 2 below, the angle of the gradient θ For example, the derivative components with respect to the corresponding axes can be G x and G y get.

[0062] Equation 2: The angle of the gradient calculated according to Equation 2 θ The value range of may correspond to [-π / 2, π / 2]. For example, the angle of the gradient calculated according to Equation 2 θ It may have a value within the range of greater than or equal to -90 degrees and less than or equal to 90 degrees.

[0063] According to one example, the number of direction types may vary according to settings. For example, the number of direction types may be set to a value greater than or equal to "1". When the number of direction types is set to a larger value, the size of the lookup table (LUT) may increase because the number of kernels corresponding to the direction types increases. When the number of direction types is set to a larger value, the method and apparatus of one or more embodiments may further improve super-resolution performance.

[0064] As an example, as shown in Table 1 below, the angle of the gradient calculated for the input image may be θ range, determining the direction type of the input image as one of four predetermined direction types.

[0065] Table 1: For example, when the gradient angle θ When the gradient angle is within the range of -22.5 degrees or more and less than 22.5 degrees, the direction type of the input image may be determined as a horizontal type. θWhen the gradient angle is within the range of 22.5 degrees or more and less than 67.5 degrees, the direction type of the input image may be determined as a diagonal line type. θ When the gradient angle is within the range of -67.5 degrees or more and less than -22.5 degrees, the direction type of the input image may be determined as a backslash type. θ When not falling within any one of the range of greater than or equal to 22.5 degrees and less than 67.5 degrees, the range of greater than or equal to -67.5 degrees and less than -22.5 degrees, and the range of greater than or equal to -67.5 degrees and less than -22.5 degrees, the direction type of the input image can be determined as a vertical type.

[0066] As an example, as shown in Table 2 below, the angle of the gradient calculated for the input image may be θ range, determining the direction type of the input image as one of two predetermined direction types.

[0067] Table 2: For example, when the gradient angle θ When the gradient angle is within the range of -45 degrees or more and less than 45 degrees, the direction type of the input image may be determined as a horizontal type. θ When not falling within the range of greater than or equal to −45 degrees and less than 45 degrees, the direction type of the input image may be determined as a vertical type.

[0068] According to an example, the number of direction types may vary according to settings. For example, the number of direction types may be set to a value greater than or equal to "1". When the number of direction types is set to a larger value, the method and apparatus of one or more embodiments may further improve super-resolution performance. As an example, the number of direction types may be a value determined independently of an upscaling factor. As an example, the number of direction types may be a value determined independently of a kernel size.

[0069] According to an example, a kernel set may be mapped to each direction type. A kernel set corresponding to an input image may be determined according to the direction type of the input image. A kernel set may correspond to a set of at least one kernel. A kernel may correspond to data indicating a receptive field of a super-resolution model, and the method and apparatus of one or more embodiments implement the super-resolution model to perform super-resolution on an LR image.

[0070] The super-resolution model according to the example may correspond to a model for outputting a super-resolution image corresponding to the LR image input thereto. For example, the super-resolution model may include a neural network trained to output a super-resolution image obtained by upscaling the input image by a preset factor. The super-resolution model may output pixels in the super-resolution image corresponding to each pixel in the input image of the super-resolution model. As an example, when the upscaling factor of the super-resolution is r, the super-resolution model may output r pixels in the super-resolution image corresponding to one pixel in the input image. 2 pixels.

[0071] According to one example, the upscaling factor of the super-resolution model may be determined differently for the width and height of the image. For example, when the upscaling factor for the width of the image is r 1 And the magnification factor for height is r 2 When , the super-resolution model can output r in the super-resolution image corresponding to a pixel in the input image. 1 × 2 pixels. 1 and r 2 Can have the same value or different values.

[0072] According to one example, the super-resolution model may be operated to output a pixel corresponding to a reference pixel in a super-resolution image based on a receptive field corresponding to a predetermined pixel in an input image (hereinafter, a reference pixel). The receptive field corresponding to the reference pixel may include the reference pixel and (one or more) pixels located around or near the reference pixel and referenced to output the pixel corresponding to the reference pixel in the super-resolution image. The super-resolution model may be operated to output a pixel corresponding to the reference pixel in the super-resolution image by referring to the reference pixel in the input image and (one or more) pixels located around the reference pixel. For example, the super-resolution model may determine the value of the pixel corresponding to the reference pixel in the super-resolution image based on an operation on the value of the reference pixel in the image and (one or more) pixels located around the reference pixel.

[0073] According to one example, the super-resolution model may determine a pixel to be referenced based on a kernel set mapped to a direction type corresponding to a reference pixel in an input image to determine a super-resolution operation result of the reference pixel. The pixel indicated by the kernel included in the kernel set may be referenced to determine the value of the pixel corresponding to the reference pixel in the super-resolution image. When a plurality of kernels are included in the kernel set, the value of the pixel corresponding to the reference pixel in the super-resolution image may be determined based on a simple sum (e.g., a simple sum of a and b may be expressed as a+b) or a weighted sum of the results obtained for each kernel.

[0074] For example, Figure 2shows an example of a kernel set mapped to a direction type. Figure 2 , a first kernel set 210 mapped to the horizontal type, a second kernel set 220 mapped to the vertical type, a third kernel set 230 mapped to the backslash type, and a fourth kernel set 240 mapped to the slash type are shown. Figure 2 The rectangular matrix in may correspond to a partial area of ​​the input image, and one unit of the rectangular matrix may correspond to a pixel unit of the input image. Figure 2 A thick line graph (eg, 211) on a rectangular matrix in may correspond to a kernel. Each of kernel sets 210, 220, 230, and 240 may include four kernels.

[0075] For example, the kernel 211 may correspond to a kernel having a size of “4” including four pixels. The shaded pixels in the kernel 211 may be reference pixels, and pixels corresponding to the reference pixels in the super-resolution image may be determined based on the values ​​of the pixels included in the kernel 211.

[0076] According to an example, a plurality of different kernels included in the same kernel set may have different kernel characteristics (such as size and / or shape). Each kernel set may include different combinations of kernels. For example, the shapes of the kernels included in the first kernel set 210 and the kernels included in the second kernel set 220 may be different. Figure 2 An example is shown in which the kernels included in each kernel set have the same size of "4" pixels. However, Figure 2 Different from the example of, each kernel set may include kernels of different sizes. For example, the size of the kernel included in the kernel set mapped to the first direction type may be 2×2 pixels, and the size of the kernel included in the kernel set mapped to the second direction type may be 3×3 pixels.

[0077] The method for super-resolution according to an example may include operation 120 of acquiring a first intermediate image as a super-resolution image corresponding to the input image based on a residual LUT corresponding to a kernel set mapped to a determined direction type.

[0078] The LUT according to the example may correspond to a data structure storing super-resolution operation results of a super-resolution model. As described above, a super-resolution operation result corresponding to a predetermined pixel may be determined based on a pixel value within a receptive field according to a kernel. The LUT may store a super-resolution operation result obtained by using a super-resolution model for a combination of all pixels corresponding to the receptive field. As described above, the receptive field may be determined by a kernel. The super-resolution operation result may include pixel values ​​of a super-resolution image mapped to each pixel value combination corresponding to the size of the kernel.

[0079] According to an example, a LUT may correspond to each kernel. A kernel set may include one or more kernels, and each kernel included in the kernel set may be mapped to a LUT. For example, referring to Figure 2 , 16 LUTs may store super-resolution operation results corresponding to the 16 kernels included in the kernel sets 210, 220, 230, and 240, respectively. As an example, a first LUT corresponding to a first kernel may include a super-resolution operation result corresponding to the first kernel, and a second LUT corresponding to a second kernel may include a super-resolution operation result corresponding to the second kernel. For the same combination of pixel values, the super-resolution operation result corresponding to the first kernel and the super-resolution operation result corresponding to the second kernel may be different from each other.

[0080] The LUT according to the example may include a residual LUT. The residual LUT may include a super-resolution operation result of a combination of (one or more) pixel values, the number of (one or more) pixel values ​​being less than "1" than the size of the corresponding kernel. The residual LUT may store a super-resolution operation result of a combination of (one or more) pixel values ​​((kernel size - 1) pixel values) excluding reference pixels from the size of the kernel. By reducing the number of input pixels by excluding reference pixels from the size of the kernel, the methods and apparatus of one or more embodiments may advantageously reduce the LUT memory size by a factor of the number of sampling points.

[0081] Operation 120 of acquiring a first intermediate image according to the example may include the following operations: determining a receptive field corresponding to a reference pixel in the input image based on a kernel included in a kernel set; updating the value of a pixel in the receptive field using a value obtained by subtracting the value of the reference pixel from the value of the pixel in the receptive field; and obtaining a value stored in a residual LUT corresponding to the kernel for a combination of updated values ​​of pixels in the receptive field other than the reference pixel.

[0082] For example, Figure 3 An example of residual lookup table (LUT) processing is shown. Figure 3 , the values ​​of pixels 311, 312, 313, and 314 included in the receptive field of the input image 310 according to the kernel 320 may be updated based on the value of the reference pixel 311. The values ​​of the pixels 311, 312, 313, and 314 included in the receptive field may be updated to a value obtained by subtracting the value of the reference pixel 311 from the original values ​​of the pixels 311, 312, 313, and 314. Referring to the updated receptive field 330, the updated value of the reference pixel 311 may be I 0 -I 0 = 0, the updated value of pixel 312 may be 1 1 -I 0 =I 1 ', the updated value of pixel 313 may be I2 -I 0 =I 2 ', and the updated value of pixel 314 may be I 3 -I 0 =I 3 '. The residual LUT 340 may include super-resolution operation results corresponding to a combination of three pixels 312, 313, and 314 excluding the reference pixel 311 from the size of the kernel (e.g., 4). For example, when a pixel has an integer value greater than or equal to "0" and less than or equal to "255", the residual LUT 340 corresponding to the kernel 320 having the size of "4" may include a combination of three pixels 312, 313, and 314 excluding the reference pixel 311 from the size of the kernel (e.g., 4). 3 The super-resolution calculation results corresponding to the combination of 256 3 The combinations are all possible combinations of three pixel values.

[0083] According to one example, the residual LUT may be based on the magnification factor r for the width 1 and the magnification factor r for height 2 , storing r for each combination of pixel values 1 × 2 The result of the super-resolution operation is that each combination of pixel values ​​has a size corresponding to the kernel. Figure 3 , when the magnification factor for width is "2" and the magnification factor for height is "3", the residual LUT 340 may store the same information as I 1 '、I 2 'and I 3 The six super-resolution calculation results corresponding to the combination of ' 0 , O 1 , O 2 , O 3 , O 4 and O 5 , I 1 '、I 2 'and I 3 The combination of ' is a combination of three pixels that is smaller than the size of the kernel 320 by "1".

[0084] As an example, the storage for I 1 '、I 2 'and I 3 'The six values ​​in the combined LUT LUT [I 1 '][I 2 '][I 3 '][0]=O 0 、LUT[I 1 '][I 2 '][I 3 '][1]=O 1 、LUT[I1 '][I 2 '][I 3 '][2]=O 2 、LUT[I 1 '][I 2 '][I 3 '][3]=O 3 、LUT[I 1 '][I 2 '][I 3 '][4]=O 4 , and LUT[I 1 '][I 2 '][I 3 '][5]=O 5 The values ​​of the six pixels 351, 352, 353, 354, 355 and 356 corresponding to the reference pixel 311 in the super-resolution image 350 may be determined respectively. 1 '、I 2 'and I 3 The combination of ' is a combination of updated values ​​of the remaining pixels 312, 313, and 314 excluding the reference pixel 311 included in the updated receptive field 330. The values ​​of the six pixels 351, 352, 353, 354, 355, and 356 in the super-resolution image 350 may correspond to the super-resolution result of the reference pixel 311. The reference pixel 311 may be determined as one of the pixels included in the input image 310. The super-resolution image 350 may correspond to an image including a super-resolution operation result obtained from the residual LUT 340 by setting each of all pixels included in the input image 310 as a reference pixel.

[0085] According to one example, based on r 1 × 2 The position of the super-resolution operation result stored in the residual LUT determines the position of the super-resolution image corresponding to the reference pixel in the input image. 1 × 2 pixel values. For example, refer to Figure 3 , for the combination of updated values ​​of the remaining pixels 312, 313, and 314 excluding the reference pixel 311 among the pixels 311, 312, 313, and 314 included in the receptive field of the reference pixel 311, I 1 '、I 2 'and I 3 ', stored in the LUT[I 1 '][I 2 '][I 3 '][0] The value O at position 0The value of the pixel 351 which may be determined to correspond to the position (0, 0) among the six pixels 351, 352, 353, 354, 355, and 356 corresponding to the reference pixel 311 in the super-resolution image 350 is stored in the LUT [I 1 '][I 2 '][I 3 '][1] The value at position O 1 The value of the pixel 352 which may be determined to correspond to the position (0, 1) among the six pixels 351, 352, 353, 354, 355, and 356 corresponding to the reference pixel 311 in the super-resolution image 350 is stored in the LUT [I 1 '][I 2 '][I 3 '][2] The value at position O 2 The value of the pixel 353 which may be determined to correspond to the position (1, 0) among the six pixels 351, 352, 353, 354, 355, and 356 corresponding to the reference pixel 311 in the super-resolution image 350 is stored in the LUT [I 1 '][I 2 '][I 3 '][3] The value O at position 3 The value of the pixel 354 which may be determined to correspond to the position (1, 1) among the six pixels 351, 352, 353, 354, 355, and 356 corresponding to the reference pixel 311 in the super-resolution image 350 is stored in the LUT [I 1 '][I 2 '][I 3 '][4] The value at position O 4 The value of the pixel 355 corresponding to the position (2, 0) among the six pixels 351, 352, 353, 354, 355, and 356 corresponding to the reference pixel 311 in the super-resolution image 350 may be determined and stored in the LUT [I 1 '][I 2 '][I 3 '][5] The value at position O 5 The value of the pixel 356 corresponding to the position (2, 1) among the six pixels 351 , 352 , 353 , 354 , 355 , and 356 corresponding to the reference pixel 311 in the super-resolution image 350 may be determined.

[0086] According to one example, the LUT corresponding to the kernel may store the super-resolution operation results of the combination of pixel values ​​in the LUT corresponding to the most significant bit (MSB) and the LUT corresponding to the least significant bit (LSB), respectively. For example, when a pixel has a value greater than or equal to "0" and less than or equal to "2 n” (where n is a natural number greater than or equal to “1”), the value of the pixel can be represented by n bits. The LUT corresponding to the MSB can store the high n bits corresponding to the n-bit pixel value. 1 The LUT corresponding to the LSB can store the data corresponding to the low n bits of the n-bit pixel value. 2 Here, n 1 and n 2 Can be determined to satisfy the condition n 1 +n 2 =n is a positive integer less than n. For example, in the case where each combination of four pixels having a pixel value represented by 8 bits will have a pixel value r represented by 8 bits. 1 × 2 When the super-resolution operation results are stored in the LUT, the LUT corresponding to the MSB can store 1 × 2 The value corresponding to the high 4 bits of the super-resolution calculation result, r 1 × 2 The upper 4 bits of the super-resolution calculation result correspond to the combination of the upper 4 bits of the pixel values ​​of the four pixels. The LUT corresponding to the LSB can store 1 × 2 The value corresponding to the lower 4 bits of the super-resolution calculation result, r 1 × 2 The lower 4 bits of the super-resolution operation result correspond to the combination of the lower 4 bits of the pixel values ​​of the four pixels. The final pixel value of the super-resolution image can be determined based on the combination of the result in the LUT corresponding to the MSB and the result in the LUT corresponding to the LSB. For example, when the predetermined pixel value of the super-resolution image obtained based on the LUT corresponding to the MSB is 0101 and the corresponding pixel value obtained based on the LUT corresponding to the LSB is 0001, the final value of the corresponding pixel can be determined as 01010001 with 0101 as four high-order values ​​and 0001 as four low-order values.

[0087] According to one example, a process of obtaining a super-resolution operation result corresponding to each pixel in the input image from a residual LUT storing a super-resolution operation result of a combination of pixel values ​​corresponding to a kernel of the input image may be referred to as a residual LUT process. The kernel of the input image may include a kernel included in a kernel set mapped to a direction type determined in operation 110.

[0088] According to one example, a first intermediate image may be obtained by performing residual LUT processing on an input image. The first intermediate image may correspond to an output result of a super-resolution model for the input image. By obtaining the first intermediate image using the operation result stored in the LUT without performing the operation of the super-resolution model on the image, the method and apparatus of one or more embodiments may reduce the time and resources used for operation.

[0089] According to one example, when a plurality of kernels are included in a kernel set of a direction type mapped to an input image, a first intermediate image may be acquired based on a super-resolution operation result of each of the plurality of kernels. The super-resolution operation result of each kernel may be obtained from a residual LUT corresponding to each kernel. As an example, when a plurality of kernels are included in the kernel set, a simple sum value of the super-resolution operation results of each kernel may be determined as a pixel value of the first intermediate image. As an example, when a plurality of kernels are included in the kernel set, an average value of the super-resolution operation results of each kernel may be determined as a pixel value of the first intermediate image.

[0090] As an example, when multiple kernels are included in the kernel set, the value of the weighted sum of the super-resolution operation results of each kernel may be determined as the pixel value of the first intermediate image. For example, the operation 120 of acquiring the first intermediate image according to the example may include an operation of acquiring the first intermediate image by weighted summing the multiple super-resolution operation results corresponding to the multiple kernels respectively based on the multiple weights determined for the multiple kernels respectively. An example of the operation of acquiring the first intermediate image based on the weighted sum of the super-resolution operation results of each kernel will be described in detail below.

[0091] The operation 120 of acquiring the first intermediate image according to the example may include an operation of updating the first intermediate image using an image acquired by overlapping a plurality of blocks corresponding to different regions of the first intermediate image. Each of the plurality of blocks of the first intermediate image may be a group of pixels corresponding to a super-resolution result of a corresponding pixel in the input image. The pixel value of the region where the plurality of blocks are overlapped in the updated first intermediate image may be determined as an average value of the pixel values ​​of the overlapped blocks. An example thereof will be described in detail below.

[0092] The method for super-resolution according to an example may include operation 130: determining a LUT intensity map of the input image based on a gradient of the input image and a preset tuning parameter. As an example, the tuning parameter may be an adjustable variable value and may be predetermined and / or set by a user.

[0093] According to one example, the LUT intensity map of the input image may include a LUT intensity value for each pixel in the input image. The LUT intensity value for each pixel in the input image may be determined based on a product of a gradient of each pixel in the input image and a tuning parameter.

[0094] For example, the LUT intensity map of the input image may be determined as expressed by Equation 3 below.

[0095] Equation 3: In equation 3, G x and G y denote the derivative components with respect to the x-axis and y-axis of the input image, respectively, and can be obtained by Equation 1. | G x |+| G y | may be the magnitude of a gradient, and may be a value indicating the degree of difference or change in pixel values.

[0096] According to Equation 3, when the magnitude of the gradient of a predetermined pixel (e.g., the first pixel) included in the input image | G x |+| G y |With tuning parameters α The product of is greater than the predetermined maximum value max When the LUT intensity value of the first pixel can be determined as the maximum value max ; When the magnitude of the gradient of the first pixel | G x |+| G y |With tuning parameters α The product of is less than the predetermined minimum value min When the LUT intensity value of the first pixel can be determined as the minimum value min ; and when the magnitude of the gradient of the first pixel | G x |+| G y |With tuning parameters α The product of is less than or equal to the maximum value max and greater than or equal to the minimum value min When , the LUT intensity value of the first pixel can be determined as the magnitude of the gradient of the first pixel | G x |+| G y |With tuning parameters αFor example, the above operation can be implemented by Clip() function. In equation 3, in one example, the maximum value max and minimum value min is a predetermined value. For example, the maximum max can be determined as "1", and the minimum value min Can be determined as "0".

[0097] According to one example, the LUT intensity map of the input image may be determined based on a value obtained by applying smoothing to the magnitude of the gradient and then multiplying the smoothed value by the tuning parameter. As an example, the LUT intensity map may be determined, for example, as in the following Equation 4.

[0098] Equation 4: In equation 4, k represents the filter used for smoothing and can be compared with the corresponding G x and G y For example, when G x and G y When the size is 3×3, k It can correspond to 3×3 filters and is included in k The values ​​of the elements in can be used to obtain the derivative component of the image with respect to the x-axis G x and the derivative component with respect to the y-axis G y The inverse of the size of the mask (e.g., Sobel mask) corresponds to (e.g., 1 / 9). It can correspond to the convolution operation.

[0099] As the value of the tuning parameter increases, the LUT intensity map may have more elements with a maximum value or a value close to the maximum value. As an example, pixels corresponding to an edge in an image may have a larger gradient magnitude than pixels not corresponding to an edge. When the value of the tuning parameter is set to a value greater than "1", the LUT intensity value of a pixel corresponding to an edge portion having a relatively large gradient magnitude may be a maximum value or a value close to the maximum value.

[0100] The method for super-resolution according to the example may include operation 140: acquiring (e.g., generating) an output image as a super-resolution image of the input image based on the LUT intensity map and the first intermediate image. As an example, the value of the pixel corresponding to the first pixel in the output image may be determined based on the product of "the LUT intensity value of a predetermined pixel (e.g., the first pixel) in the input image included in the LUT intensity map" and "the value of the pixel corresponding to the first pixel in the first intermediate image". The value of the pixel corresponding to the first pixel in the output image may correspond to the super-resolution result of the first pixel. As an example, the output image may be acquired based on pixel-by-pixel multiplication between the LUT intensity map and the first intermediate image. When the LUT intensity map has a resolution corresponding to the resolution of the input image (the resolution of the input image is lower than the resolution of the first intermediate image), the LUT intensity map enlarged to the resolution of the first intermediate image may be used. The enlarged LUT intensity map may include the same LUT intensity value for the same pixel.

[0101] The extent to which the first intermediate image is applied to the output image may be determined for each pixel by the LUT intensity map. According to the LUT intensity map, since pixels corresponding to edges have larger LUT intensity values ​​than pixels not corresponding to edges, the values ​​of the first intermediate image may be applied to the output image more in areas corresponding to edges than in areas not corresponding to edges. By applying the values ​​of the first intermediate image more to areas corresponding to edges, the methods and apparatuses of one or more embodiments may generate an output image with increased sharpness in areas corresponding to edges. By applying the values ​​of the first intermediate image less to areas not corresponding to edges (e.g., flat areas with relatively small gradient magnitudes), the methods and apparatuses of one or more embodiments may generate an output image with reduced artifacts in areas not corresponding to edges.

[0102] Operation 140 of acquiring an output image according to an example may include operations of acquiring a second intermediate image based on the LUT intensity map and the first intermediate image, and acquiring an output image based on the second intermediate image and a baseline image acquired by applying interpolation for super-resolution to the input image.

[0103] As an example, the second intermediate image may be obtained based on a pixel-by-pixel multiplication of the LUT intensity map and the first intermediate image.

[0104] The baseline image may correspond to a super-resolution image of an input image acquired by interpolation for super-resolution. Interpolation for super-resolution may be a method of estimating new pixel values ​​included in a super-resolution image using a function derived from positions and values ​​of pixels in the input image, and may include, for example, linear interpolation, bilinear interpolation, nearest neighbor interpolation, bicubic interpolation, inverse distance weighted (IDW) interpolation, and / or kriging interpolation. As an example, the baseline image may correspond to a super-resolution image acquired by enlarging the input image by a preset factor.

[0105] According to one example, an output image may be obtained based on a pixel-by-pixel sum of pixel values ​​of a baseline image and a second intermediate image. By adding the second intermediate image to the baseline image obtained by interpolation, the method and apparatus of one or more embodiments may generate a super-resolution image with improved quality and reduced artifacts.

[0106] Figure 4 An example of a framework showing a method for super-resolution.

[0107] Reference Figure 4 , a method for super-resolution according to an example may include an interpolation process 410 for acquiring a baseline image by applying interpolation for super-resolution to an input image 401 .

[0108] The method for super-resolution according to an example may include a gradient calculation process 440. The gradient calculation process 440 may include an operation of calculating an angle of a gradient of each pixel included in the input image 401. The angle of the gradient may be calculated as in Equation 2. As an example, the gradient calculation process 440 may include an operation of calculating a magnitude of a gradient of the input image 401.

[0109] The method for super-resolution according to an example may include a residual LUT selection process 450. The residual LUT selection process 450 may include an operation of selecting a residual LUT corresponding to a direction type corresponding to each pixel in the input image. The direction type corresponding to each pixel in the input image may be determined based on the gradient calculated in the gradient calculation process 440. As described above, the direction type may be mapped to a kernel set. In the residual LUT selection process 450, a residual LUT corresponding to each kernel in the kernel set mapped to the direction type may be selected.

[0110] The method for super-resolution according to an example may include a residual pixel calculation process 420. The residual pixel calculation process 420 may include an operation of updating the value of a pixel in the receptive field using a value obtained by subtracting a value of a reference pixel from a value of a pixel in the receptive field. As described above, the receptive field may be determined by a kernel included in a kernel set mapped to a direction type corresponding to the reference pixel.

[0111] The method for super-resolution according to the example may include a residual LUT process 430. As described above, the residual LUT process 430 may include obtaining a super-resolution operation result corresponding to each pixel in the input image from the residual LUT, and the super-resolution operation result of the pixel value combination corresponding to the kernel of the input image is stored in the residual LUT. The residual LUT used to obtain the super-resolution operation result may be a residual LUT selected by the residual LUT selection process 450. A combination of the remaining pixel values ​​excluding the reference pixel in the receptive field updated by the residual pixel calculation process 420 may be obtained. For the combination of the remaining pixel values ​​obtained by the residual LUT process 430, a super-resolution operation result of the reference pixel stored in the residual LUT may be obtained.

[0112] When a plurality of kernels are included in the kernel set, the residual LUT processing 430 according to the example may include an operation of calculating a simple sum or a weighted sum of a plurality of super-resolution results obtained from a plurality of residual LUTs respectively corresponding to the plurality of kernels. For example, for a first pixel in the input image detected as a corner, a weighted sum of a plurality of super-resolution operation results respectively corresponding to the plurality of kernels may be obtained, and for a second pixel in the input image not detected as a corner, a simple sum of a plurality of super-resolution operation results respectively corresponding to the plurality of kernels may be obtained.

[0113] The super-resolution image acquired through the residual LUT process 430 may be referred to as a first intermediate image.

[0114] According to an example, the output image 402 may be obtained based on the first intermediate image obtained by the residual LUT process 430 and the baseline image obtained by the interpolation process 410. As an example, the output image 402 may be obtained by adding the baseline image to the first intermediate image pixel by pixel.

[0115] Figure 5 An example of a framework showing a method for super-resolution.

[0116] Reference Figure 5 , the method for super-resolution may include a LUT intensity map calculation process 560. The LUT intensity map calculation process 560 may be the same as that described above with reference to Figure 1The LUT intensity map calculation process 560 may include an operation of determining a LUT intensity map including a LUT intensity value of each pixel in the input image 501 based on the gradient of the input image 501 and the preset tuning parameter 502. As an example, the tuning parameter 502 may be determined as a value input by a user or a predetermined value.

[0117] According to an example, the output image 503 may be acquired based on the LUT intensity map obtained through the LUT intensity map calculation process 560 , the first intermediate image acquired through the residual LUT process 430 , and the baseline image acquired through the interpolation process 410 .

[0118] As an example, the output image 503 may be obtained by multiplying the “image generated by adding the baseline image to the first intermediate image pixel by pixel” by the LUT intensity map. The multiplication of the image and the LUT intensity map may be the multiplication of the value of a pixel in the image and the LUT intensity value corresponding to the pixel, or the pixel-by-pixel multiplication of the image and the LUT intensity map.

[0119] As an example, the output image 503 may be obtained by adding the baseline image to a “second intermediate image generated by multiplying the first intermediate image by the LUT intensity map.” The sum of the second intermediate image and the baseline image may be the sum of the value of a pixel in the second intermediate image and the value of a pixel corresponding to the pixel in the baseline image, or the pixel-by-pixel sum of the second intermediate image and the baseline image.

[0120] Figure 6 An example of a method for super-resolution is shown. Figure 6 Operations 610 to 630 of the embodiment may be performed in the order and manner shown. However, without departing from the spirit and scope of the illustrated examples, the order of one or more of the operations may be changed and / or one or more of the operations may be performed in parallel or simultaneously.

[0121] Reference Figure 6 , the method for super-resolution according to the example may include operation 610: determining the direction type of the input image based on the gradient of the input image. Operation 610 may be the same as described above. Figure 1 The operation 110 corresponds to the above.

[0122] The method for super-resolution according to an example may include operation 620 of obtaining a plurality of super-resolution operation results respectively corresponding to a plurality of kernels based on a plurality of residual LUTs respectively corresponding to a plurality of kernels included in a kernel set of a direction type mapped to a determined input image.

[0123] According to one example, operation 620 of obtaining a super-resolution operation result may include operations such as determining a receptive field corresponding to a reference pixel in an input image based on a kernel included in a kernel set; updating the value of a pixel in the receptive field using a value obtained by subtracting the value of the reference pixel from the value of the pixel in the receptive field; and obtaining a value stored in a residual LUT corresponding to the kernel for a combination of updated values ​​of pixels other than the reference pixel in the receptive field.

[0124] The method for super-resolution according to an example may include operation 630 of acquiring an output image as a super-resolution image of the input image based on weighted summing of a plurality of first intermediate images respectively corresponding to a plurality of kernels by a plurality of weights determined respectively for the plurality of kernels.

[0125] The weight according to the example may be determined for each kernel. As an example, a first weight determined for a first kernel may be determined based on a relative positional relationship between a reference pixel and other pixels in a receptive field indicated by a first kernel among a plurality of kernels included in a kernel set. When a relative positional relationship between a reference pixel and other pixels in a receptive field indicated by a second kernel among a plurality of kernels is different from a relative positional relationship indicated by the first kernel, a second weight determined for the second kernel may be different from the first weight.

[0126] For example, Figure 7 An example of determining weights is shown. Figure 7 , when the kernel set 710 corresponding to the reference pixel in the input image is determined as a kernel set mapped to the backslash type, super-resolution operation results 720 (e.g., output 1 to output 4) corresponding to the four kernels 711, 712, 713, and 714 included in the kernel set 710, respectively, may be obtained. The super-resolution operation result 720 may include a super-resolution operation result corresponding to kernel E2 711, a super-resolution operation result corresponding to kernel E1 712, a super-resolution operation result corresponding to kernel G2 713, and a super-resolution operation result corresponding to kernel G1 714. The super-resolution operation result corresponding to each kernel may be a super-resolution operation result of a reference pixel stored in a residual LUT of each kernel, and may correspond to an area of ​​the super-resolution image corresponding to the reference pixel in the input image. As an example, when the magnification factor is 3×3, the super-resolution operation result of the reference pixel may include nine pixel values.

[0127] Referring to the weight determination process 730 according to the example, a convolution operation between the input image and the first gradient kernel 731, a convolution operation between the input image and the second gradient kernel 732, a convolution operation between the input image and the third gradient kernel 733, and a convolution operation between the input image and the fourth gradient kernel 734 may be performed to determine weights corresponding to the kernels 711, 712, 713, and 714 included in the kernel set 710, respectively. The convolution operation between the input image and the gradient kernels 731, 732, 733, and 734 may be performed based on a reference pixel 735 of the input image. A result of the convolution operation between the input image and the first gradient kernel 731 may be indicated as H1, a result of the convolution operation between the input image and the second gradient kernel 732 may be indicated as H2, a result of the convolution operation between the input image and the third gradient kernel 733 may be indicated as V1, and a result of the convolution operation between the input image and the fourth gradient kernel 734 may be indicated as V2.

[0128] According to one example, the top left weight w may be determined based on the result of the convolution operation between the input image and the gradient kernels 731, 732, 733, and 734. Up-Left , upper right weight w Up-Right , lower left weight w Down-Left and the lower right weight w Down-Right The diagonal weight is 736.

[0129] As an example, the upper left weight w may be determined as expressed by the following equations 5 to 8, respectively: Up-Left , upper right weight w Up-Right , lower left weight w Down-Left and the lower right weight w Down-Right .

[0130] Equation 5: Equation 6: Equation 7: Equation 8: According to one example, weights corresponding to the four kernels 711, 712, 713, and 714 included in the kernel set 710 may be determined. The weights corresponding to the kernels 711, 712, 713, and 714 may be determined based on the positional relationship between the reference pixel in the receptive field and the remaining (one or more) pixels. The positional relationship between the reference pixel in the receptive field and the remaining (one or more) pixels may be determined as the direction in which the remaining (one or more) pixels are located relative to the reference pixel (e.g., the right (rightward) direction, the left (leftward) direction, the upper (upward) direction, or the lower (downward) direction). When the value of the super-resolution operation result obtained from the residual LUT corresponding to the kernel decreases (that is, when the value difference between the reference pixel in the receptive field and the remaining (one or more) pixels decreases), the weight determined based on the positional relationship between the reference pixel in the receptive field indicated by the kernel and the remaining (one or more) pixels may be determined to be a higher value.

[0131] As an example, since in kernel G1 714, the remaining pixels in the receptive field are located in the lower direction and the right direction relative to the reference pixel, the relative position relationship between the reference pixel and the other pixels in the receptive field indicated by kernel G1 714 may correspond to the lower right direction. Down-Right To determine the weight w corresponding to the kernel G1 714 in the lower right direction G1 .

[0132] As an example, since in kernel G2 713, the remaining pixels in the receptive field are located in the upper direction and the left direction relative to the reference pixel, the relative position relationship between the reference pixel and the other pixels in the receptive field indicated by kernel G2 713 may correspond to the upper left direction. Up-Left To determine the weight w corresponding to the kernel G2 713 in the upper left direction G2 .

[0133] As an example, since in kernel E1 712, the remaining pixels in the receptive field are located in the lower direction and the left direction relative to the reference pixel, the relative position relationship between the reference pixel and the other pixels in the receptive field indicated by kernel E1 712 may correspond to the lower left direction. Down-Left To determine the weight w corresponding to the kernel E1 712 in the lower left direction E1 .

[0134] As an example, since in kernel E2 711, the remaining pixels in the receptive field are located in the upper direction and the right direction relative to the reference pixel, the relative position relationship between the reference pixel and the other pixels in the receptive field indicated by kernel E2 711 may correspond to the upper right direction. Up-Right To determine the weight w corresponding to the kernel E2 711 in the upper right direction E2 .

[0135] As an example, the weight w corresponding to the kernel G1 714 may be determined based on the function f(x) represented by the following equations 9 to 12, respectively. G1 , weight w corresponding to kernel G2 713 G2 , the weight w corresponding to kernel E1 712 E1 and the weight w corresponding to kernel E2 711 E2 As an example, the function may be the function "f(x)=1 / x". As an example, the function may be the function "f(x)=clip(-x+1,0,1)".

[0136] Equation 9: Equation 10: Equation 11: Equation 12: According to one example, weighted summation of the super-resolution operation result 720 based on weights respectively corresponding to four kernels 711 , 712 , 713 , and 714 included in the kernel set 710 may be defined, for example, as represented by the following Equation 13.

[0137] Equation 13: and Figure 7 Different from the example of, when the direction type of the reference pixel in the input image is a diagonal line type, it can be determined that the direction type used to include Figure 2 The fourth kernel set 240 has the same kernel configuration as the kernel set 230 mapped to the backslash type. The kernel weights in the fourth kernel set 240 are shown in FIG. ... Figure 7 The upper left weight w is shown in Up-Left , upper right weight w Up-Right , lower left weight w Down-Left and the lower right weight wDown-Right The weight of each kernel included in the fourth kernel set 240 is determined by using one of the diagonal weights 736. As an example, the weight of the kernel included in the fourth kernel set 240 may be determined as the upper left weight w Up-Left , upper right weight w Up-Right , lower left weight w Down-Left or the lower right weight w Down-Right The reciprocal of .

[0138] and Figure 7 Different from the example of, when the direction type of the reference pixel in the input image is a horizontal type, it can be determined that the direction type used to include Figure 2 The positional relationship between the reference pixel in the receptive field indicated by each kernel included in the first kernel set 210 and the remaining (one or more) pixels may correspond to one of the left direction, the right direction, the downward direction, and the upward direction. When the positional relationship between the reference pixel in the receptive field indicated by the kernel included in the first kernel set 210 and the remaining (one or more) pixels is the left direction, w may be determined. Left = |H2|, and can be based on f(w Left ) to determine the weight of the corresponding kernel. When the positional relationship between the reference pixel in the receptive field indicated by the kernel included in the first kernel set 210 and the remaining (one or more) pixels is in the right direction, w may be determined. Right = |H1|, and can be based on f(w Right ) to determine the weight of the corresponding kernel. When the positional relationship between the reference pixel in the receptive field indicated by the kernel included in the first kernel set 210 and the remaining (one or more) pixels is in the downward direction, w may be determined. Down =|V1|, and can be based on f(w Down ) to determine the weight of the corresponding kernel. When the positional relationship between the reference pixel in the receptive field indicated by the kernel included in the first kernel set 210 and the remaining (one or more) pixels is in the upward direction, w may be determined. Up =|V2|, and can be based on f(w Up ) to determine the weights of the corresponding kernels.

[0139] and Figure 7 Different from the example of, when the direction type of the reference pixel in the input image is a vertical type, it can be determined that the direction type used to include Figure 22. The weights of the kernels in the second kernel set 220 shown in FIG. 2. Similar to the first kernel set 210, the positional relationship between the reference pixel in the receptive field indicated by each kernel included in the second kernel set 220 and the remaining (one or more) pixels may correspond to one of the left direction, the right direction, the downward direction, and the upward direction. Based on w Left 、w Right 、w Down and w Up The weight of each kernel included in the second kernel set 220 is determined by one of the kernels.

[0140] According to the example, operation 630 of obtaining an output image may include the following operations: obtaining a first intermediate image as a super-resolution image corresponding to an input image by weighted summing a plurality of super-resolution operation results corresponding to a plurality of kernels respectively based on a plurality of weights determined for the plurality of kernels respectively; and obtaining an output image based on the first intermediate image and a baseline image obtained by applying interpolation for super-resolution to the input image.

[0141] The operation 630 of acquiring the output image according to the example may include an operation of updating the super-resolution operation result corresponding to the first kernel by overlapping a plurality of blocks corresponding to different regions of the super-resolution operation result corresponding to the first kernel among the plurality of kernels. Each of the plurality of blocks of the super-resolution operation result corresponding to the first kernel may be a set of super-resolution results of each pixel in the input image stored in the residual LUT corresponding to the first kernel. The pixel value of the region where the plurality of blocks are overlapped in the updated super-resolution operation result corresponding to the first kernel may be determined as the average value of the pixel values ​​of the overlapped blocks.

[0142] Figure 8 An example of a framework showing a method for super-resolution.

[0143] Reference Figure 8 , the method for super-resolution may include a weighted summation process 810. As an example, the weighted summation process 810 may be the same as that described above with reference to Figure 6 The weighted summation process 810 may include the following operations: based on the multiple weights determined for the multiple kernels in the kernel set corresponding to the reference pixel, a weighted sum of the multiple super-resolution operation results corresponding to the multiple kernels is calculated. A first intermediate image may be generated as a result of the weighted summation. As an example, the weighted summation process may be performed by referring to the above reference pixel. Figure 7 The depicted weight determination process 730 determines the weights of the various kernels used in the weighted summation.

[0144] According to one example, the output image 802 may be acquired based on the first intermediate image acquired through the weighted sum process 810 and the baseline image acquired through the interpolation process 410. For example, the output image 802 may be a super-resolution image of the input image 801. As an example, the output image 802 may be acquired by adding the baseline image to the first intermediate image pixel by pixel.

[0145] The method and apparatus of one or more embodiments may generate an output image 802 with reduced artifacts using a first intermediate image generated by applying weights to a plurality of super-resolution operation results obtained for a plurality of kernels.

[0146] Fig. 9 An example of a framework showing a method for super-resolution.

[0147] Reference Fig. 9 , an output image 902 according to the example may be acquired based on the LUT intensity map obtained through the LUT intensity map calculation process 560, the first intermediate image acquired through the weighted summation process 810, and the baseline image acquired through the interpolation process 410. For example, the output image 902 may be a super-resolution image of the input image 901.

[0148] As an example, the output image 902 may be obtained by multiplying the “image generated by adding the baseline image to the first intermediate image pixel by pixel” by the LUT intensity map. The multiplication of the image and the LUT intensity map may be the multiplication of the value of a pixel in the image and the LUT intensity value corresponding to the pixel, or the pixel-by-pixel multiplication of the image and the LUT intensity map.

[0149] As an example, the output image 902 may be obtained by adding the baseline image to a “second intermediate image generated by multiplying the first intermediate image by the LUT intensity map.” The sum of the second intermediate image and the baseline image may be the sum of “the value of a pixel in the second intermediate image” and “the value of a pixel corresponding to the pixel in the baseline image,” or the pixel-by-pixel sum of the second intermediate image and the baseline image.

[0150] Fig.10 An example of a method for super-resolution based on the average value of overlapping areas is shown.

[0151] Reference Fig.10 According to an example, a method for super-resolution based on an average value of an overlapping area may include such an operation: performing super-resolution processing (abbreviated as "SR") 1010 on an input image 1001 by using an enlargement factor greater than a target resolution to obtain an intermediate super-resolution image 1002. The super-resolution processing 1010 may include the residual LUT processing described above or the above reference Figures 1 to 9The method for super-resolution described above. The target resolution may correspond to a desired resolution of an output image ultimately obtained by performing super-resolution on the input image 1001. The intermediate super-resolution image 1002 obtained by the super-resolution process 1010 may correspond to an image having a resolution higher than the target resolution. For example, when the target resolution is 3×3, the resolution of the intermediate super-resolution image 1002 may be 5×5.

[0152] The method for super-resolution based on an average value of overlapping areas according to an example may include an operation of acquiring a super-resolution image 1003 of a target resolution by overlapping blocks 1021 , 1022 , 1023 , and 1024 of the acquired intermediate super-resolution image 1002 .

[0153] Blocks 1021, 1022, 1023, and 1024 may be sets of super-resolution results of corresponding pixels in the input image. As an example, the first block 1021 may be a set of super-resolution results of the first pixel 1011 in the input image 1001, and may include 5×5 pixels obtained by super-resolution of the first pixel 1011. As an example, the second block 1022 may be a set of super-resolution results of the second pixel 1012 in the input image 1001, and may include 5×5 pixels obtained by super-resolution of the second pixel 1012. As an example, the third block 1023 may be a set of super-resolution results of the third pixel 1013 in the input image 1001, and may include 5×5 pixels obtained by super-resolution of the third pixel 1013. As an example, the fourth block 1024 may be a set of super-resolution results of the fourth pixel 1014 in the input image 1001, and may include 5×5 pixels obtained by super-resolution of the fourth pixel 1014.

[0154] The super-resolution image 1003 may correspond to an image of a target resolution obtained by overlapping blocks 1021, 1022, 1023, and 1024 of the intermediate super-resolution image 1002. The super-resolution image 1003 may include super-resolution results corresponding to pixels 1011, 1012, 1013, and 1014 in the input image 1001. As an example, the super-resolution image 1003 may include 3×3 pixels corresponding to each of the pixels 1011, 1012, 1013, and 1014 obtained by super-resolving the input image 1001.

[0155] According to an example, a super-resolution image 1003 having a lower resolution than the intermediate super-resolution image 1002 may be obtained by overlapping blocks of the intermediate super-resolution image 1002. The pixel value of the area where the blocks are overlapped may be determined as the average value of the pixel values ​​of the overlapped blocks. The pixel value of the area where the partial areas of two or more blocks are overlapped in the super-resolution image 1003 may be determined as the average value of the pixel values ​​of the overlapping areas of the overlapped blocks. For example, the pixel value of one pixel in the area where the partial areas of two or more blocks are overlapped in the super-resolution image 1003 may be determined as the average value of the pixel values ​​of the corresponding pixels in the partial areas of the two or more blocks. As an example, a partial area 1031 of the first block 1021 corresponding to the first pixel 1011 and a partial area 1032 of the second block 1022 corresponding to the second pixel 1012 adjacent to the first pixel 1011 may be overlapped to generate a corresponding area of ​​the super-resolution image 1003. The pixel value of the area where the partial area 1031 of the first block 1021 and the partial area 1032 of the second block 1022 overlap in the super-resolution image 1003 can be determined as the average value of the pixel value of the partial area 1031 of the first block 1021 and the pixel value of the partial area 1032 of the second block 1022.

[0156] Hereinafter, the operation of acquiring the super-resolution image 1003 by overlapping a plurality of blocks of the intermediate super-resolution image 1002 may be referred to as an overlapped region average calculation process. The overlapped region average calculation process may be performed on the super-resolution image acquired for the input image.

[0157] As an example, Fig.11 An example of a framework for a method for super-resolution including an overlap region average calculation process is shown. Fig.11 , the overlap area average calculation process 1110 may be performed on the first intermediate image obtained by the residual LUT process 430. The first intermediate image may correspond to an image having a resolution higher than the target resolution. The image corresponding to the target resolution may be obtained by performing the overlap area average calculation process 1110 on the first intermediate image. As described above, the first intermediate image may include an image generated by summing or weighted summing a plurality of super-resolution results corresponding to a plurality of kernels.

[0158] As an example, the overlap area average calculation process 1110 may be performed on an image generated based on summing the first intermediate image acquired through the residual LUT process 430 and the baseline image acquired through the interpolation process 410. The first intermediate image acquired through the residual LUT process 430 and the baseline image acquired through the interpolation process 410 may correspond to an image having a resolution higher than a target resolution. By performing the overlap area average calculation process 1110 on the image generated based on summing the first intermediate image acquired through the residual LUT process 430 and the baseline image acquired through the interpolation process 410, an image corresponding to the target resolution may be acquired.

[0159] As an example, the overlap area average calculation process 1110 may be performed on the second intermediate image generated based on multiplying the first intermediate image acquired through the residual LUT process 430 and the LUT intensity map obtained through the LUT intensity map calculation process 560 .

[0160] As an example, the overlap area average calculation process 1110 may be performed on an image generated based on summing the second intermediate image and the baseline image acquired through the interpolation process 410 .

[0161] As an example, the output image 1102 may correspond to the image acquired through the overlap region average calculation process 1110. As an example, the output image 1102 may be acquired based on the image acquired through the overlap region average calculation process 1110 and the baseline image acquired through the interpolation process 410. As an example, the output image 1102 may be acquired based on the image acquired through the overlap region average calculation process 1110 and the LUT intensity map acquired through the LUT intensity map calculation process 560. As an example, the output image 1102 may be acquired based on the image acquired through the overlap region average calculation process 1110, the baseline image acquired through the interpolation process 410, and the LUT intensity map acquired through the LUT intensity map calculation process 560.

[0162] although Fig.11 Although not shown in the figure, at least one of the residual pixel calculation process 420, the gradient calculation process 440, the residual LUT selection process 450, and the LUT intensity map calculation process 560 described above may be further performed.

[0163] As an example, Fig.12 An example of a framework for a method for super-resolution including an overlap region average calculation process is shown. Fig.12, an overlapped area average calculation process 1210 may be performed on a super-resolution operation result obtained from a residual LUT corresponding to each kernel included in the kernel set (e.g., LUT 1, LUT 2, ..., LUT N). The super-resolution operation result obtained from the residual LUT corresponding to each kernel may correspond to an image having a resolution higher than the target resolution. A first intermediate image of the target resolution may be acquired based on a sum or a weighted sum of LUT outputs 1221 (e.g., LUT output 1), LUT outputs 1222 (e.g., LUT output 2), and LUT outputs 1223 (e.g., LUT output N) obtained by performing the overlapped area average calculation process 1110 on the super-resolution operation result corresponding to each kernel.

[0164] Fig.13 An example of a method for super-resolution that is repeatedly performed is shown.

[0165] Reference Fig.13 The method for super-resolution according to the example may be repeated multiple times. The method for super-resolution may be the same as that described above. Figures 1 to 12 The method for super-resolution described above for obtaining an output image corresponding to a target resolution of an input image corresponds to the method described above. The residual LUT-based super-resolution model 1310 may be used to perform the above reference Figures 1 to 12 The model of operation of the super-resolution method is described. Fig.13 The weighted summation process 810 is not shown in the figure, but the residual LUT-based super-resolution model 1310 may also include the weighted summation process 810 for acquiring the first intermediate image.

[0166] As an example, the image 1302 input into the residual LUT-based super-resolution model 1310 in the first iteration (i=1) may correspond to the input image 1301. The (i+1)th image 1303 output from the residual LUT-based super-resolution model 1310 in the first iteration (i=1) may be input into the residual LUT-based super-resolution model 1310 in the second iteration (i=2). For example, for the i-th image 1302 in the i-th iteration, the (i+1)th image 1303 output from the residual LUT-based super-resolution model 1310 may correspond to the input of the residual LUT-based super-resolution model 1310 in the (i+1)-th iteration. When n iterations (n ​​is a natural number greater than or equal to “1”) are set, the image 1303 output from the residual LUT-based super-resolution model 1310 for the nth input image may correspond to the output image 1304 output as the final super-resolution operation result for the input image 1301.

[0167] According to one example, the number of iterations n of the method for super-resolution may be determined according to the upscaling factor of the residual LUT-based super-resolution model 1310 and the target resolution. For example, when the upscaling factor of the residual LUT-based super-resolution model 1310 is r 1 × 2 And the target resolution is r 1 k × 2 k When , the number of iterations can be determined as k.

[0168] According to one example, the residual LUT-based super-resolution model 1310 may include multiple models with different magnification factors. When the residual LUT-based super-resolution model 1310 includes a first model and a second model with different magnification factors, the input image may be input into the first model or the second model in each iteration.

[0169] FIG. 14A to FIG. 14C An example of a method for super-resolution for controlling artifacts is shown.

[0170] Reference Fig.14A , a method for super-resolution for artifact control of one or more embodiments may include the following operations: performing super-resolution based on a residual LUT-based model 1410 for super-resolution; and performing artifact reduction based on a residual LUT-based model 1420 for artifact reduction.

[0171] According to one example, the operation of performing super-resolution can be performed as described above. Figures 1 to 12 The super-resolution method described above for obtaining an output image corresponding to a target resolution of an input image corresponds to the residual LUT-based model 1410 for super-resolution can be used to perform the above reference Figures 1 to 12 The model of operation of the method for super-resolution is described. The residual LUT process 1411 for super-resolution may correspond to the residual LUT process 430 described above.

[0172] According to one example, the operation of performing artifact reduction may include a residual LUT process for artifact reduction 1421. Similar to the residual LUT process 430 described above, the residual LUT process for artifact reduction 1421 may include a process of obtaining an artifact reduction operation result corresponding to each pixel in the input image from the residual LUT for artifact reduction, the residual LUT for artifact reduction storing an artifact reduction operation result of a combination of pixel values ​​corresponding to a kernel of the input image.

[0173] The residual LUT processing 1421 for artifact reduction is different from the residual LUT processing 1411 for super-resolution in that the residual LUT processing 1421 for artifact reduction obtains a value stored in a residual LUT of a different type from the residual LUT processing 1411 for super-resolution as an operation result of artifact reduction. The residual LUT for artifact reduction may correspond to a data structure storing the operation result of artifact reduction of a model for artifact reduction. As an example, the model for artifact reduction may include a neural network trained to output an image with reduced artifacts from an input image.

[0174] Similar to the residual LUT processing 430, the residual LUT processing 1421 for artifact reduction may include the following operations: calculating the angle of the gradient of each pixel through the gradient calculation processing 440, and selecting the residual LUT for artifact reduction corresponding to the direction type determined based on the gradient calculated in the gradient calculation processing 440 through the residual LUT selection processing 450.

[0175] According to one example, the operation of performing super-resolution may include a LUT intensity map calculation process 561. According to one example, the operation of performing artifact reduction may include a LUT intensity map calculation process 562. The LUT intensity map calculation process 561 in the operation of performing super-resolution and the LUT intensity map calculation process 562 in the operation of performing artifact reduction may correspond to the LUT intensity map calculation process 560 described above. As an example, the tuning parameters input during the LUT intensity map calculation process 561 in the operation of performing super-resolution and the tuning parameters input during the LUT intensity map calculation process 562 in the operation of performing artifact reduction may be the same as or different from each other.

[0176] According to one example, the operation of performing super-resolution may include a weighted summation process 811. According to one example, the operation of performing artifact reduction may include a weighted summation process 812. The weighted summation process 811 in the operation of performing super-resolution and the weighted summation process 812 in the operation of performing artifact reduction may correspond to the weighted summation process 810 described above. As an example, since the residual LUT for super-resolution in the operation of performing super-resolution and the residual LUT for artifact reduction in the operation of performing artifact reduction are different LUTs, the multiple weights respectively corresponding to the multiple kernels determined by the weighted summation process 811 in the operation of performing super-resolution and the multiple weights respectively corresponding to the multiple kernels determined by the weighted summation process 812 in the operation of performing artifact reduction may be different from each other.

[0177] According to an example, the image input into the residual LUT-based model 1420 for artifact reduction may be an image output from the residual LUT-based model 1410 for super-resolution upon receiving the input image 1401 .

[0178] According to an example, the image output from the residual LUT-based model 1420 for artifact reduction may correspond to the final output image 1402. As an example, the baseline image obtained through the interpolation process 410 may be added to the image output from the residual LUT-based model 1410 for super-resolution, or may be added to the image output from the residual LUT-based model 1420 for artifact reduction.

[0179] Reference Fig. 14B , and reference Fig.14A Different from the description provided, in order to obtain an output image 1402 corresponding to the input image 1401 as a super-resolution image for artifact control, an operation of performing artifact reduction may first be performed based on a residual LUT-based model 1420 for artifact reduction, and an operation of performing super-resolution may be performed based on a residual LUT-based model 1410 for super-resolution.

[0180] According to an example, the image input into the residual LUT-based model 1410 for super-resolution may be an image output from the residual LUT-based model 1420 for artifact reduction upon receiving the input image 1401 .

[0181] According to an example, the image output from the residual LUT-based model 1410 for super-resolution may correspond to the final output image 1402. As an example, the baseline image obtained through the interpolation process 410 may be added to the image output from the residual LUT-based model 1410 for super-resolution, or may be added to the image output from the residual LUT-based model 1420 for artifact reduction.

[0182] Reference Fig. 14C , and reference Fig.14A and Fig. 14B Different from the description provided, in order to obtain an output image 1402 corresponding to the input image 1401 as a super-resolution image for artifact control, an operation of performing super-resolution based on a residual LUT-based model 1410 for super-resolution and an operation of performing artifact reduction based on a residual LUT-based model 1420 for artifact reduction can be performed in parallel.

[0183] According to an example, the input image 1401 may be input to the residual LUT-based model 1410 for super-resolution and the residual LUT-based model 1420 for artifact reduction. The output image 1402 may be acquired based on the sum or weighted sum of the image output from the residual LUT-based model 1410 for super-resolution and the image output from the residual LUT-based model 1420 for artifact reduction. As an example, the baseline image acquired through the interpolation process 410 may be added to an image acquired by adding the image output from the residual LUT-based model 1410 for super-resolution and the image output from the residual LUT-based model 1420 for artifact reduction to acquire the output image 1402.

[0184] FIG. 15A to FIG. 15C An example of a method for super-resolution for corner case handling is shown.

[0185] Reference Fig.15A , a corner detection process 1510 may be performed on an input image 1501. A corner may correspond to an area corresponding to an edge in the image.

[0186] The method for super-resolution for corner case processing according to the example may include the following operations: performing super-resolution based on a residual LUT-based model 1410 for super-resolution; and processing the corner case based on a residual LUT-based model 1520 for the corner case for the corner region detected by the corner detection processing 1510.

[0187] According to one example, the operation of processing the corner case may include a residual LUT processing for the corner case 1521. Similar to the residual LUT processing 430 described above, the residual LUT processing for the corner case 1521 may include a process of obtaining an operation result of the corner case processing corresponding to each pixel in the input image from the residual LUT for the corner case processing, the residual LUT for the corner case processing storing the operation result of the corner case processing combined with the pixel value corresponding to the kernel of the input image.

[0188] The residual LUT processing 1521 for corner case processing is different from the residual LUT processing 1411 for super-resolution in that the residual LUT processing 1521 for corner case processing obtains a value stored in a residual LUT of a different type from the residual LUT processing 1411 for super-resolution as an operation result of the corner case processing. The residual LUT for corner case processing may correspond to a data structure storing the operation result of the corner case processing of the model for corner case processing. As an example, the model for corner case processing may include a neural network trained to output an image with corrected corner cases from an input image.

[0189] Similar to the residual LUT processing 430, the residual LUT processing 1521 for angular case processing may include the following operations: calculating the angle of the gradient of each pixel through the gradient calculation processing 440, and selecting the residual LUT for angular case processing corresponding to the direction type determined based on the gradient calculated in the gradient calculation processing 440 through the residual LUT selection processing 450.

[0190] According to an example, the operation of performing the angular case processing may include a LUT intensity map calculation process 563. The LUT intensity map calculation process 563 in the operation of performing the angular case processing may correspond to the LUT intensity map calculation process 560 described above. As an example, the tuning parameters input during the LUT intensity map calculation process 561 in the operation of performing the super-resolution and the tuning parameters input during the LUT intensity map calculation process 563 in the operation of performing the angular case processing may be the same as or different from each other.

[0191] According to an example, the operation of performing the corner case processing may include a weighted summation process 813. The weighted summation process 813 in the operation of performing the corner case processing may correspond to the weighted summation process 810 described above. As an example, since the residual LUT for super-resolution in the operation of performing the super-resolution and the residual LUT for the corner case processing in the operation of performing the corner case processing are different LUTs, the multiple weights respectively corresponding to the multiple kernels determined by the weighted summation process 811 in the operation of performing the super-resolution and the multiple weights respectively corresponding to the multiple kernels determined by the weighted summation process 813 in the operation of performing the corner case processing may be different from each other.

[0192] According to an example, the image input into the residual LUT based model 1520 for corner case processing may be an image output from the residual LUT based model 1410 for super-resolution upon receiving the input image 1501 .

[0193] According to an example, the image output from the residual LUT-based model 1520 for corner case processing may correspond to the final output image 1502. As an example, the baseline image obtained through the interpolation process 410 may be added to the image output from the residual LUT-based model 1410 for super-resolution, or may be added to the image output from the residual LUT-based model 1520 for corner case processing.

[0194] Reference Fig. 15B , and reference Fig.15ADifferent from the description provided, in order to obtain an output image 1502 corresponding to the input image 1501 as a super-resolution image for corner case processing, the operation of processing the corner case can first be performed based on the residual LUT-based model 1520 for corner case processing, and the operation of performing super-resolution can be performed based on the residual LUT-based model 1410 for super-resolution.

[0195] According to an example, the image input into the residual LUT based model 1410 for super-resolution may be an image output from the residual LUT based model 1520 for corner case processing when the input image 1501 is received.

[0196] According to an example, the image output from the residual LUT-based model 1410 for super-resolution may correspond to the final output image 1502. As an example, the baseline image obtained through the interpolation process 410 may be added to the image output from the residual LUT-based model 1410 for super-resolution, or may be added to the image output from the residual LUT-based model 1520 for corner case processing.

[0197] Reference Fig. 15C , and reference Fig.15A and Fig. 15B Different from the description provided, in order to obtain an output image 1502 corresponding to the input image 1501 as a super-resolution image for corner case processing, operations of performing super-resolution based on the residual LUT-based model 1410 for super-resolution and operations of processing corner cases based on the residual LUT-based model 1520 for corner case processing can be performed in parallel.

[0198] According to an example, an input image 1501 may be input to the residual LUT-based model 1410 for super-resolution and the residual LUT-based model 1520 for corner case processing. An output image 1502 may be acquired based on a sum or a weighted sum of an image output from the residual LUT-based model 1410 for super-resolution and an image output from the residual LUT-based model 1520 for corner case processing. As an example, a baseline image acquired through the interpolation process 410 may be added to an image acquired by adding an image output from the residual LUT-based model 1410 for super-resolution and an image output from the residual LUT-based model 1520 for corner case processing to acquire the output image 1502.

[0199] Fig.16 An example of the configuration of a device is shown.

[0200] Reference Fig.16According to the example, the device 1600 may include a processor 1601 (eg, one or more processors), a memory 1603 (eg, one or more memories), and a communication module 1605. According to the example, the device 1600 may be a device for super-resolution image processing, and may include a device for performing the above reference. Figures 1 to 15C The apparatus 1600 may include various devices for image processing (such as a smart phone, a camera, a medical imaging device, a virtual reality (VR) device, and a semiconductor image measuring device). For example, the apparatus 1600 may be applied to hardware image signal processing (ISP).

[0201] According to the example, the processor 1601 may execute the above reference Figures 1 to 15C For example, the processor 1601 may perform at least one of the following operations: determining a direction type of the input image based on a gradient of the input image; acquiring a first intermediate image as a super-resolution image corresponding to the input image based on a residual LUT corresponding to a kernel set mapped to the determined direction type; determining a LUT intensity map of the input image based on the gradient of the input image and a preset tuning parameter; and acquiring an output image as a super-resolution image of the input image based on the LUT intensity map and the first intermediate image.

[0202] For example, the processor 1601 may perform the following operations: determine the directional type of the input image based on the gradient of the input image; obtain multiple super-resolution operation results corresponding to the multiple kernels based on multiple residual LUTs corresponding to the multiple kernels included in the kernel set mapped to the determined directional type of the input image; and obtain an output image as a super-resolution image of the input image based on weighted summation of the multiple super-resolution operation results corresponding to the multiple kernels based on multiple weights determined for the multiple kernels.

[0203] The memory 1603 according to the example may be a volatile memory or a non-volatile memory, and may store the same as described above. Figures 1 to 15C As an example, the memory 1603 may store data generated during the process of executing the method for super-resolution or data required to execute the method for super-resolution. In one example, the memory 1603 may be or include a non-transitory computer-readable storage medium storing instructions that, when executed by the processor 1601, configure the processor 1601 to execute the instructions described herein. Figures 1 to 16 Any one, any combination, or all of the operations and methods described.

[0204] The communication module 1605 according to an example may provide the device 1600 with a function of communicating with another electronic device or another server through a network. For example, the device 1600 may be connected to an external device (eg, a user's terminal, a server, or a network) through the communication module 1605 and exchange data with the external device.

[0205] According to one example, the memory 1603 may not be a component of the device 1600, and may be included in an external device accessible to the device 1600. In this case, the device 1600 may receive data stored in the memory included in the external device, and transmit the data to be stored in the memory through the communication module 1605.

[0206] According to an example, the memory 1603 may store a program configured to implement the above reference Figures 1 to 15C The processor 1601 may execute the program stored in the memory 1603 and control the device 1600. The code of the program executed by the processor 1601 may be stored in the memory 1603.

[0207] The device 1600 according to the example may also include other components not shown in the drawings. For example, the device 1600 may also include an input / output interface including an input device and an output device as a device for interfacing with the communication module 1605. In addition, for example, the device 1600 may also include other components (such as a transceiver, various sensors, and a database).

[0208] about Figures 1 to 16The devices, processors, memories, communication modules, devices 1600, processors 1601, memories 1603, communication modules 1605, and other devices, devices, units, modules, and components disclosed and described herein are implemented by hardware components or represent hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as, operating system (OS) and one or more software applications running on OS) to perform the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or multiple computers can be used, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components. As described above, or in addition to the description above, the example hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0209] The operations described in this application are performed in Figures 1 to 16 The method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented to execute instructions or software as described above to perform the operations performed by the method described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0210] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly run by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0211] The processor or computer used to implement the hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, and are therefore not signals themselves. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include any one or more of the following: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type memory (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner, and provide the instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0212] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered to be merely descriptive and not for limiting purposes. The description of features or aspects in each example is to be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0213] Therefore, in addition to the disclosure above and throughout the accompanying drawings, the scope of the disclosure also includes the claims and their equivalents (that is, all changes within the scope of the claims and their equivalents should be construed as being included in the disclosure).

Claims

1. A method for super-resolution, the method comprising: Determining a direction type of the input image based on a gradient of the input image; Acquire a first intermediate image based on a residual lookup table corresponding to a kernel set mapped to the determined direction type, the first intermediate image being a super-resolution image corresponding to the input image; determining a lookup table intensity map of the input image based on a gradient of the input image and a preset tuning parameter; as well as An output image is generated based on the lookup table intensity map and the first intermediate image, the output image being a super-resolution image of the input image.

2. The method according to claim 1, wherein: The step of acquiring a first intermediate image comprises: determining a receptive field corresponding to a reference pixel in the input image based on the kernels included in the kernel set; updating the value of the pixel in the receptive field using a value obtained by subtracting the value of the reference pixel from the value of the pixel in the receptive field; and A value stored in a residual lookup table corresponding to the kernel is obtained for a combination of updated values ​​of pixels other than the reference pixel in the receptive field to obtain a first intermediate image.

3. The method according to claim 1, wherein: The steps to generate the output image are: acquiring a second intermediate image based on the lookup table intensity map and the first intermediate image; and An output image is generated based on the second intermediate image and a baseline image obtained by applying interpolation for super-resolution to the input image.

4. The method according to claim 1, wherein: The lookup table intensity map for the input image includes a lookup table intensity value for each pixel in the input image determined based on a product of a gradient for each pixel in the input image and a tuning parameter.

5. The method according to claim 4, wherein: In response to a product of a magnitude of the gradient of the first pixel and the tuning parameter being greater than a predetermined maximum value, a lookup table intensity value for the first pixel included in the input image is determined to be a predetermined maximum value, In response to the product of the magnitude of the gradient of the first pixel and the tuning parameter being less than a predetermined minimum value, the lookup table intensity value of the first pixel included in the input image is determined to be a predetermined minimum value, and In response to the product of the magnitude of the gradient of the first pixel and the tuning parameter being less than or equal to a predetermined maximum value and greater than or equal to a predetermined minimum value, a lookup table intensity value for the first pixel included in the input image is determined as the product of the magnitude of the gradient of the first pixel and the tuning parameter.

6. The method according to claim 1, wherein A kernel set includes multiple kernels, and The step of acquiring a first intermediate image comprises: A first intermediate image is acquired by weighted summing a plurality of super-resolution operation results respectively corresponding to the plurality of kernels based on a plurality of weights determined respectively for the plurality of kernels.

7. The method according to claim 1, wherein The step of acquiring a first intermediate image comprises: updating the first intermediate image using an image obtained by overlapping a plurality of blocks corresponding to different areas of the first intermediate image, Each of the plurality of blocks of the first intermediate image is a group of pixels corresponding to a super-resolution result of a corresponding pixel in the input image, and The pixel value of the region where the plurality of blocks overlap in the updated first intermediate image is determined as an average value of the pixel values ​​of the overlapped blocks.

8. The method according to claim 1, wherein The steps of determining a lookup table intensity map for an input image include: determining a lookup table intensity value corresponding to a pixel in the input image detected as a corner based on the tuning parameters and the gradient of the pixel in the input image detected as a corner, and The step of generating an output image includes: in response to a first pixel in the first intermediate image corresponding to a pixel detected as a corner in the input image, generating the output image based on the first intermediate image and a lookup table intensity value corresponding to the pixel detected as a corner in the input image.

9. The method according to claim 1, further comprising: The output image is set as the input image, and the steps of determining the direction type of the input image, acquiring the first intermediate image, determining the lookup table intensity map, and generating the output image are repeated.

10. A method for super-resolution, the method comprising: Determining a direction type of the input image based on a gradient of the input image; obtaining a plurality of super-resolution operation results respectively corresponding to a plurality of kernels included in a kernel set based on a plurality of residual lookup tables respectively corresponding to the plurality of kernels, the kernel set being mapped to a determined direction type of the input image; as well as generating an output image by weighted summing the plurality of super-resolution operation results respectively corresponding to the plurality of kernels based on a plurality of weights determined respectively for the plurality of kernels, the output image being a super-resolution image of the input image; The plurality of residual lookup tables include super-resolution operation results of combinations of pixel values, the number of which is less than "1" than the size of the corresponding kernel.

11. The method according to claim 10, wherein: The steps to generate the output image are: acquiring a first intermediate image by weighted summing the plurality of super-resolution operation results respectively corresponding to the plurality of kernels based on the plurality of weights determined for the plurality of kernels respectively, the first intermediate image being a super-resolution image corresponding to the input image; and An output image is generated based on the first intermediate image and a baseline image obtained by applying interpolation for super-resolution to the input image.

12. The method according to claim 10, wherein: The step of obtaining the plurality of super-resolution operation results comprises: determining a receptive field corresponding to a reference pixel in the input image based on the kernels included in the kernel set; updating the value of the pixel in the receptive field using a value obtained by subtracting the value of the reference pixel from the value of the pixel in the receptive field; and The values ​​stored in the residual lookup table corresponding to the kernel are obtained for combinations of updated values ​​of pixels other than the reference pixel in the receptive field to obtain the plurality of super-resolution operation results.

13. The method according to claim 10, wherein: The first weight determined for a first kernel among the plurality of kernels is determined based on a relative positional relationship between a reference pixel and another pixel in a receptive field indicated by the first kernel.

14. The method according to claim 10, wherein The steps to generate the output image are: updating the super-resolution operation result corresponding to the first kernel among the plurality of kernels by overlapping a plurality of blocks corresponding to different regions of the super-resolution operation result corresponding to the first kernel, Each of the plurality of blocks corresponding to the super-resolution operation results of the first kernel is a set of super-resolution results for each pixel in the input image stored in the residual lookup table corresponding to the first kernel, and The pixel value of the region where the plurality of blocks overlap in the updated super-resolution operation result corresponding to the first kernel is determined as an average value of the pixel values ​​of the overlapped blocks.

15. The method according to claim 10, wherein: The steps to generate the output image are: For a first pixel in the input image that is detected as a corner, determining a weighted sum of the plurality of super-resolution operation results respectively corresponding to the plurality of kernels; and For a second pixel in the input image that is not detected as a corner, a simple sum of the plurality of super-resolution operation results respectively corresponding to the plurality of kernels is determined.

16. The method according to claim 10, further comprising: The output image is set as the input image, and the steps of determining the direction type of the input image, obtaining the multiple super-resolution operation results respectively corresponding to the multiple kernels, and generating the output image are repeated.

17. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 16.

18. A device for super-resolution image processing, the device comprising: One or more processors configured to: Determine the direction type of the input image based on the gradient of the input image, A first intermediate image is obtained based on a residual lookup table corresponding to a kernel set mapped to the determined direction type, the first intermediate image being a super-resolution image corresponding to the input image, determining a lookup table intensity map for the input image based on a gradient of the input image and a preset tuning parameter, and An output image is generated based on the lookup table intensity map and the first intermediate image, the output image being a super-resolution image of the input image.

19. The apparatus according to claim 18, wherein A kernel set includes multiple kernels, and In order to obtain the first intermediate image, the one or more processors are further configured to obtain the first intermediate image by weighted summing up multiple super-resolution operation results corresponding to the multiple kernels respectively based on multiple weights determined for the multiple kernels respectively.

20. A device for super-resolution image processing, the device comprising: One or more processors configured to: Determine the direction type of the input image based on the gradient of the input image, obtaining a plurality of super-resolution operation results respectively corresponding to the plurality of kernels included in a kernel set based on a plurality of residual lookup tables respectively corresponding to the plurality of kernels, the kernel set being mapped to the determined direction type of the input image, and generating an output image by weighted summing the plurality of super-resolution operation results respectively corresponding to the plurality of kernels based on a plurality of weights determined respectively for the plurality of kernels, the output image being a super-resolution image of the input image; The plurality of residual lookup tables include super-resolution operation results of combinations of pixel values, the number of which is less than "1" than the size of the corresponding kernel.

Images