Image processing apparatus and image signal processing method

Abstract

To generate an image with higher resolution by using data of an input image.SOLUTION: An image processing device 100 according to an embodiment includes a first image generator 110 configured to generate a first image 710 and a second image 720 by using an input image 700, and search the first image 710 for a first reference region 712 that is most similar to a target region 702 of the input image 700. The image processing apparatus may include a reference area search unit 120 configured to search for a second reference area 722 most similar to the target area 702 in the second image 720, a correction area determination unit 130 configured to determine a target correction area more similar to the target area 702 among a first correction area 704 located in an area corresponding to the first reference area 712 in the input image 700 and a second correction area 706 located in an area corresponding to the second reference area 722 in the input image 700, and a second image generation unit 140 configured to generate a third image 730 using the target correction area.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to processing devices, and more particularly to image processing devices that increase the resolution of an input image to generate a high-resolution image. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical micro cameras.

[0003] When processing an optical image, an operation of increasing the resolution of the optical image captured by the image sensing device may be performed using an external database or by referring to an image other than the image captured by the image sensing device. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical idea of ​​the present invention aims to provide an image processing device that improves the resolution of an input image.

[0005] Another object of the present invention is to provide an image processing device that can generate a higher resolution image using input image data.

[0006] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] An image signal processor according to an embodiment of the present invention disclosed in this document may include a first image generation unit that generates a first image and a second image using an input image; a reference area search unit that searches for a first reference area in the first image that is most similar to a target area of ​​the input image and searches for a second reference area in the second image that is most similar to the target area; a correction area determination unit that determines a target correction area that is more similar to the target area from among a first correction area located in an area corresponding to the first reference area in the input image and a second correction area located in an area corresponding to the second reference area in the input image; and a second image generation unit that generates a third image using the target correction area.

[0008] An image signal processor according to one embodiment of the present invention disclosed in this document may include a first image generation unit that generates a low-resolution image using an input image, a reference area search unit that searches for a reference area in the low-resolution image that is similar to a target area of ​​the input image, a similarity determination unit that determines the similarity between a correction area located at coordinates obtained by multiplying the low-resolution image coordinates of the reference area among the coordinates of the input image by a scaling coefficient and the target area, and a second image generation unit that generates a high-resolution image using the correction area based on the similarity between the correction area and the target area.

[0009] An image signal processing method according to one embodiment of the present invention disclosed in this document may include generating a first image and a second image using an input image, searching for a first reference area in the first image that is most similar to a target area of ​​the input image, searching for a second reference area in the second image that is most similar to the target area, determining a target correction area that is more similar to the target area from among a first correction area located in an area corresponding to the first reference area of ​​the input image and a second correction area located in an area corresponding to the second reference area of ​​the input image, and generating a third image using the target correction area. [Effects of the Invention]

[0010] According to the embodiments disclosed herein, a high-resolution image can be generated from a single input image and a low-resolution image generated from it, and a high-quality high-resolution image can be generated without using a separate external database.

[0011] According to the embodiments disclosed herein, it is possible to well preserve detail components even in high-resolution images, and obtain a super-resolution (SR) effect with excellent performance without miscorrection or color defects.

[0012] In addition, various other effects may be provided that can be grasped directly or indirectly through this document. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2a] FIG. 1 is a block diagram showing a detailed configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2b] FIG. 1 is a block diagram showing a detailed configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2c]FIG. 2 is a block diagram illustrating a second image generating unit according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a method for generating a low-resolution image according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating an operation of an image processing device according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates a method for generating a high-resolution image according to an embodiment of the present disclosure. [Figure 6] 1 is a diagram illustrating a high-resolution image generation method according to an embodiment of the present disclosure in relation to an actual image; [Figure 7] FIG. 1 illustrates a target region, a reference region, and a correction region according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a block diagram showing an example of a computer device that corresponds to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0014] Various embodiments will be described below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the specific embodiments and includes various modifications, equivalents, and / or alternatives of the embodiments. The embodiments of the present disclosure can provide various advantages that can be recognized directly or indirectly through the present disclosure.

[0015] FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure.

[0016] Referring to FIG. 1, the image processing device 100 may include a first image generating unit 110, a reference area searching unit 120, a correction area determining unit 130, and a second image generating unit 140. In one example, the image processing device 100 may generate high-resolution image data (HR image) by performing at least one image signal processing on an input image. The configuration of the image processing device 100 according to the present disclosure is merely an example, and some components may be combined, added, or omitted from the configuration of the image processing device 100. For example, the first image generating unit 110 and the second image generating unit 140 may be combined into a single module (e.g., an image generating unit (not shown)). For example, the first image generating unit 110 may be omitted, and the image processing device 100 may receive an input image or a low-resolution image LR from an external device (e.g., an external server (not shown)). In one example, the first image generating unit 110 may correspond to a low-resolution image generating unit that generates a low-resolution image, and the second image generating unit 140 may correspond to a high-resolution image generating unit that generates a high-resolution image.

[0017] According to an embodiment, the image processing device 100 may generate an image file by compressing image data that has undergone image signal processing for improving image quality, or may restore image data from the image file. The image compression format may be a lossless format or a lossy format. For example, in the case of a still image, the Joint Photographic Experts Group (JPEG) format or the JPEG2000 format may be used. In addition, in the case of a moving image, a moving image file may be generated by compressing multiple frames according to the Moving Picture Experts Group (MPEG) standard.

[0018] According to one embodiment, the image processing device 100 may use a single image super resolution (SISR) method, a multi image super resolution (MISR) method, and / or a video super resolution (VSR) method to convert a low-resolution image into a high-resolution image. For example, the image processing device 100 may generate an image having a higher resolution than the resolution of a single input image by using the SISR method on the single input image without referring to an external database. In one example, the input image may correspond to image data generated by an image sensing device.

[0019] According to one embodiment, the input image may be generated by an image sensing device that captures an optical image of a scene, although the scope of the present invention is not limited thereto. The image sensing device may include a pixel array including a plurality of pixels for sensing light incident from a scene, a control circuit for controlling the pixel array, and a readout circuit that converts analog pixel signals received from the pixel array into a digital input image and outputs the digital input image. In this disclosure, it is assumed that the input image is generated by the image sensing device and that the image processing device 100 receives the input image.

[0020] According to one embodiment, the first image generation unit 110 may receive an input image and generate a low-resolution image LR. In one example, the low-resolution image LR may correspond to an image obtained by downscaling the input image. For example, the downscaling may include at least one of warping, blurring, or downsampling. A more detailed description of the operation of the first image generation unit 110 will be provided below with reference to FIG. 3.

[0021] According to one embodiment, the reference area search unit 120 may receive a low-resolution image LR and generate reference area data RA. In the present disclosure, a certain area may be understood as a concept corresponding to at least a portion of an image. In addition, the certain area data may include data for the certain area. It is assumed that the data for the certain area includes coordinates and pixel data of each pixel included in the certain area. For example, the 1_1 reference area data (RA1_1) may include data for the 1_1 reference area. For example, the data for the 1_1 reference area may include coordinates and pixel data of each pixel included in the 1_1 reference area.

[0022] According to one embodiment, the reference area data RA may include coordinates and pixel data of each pixel included in the reference area. In one example, the reference area may correspond to a region of the low-resolution image LR that is most similar to the target region of the input image. In one example, the target region may correspond to a region of the input image whose resolution is to be improved. For example, the image processing apparatus 100 may set a portion of the region of the input image as the target region. In one example, the image processing apparatus 100 may divide the input image 500 into one or more regions and set at least one of the one or more regions as the target region 502.

[0023] According to an embodiment, the reference region search unit 120 may search for a reference region that is most similar to the target region among regions in the low-resolution image LR based on a sum of absolute difference (SAD), normalized cross correlation (NCC), or mean square error (MSE) method. A more detailed description of the operation of the reference region search unit 120 will be provided below with reference to Figures 2a and 2b.

[0024] According to an embodiment, the correction area determination unit 130 may receive the reference area data RA and generate the correction area data CA. In an example, the correction area data CA may include coordinates and pixel data of each pixel included in the correction area. In an example, the correction area may correspond to an area located in an area of ​​an input image corresponding to the reference area. In an example, the correction area determination unit 130 may determine a target correction area that is more similar to the reference area from among a first correction area corresponding to the first low-resolution image and a second correction area corresponding to the second low-resolution image. In an example, the correction area determination unit 130 may determine a correction area that is more similar to the reference area from among the first correction area and the second correction area as the target correction area and transmit the determined target correction area to the second image generation unit. A more detailed description of the operation of the correction area determination unit 130 will be provided below with reference to FIG. 7.

[0025] According to one embodiment, the second image generation unit 140 may receive the correction area data CA and generate a high resolution image (HR image). In one example, some areas of the high resolution image (HR image) may be generated based on one or more correction area data CA. For example, the second image generation unit 140 may combine the first through tenth correction areas to form the high resolution image (HR image). A more detailed description of the operation of the second image generation unit 140 will be provided below with reference to FIG. 2c.

[0026] According to one embodiment, the first image generating unit 110 may generate the first image and the second image, and the second image generating unit 140 may generate the third image. In one example, the first image and the second image may correspond to a lower resolution than the third image.

[0027] 2a and 2b are block diagrams showing a detailed configuration of an image processing device according to an embodiment of the present disclosure.

[0028] Referring to FIG. 2a, the image processing device 100 may include a first image generation unit 110, a reference area search unit 120, and a correction area determination unit 130. In one example, the image processing device 100 receives an input image including a first target area, and may generate data (e.g., CA1_3) corresponding to a correction area most similar to the first target area using the first image generation unit 110, the reference area search unit 120, and the correction area determination unit 130. In one example, the correction area most similar to the first target area (e.g., correction area 1_3) may correspond to the first target correction area. FIG. 2a will be described assuming that the image processing device 100 sets a first target area from among areas of the input image. For example, the image processing device 100 may set a first target area among areas of an input image, generate data (e.g., CA1_3) corresponding to a correction area that is most similar to the first target area, and generate an area of ​​a high-resolution image (HR image) that corresponds to the first target area based on the correction area (e.g., 1_3 correction area) that is most similar to the first target area.

[0029] According to one embodiment, the first image generation unit 110 may receive an input image and generate a first low-resolution image LR1, a second low-resolution image LR2, a third low-resolution image LR3, and a fourth low-resolution image LR4. In the present disclosure, it is assumed that the first image generation unit 110 generates four low-resolution images (e.g., LR1, LR2, LR3, and LR4) for the input image, but the number of low-resolution images is not limited thereto. For example, the first image generation unit 110 may receive an input image and generate first to tenth low-resolution images.

[0030] According to one embodiment, the first image generation unit 110 may generate a first low-resolution image LR1 by applying a first downscaling to an input image. In one example, the first image generation unit 110 may generate a second low-resolution image LR2 by applying a second downscaling to the input image. The third low-resolution image LR3 and the fourth low-resolution image LR4 may also be generated in a similar manner.

[0031] According to one embodiment, the reference area search unit 120 may receive low-resolution images (e.g., LR1, LR2, LR3, and / or LR4) and generate reference area data (e.g., RA1_1, RA1_2, RA1_3, and / or RA1_4). In FIGS. 2A and 2B, it is assumed that the reference area search unit 120 receives four low-resolution images and generates four reference areas. However, the number of low-resolution images and the number of reference areas are not limited thereto. In one example, the reference area may be expressed as RAm_n. In this case, an input image may be composed of M target areas, where m may be any number from 1 to M. Also, N low-resolution images may be generated for the input image, where n may be any number from 1 to N. That is, the m_nth reference area may be expressed as RAm_n, and RAm_n may correspond to the area of ​​the nth low-resolution image that is searched by the reference area search unit 120 as the reference area most similar to the mth target area.

[0032] According to one embodiment, the reference area search unit 120 can generate 1_1 reference area data (RA1_1), 1_2 reference area data (RA1_2), 1_3 reference area data (RA1_3), and 1_4 reference area data (RA1_4) based on the results of comparing the first target area of ​​the input image with each of the first low-resolution image LR1, the second low-resolution image LR2, the third low-resolution image LR3, and the fourth low-resolution image LR4.

[0033] According to one embodiment, the reference area search unit 120 may search for a region in the first low-resolution image LR1 that is most similar to the first target region to identify a 1_1 reference area, and may transmit the 1_1 reference area data (RA1_1) to the correction area conversion unit 132. In one example, the reference area search unit 120 may search for a region in the second low-resolution image LR2 that is most similar to the first target region to identify a 1_2 reference area, and may transmit the 1_2 reference area data (RA1_2) to the correction area conversion unit 132. In one example, the reference area search unit 120 may search for a region in the third low-resolution image LR3 that is most similar to the first target region to identify a 1_3 reference area, and may transmit the 1_3 reference area data (RA1_3) to the correction area conversion unit 132. In one example, the reference area search unit 120 can search for the area in the fourth low-resolution image LR4 that is most similar to the first target area to identify the 1_4 reference area, and can transmit the 1_4 reference area data (RA1_4) to the correction area conversion unit 132.

[0034] According to an embodiment, the size of the reference area (e.g., the 1_1 reference area, the 1_2 reference area, the 1_3 reference area, and / or the 1_4 reference area) may be the same as the size of the first target area. For example, if the first target area is a rectangular area including points located at (6, 7), (6, 6), (7, 6), and (7, 7) in the coordinates of the input image, the 1_1 reference area may correspond to a rectangular area including points located at (1, 2), (1, 1), (2, 1), and (2, 2) in the coordinates of the first low-resolution image LR1, and the 1_2 reference area may correspond to a rectangular area including points located at (0, 1), (0, 0), (1, 0), and (1, 1) in the coordinates of the second low-resolution image LR2. The 1_3 reference area may correspond to a rectangular area having vertices at points (5, 6), (5, 5), (6, 5), and (6, 6) among the coordinates of the third low-resolution image LR3, and the 1_4 reference area may correspond to a rectangular area having vertices at points (0, 11), (0, 10), (1, 10), and (1, 11) among the coordinates of the fourth low-resolution image LR4.

[0035] According to one embodiment, the reference area search unit 120 may use a sum of absolute difference (SAD), zero-mean version SAD (ZSAD), normalized cross correlation (NCC), and / or zero-version NCC (ZNCC) method to search for a reference area (e.g., 1_1 reference area, 1_2 reference area, 1_3 reference area, and / or 1_4 reference area) that is most similar to the first target area among areas in the low-resolution image (e.g., LR1, LR2, LR3, and / or LR4). In one example, the reference area search unit 120 may compare an average mean square error (MSE) of pixel data of pixels included in the first target area with an average MSE of pixel data of pixels included in an area of ​​the same size as the first target area among areas in the low-resolution image (e.g., LR1, LR2, LR3, and / or LR4) to search for an area with the least difference.

[0036] According to an embodiment, the correction area determination unit 130 may include a correction area conversion unit 132 and a similarity determination unit 134. In one example, the correction area determination unit 130 receives the first_1 reference area data (RA1_1), the first_2 reference area data (RA1_2), the first_3 reference area data (RA1_3), and the first_4 reference area data (RA1_4), and may identify the correction area (e.g., the first_3 correction area) that is most similar to the first target area using the correction area conversion unit 132 and the similarity determination unit 134.

[0037] According to one embodiment, the correction area conversion unit 132 can receive the 1_1 reference area data (RA1_1), the 1_2 reference area data (RA1_2), the 1_3 reference area data (RA1_3), and / or the 1_4 reference area data (RA1_4) and generate the 1_1 correction area data (CA1_1), the 1_2 correction area data (CA1_2), the 1_3 correction area data CA1_3, and / or the 1_4 correction area data (CA1_4). In one example, the correction area conversion unit 132 can identify areas of the input image that are located in areas corresponding to reference areas (e.g., the 1_1 reference area, the 1_2 reference area, the 1_3 reference area, and / or the 1_4 reference area) as correction areas (e.g., the 1_1 correction area, the 1_2 correction area, the 1_3 correction area, and / or the 1_4 correction area), and can transmit the identified correction area data (e.g., CA1_1, CA1_2, CA1_3, and / or CA1_4) to the similarity determination unit 134.

[0038] According to one embodiment, the 1_1 correction area may correspond to an area located at coordinates of the first low-resolution image LR1 of the 1_1 reference area among the coordinates of the input image multiplied by a first coefficient. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the first low-resolution image LR1. For example, if the 1_1 reference area is a quadrangular area including vertices at points (1,2), (1,1), (2,1), and (2,2) among the coordinates of the first low-resolution image LR1, and the input image has a resolution twice as high as the first low-resolution image LR1, the 1_1 correction area may correspond to a quadrangular area including vertices at points (2,4), (2,2), (4,2), and (4,4) among the coordinates of the input image.

[0039] According to one embodiment, the first_2 correction area may correspond to an area located at coordinates of the second low-resolution image LR2 of the first_2 reference area among the coordinates of the input image multiplied by a first coefficient. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the second low-resolution image LR2. For example, if the first_2 reference area is a quadrangular area including vertices at points (0,1), (0,0), (1,0), and (1,1) among the coordinates of the second low-resolution image LR2, and the input image has a resolution twice as high as the second low-resolution image LR2, the first_2 correction area may correspond to a quadrangular area including vertices at points (0,2), (0,0), (2,0), and (2,2) among the coordinates of the input image.

[0040] According to one embodiment, the 1_3 correction area may correspond to an area located at coordinates obtained by multiplying the coordinates of the third low-resolution image LR3 of the 1_3 reference area by a first coefficient among the coordinates of the input image. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the third low-resolution image LR3. For example, if the 1_3 reference area is a rectangular area including vertices at points (5,6), (5,5), (6,5), and (6,6) among the coordinates of the third low-resolution image LR3, and the input image has a resolution twice as high as the third low-resolution image LR3, the 1_3 correction area may correspond to a rectangular area including vertices at points (10,12), (10,10), (12,10), and (12,12) among the coordinates of the input image.

[0041] According to one embodiment, the 1_4 correction area may correspond to an area located at coordinates obtained by multiplying the coordinates of the fourth low-resolution image LR4 of the 1_4 reference area by a first coefficient among the coordinates of the input image. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the fourth low-resolution image LR4. For example, if the 1_4 reference area is a rectangular area including vertices at points (0, 11), (0, 10), (1, 10), and (1, 11) among the coordinates of the fourth low-resolution image LR4, and the input image has a resolution twice as high as the fourth low-resolution image LR4, the 1_4 correction area may correspond to a rectangular area including vertices at points (0, 22), (0, 20), (2, 20), and (2, 22) among the coordinates of the input image.

[0042] According to one embodiment, the similarity determination unit 134 receives the 1_1 correction area data (CA1_1), the 1_2 correction area data (CA1_2), the 1_3 correction area data (CA1_3), and the 1_4 correction area data (CA1_4), and determines the correction area that is most similar to the first target area among the correction areas corresponding to the received 1_1 correction area data (CA1_1), the 1_2 correction area data (CA1_2), the 1_3 correction area data (CA1_3), and the 1_4 correction area data (CA1_4) as the first target correction area, and transmits the corresponding correction area data to an external device (e.g., the second image generation unit 140).

[0043] According to one embodiment, the similarity determination unit 134 may downscale the received correction regions (e.g., the 1_1 correction region, the 1_2 correction region, the 1_3 correction region, and / or the 1_4 correction region) to determine the correction region that is most similar to the first target region. In one example, the downscaling of the correction regions may include at least one of warping, blurring, or downsampling.

[0044] According to an embodiment, the similarity determination unit 134 may first downscale the 1_1 correction region to generate a 1_1 downscaled correction region, second downscale the 1_2 correction region to generate a 1_2 downscaled correction region, third downscale the 1_3 correction region to generate a 1_3 downscaled correction region, and fourth downscale the 1_4 correction region to generate a 1_4 downscaled correction region. In one example, the first to fourth downscalings may correspond to downscalings applied to generate the first to fourth low-resolution images (LR1 to LR4).

[0045] According to an embodiment, the similarity determination unit 134 may determine the most similar correction area based on a result of comparing pixel data of the first target area with pixel data of each of the 1_1 downscaling correction area, the 1_2 downscaling correction area, the 1_3 downscaling correction area, and the 1_4 downscaling correction area. For example, if the average pixel data value of the first target area is 10, and the average pixel data value of the 1_1 downscaling correction area is 15, the average pixel data value of the 1_2 downscaling correction area is 20, the average pixel data value of the 1_3 downscaling correction area is 7, and the average pixel data value of the 1_4 downscaling correction area is 25, the similarity determination unit 134 may determine that the 1_3 correction area corresponding to the 1_3 downscaling correction area is the most similar correction area to the first target area. For example, the similarity determination unit 134 may determine that the 1_3 correction area is the first target correction area and transmit the 1_3 correction area data CA1_3 to the second image generation unit 140.

[0046] According to an embodiment, the similarity determination unit 134 may determine the most similar correction area based on a result of comparing pixel data of the first target area with pixel data of each of the 1_1 correction area, the 1_2 correction area, the 1_3 correction area, and the 1_4 correction area. For example, if the average pixel data value of the first target area is 10, and the average pixel data value of the 1_1 correction area is 15, the average pixel data value of the 1_2 correction area is 20, the average pixel data value of the 1_3 correction area is 7, and the average pixel data value of the 1_4 correction area is 25, the similarity determination unit 134 may determine that the 1_3 correction area is the most similar correction area to the first target area.

[0047] According to an embodiment, the similarity determination unit 134 may determine the similarity between the correction area and the first target area among the coordinates of the input image. In one example, the similarity determination unit 134 may determine that the correction area and the first target area are similar if a difference value between pixel data of at least one pixel included in an area obtained by downscaling the correction area and pixel data of at least one pixel included in the first target area is equal to or less than a predetermined threshold value.

[0048] Referring to FIG. 2b, the image processing device 100 may include a first image generation unit 110, a reference area search unit 120, and a correction area determination unit 130. In one example, the image processing device 100 receives an input image including a second target area and generates a correction area (e.g., a 2_1 correction area) that is most similar to the second target area using the first image generation unit 110, the reference area search unit 120, and the correction area determination unit 130. In one example, the correction area (e.g., the 2_1 correction area) that is most similar to the second target area may correspond to the second target correction area. FIG. 2b will be described assuming that the image processing device 100 sets a second target area from among areas of the input image. For example, the image processing device 100 may set a second target area among areas of an input image, generate a correction area (e.g., 2_1 correction area) that is most similar to the second target area, and generate an area of ​​a high resolution image (HR image) that corresponds to the second target area based on the correction area (e.g., 2_1 correction area) that is most similar to the second target area.

[0049] According to one embodiment, the second target area may correspond to an area other than the first target area in Fig. 2a. In one example, the operation of generating an area corresponding to the first target area among areas of the high resolution image (HR image) based on the correction area most similar to the first target area (e.g., the 1_3 correction area) and the operation of generating an area corresponding to the second target area among areas of the high resolution image (HR image) based on the correction area most similar to the second target area (e.g., the 2_1 correction area) may be performed simultaneously or sequentially by the image processing device 100.

[0050] According to one embodiment, the first image generation unit 110 may receive an input image and generate a first low-resolution image LR1, a second low-resolution image LR2, a third low-resolution image LR3, and a fourth low-resolution image LR4. In the present disclosure, it is assumed that the first image generation unit 110 generates four low-resolution images (e.g., LR1, LR2, LR3, and LR4) for the input image, but the number of low-resolution images is not limited thereto. For example, the first image generation unit 110 may receive an input image and generate first to tenth low-resolution images.

[0051] According to one embodiment, the first image generation unit 110 may generate a first low-resolution image LR1 by applying a first downscaling to an input image. In one example, the first image generation unit 110 may generate a second low-resolution image LR2 by applying a second downscaling to the input image. The third low-resolution image LR3 and the fourth low-resolution image LR4 may also be generated in a similar manner.

[0052] According to one embodiment, the reference area search unit 120 may receive a first low-resolution image LR1, a second low-resolution image LR2, a third low-resolution image LR3, and a fourth low-resolution image LR4. In one example, the reference area search unit 120 may generate 2_1 reference area data (RA2_1), 2_2 reference area data (RA2_2), 2_3 reference area data (RA2_3), and 2_4 reference area data (RA2_4) based on results of comparing the second target area of ​​the input image with the first low-resolution image LR1, the second low-resolution image LR2, the third low-resolution image LR3, and the fourth low-resolution image LR4, respectively.

[0053] According to one embodiment, the reference area search unit 120 may search for an area similar to the second target area among areas in the first low resolution image LR1 to identify a 2_1 reference area, and may transmit the 2_1 reference area data (RA2_1) to the correction area conversion unit 132. In one example, the reference area search unit 120 may search for an area similar to the second target area among areas in the second low resolution image LR2 to identify a 2_2 reference area, and may transmit the 2_2 reference area data (RA2_2) to the correction area conversion unit 132. In one example, the reference area search unit 120 may search for an area similar to the second target area among areas in the third low resolution image LR3 to identify a 2_3 reference area, and may transmit the 2_3 reference area data (RA2_3) to the correction area conversion unit 132. In one example, the reference area search unit 120 can search for an area in the fourth low-resolution image LR4 that is similar to the second target area to identify the second_4 reference area, and can transmit the second_4 reference area data (RA2_4) to the correction area conversion unit 132.

[0054] According to an embodiment, the size of the reference area (e.g., 2_1 reference area, 2_2 reference area, 2_3 reference area, and / or 2_4 reference area) may be the same as the size of the second target area. For example, if the second target area is a rectangular area including points located at (1,1), (1,0), (2,0), and (2,1) coordinates in the input image as vertices, the 2_1 reference area may correspond to a rectangular area including points located at (3,6), (3,5), (4,5), and (4,6) coordinates in the first low-resolution image LR1 as vertices, and the 2_2 reference area may correspond to a rectangular area including points located at (2,5), (2,4), (3,4), and (3,5) coordinates in the second low-resolution image LR2 as vertices. The 2_3 reference area may correspond to a rectangular area including points located at (7, 10), (7, 9), (8, 9), and (8, 10) as vertices in the coordinates of the third low-resolution image LR3, and the 2_4 reference area may correspond to a rectangular area including points located at (2, 15), (2, 14), (3, 14), and (3, 15) as vertices in the coordinates of the fourth low-resolution image LR4.

[0055] According to an embodiment, the correction area determination unit 130 may include a correction area conversion unit 132 and a similarity determination unit 134. In one example, the correction area determination unit 130 receives the 2_1 reference area data (RA2_1), the 2_2 reference area data (RA2_2), the 2_3 reference area data (RA2_3), and the 2_4 reference area data (RA2_4), and may identify the correction area (e.g., the 2_1 correction area) that is most similar to the second target area using the correction area conversion unit 132 and the similarity determination unit 134.

[0056] According to one embodiment, the correction area conversion unit 132 can receive the 2_1 reference area data (RA2_1), the 2_2 reference area data (RA2_2), the 2_3 reference area data (RA2_3), and / or the 2_4 reference area data (RA2_4) and generate the 2_1 correction area data CA2_1, the 2_2 correction area data (CA2_2), the 2_3 correction area data (CA2_3), and / or the 2_4 correction area data (CA2_4). In one example, the correction area conversion unit 132 can identify areas of the input image that are located in areas corresponding to reference area data (e.g., RA2_1, RA2_2, RA2_3, and / or RA2_4) as correction areas (e.g., 2_1 correction area, 2_2 correction area, 2_3 correction area, and / or 2_4 correction area), and can transmit the identified correction areas (e.g., 2_1 correction area, 2_2 correction area, 2_3 correction area, and / or 2_4 correction area) to the similarity determination unit 134.

[0057] According to one embodiment, the 2_1 correction area may correspond to an area located at coordinates of the first low-resolution image LR1 of the 2_1 reference area among the coordinates of the input image multiplied by a second coefficient. In one example, the second coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the first low-resolution image LR1. For example, if the 2_1 reference area is a rectangular area including vertices at points (3, 6), (3, 5), (4, 5), and (4, 6) among the coordinates of the first low-resolution image LR1, and the input image has a resolution twice as high as the first low-resolution image LR1, the 2_1 correction area may correspond to a rectangular area including vertices at points (6, 12), (6, 10), (8, 10), and (8, 12) among the coordinates of the input image.

[0058] According to one embodiment, the 2_2 correction area may correspond to an area located at coordinates obtained by multiplying the coordinates of the second low-resolution image LR2 of the 2_2 reference area by a second coefficient among the coordinates of the input image. In one example, the second coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the second low-resolution image LR2. For example, if the 2_2 reference area is a rectangular area including vertices at points (2, 5), (2, 4), (3, 4), and (3, 5) among the coordinates of the second low-resolution image LR2, and the input image has a resolution twice as high as the second low-resolution image LR2, the 2_2 correction area may correspond to a rectangular area including vertices at points (4, 10), (4, 8), (6, 8), and (6, 10) among the coordinates of the input image.

[0059] According to one embodiment, the 2_3 correction area may correspond to an area located at coordinates obtained by multiplying the coordinates of the third low-resolution image LR3 of the 2_3 reference area by a second coefficient among the coordinates of the input image. In one example, the second coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the third low-resolution image LR3. For example, if the 2_3 reference area is a rectangular area including vertices at points (7, 10), (7, 9), (8, 9), and (8, 10) among the coordinates of the third low-resolution image LR3, and the input image has a resolution twice as high as the third low-resolution image LR3, the 2_3 correction area may correspond to a rectangular area including vertices at points (14, 20), (14, 18), (16, 18), and (16, 20) among the coordinates of the input image.

[0060] According to one embodiment, the 2_4 correction area may correspond to an area located at coordinates obtained by multiplying the coordinates of the fourth low-resolution image LR4 of the 2_4 reference area by a second coefficient among the coordinates of the input image. In one example, the second coefficient may correspond to a value obtained by dividing the resolution of the input image by the resolution of the fourth low-resolution image LR4. For example, if the 2_4 reference area is a rectangular area having vertices located at points (2, 15), (2, 14), (3, 14), and (3, 15) among the coordinates of the fourth low-resolution image LR4, and the input image has a resolution twice as high as the fourth low-resolution image LR4, the 2_4 correction area may correspond to a rectangular area having vertices located at points (4, 30), (4, 28), (6, 28), and (6, 30) among the coordinates of the input image. In one example, referring also to FIG. 2a, the second coefficient may correspond to the same value as the first coefficient.

[0061] According to one embodiment, the similarity determination unit 134 receives the 2_1 correction area data CA2_1, the 2_2 correction area data CA2_2, the 2_3 correction area data CA2_3, and the 2_4 correction area data CA2_4, and determines the area most similar to the second target area among the correction areas corresponding to the received 2_1 correction area data CA2_1, the 2_2 correction area data CA2_2, the 2_3 correction area data CA2_3, and the 2_4 correction area data CA2_4 as the second target correction area, and transmits the second target correction area to an external device (e.g., the second image generation unit 140).

[0062] According to one embodiment, the similarity determination unit 134 may downscale the received correction regions (e.g., the 2_1 correction region, the 2_2 correction region, the 2_3 correction region, and / or the 2_4 correction region) to determine the correction region that is most similar to the second target region. In one example, the similarity determination unit 134 may first downscale the 2_1 correction region to generate the 2_1 downscaled correction region, second downscale the 2_2 correction region to generate the 2_2 downscaled correction region, third downscale the 2_3 correction region to generate the 2_3 downscaled correction region, and fourth downscale the 2_4 correction region to generate the 2_4 downscaled correction region.

[0063] According to an embodiment, the similarity determination unit 134 may determine the most similar correction area based on a result of comparing pixel data of the second target area with pixel data of each of the 2_1 downscaling correction area, the 2_2 downscaling correction area, the 2_3 downscaling correction area, and the 2_4 downscaling correction area. For example, if the average pixel data value of the second target area is 0, and the average pixel data value of the 2_1 downscaling correction area is 5, the average pixel data value of the 2_2 downscaling correction area is 20, the average pixel data value of the 2_3 downscaling correction area is 10, and the average pixel data value of the 2_4 downscaling correction area is 25, the similarity determination unit 134 may determine that the 2_1 correction area corresponding to the 2_1 downscaling correction area is the most similar correction area to the second target area. For example, the similarity determination unit 134 may determine that the 2_1 correction area is the second target correction area and transmit the 2_1 correction area data CA2_1 to the second image generation unit 140.

[0064] According to an embodiment, the similarity determination unit 134 may determine the most similar correction area based on a result of comparing pixel data of the second target area with pixel data of each of the 2_1 correction area, the 2_2 correction area, the 2_3 correction area, and the 2_4 correction area. For example, if the average pixel data value of the second target area is 10, and the average pixel data value of the 2_1 correction area is 15, the average pixel data value of the 2_2 correction area is 20, the average pixel data value of the 2_3 correction area is 0, and the average pixel data value of the 2_4 correction area is 25, the similarity determination unit 134 may determine that the 2_1 correction area is the most similar correction area to the second target area.

[0065] According to an embodiment, the similarity determination unit 134 may determine the similarity between the correction area and the second target area among the coordinates of the input image. In one example, the similarity determination unit 134 may determine that the correction area and the second target area are similar if a difference value between pixel data of at least one pixel included in an area obtained by downscaling the correction area and pixel data of at least one pixel included in the second target area is equal to or less than a predetermined threshold value.

[0066] FIG. 2c is a block diagram illustrating a second image generator according to one embodiment of the present disclosure.

[0067] Referring to FIG. 2c, the second image generating unit 140 may receive the first_3 correction area data CA1_3 and / or the second_1 correction area data CA2_1 and generate a high resolution image (HR image). In one example, the second image generating unit 140 may generate a high resolution image by positioning a correction area most similar to the target area in a region of the high resolution image corresponding to the target area. In one example, the second image generating unit 140 may determine a region of the high resolution image corresponding to the target area as the correction area most similar to the target area. For example, the second image generating unit 140 may position a target correction area in a region of the high resolution image corresponding to the target area. In FIG. 2c, the description will be made assuming that the second image generating unit 140 generates a high resolution image (HR image) based on two correction areas, but the number of correction areas is not limited thereto. For example, the second image generation unit 140 may receive 10 correction areas based on 10 different target areas, and may generate a high resolution image (HR image) by positioning the received 10 correction areas in areas of the high resolution image that correspond to the 10 target areas.

[0068] According to one embodiment, the second image generation unit 140 receives the first_3 correction area data CA1_3 and / or the second_1 correction area data CA2_1 and generates a high resolution image (HR image) by combining the first_3 correction area corresponding to the received first_3 correction area data CA1_3 and / or the second_1 correction area corresponding to the received second_1 correction area data CA2_1. In one example, the first_3 correction area may correspond to the correction area most similar to the first target area based on the operation described with reference to FIG. 2a. For example, the first_3 correction area may correspond to the first target correction area. In one example, the second_1 correction area may correspond to the correction area most similar to the second target area based on the operation described with reference to FIG. 2b. For example, the second_1 correction area may correspond to the second target correction area.

[0069] According to one embodiment, the first-third correction area may be located at coordinates of an HR image yet to be generated, which are obtained by multiplying the input image coordinates of the first target area by a third coefficient, and may constitute at least a portion of the HR image. In one example, the third coefficient may correspond to a value obtained by dividing the resolution of the HR image by the resolution of the input image. For example, if the first target area is a rectangular area having vertices at points (6, 7), (6, 6), (7, 6), and (7, 7) in the coordinates of the input image, and the high-resolution image (HR image) has a resolution twice higher than the input image, the first-third correction area may be located in a rectangular area having vertices at points (12, 14), (12, 12), (14, 12), and (14, 14) in the high-resolution image (HR image) that has not yet been generated, and may constitute at least a portion of the high-resolution image (HR image).

[0070] According to one embodiment, the 2_1 correction region may be located at coordinates of the high resolution image (HR image) yet to be generated, which are obtained by multiplying the input image coordinates of the second target region by a third coefficient, and may constitute a portion of the high resolution image (HR image). In one example, the third coefficient may correspond to a value obtained by dividing the resolution of the high resolution image (HR image) by the resolution of the input image. For example, if the second target region is a rectangular region having vertices at points (1,1), (1,0), (2,0), and (2,1) of the input image, and the high resolution image (HR image) has a resolution twice higher than the input image, the 2_1 correction region may be located at a rectangular region having vertices at points (2,2), (2,0), (4,0), and (4,2) of the high resolution image (HR image) yet to be generated, and may constitute at least a portion of the high resolution image (HR image).

[0071] FIG. 3 illustrates a method for generating a low-resolution image according to one embodiment of the present disclosure.

[0072] 3, the first image generator 110 may generate a low-resolution image (e.g., LR1, LR2, LR3, or LR4) by downscaling the input image 300. In one example, the downscaling may include applying at least one of warping (e.g., W1, W2, W3, or W4), blurring (e.g., B1, B2, B3, or B4), or downsampling (e.g., D1, D2, D3, or D4) to the input image 300, and / or adding noise (e.g., N1, N2, N3, or N4). In one example, in the downscaling operation, the order of warping (e.g., W1, W2, W3, or W4), blurring (e.g., B1, B2, B3, or B4), or downsampling (e.g., D1, D2, D3, or D4) to the input image 300 may be changed.

[0073] According to one embodiment, the low-resolution images (e.g., LR1, LR2, LR3, or LR4) may satisfy a matrix relation such as Equation 1. In one example, LR may correspond to a low-resolution image, CR may correspond to a reference image, D may correspond to a downsampling function, B may correspond to a blurring function, W may correspond to a warping function, N may correspond to noise, and n may correspond to the number of low-resolution images. For example, CR may be substituted with the input image 300.

number

[0074] According to one embodiment, warping (e.g., W1, W2, W3, or W4) can include an operation in which pixel positions of an input image are changed. In one example, warping (e.g., W1, W2, W3, or W4) can include scaling and / or rotation operations.

[0075] According to one embodiment, when warping (e.g., W1, W2, W3, or W4) is applied to the input image 300, the coordinates (x, y) of the input image 300 may satisfy matrix relations such as Equation 2 and Equation 3. In one example, x may correspond to the X-axis coordinate of the input image 300, y may correspond to the Y-axis coordinate of the input image 300, x' may correspond to a value obtained by performing a scaling and rotation operation on x, y' may correspond to a value obtained by performing a scaling and rotation operation on y, a may correspond to an x-component noise, b may correspond to a y-component noise, x_W may correspond to a value obtained by performing a warping operation on x, and y_W may correspond to a value obtained by performing a warping operation on y.

number

number

[0076] According to one embodiment, blurring (e.g., B1, B2, B3, or B4) can include filtering pixel data of an input image. In one example, blurring (e.g., B1, B2, B3, or B4) can include smoothing pixel data of an input image.

[0077] According to one embodiment, when blurring (e.g., B1, B2, B3, or B4) is applied to an input image, the coordinates (x, y) of the input image may satisfy a relationship such as the following equation: In one example, f[x, y] may correspond to an image after blurring, g[k, l] may correspond to an image before blurring, and h[x, y; k, l] may correspond to a response value at [x, y] to an impulse of a pixel located at [k, l].

number

[0078] According to one embodiment, downsampling (e.g., D1, D2, D3, or D4) can include reducing the size of an input image. In one example, downsampling (e.g., D1, D2, D3, or D4) can include reducing the resolution of an input image.

[0079] According to one embodiment, when downsampling (e.g., D1, D2, D3, or D4) is applied to an input image, the input image i[n] may satisfy the following relationship: In one example, i[n] may correspond to the input image, d[n] may correspond to the downsampled image, h[k] may correspond to the impulse at k, K may correspond to the impulse length, and M may correspond to the reduction ratio.

number

[0080] According to one embodiment, the operation of adding noise (e.g., N1, N2, N3, or N4) to the input image may include an operation of adding additive white Gaussian noise (AWGN) and / or shot noise to the input image. In one example, the noise (e.g., N1, N2, N3, or N4) added to the input image may correspond to a and / or b in [Equation 2] and [Equation 3].

[0081] According to one embodiment, the first image generation unit 110 may perform a first downscaling on the input image 300 to generate a first low-resolution image LR1, may perform a second downscaling on the input image 300 to generate a second low-resolution image LR2, may perform a third downscaling on the input image 300 to generate a third low-resolution image LR3, and may perform a fourth downscaling on the input image 300 to generate a fourth low-resolution image LR4.

[0082] According to one embodiment, the first downscaling may include applying at least one of a first warping W1, a first blurring B1, and a first downsampling D1, and / or adding a first noise N1, to the input image 300. In one example, the second downscaling may include applying at least one of a second warping W2, a second blurring B2, and a second downsampling D2, and / or adding a second noise N2, to the input image 300. Similar terms may be used for the third downscaling and the fourth downscaling.

[0083] FIG. 4 is a flowchart illustrating the operation of the image processing device according to an embodiment of the present disclosure.

[0084] Referring to FIG. 4, the image processing device 100 may generate a first image and a second image using an input image (S100). In one example, referring also to FIG. 1, the first image generation unit 110 may generate the first image and the second image, and the first image and the second image may correspond to images corresponding to a lower resolution than a third image (described later). In one example, the first image generation unit 110 of the image processing device 100 may generate the first image by applying a first downscaling to the input image. Also, the first image generation unit 110 may generate the second image by applying a second downscaling to the input image. In one example, the first image and the second image may correspond to images having degraded resolution, clarity, and / or contrast compared to the input image.

[0085] According to an embodiment, the image processing device 100 may search for a first reference area and a second reference area (S110). In one example, the reference area search unit 120 of the image processing device 100 may search for an area in a first image that is most similar to a target area in an input image. The reference area search unit 120 may identify the area found to be most similar to the target area as the first reference area. In one example, the reference area search unit 120 of the image processing device 100 may search for an area in a second image that is most similar to the target area in an input image. The reference area search unit 120 may identify the area found to be most similar to the target area as the second reference area.

[0086] According to an embodiment, the image processing device 100 may determine a correction area that is more similar to the target area from among the first correction area and the second correction area (S120). In one example, the correction area determination unit 130 may identify the first correction area based on the first reference area and the second correction area based on the second reference area. In one example, the first correction area may correspond to an area located in an area of ​​the input image that corresponds to the first reference area. Also, the second correction area may correspond to an area located in an area of ​​the input image that corresponds to the second reference area. In one example, the correction area determination unit 130 of the image processing device 100 may determine a target correction area that is more similar to the target area from among the first correction area located in an area of ​​the input image that corresponds to the first reference area and the second correction area located in an area of ​​the input image that corresponds to the second reference area.

[0087] According to an embodiment, the correction region determination unit 130 may generate a first downscaling correction region by performing a first downscaling on the first correction region and a second downscaling correction region by performing a second downscaling on the second correction region, and may determine that a correction region corresponding to a downscaling correction region determined to be more similar to the target region among the first downscaling correction region and the second downscaling correction region is a more similar correction region. For example, the correction region determination unit 130 may determine that a correction region corresponding to a downscaling correction region determined to be more similar to the target region among the first downscaling correction region and the second downscaling correction region is a target correction region.

[0088] According to one embodiment, the image processing device 100 may generate a third image using a more similar correction area (S130). In one example, referring to FIG. 1 , the second image generation unit 140 may generate the third image, and the third image may correspond to an image corresponding to a higher resolution than the first and second images. In one example, the second image generation unit 140 may generate the third image using a target correction area. In one example, the third image may correspond to an image having improved resolution, clarity, and / or contrast compared to the input image. In one example, the second image generation unit 140 of the image processing device 100 may generate the third image using the correction area determined to be the more similar correction area through step S120. In one example, the second image generation unit 140 may generate at least a portion of the third image by positioning the correction area determined to be the more similar correction area in a region of the third image corresponding to the target region. In one example, the second image generating unit 140 may determine a region of the third image corresponding to the target region as a correction region determined to be a more similar correction region.

[0089] FIG. 5 illustrates a method for generating a high resolution image according to one embodiment of the present disclosure.

[0090] 5, the image processing apparatus 100 may receive an input image 500 and set a target region 502 of the input image 500. In one example, the target region 502 may correspond to a region of the input image 500 whose resolution is to be improved. For example, the image processing apparatus 100 may set at least a portion of the input image 500 as the target region 502. In one example, the image processing apparatus 100 may divide the input image 500 into NxM frames and set one of the NxM frames as the target region 502. N and M may correspond to natural numbers. For example, the image processing apparatus 100 may divide the input image 500 into NxM frames, sequentially set each frame as a target region, and then generate a high-resolution image corresponding to the set target region, thereby generating a high-resolution image 550 for the entire input image 500.

[0091] According to an embodiment, the first image generating unit 110 of the image processing device 100 may generate a low-resolution image 520 based on an input image 500. For example, the first image generating unit 110 may generate the low-resolution image 520 by performing a downscaling operation on the input image 500.

[0092] According to one embodiment, the reference area search unit 120 of the image processing device 100 can search for the area in the low-resolution image 520 that is most similar to the target area 502 and identify the most similar area as the reference area 522 (510).

[0093] In one example, the size of the reference region 522 may be the same as the size of the target region 502. For example, if the target region 502 corresponds to a rectangular region having vertices at coordinates (100, 100), (100, 80), (120, 80), and (120, 100) of the input image 500, the reference region 522 may correspond to a rectangular region having vertices at coordinates (5, 25), (5, 5), (25, 5), and (25, 25) of the low-resolution image 520.

[0094] According to one embodiment, the correction area determination unit 130 of the image processing device 100 may identify 530 a region of the input image 500 that corresponds to the reference area 522 as the correction area 504. In one example, the correction area 504 may correspond to a region located at coordinates obtained by multiplying the coordinates of the low-resolution image 520 of the reference area 522 among the coordinates of the input image 500 by a first coefficient. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image 500 by the resolution of the low-resolution image 520. For example, if the first coefficient is 2 and the reference region 522 corresponds to a rectangular region having vertices (5,25), (5,5), (25,5), and (25,25) among the coordinates of the low-resolution image 520, the correction region 504 may correspond to a rectangular region having vertices (10,50), (10,10), (50,10), and (50,50) among the coordinates of the input image 500. For convenience of explanation, in FIG. 5, it is assumed that the correction region 504 is determined as the correction region most similar to the target region 502.

[0095] According to one embodiment, the second image generation unit 140 of the image processing device 100 may generate at least a portion of the high-resolution image 550 by positioning a correction region 504 in a region 552 of the high-resolution image 550 that corresponds to the target region 502 (540). In one example, the second image generation unit 140 may determine the region 552 of the high-resolution image 550 that corresponds to the target region 502 as the correction region 504. In one example, the region 552 corresponding to the target region 502 may correspond to a region located at coordinates obtained by multiplying the input image 500 coordinates of the target region 502 by a third coefficient among the coordinates of the high-resolution image 550 that is not yet generated. In one example, the third coefficient may correspond to a value obtained by dividing the resolution of the high-resolution image 550 by the resolution of the input image 500. For example, if the third coefficient is 2 and the target region 502 corresponds to a quadrilateral region having vertices at (100, 100), (100, 80), (120, 80), and (120, 100) of the coordinates of the input image 500, the region 552 corresponding to the target region 502 may correspond to a quadrilateral region having vertices at (200, 200), (200, 160), (240, 160), and (240, 200) of the coordinates of the high-resolution image 550 that has not yet been generated. In this case, the second image generation unit 140 may generate at least a portion of the high-resolution image 550 by copying the correction region 504 to the region 552 corresponding to the target region 502.

[0096] FIG. 6 is a diagram illustrating a high-resolution image generating method according to an embodiment of the present disclosure in correspondence with an actual image.

[0097] 6, the image processing device 100 may receive an input image 600 and set a target region 602 of the input image 600. In one example, the first image generation unit 110 of the image processing device 100 may generate a low-resolution image 620 based on the input image 600.

[0098] According to one embodiment, the reference area search unit 120 of the image processing device 100 may search for an area in the low-resolution image 620 that is most similar to the target area 602, and identify the most similar area as the reference area 622 (610). In one example, the size of the reference area 622 may be the same as the size of the target area 602. For example, the reference area 622 may be the same size as the target area 602, but may correspond to a different image.

[0099] According to an embodiment, the correction area determination unit 130 of the image processing apparatus 100 may identify 630 a region of the input image 600 that corresponds to the reference area 622 as the correction area 604. In one example, the correction area 604 may correspond to a region located at coordinates obtained by multiplying the coordinates of the low-resolution image 620 of the reference area 622 among the coordinates of the input image 600 by a first coefficient. In one example, the first coefficient may correspond to a value obtained by dividing the resolution of the input image 600 by the resolution of the low-resolution image 620. For convenience of explanation, in FIG. 6, it is assumed that the correction area 604 is determined as the correction area most similar to the target area 602.

[0100] According to one embodiment, the second image generation unit 140 of the image processing device 100 may generate at least a portion of the high-resolution image 650 by positioning a correction region 604 in a region 652 of the high-resolution image 650 that corresponds to the target region 602 (640). In one example, the second image generation unit 140 may determine the region 652 of the high-resolution image 650 that corresponds to the target region 602 as the correction region 604. In one example, the region 652 corresponding to the target region 602 may correspond to a region located at coordinates obtained by multiplying the input image 600 coordinates of the target region 602 by a third coefficient among the coordinates of the high-resolution image 650 that is not yet generated. In one example, the third coefficient may correspond to a value obtained by dividing the resolution of the high-resolution image 650 by the resolution of the input image 600. For example, the correction region 604 located in the region 652 corresponding to the target region 602 may correspond to a region that improves the resolution, clarity, and / or contrast of the target region 602. For example, in the completed high-resolution image, the image of the region 652 corresponding to the target region 602 and the image of the correction region 604 may correspond to the same image.

[0101] FIG. 7 is a diagram illustrating a target region, a reference region, and a correction region according to one embodiment of the present disclosure.

[0102] 7, the image processing device 100 may receive an input image 700 and set a target region 702 of the input image 700. In one example, the first image generation unit 110 of the image processing device 100 may generate a first low-resolution image 710, a second low-resolution image 720, and a third low-resolution image 730 based on the input image 700. Although three low-resolution images are shown in FIG. 7 for convenience of explanation, the first image generation unit 110 may generate more low-resolution images based on the input image 700.

[0103] According to one embodiment, the reference area search unit 120 of the image processing device 100 may search for an area in the first low-resolution image 710 that is most similar to the target area 702 and identify the most similar area as the first reference area 712. In one example, the reference area search unit 120 may search for an area in the second low-resolution image 720 that is most similar to the target area 702 and identify the most similar area as the second reference area 722. In one example, the reference area search unit 120 may search for an area in the third low-resolution image 730 that is most similar to the target area 702 and identify the most similar area as the third reference area 732.

[0104] According to an embodiment, the correction region determination unit 130 of the image processing device 100 may identify a region of the input image 500 that corresponds to a reference region (e.g., 712, 722, or 732) as a correction region (e.g., 704, 706, or 708). In one example, the correction region determination unit 130 may identify a region of the input image 700 that corresponds to the first reference region 712 as a first correction region 704. In one example, the correction region determination unit 130 may identify a region of the input image 700 that corresponds to the second reference region 722 as a second correction region 706. In one example, the correction region determination unit 130 may identify a region of the input image 700 that corresponds to the third reference region 732 as a third correction region 708.

[0105] According to an embodiment, the similarity determination unit 134 of the image processing device 100 may determine one of the plurality of correction regions (e.g., 704, 706, and 708) that is most similar to the target region 702. For example, the similarity determination unit 134 may determine one of the plurality of correction regions (e.g., 704, 706, and 708) that is most similar to the target region 702 as the target correction region. In one example, the similarity determination unit 134 may perform a first downscaling on the first correction region 704 to generate a first downscaling correction region, a second downscaling on the second correction region 706 to generate a second downscaling correction region, and a third downscaling on the third correction region 708 to generate a third downscaling correction region. In one example, the first downscaling correction region, the second downscaling correction region, and the third downscaling correction region may have the same size as the target region 702.

[0106] According to one embodiment, the similarity determination unit 134 may determine the correction area (e.g., 704, 706, or 708) corresponding to the downscaling correction area having the smallest difference value among the difference value between the pixel data of the first downscaling correction area and the pixel data of the target area 702, the difference value between the pixel data of the second downscaling correction area and the pixel data of the target area 702, and the difference value between the pixel data of the third downscaling correction area and the pixel data of the target area 702, as the target correction area to be used for generating a high-resolution image (HR image).

[0107] According to one embodiment, the target region 702, the correction region (e.g., 704, 706, or 708), and the downscaling correction region may satisfy a relationship such as Equation 6. In one example, TA is the target region, CA is the correction region, argmin is a function that outputs the smallest reference value among Euclidean distances of norm 2, A*CA is the downscaling correction region, w is a weighting constant, TV (total variation) is a global deformation factor, A is a function that applies downsampling, blurring, and warping operations, and HCA may correspond to the correction region determined as the correction region most similar to the target region. For example, HCA may correspond to the target correction region.

number

[0108] According to one embodiment, the preceding term in [Equation 6] may correspond to a term for identifying a correction region (e.g., 704, 706, or 708) that is most similar to the target region 702, and the following term in [Equation 6] may correspond to a term that applies weight variables of the correction regions (e.g., 704, 706, or 708) and combines them with the loss function of the preceding term in [Equation 6]. In one example, argmin may correspond to a function that outputs a CA value that is the smallest between a pixel data value at the same pixel position of TA and a pixel data value at the same pixel position of A*CA.

[0109] According to an embodiment, the similarity determination unit 134 may use [Equation 6] to determine one correction region that is most similar to the target region 702 among a plurality of correction regions (e.g., 704, 706, and 708). For example, the plurality of correction regions (e.g., 704, 706, and 708) may correspond to CA in [Equation 6], the target region 702 may correspond to TA in [Equation 6], and the most similar correction region may correspond to HCA in [Equation 6].

[0110] According to one embodiment, the target region 702 and the correction region (e.g., 704, 706, or 708) may satisfy the following relationships: [Equation 7], [Equation 8], and [Equation 9]. In one example, x represents the target region, y represents the correction region, n represents the number of target regions, i represents the x-axis coordinate, j represents the y-axis coordinate, V represents a function that sums up the amount of variation between pixels in the y direction, E represents the average MSE of x and y, and min[] represents a function that outputs a reference value that minimizes the equation in parentheses.

number

number

number

[0111] According to one embodiment, the similarity determination unit 134 may determine one correction area (e.g., 704, 706, and 708) that is most similar to the target area 702 and to which TV (total variation) is applied, using [Equation 7], [Equation 8], and [Equation 9].

[0112] According to one embodiment, the first image generation unit 110 may generate a plurality of low-resolution images (e.g., 710, 720, and 730). When a plurality of reference regions (e.g., 712, 722, and 732) and a plurality of corresponding correction regions (e.g., 704, 706, and 708) are searched based on the plurality of low-resolution images (e.g., 710, 720, and 730), the relationship between the low-resolution images (e.g., 710, 720, and 730) and the high-resolution images may correspond to a relationship of simultaneous equations with an infinite number of solutions. In one example, the operation of the similarity determination unit 134 may correspond to an operation of applying an objective function that finds an optimal solution from an infinite number of solutions by determining an optimal correction region from a number of candidate correction regions (e.g., 704, 706, and 708).

[0113] FIG. 8 is a block diagram showing an example of a computer device that corresponds to the image processing device of FIG.

[0114] Referring to FIG. 8, a computing device 800 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 100 of FIG.

[0115] The computing device 800 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 800 is mounted may be embodied in a single package, for example, a multi-chip package (MCP), although the scope of the present invention is not limited thereto.

[0116] Furthermore, the internal configurations or arrangements of the computer device 800 and the image sensing device may vary depending on the embodiment. For example, at least a portion of the configuration of the image sensing device may be included in the computer device 800. Alternatively, at least a portion of the configuration of the computer device 800 may be included in the image sensing device. In this case, at least a portion of the configuration of the computer device 800 may be mounted on a chip on which the image sensing device is mounted.

[0117] The computing device 800 may include a processor 810 , a memory 820 , an input / output interface 830 , and a communication interface 840 .

[0118] The processor 810 can process data and / or instructions necessary to perform the operations of the components 110 and 120 of the image processing device 100 described in Fig. 1. In other words, the processor 810 can refer to the image processing device 100 itself, although the scope of the present invention is not limited thereto.

[0119] The memory 820 may store data and / or instructions necessary to perform the operations of the components 110, 120, and 130 of the image processing device 100, and may be accessed by the processor 810. For example, the memory 820 may be embodied as a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read only memory (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, etc.).

[0120] That is, a computer program for performing the operations of the image processing device 100 disclosed in this document may be recorded in the memory 820 and executed and processed by the processor 810 to implement the operations of the image processing device 100.

[0121] The input / output interface 830 may provide an interface for connecting an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 810 so that data can be sent and received.

[0122] The communication interface 840 is configured to be capable of transmitting and receiving various data to and from an external device (for example, an application processor, an external memory, etc.), and may be a device capable of supporting wired or wireless communication.

Claims

1. a first image generator that generates a first image and a second image using an input image; a reference area search unit that searches the first image for a first reference area that is most similar to the target area of ​​the input image, and searches the second image for a second reference area that is most similar to the target area; a correction area determination unit that determines a target correction area that is more similar to the target area from among a first correction area located in an area corresponding to the first reference area in the input image and a second correction area located in an area corresponding to the second reference area in the input image; and an image processing device including a second image generating unit that generates a third image using the target correction area;

2. The first image and the second image are an image of lower resolution than the input image; The third image is The image processing apparatus of claim 1 , wherein the input image is a higher resolution image.

3. The first image generation unit applying a first down-scaling to the input image to generate the first image; The image processing apparatus of claim 2 , further comprising: applying a second downscaling to the input image to generate the second image.

4. The first downscaling or the second downscaling is 4. The image processing apparatus of claim 3, further comprising at least one of warping, blurring, or downsampling.

5. The correction area determination unit a first downscaling correction area obtained by applying the first downscaling to the first correction area; a second downscaling correction area obtained by applying the second downscaling to the second correction area, The image processing device of claim 3 , further comprising: a step of comparing the target area to determine the target correction area;

6. The correction area determination unit a difference value between pixel data of the first downscaling correction area and pixel data of the target area; Among the difference values ​​between the pixel data of the second down-scaling correction area and the pixel data of the target area, The image processing apparatus according to claim 5 , wherein the correction area having the smaller difference value is determined as the target correction area.

7. The reference region search unit The image processing apparatus of claim 1 , further comprising: searching for a reference region that is most similar to the target region based on a sum of absolute difference (SAD), normalized cross correlation (NCC), or mean square error (MSE) method.

8. The first correction area is a region located at coordinates obtained by multiplying the first image coordinates of the first reference region by a first coefficient among the coordinates of the input image, The second correction area is The image processing device according to claim 1 , wherein the coordinates of the input image are an area located at coordinates obtained by multiplying second image coordinates of the second reference area by a second coefficient.

9. The first coefficient is a resolution of the input image divided by a resolution of the first image; The second coefficient is The image processing device according to claim 8 , wherein the resolution of the input image is a value obtained by dividing the resolution of the second image.

10. The second coefficient is The image processing device according to claim 9 , wherein the first coefficient is the same value as the second coefficient.

11. The second image generation unit The image processing apparatus of claim 1 , wherein a region of the third image corresponding to the target region is determined as the target correction region.

12. The area corresponding to the target area is: The image processing device of claim 11 , wherein the coordinates of the third image are an area located at coordinates obtained by multiplying the input image coordinates of the target area by a third coefficient.

13. The third coefficient is The image processing apparatus of claim 12 , wherein the resolution of the third image is a value obtained by dividing the resolution of the input image.

14. The third coefficient is The image processing device according to claim 12 , wherein the resolution of the input image is a value obtained by dividing the resolution of the first image or the resolution of the second image.

15. generating a first image and a second image using the input image; searching for a first reference area in the first image that is most similar to a target area of ​​the input image, and searching for a second reference area in the second image that is most similar to the target area; determining a target correction area that is more similar to the target area from among a first correction area located in an area corresponding to the first reference area of ​​the input image and a second correction area located in an area corresponding to the second reference area of ​​the input image; and generating a third image using the target correction area;

16. The step of generating the first image and the second image comprises:

16. The method of claim 15, further comprising applying at least one of warping, blurring, or downsampling to the input image to generate the first image and the second image.

17. The step of determining the target correction area comprises: applying at least one of warping, blurring, or downsampling to the first and second correction regions to generate first and second downscaling correction regions; and 16. The image signal processing method of claim 15, further comprising determining, as the target correction area, a correction area corresponding to a downscaling correction area having a smaller difference value between a difference value between pixel data of the first downscaling correction area and pixel data of the target area and a difference value between pixel data of the second downscaling correction area and pixel data of the target area.

18. The step of generating the third image comprises: The image signal processing method of claim 15, further comprising determining a region of the third image corresponding to the target region as the target correction region.

19. a first image generator for generating a low-resolution image using an input image; a reference region search unit that searches the low-resolution image for a reference region similar to the target region of the input image; a similarity determination unit that determines similarity between the target area and a correction area located at coordinates obtained by multiplying the low-resolution image coordinates of the reference area by a scaling coefficient among the coordinates of the input image; and a second image generator configured to generate a high-resolution image using the correction region based on the similarity between the correction region and the target region.

20. The second image generation unit 20. The image processing device of claim 19, wherein the high-resolution image is generated if a difference value between pixel data of at least one pixel included in an area obtained by downscaling the correction area and pixel data of at least one pixel included in the target area is equal to or less than a predetermined threshold value.

Images