Apparatus and method for efficient color re-interpolation based on kernel learning

Abstract

The present invention relates to a technology for providing an efficient remosaicing algorithm while maintaining performance, and an apparatus for color re-interpolation based on kernel learning according to an embodiment may comprise: a feature map extraction unit for extracting a feature map from an input image; a kernel learning-based interpolation unit for generating, by applying features extracted through the extracted feature map, a local intermediate Bayer image and a non-local intermediate Bayer image through local interpolation or non-local interpolation from the input image; and a spatial weighted synthesis unit which learns a spatial weight map and generates a reconstructed Bayer image by synthesizing the generated local intermediate Bayer image and non-local intermediate Bayer image.

Description

Efficient color reinterpolation device and method based on kernel learning

[0001] The present invention relates to an efficient color reinterpolation technique based on kernel learning, and more particularly, to a technique for providing an efficient remosaicing algorithm while maintaining performance.

[0002] Despite recent advances in digital imaging systems, obtaining high-quality images in low-light environments remains a challenge due to the limited sensitivity of the camera's Bayer color filter array (CFA).

[0003] To overcome these limitations, a CFA pattern, namely RGBW CFA, was developed that includes a white (W) pixel in the existing RGB pixel to increase the amount of light reaching the pixel and thus increase the transmittance of visible light.

[0004] Since W pixels exhibit higher light efficiency than RGB pixels, the signal-to-noise ratio of images captured with RGBW CFA is increased, improving image quality in low-light environments.

[0005] However, despite the advantage of higher luminous efficiency of RGBW CFA compared to Bayer CFA, the image signal processor (ISP) for color interpolation was developed and optimized for Bayer CFA.

[0006] To improve the practicality of RGBW CFA, various demosaicing algorithms have been developed to reconstruct full-color images from RGBW images.

[0007] A straightforward approach to RGBW CFA is to develop an RGBW demosaicing algorithm. However, RGBW demosaicing is more challenging than Bayer demosaicing. This is because using W pixels without color information reduces the spatial resolution of the RGB information, making it more difficult to reconstruct the color information at each pixel location.

[0008] Nonetheless, many RGBW demosaicing algorithms have been developed due to their practical importance. For example, color restoration techniques for RGBW demosaicing have been developed by exploiting the inter-channel correlation of the RGBW CFA and the statistical characteristics of the image.

[0009] However, despite their superior color restoration performance, these methods require high computational complexity to solve the optimization problem. Another approach to RGBW demosaicing is to transform the color restoration problem into a composite of W and RGB images, solving it as a pansharpending problem. However, this approach relies heavily on the performance of the pansharpening algorithm, making it difficult to deploy on ISPs.

[0010] The present invention aims to use RGBW CFA in commercial cameras.

[0011] The present invention aims to use RGBW CFA in a commercial camera by converting an RGBW image into a Bayer image and then generating a color image using an existing ISP pipeline.

[0012] The present invention aims to provide a remosaicing algorithm based on the characteristics of each CFA by reusing an existing ISP pipeline.

[0013] The present invention aims to develop an efficient remosaicing algorithm while maintaining performance so that it can be used in commercial cameras such as smartphones.

[0014] A kernel learning-based color re-interpolation device according to one embodiment may include a feature map extraction unit that extracts a feature map from an input image, a kernel learning-based interpolation unit that generates a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the features extracted through the extracted feature map, and a spatial weighted synthesis unit that learns a spatial weight map and synthesizes the generated local intermediate Bayer image and the global intermediate Bayer image to generate a reconstructed Bayer image.

[0015] According to one embodiment, the kernel learning-based interpolation unit can generate a local intermediate Bayer image through local convolution (LC) for each local pixel of the input image, and generate a global intermediate Bayer image by synthesizing each non-local pixel of the input image.

[0016] The kernel learning-based interpolator according to one embodiment of the present invention is a local intermediate Bayer image ( ) can be created.

[0017]

[0018] [Mathematical Formula 1]

[0019]

[0020]

[0021] Here, (i, j) represents the local coordinates around (x, y), and the filter coefficients are Normalized as follows

[0022] The kernel learning-based interpolator according to one embodiment is, through [Mathematical Formula 2], the global intermediate Bayer image ( ) can be created.

[0023]

[0024] [Equation 2]

[0025]

[0026]

[0027] W is the coefficient of the estimated filter in the deformable convolution block, is the offset, stands for modulation mask

[0028] According to one embodiment, the feature map extraction unit can extract a feature map for local and global interpolation using a feature map extractor including an encoder and a decoder.

[0029] According to an embodiment, the encoder extracts a four-stage feature map pyramid having 32, 64, 128, and 256 channels from an RGBW image, and the decoder can receive the four-stage feature map as input through skip-connection and generate a feature map.

[0030] In one embodiment, the encoder may use a convolutional layer with a 2Х2 kernel and a stride of 2 for downsampling in the process of extracting a feature map pyramid, and the decoder may use a convolutional layer including a pixel shuffle operator in the process of upsampling.

[0031] An operating method of a color re-interpolation device based on kernel learning according to an embodiment may include a step of extracting a feature map from an input image in a feature map extraction unit, a step of generating a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the extracted feature map in a kernel learning-based interpolation unit, and a step of generating a reconstructed Bayer image by learning a spatial weight map in a spatial weight synthesis unit and synthesizing the generated local intermediate Bayer image and global intermediate Bayer image based on the learned spatial weight map.

[0032] The step of generating the local intermediate Bayer image and the global intermediate Bayer image according to one embodiment may include the step of generating the local intermediate Bayer image through local convolution (LC) for each local pixel of the input image, and the step of generating the global intermediate Bayer image by synthesizing each non-local pixel of the input image.

[0033] According to an embodiment, a simple yet effective RGBW remosaicing algorithm can be provided through kernel learning-based interpolation.

[0034] In one embodiment, RGBW CFA can be used in commercial cameras.

[0035] In one embodiment, to use an RGBW CFA in a commercial camera, an RGBW image can be converted to a Bayer image and then a color image can be generated using a conventional ISP pipeline.

[0036] In one embodiment, the existing ISP pipeline can be reused to provide a remosaicing algorithm based on the characteristics of each CFA.

[0037] In one embodiment, an efficient remosaicing algorithm can be developed while maintaining performance so that it can be used in commercial cameras such as smartphones.

[0038] FIG. 1 is a drawing illustrating an operating device of a color re-interpolation device based on kernel learning according to an embodiment.

[0039] Figure 2 is a diagram illustrating a local interpolation block structure.

[0040] Figure 3 is a drawing explaining the global interpolation block structure.

[0041] Figure 4 is a diagram explaining the structure of a weight learning network.

[0042] Figure 5 is a diagram showing an RGBW image, a demosaiced full-color image, and a ground-truth image.

[0043] FIG. 6 is a diagram illustrating an operation method of a color re-interpolation device based on kernel learning according to an embodiment.

[0044] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0045] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0046] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0047] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0048] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0050]

[0051] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0052] FIG. 1 is a drawing illustrating an operating device (100) of a color re-interpolation device based on kernel learning according to an embodiment.

[0053] An operating device (100) of a color re-interpolation device based on kernel learning according to an embodiment may include a feature map extraction unit (110), a kernel learning-based interpolation unit (120), and a spatial weighted synthesis unit (130).

[0054] The feature map extraction unit (110) can extract a feature map from an input image. For example, the feature map extraction unit (110) can extract a feature map for local and global interpolation using a feature map extractor including an encoder and a decoder.

[0055] One of the operating principles of the feature map extraction unit (110) may utilize a feature map extractor including an encoder and a decoder. This feature map extractor functions to convert an input image into high-dimensional features, which means extracting features by considering both local and global information.

[0056] For example, an encoder compresses an input image into low-dimensional features, and a decoder expands these into high-dimensional features, interpolating the original image's features. The resulting feature maps effectively reflect patterns and features appearing across various parts of the input image, enabling the abstraction and interpretation of visual information.

[0057] The feature map extraction unit (110) can utilize these feature maps for image recognition, object detection, segmentation, and other advanced image processing tasks. Therefore, the system can enhance performance in computer vision and artificial intelligence applications by extracting and processing meaningful information from input images.

[0058] The encoder extracts a four-level feature map pyramid with 32, 64, 128, and 256 channels from an RGBW image, and the decoder can receive the four-level feature map as input through skip-connection and generate a feature map.

[0059] The encoder extracts high-level features from RGBW images (typically red, green, blue, and white channels). It creates a four-level feature map pyramid with 32, 64, 128, and 256 channels. Each level progressively abstracts the input image and extracts features containing varying levels of detail. This hierarchical structure captures various spatial and semantic characteristics of the input image, resulting in a high-level, meaningful representation.

[0060] The decoder takes these four feature maps as input and uses skip connections to generate feature maps. Skip connections represent connections between the encoder and decoder, connecting feature maps from intermediate stages to the feature maps from the immediately preceding and subsequent stages.

[0061] This helps the decoder restore the original image while preserving the details extracted at each stage of the encoder. This structure enables the restoration of the input image while maintaining high-resolution features, making it effective in image reconstruction and other applications.

[0062] Additionally, the encoder may use a convolutional layer with a 2Х2 kernel and a stride of 2 for downsampling during the process of extracting the feature map pyramid, and the decoder may use a convolutional layer with a pixel shuffle operator during upsampling.

[0063] Next, the kernel learning-based interpolation unit (120) can generate a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the extracted features through the extracted feature map.

[0064] Local interpolation refers to generating intermediate Bayer images for localized regions within an input image. It can be used to obtain high-resolution intermediate Bayer images while emphasizing and preserving specific regional features. Local interpolation can contribute to the creation of natural-looking images, for example, by emphasizing object boundaries or detailed structures.

[0065] Global interpolation, on the other hand, involves generating an intermediate Bayer image by considering the entire input image. This can help achieve high-resolution results while maintaining the overall structure and characteristics of the input image. Global interpolation is suitable for interpolating images while maintaining color and texture consistency across a large area.

[0066] The kernel learning-based interpolation unit (120) utilizes these local and global interpolation techniques to make the most of the extracted features, thereby generating an intermediate Bayer image of the input image.

[0067] For example, the kernel learning-based interpolator (120) can generate a local intermediate Bayer image through local convolution (LC) for each local pixel of the input image. In addition, it can generate a global intermediate Bayer image by synthesizing each non-local pixel of the input image.

[0068] The kernel learning-based interpolator (120) uses [Mathematical Formula 1] to calculate the local intermediate Bayer image ( ) can be created.

[0069]

[0070] [Mathematical Formula 1]

[0071]

[0072]

[0073] Here, (i, j) represents the local coordinates around (x, y), and the filter coefficients are It is normalized as follows.

[0074] Meanwhile, the kernel learning-based interpolator (120) uses [Mathematical Formula 2] to calculate the global intermediate Bayer image ( ) can be created.

[0075]

[0076] [Equation 2]

[0077]

[0078]

[0079] In [Mathematical Formula 2], W is the coefficient of the estimated filter in the deformable convolution block, is the offset, stands for modulation mask.

[0080] In addition, the spatial weighted synthesis unit (130) can learn a spatial weight map and synthesize the generated local intermediate Bayer image and global intermediate Bayer image to generate a reconstructed Bayer image.

[0081] A spatial weight map is learned as a map indicating the relative importance of each pixel in the input image. These weights reflect information about various parts of the image and can be used to determine the importance of specific regions or features in the reconstructed Bayer image.

[0082] The local intermediate Bayer image and the global intermediate Bayer image are the results generated by the kernel learning-based interpolation unit (120) mentioned above, and can form intermediate Bayer images by considering local features and global structures, respectively. The spatial weighted synthesis unit (130) calculates the spatial weighted sum of each pixel using these intermediate Bayer images to generate a final reconstructed Bayer image.

[0083] Through this process, the spatial weighted synthesis unit (130) can appropriately combine local and global features of the input image to obtain a high-quality reconstructed Bayer image.

[0084] Figure 1 shows a schematic diagram of the proposed algorithm. First, the input image is RGBW image I. in Feature maps can be extracted from the extracted features through local and non-local interpolation blocks. class is used to obtain. Finally, the spatially weighted fusion module is class Adaptively synthesized to produce the final remosaiced Bayer image I R Synthesizes.

[0085] Figure 1 illustrates an overview of the proposed algorithm, which consists of three modules: feature map extraction, kernel learning-based interpolation, and spatial weighted synthesis module.

[0086] The feature map extraction unit (110) extracts a feature map from the RGBW image, which is the input image, and the kernel learning-based interpolation unit (120) simultaneously learns dynamic weights and offsets through local and global interpolation to generate two intermediate Bayer images. class Finally, the spatial weighted synthesis unit (130) learns the spatial weight map. class and the final reconstructed Bayer image I R Get .

[0087] Figure 2 is a diagram illustrating a local interpolation block structure.

[0088] The feature map extraction unit (110) extracts a feature map for local and global interpolation from an RGBW input image. As illustrated in Fig. 1, the present invention uses a feature map extractor including an encoder and a decoder. In particular, the encoder extracts an RGBW image I in A 4-level feature map pyramid with 32, 64, 128, and 256 channels is extracted from the input. A 2Х2 kernel and a convolutional layer with a stride of 2 are used for downsampling. Then, the decoder receives the 4-level feature map as input through skip connection and generates a feature map. A convolutional layer with a pixel shuffle operator is used for upsampling.

[0089] Since RGBW CFA has a high proportion of W pixels containing useful information in the input, it is essential to effectively extract meaningful information from W pixels for effective remosaicing.

[0090] The present invention provides a local interpolation block that enables a network to dynamically estimate the relative importance of adjacent pixels.

[0091] Figure 2 shows the extracted feature map and RGBW image I inIt takes as input an intermediate Bayer image and adaptively combines it with adjacent pixels. The structure of a local interpolation block that outputs is shown.

[0092] The proposed local interpolation block has filter coefficients It consists of four convolutional layers for estimating , where W and H represent the width and height of the input image, respectively.

[0093] The filter coefficients are generated using three 3Х3 convolutional layers and one 3Х3 convolutional layer using the GELU activation function.

[0094] At this time, the number of channels in the first three convolutional layers is 32, and the number of channels in the last convolutional layer is 9. Therefore, the local interpolation block is an RGBW image I in Adaptively fuse adjacent pixels of size 3Х3 in .

[0095] Specifically I in For each pixel (x, y) of [Mathematical Formula 1], the intermediate Bayer image is obtained through local convolution (LC) as follows. Get .

[0096]

[0097] [Mathematical Formula 1]

[0098]

[0099]

[0100] Here, (i, j) represents the local coordinates around (x, y), and the filter coefficients are It is normalized as follows.

[0101] Figure 3 is a drawing explaining the global interpolation block structure.

[0102] The RGBW CFA contains more W pixels than R, G, and B pixels, and more G pixels than R and B pixels. Therefore, interpolation using a fixed kernel may fail to accurately estimate the corresponding pixel due to insufficient surrounding color information.

[0103] To compensate for this, we develop a global interpolation algorithm based on deformable convolution, which allows the network to use pixels outside the rectangular grid of the kernel window.

[0104] Figure 3 illustrates a global interpolation block according to one embodiment, which consists of three branches having the same structure.

[0105] Each branch estimates a different element for the deformable convolution. The first branch estimates the global coefficients for the input pixels. , and the second branch estimates the 2D offset for every pixel. and learn here and represent horizontal and vertical offsets respectively.

[0106] This offset allows global interpolation by resampling pixels outside the rectangular window into the convolution.

[0107] Finally, the third branch is a modulation mask that allows the network to focus on more important pixels. improves the performance of deformable convolution by learning. In order to obtain a global kernel, global average pooling (GAP) is used in the first branch, and the sigmoid function is used in the third branch. Limits values ​​to the range [0, 1].

[0108] Similar to the local interpolation block, the number of channels in the first three convolutional layers of each branch is 32. Then, the deformable convolution block of Fig. 3 calculates the coefficients W of the estimated filter and the offset and modulation mask Using I in Adaptively synthesize the global pixels of the intermediate Bayer image at pixel location (x, y) as follows: Creates.

[0109]

[0110] [Equation 2]

[0111]

[0112]

[0113] Two intermediate images in Fig. 1 class are interpolated using local and global interpolation blocks, respectively. Since the two images are interpolated independently using different blocks, they have different features with complementary information. Therefore, the two images class The final Bayer image I is combined to effectively preserve complementary information. R Synthesizes.

[0114] For this purpose class Dynamic filter coefficients k that determine the adaptive weights between fuse We develop a spatial weighted synthesis module that estimates .

[0115] Figure 4 is a diagram explaining the structure of a weight learning network.

[0116] Figure 4 illustrates the structure of the weight learning network. It includes seven 3X3 convolutional layers with symmetric skip connections, at each pixel location (x, y). , where d represents the size of the kernel, and dxd adjacent pixels of each image are synthesized.

[0117] Each convolutional layer except the last uses the GELU activation function. The first six convolutional layers have 32 channels, and the last convolutional layer has 2 channels. Finally, the synthesized Bayer image I R It can be obtained through LC as shown in [Mathematical Formula 3] below.

[0118]

[0119] [Equation 3]

[0120]

[0121]

[0122] Here (i, j) is Represents the relative coordinates around (x, y), and the coefficients are is normalized to .

[0123] To train the proposed algorithm, the cost function (Loss Functions) L total As in [Equation 4], Bayer loss L Bayer , reconstruction loss L r , SSIM loss L SSIM , perceptual loss L per It is defined as the weighted sum of .

[0124]

[0125] [Equation 4]

[0126]

[0127]

[0128] In [Equation 4] is a hyperparameter to control the balance of the four losses.

[0129] Bayer loss L Bayer is a reconstructed Bayer image I Rand ground-truth video I gt The difference between them is measured as in [Mathematical Formula 5].

[0130]

[0131] [Equation 5]

[0132]

[0133]

[0134] Estimated Bayer Image I R Since the color images obtained by demosaicing must be color interpolated for use in real applications, the loss is calculated using the color images obtained by demosaicing.

[0135] For this, I as in [Equation 6] R Wow I gt Obtain color images for each.

[0136]

[0137] [Equation 6]

[0138]

[0139]

[0140] In [Equation 6], G can be calculated using the existing demosaicing algorithm.

[0141] In [Equation 6], r is set to 2.2, and the L1-norm is calculated as the reconstruction loss as in [Equation 7].

[0142]

[0143] [Equation 7]

[0144]

[0145]

[0146] SSIM loss L SSIM is defined as in [Mathematical Formula 8].

[0147]

[0148] [Equation 8]

[0149]

[0150]

[0151] In [Equation 8], SSIM(.) is a structural similarity index. Finally, the perceptual loss Lper is used to compare high-level differences between the two images, as in [Equation 9].

[0152]

[0153] [Equation 9]

[0154]

[0155]

[0156] LPIPS(.) represents the LPIPS score using pre-trained AlexNet.

[0157] FIG. 5 is a diagram illustrating an RGBW image (510), a demosaiced full-color image (520), and a ground-truth image (530).

[0158] The RGBW image (510) can be interpreted as an input image.

[0159] An RGBW image (510) is an image that includes color channels representing Red, Green, Blue, and White, and in an RGBW image, the white channel provides additional luminance information and is generally used for color reproduction.

[0160] A demosaiced full-color image (520) is a full-color image reconstructed with all color channels using information extracted from a raw Bayer pattern or other color filter array through a demosaicing technique. This represents an image in which color information is fully restored.

[0161] A ground-truth image (530) is an image that represents accurate colors and details observed in real life. The ground-truth image (530) is generally used for experiments or comparisons, and ideally should contain accurate colors and details when compared to other images.

[0162] FIG. 6 is a diagram illustrating an operation method of a color re-interpolation device based on kernel learning according to an embodiment.

[0163] According to an embodiment, a method of operating a color re-interpolation device based on kernel learning can extract a feature map from an input image in a feature map extraction unit (step 601).

[0164] In one embodiment, a method of operating a color re-interpolation device based on kernel learning can generate a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the extracted features through the extracted feature map in the kernel learning-based interpolation unit (step 602).

[0165] In the process of generating the above local intermediate Bayer image and global intermediate Bayer image, a local intermediate Bayer image can be generated through local convolution (LC) for each local pixel of the input image, and a global intermediate Bayer image can be generated by synthesizing each global (non-local) pixel of the input image.

[0166] In one embodiment, a method of operating a color re-interpolation device based on kernel learning can be performed in a spatial weighted synthesis unit, wherein a spatial weight map is learned, and a local intermediate Bayer image and a global intermediate Bayer image generated based on the learned spatial weight map are synthesized to generate a reconstructed Bayer image (step 603).

[0167] Ultimately, the present invention provides a simple yet effective RGBW remosaicing algorithm through kernel learning-based interpolation, and enables the use of RGBW CFA in commercial cameras.

[0168] Furthermore, to use RGBW CFAs in commercial cameras, RGBW images can be converted to Bayer images and then color images can be generated using existing ISP pipelines. This existing ISP pipeline can be reused to provide remosaicing algorithms based on the characteristics of each CFA. Furthermore, an efficient remosaicing algorithm can be developed while maintaining performance for use in commercial cameras such as smartphones.

[0169]

[0170] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0171] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0172] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0173] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0174] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A feature map extraction unit that extracts a feature map from an input image; A kernel learning-based interpolation unit that generates a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the features extracted through the above-mentioned extracted feature map; and A spatial weighted synthesis unit that learns a spatial weight map and synthesizes the generated local intermediate Bayer image and global intermediate Bayer image to generate a reconstructed Bayer image. A color re-interpolation device based on kernel learning, characterized by including:

2. In paragraph 1, The above kernel learning-based interpolator, For each local pixel of the input image above, a local intermediate Bayer image is generated through local convolution (LC), A kernel learning-based color re-interpolation device characterized in that it generates a global intermediate Bayer image by synthesizing each non-local pixel of the input image.

3. In paragraph 2, The above kernel learning-based interpolator, [Mathematical expression 1] The local intermediate Bayer image ( ) is characterized by generating a kernel learning-based color re-interpolation device. [Mathematical Formula 1] Here, (i, j) represents the local coordinates around (x, y), and the filter coefficients are Normalized as follows 4. In paragraph 3, The above kernel learning-based interpolator, Through [Mathematical Formula 2], the global intermediate Bayer image ( ) is characterized by generating a kernel learning-based color re-interpolation device. [Equation 2] W is the coefficient of the estimated filter in the deformable convolution block, is the offset, stands for modulation mask 5. In paragraph 1, The above feature map extraction unit, A kernel learning-based color re-interpolation device characterized by extracting feature maps for local and global interpolation using a feature map extractor including an encoder and a decoder.

6. In paragraph 5, The above encoder extracts a four-level feature map pyramid with 32, 64, 128, and 256 channels from an RGBW image, The above decoder is a kernel learning-based color re-interpolation device characterized in that it receives a four-stage feature map as input through skip-connection and generates a feature map.

7. In paragraph 6, The above encoder uses a 2Х2 kernel and a convolutional layer with a stride of 2 for downsampling in the process of extracting the feature map pyramid. The above decoder is a kernel learning-based color re-interpolator characterized in that it uses a convolutional layer including a pixel shuffle operator during upsampling.

8. In the feature map extraction unit, a step of extracting a feature map from the input image; In a kernel learning-based interpolation unit, a step of generating a local intermediate Bayer image and a global intermediate Bayer image through local interpolation or global (non-local) interpolation from the input image by reflecting the features extracted through the extracted feature map; and In a spatial weighted synthesis unit, a step of learning a spatial weight map and generating a reconstructed Bayer image by synthesizing the generated local intermediate Bayer image and the global intermediate Bayer image based on the learned spatial weight map. A method of operating a color re-interpolation device based on kernel learning, characterized in that it includes:

9. In paragraph 8, The step of generating the above local intermediate Bayer image and global intermediate Bayer image is: A step of generating a local intermediate Bayer image through local convolution (LC) for each local pixel of the input image; A step of generating a global intermediate Bayer image by synthesizing each global (non-local) pixel of the above input image. A method of operating a color re-interpolation device based on kernel learning, characterized in that it includes:

Images