Method and apparatus for improving clarity of image obtained by compound eye camera

The image generation device improves compound eye camera image quality by merging, equalizing, and filtering images to enhance clarity and resolution, addressing the reliance on GPU resources.

KR103043750B1Active Publication Date: 2026-11-06REPUBLIC OF KOREADEFENSE ACQUISITION PROGRAM ADMINISTATION
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020240003384
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-01-09
Publication Date
2026-11-06
Estimated Expiration
2044-01-09

AI Technical Summary

Technical Problem

Existing compound eye cameras produce images with relatively low resolution and quality, and existing AI-based enhancement methods rely heavily on GPU computing resources, limiting their effectiveness without sufficient resources.

Method used

An image generation device and method that includes a merging unit to combine images based on clarity, an equalization unit to separate and equalize RGB channels, and a correction unit to apply filters like median and USM filters to improve image quality.

Benefits of technology

Enhances image clarity and quality without relying on GPUs, making it suitable for systems with limited computing resources, such as MCU-equipped systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024002933917-PAT00001_ABST
    Figure 112024002933917-PAT00001_ABST
Patent Text Reader

Abstract

According to one embodiment of the present invention, an image generation device based on a compound eye image camera may include a merging unit configured to generate a merged image by merging a plurality of images acquired by the compound eye image camera, a smoothing unit configured to separate the merged image into a red channel, a green channel, and a blue channel and smooth the image for each channel, and a correction unit configured to re-merge the smoothed images for each channel and apply at least one filter to the re-merged image to correct the re-merged image.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a compound eye imaging camera, and more specifically, to a method and apparatus for improving the clarity of an image obtained by a compound eye imaging camera. Background Technology

[0002] A compound eye camera is a camera that mimics the compound eye found in ecosystems, such as in dragonflies or spiders, where multiple eyes are clustered together to appear as a single eye. A compound eye camera is configured by arranging multiple lenses (or lens arrays) on a single image sensor and simultaneously capturing sub-aperture images corresponding to the number of lenses.

[0003] The advantages of reconstructed imaging cameras include the ability to acquire depth information by utilizing the disparity between sub-apertures, the capability for high-resolution image reconstruction by superimposing sub-aperture images, and the ability to secure a wide field of view. However, the individual images formed through the sub-apertures of the reconstructed imaging camera have relatively lower resolution and quality. Consequently, while much research has recently been conducted on AI-based image enhancement algorithms to improve the quality of reconstructed images, this presents a problem in that image enhancement performance and processing speed are dependent on the performance of the GPU (Graphic Processing Unit) in terms of computing resources. Therefore, there is a need for a method to improve the quality of images acquired by reconstructed imaging cameras even without a GPU. Prior art literature

[0004] Republic of Korea Published Patent Application No. 10-2017-0032227 (Date of publication: March 22, 2017) The problem to be solved

[0005] The technical problem of the present invention is to provide a method and apparatus capable of generating a clear image based on an image acquired by a compound eye camera.

[0006] Another technical objective of the present invention is to provide a method and apparatus that can improve the quality of images acquired by a compound eye camera even with limited computing resources.

[0007] The problems that the present invention aims to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0008] According to one embodiment of the present invention, an image generation device based on a compound eye camera may include a merging unit configured to generate a merged image by merging a plurality of images acquired by the compound eye camera, an equalization unit configured to separate the merged image into a red channel, a green channel, and a blue channel and equalize the image for each channel, and a correction unit configured to re-merge the equalized images for each channel and apply at least one filter to the re-merged image to correct the re-merged image.

[0009] Here, the merging unit can select a first image based on the clarity of each of the plurality of images and merge the plurality of images by adding a weight to each of the images based on the clarity of the first image.

[0010] In addition, the smoothing unit can smooth the channel-specific image based on histograms for each component of the red channel, the green channel, and the blue channel.

[0011] In addition, the correction unit can filter the re-merged image with a median filter to remove noise from the re-merged image, and apply a USM (UnSharp Masking) filter to the noise-removed image to correct the noise-removed image.

[0012] According to another embodiment of the present invention, an image generation method using an image generation device based on a compound eye camera may include the steps of: merging a plurality of images acquired by the compound eye camera to generate a merged image; separating the merged image into a red channel, a green channel, and a blue channel and smoothing the image for each channel; and re-merging the smoothed images for each channel and applying at least one filter to the re-merged image to correct the re-merged image.

[0013] According to another embodiment of the present invention, a computer program stored in a computer-readable recording medium may include instructions for the processor to perform an image generation method, which, when executed by the processor, comprises the steps of: merging a plurality of images acquired by a compound eye camera to generate a merged image; separating the merged image into a red channel, a green channel, and a blue channel and smoothing the image for each channel; and re-merging the smoothed images for each channel and applying at least one filter to the re-merged image to correct the re-merged image. Effects of the invention

[0014] According to one embodiment of the present invention, since a clear image can be generated from an image acquired by a compound eye camera without a Graphic Processing Unit (GPU), image quality can be improved at a low cost, and it can also be applied to a system equipped with a Micro Controller Unit (MCU). Brief explanation of the drawing

[0015] FIG. 1 is a drawing showing an image generation device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating a process for improving the quality of an image obtained by a compound eye camera according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an image generation method according to an embodiment of the present invention. Specific details for implementing the invention

[0016] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Accordingly, in some embodiments, well-known processing steps, well-known device structures, and well-known techniques are not specifically described to avoid the present invention being interpreted ambiguously.

[0017] The terms used in this specification have been selected from among currently widely used general terms while considering their functions in the present invention; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this specification should be interpreted not merely by their names, but based on their meanings and the overall content of the present invention.

[0018] Throughout this specification, when a part is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0019] Additionally, in this specification, the components of an image generation device may refer to software or hardware components such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and perform at least one function or operation. However, the components are not limited to software or hardware. The components may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, by example, the components include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided by the components of the present invention may be combined into a smaller number of components or further separated into additional components.

[0020] Additionally, while terms such as first, second, third, etc. may be used in this specification to describe various components, these components are not limited by said terms. These terms are used for the purpose of distinguishing one component from other components.

[0021] Hereinafter, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. In addition, parts unrelated to the description are omitted in the drawings to clearly explain the invention. Throughout the specification, the same reference numerals refer to the same components.

[0022] FIG. 1 is a drawing showing an image generation device according to one embodiment of the present invention.

[0023] Referring to FIG. 1, an image generating device (100) according to one embodiment of the present invention may include a merging unit (110), a smoothing unit (120), and a correction unit (130).

[0024] The merging unit (110) may be configured to generate a merged image by merging or fusing multiple images acquired by a compound eye camera. In the following description, the image acquired by the compound eye camera may be referred to as a sub-aperture image or an original image.

[0025] The merging unit (110) may, for example, select a first image based on the clarity of each of a plurality of original images, add weights to each of the plurality of original images based on the clarity of the first image, and then merge or fuse the original images using a preset image fusion algorithm according to the weights. Here, the first image may be the image with the highest clarity among the original images.

[0026] The equalization unit (120) may be configured to separate the image merged by the merging unit (110) into a red channel, a green channel, and a blue channel, and to equalize the image for each channel. For this purpose, as an example, the equalization unit (120) may equalize the image for each channel based on a histogram for each component of the red channel, the green channel, and the blue channel.

[0027] The correction unit (130) may be configured to re-merge each channel image smoothed by the smoothing unit (120) and to correct the re-merged image by applying at least one filter to the re-merged image. For example, the correction unit (130) may remove noise from the image by filtering the re-merged image using a median filter. Additionally, the correction unit (130) may improve the sharpness (quality or image quality) of the image by applying a USM (UnSharp Masking) filter to the image from which noise has been removed.

[0028] An image generation device (100) according to one embodiment of the present invention may be included in a compound eye image camera or in a system including a compound eye image sensor, and may be implemented as a separate device for compound eye image processing as needed.

[0029] FIG. 2 is a diagram illustrating a process for improving the quality of an image obtained by a compound eye camera according to an embodiment of the present invention.

[0030] Hereinafter, with reference to FIG. 2, the process of improving the clarity of an image obtained by a compound eye camera using an image generating device according to one embodiment of the present invention will be described in more detail.

[0031] Compound image cameras generally feature apertures of various sizes to enable shooting even when light is saturated or insufficient. Therefore, since the dynamic ranges of sub-aperture images acquired by compound image cameras differ depending on the amount of light, weighted merging is required before image enhancement.

[0032] For example, if the compound eye imaging camera includes a 3x3 lens array, nine sub-aperture images (200) with different sharpness and pixel information can be obtained by the compound eye imaging camera as shown in FIG. 3. Accordingly, the image generating device can select the image with the highest sharpness among the nine sub-aperture images (200), multiply each sub-aperture image by a different weight based on this, and then merge or fuse the images into one image (210). The weight can be derived, for example, based on the contrast, saturation, and exposedness of each sub-aperture image.

[0033] Subsequently, the image generating device separates the merged image (210) by R (Red), G (Green), and B (Blue) channels to derive channel-specific images (220), and can obtain histograms for each RGB component by verifying the channel-specific images (220). Then, by equalizing each channel-specific image (220) based on the RGB component histograms, a channel-specific image (230) with improved clarity can be derived. To this end, for example, the image generating device may apply CLAHE (Contrast Limited Adaptive Histogram Equalization) to each channel-specific image (220).

[0034] Additionally, the image generation device can generate a re-merged image (240) by re-merging images (230) separated by channel, and can generate an image (250) from which impulse noise or shot noise has been removed by applying a median filter to the image. Furthermore, the image generation device can generate a result image (260) by filtering with a USM filter to emphasize the edge portions of the noise-removed image (250). The USM is a filter that makes the edge portions of the image stand out and emphasizes the overall contrast of the image, and its performance varies depending on the parameters of the unsharp mask, which is the difference between the input image and the blurred image. Therefore, the image generation device can obtain a result image (260) with improved sharpness by multiplying the parameter values ​​derived through iterative simulation with the unsharp mask and adding this to the noise-removed image (250).

[0035] FIG. 3 is a diagram illustrating an image generation method according to an embodiment of the present invention.

[0036] Referring to FIG. 3, an image generation device according to one embodiment of the present invention can generate a merged image by merging a plurality of images acquired by a compound eye camera into a single image (S300). To do this, for example, the image generation device can measure the clarity of each of the plurality of images and select the image with the highest clarity. Then, the image generation device can calculate a weight to be added to each of the remaining images based on the selected image, and merge the plurality of images accordingly.

[0037] Subsequently, the image generation device separates the merged image into a red channel, a green channel, and a blue channel, and then equalizes the image for each channel using a histogram equalization technique such as CLAHE (S310). Then, the image generation device re-merges the equalized images for each channel and then corrects the re-merged image using at least one filter (S320). For example, the image generation device can remove noise from the image by filtering the re-merged image with a noise removal filter such as a median filter, and can improve the sharpness of the image by filtering the noise-removed image with a USM filter.

[0038] Meanwhile, each step included in the image generation method by the image generation device according to the above-described embodiment may be implemented as a computer program including instructions for a processor to perform such steps.

[0039] In addition, each step included in the image generation method by the image generation device according to the above-described embodiment may be implemented on a computer-readable recording medium having a computer program recorded thereon that includes instructions for a processor to perform such steps.

[0040] Combinations of each step of the flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in each step of the flowchart. Since these computer program instructions may also be stored in a computer-available or computer-readable recording medium that can be oriented toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in the computer-available or computer-readable recording medium may also produce a manufactured item containing instruction means for performing the function described in each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each step of the flowchart.

[0041] Additionally, each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Furthermore, in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps described in succession may be performed substantially simultaneously, or the steps may be performed in reverse order depending on the corresponding function.

[0042] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0043] 100: Image generation device 110: Merging section 120: Smoothing part 130: Correction section

Claims

Claim 1 An image generation device based on a compound eye camera, comprising: a merging unit configured to generate a merged image by merging a plurality of images acquired by the compound eye camera; and an equalization unit configured to separate the merged image into a red channel, a green channel, and a blue channel, and equalize the image for each channel. The correction unit is configured to re-merge the smoothed images for each channel and apply at least one filter to the re-merged images to correct the re-merged images. The merging unit is configured to generate the merged image by selecting the image with the highest clarity among the plurality of images as the first image based on the clarity of each of the plurality of images, adding weights to each of the plurality of images based on the clarity of the first image, and merging the plurality of images using a preset image fusion algorithm according to the weights. The weights are derived based on the contrast, saturation, and exposure of each of the plurality of images. The smoothing unit is configured to smooth the images for each channel based on histograms for the components of the red channel, the green channel, and the blue channel. The correction unit is configured to remove noise from the re-merged images by filtering the re-merged images with a median filter and to apply a USM (UnSharp Masking) filter to the noise-removed images. An image generation device characterized by correcting a noise-removed image. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A method for generating an image using an image generation device based on a compound eye camera comprises: a step of generating a merged image by merging a plurality of images acquired by the compound eye camera; a step of separating the merged image into a red channel, a green channel, and a blue channel and equalizing the image for each channel; and a step of re-merging the equalized images for each channel and correcting the re-merged image by applying at least one filter to the re-merged image, wherein the merging step comprises a step of selecting the image with the highest clarity among the plurality of images as a first image based on the clarity of each of the plurality of images. The image generation method comprises the step of generating a merged image by adding weights to each of the plurality of images based on the sharpness of the first image and merging the plurality of images using a preset image fusion algorithm according to the weights, wherein the weights are derived based on contrast, saturation, and exposure for each of the plurality of images, and the smoothing step comprises the step of smoothing the channel-specific images based on histograms for each component of the red channel, the green channel, and the blue channel, and the correction step comprises the step of removing noise from the re-merged image by filtering the re-merged image with a median filter; and the step of applying a USM (UnSharp Masking) filter to the noise-removed image. Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 A computer program stored on a computer-readable recording medium, wherein the computer program includes instructions for the processor to perform an image generation method comprising the steps of: merging a plurality of images acquired by a compound eye camera to generate a merged image when executed by a processor; separating the merged image into a red channel, a green channel, and a blue channel and equalizing the image for each channel; and re-merging the equalized images for each channel and applying at least one filter to the re-merged image to correct the re-merged image, wherein the step of generating the merged image comprises the step of selecting the image with the highest clarity among the plurality of images as a first image based on the clarity of each of the plurality of images; A computer program comprising the step of generating a merged image by adding weights to each of the plurality of images based on the sharpness of the first image and merging the plurality of images using a preset image fusion algorithm according to the weights, wherein the weights are derived based on contrast, saturation, and exposedness for each of the plurality of images, and the smoothing step comprises the step of smoothing the channel-specific images based on histograms for each component of the red channel, the green channel, and the blue channel, and the correction step comprises the step of removing noise from the re-merged image by filtering the re-merged image with a median filter; and the step of correcting the noise-removed image by applying a USM (UnSharp Masking) filter to the noise-removed image.

Citation Information

Patent Citations

  • Apparatus and method for enhancing image quality of image captured by using multiple color-filter aperture

    KR1020120039352A

  • Method and Apparatus for Processing Image to Simultaneously Enhance Contrast and Saturation of Image

    KR1020180057365A

  • Artificial intelligence-based method and appratus for automatically cerating digital image

    KR1020230090887A

  • Binocular pedestrian detection system having dual-stream deep learning neural network and the methods of using the same

    KR102182356B1