Image processing apparatus and method

Abstract

To provide an image processing apparatus for synthesizing images with different brightness.SOLUTION: An image processing device according to an exemplary embodiment of the present disclosure includes a brightness correction unit configured to generate a third image by selectively replacing a second image with a first image according to a magnitude relationship between pixel values at the same position in the first image and the second image, and an image synthesis unit configured to generate a fourth image by synthesizing the first image and the third image using a brightness value for each color of the first image or the third image and a different threshold value for each color.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical concepts of the present disclosure relate to an apparatus and method for processing images. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields, such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical microcameras.

[0003] An image captured by an image sensing device may differ depending on exposure time, gain, etc., and image distortion may occur due to a flicker phenomenon, etc. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an image processing device that combines images with different brightness levels. An object of the present disclosure is to provide an image processing device that synthesizes an image using different threshold values ​​for each color and different brightness values ​​for each color.

[0005] An object of the present disclosure is to provide an image processing device that corrects brightness by selectively replacing an image with another image. An object of the present disclosure is to provide an image processing device that synthesizes an image using pixel values ​​of another image when pixel values ​​of the image are saturated.

[0006] The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the following description. [Means for solving the problem]

[0007] An image processing device according to an exemplary embodiment of the present disclosure may include a brightness correction unit that generates a third image by selectively replacing a second image with a first image depending on the magnitude relationship between pixel values ​​at the same position in the first image and the second image, and an image synthesis unit that generates a fourth image by synthesizing the first image and the third image using brightness values ​​for each color of the first image or the third image and different thresholds for each color.

[0008] According to one embodiment, the image sensor further includes a gain control unit that generates a first image by correcting a first input image using a first gain and generates a second image by correcting a second input image using a second gain greater than the first gain, and the first input image may be an image captured during a longer exposure time than the second input image.

[0009] According to one embodiment, the image synthesis unit generates a fourth image using the pixel value of the first green pixel when the brightness value of the first green pixel of the first image is less than the first green threshold; generates a fourth image by adding the pixel value of the first green pixel and the pixel value of the second green pixel of the third image in a predetermined ratio when the brightness value of the first green pixel is greater than or equal to the first green threshold and less than the second green threshold; and generates a fourth image using the pixel value of the second green pixel when the brightness value of the first green pixel is greater than or equal to the second green threshold.

[0010] According to one embodiment, the image synthesis unit generates a fourth image using the pixel value of the first red pixel when the brightness value of the first red pixel of the first image is less than the first red threshold; generates a fourth image by adding the pixel value of the first red pixel and the pixel value of the second red pixel of the third image at a predetermined ratio when the brightness value of the first red pixel is greater than or equal to the first red threshold and less than the second red threshold; and generates a fourth image using the pixel value of the second red pixel when the brightness value of the first red pixel is greater than or equal to the second red threshold.

[0011] According to one embodiment, the image synthesis unit generates a fourth image using the pixel value of the first blue pixel when the brightness value of the first blue pixel of the first image is less than the first blue threshold; generates a fourth image by adding the pixel value of the first blue pixel and the pixel value of the second blue pixel of the third image in a predetermined ratio when the brightness value of the first blue pixel is greater than or equal to the first blue threshold and less than the second blue threshold; and generates a fourth image using the pixel value of the second blue pixel when the brightness value of the first blue pixel is greater than or equal to the second blue threshold.

[0012] According to one embodiment, the predetermined ratio may vary linearly as the brightness value of the first green pixel increases from the first green threshold value to the second green threshold value.

[0013] In one embodiment, a first green pixel may be saturated if the brightness value of the first green pixel is equal to the second green threshold value. According to one embodiment, the brightness value of the first green pixel may correspond to the pixel value of the first green pixel.

[0014] According to one embodiment, the first and second green thresholds may be determined at least in part by the spectral sensitivity characteristics of the color filters and a white balance value.

[0015] According to one embodiment, the brightness correction unit may generate a third image by replacing the second pixel value of the second image with the first pixel value if the first pixel value of the first image is greater than or equal to the second pixel value of the second image.

[0016] An image processing device according to an exemplary embodiment of the present disclosure may include an image synthesis unit that generates a third image by synthesizing a pixel value of a first green pixel of a first image with a pixel value of a second green pixel of a second image according to the magnitude relationship among the brightness value of the first green pixel of the first image, a first green threshold, and a second green threshold; that generates a third image by synthesizing a pixel value of a first blue pixel of a first image with a pixel value of a second blue pixel of a second image according to the magnitude relationship among the brightness value of the first blue pixel of the first image, a first blue threshold, and a second blue threshold; and that generates the third image by synthesizing a pixel value of a first red pixel of a first image with a pixel value of a second red pixel of a second image according to the magnitude relationship among the brightness value of the first red pixel of the first image, a first red threshold, and a second red threshold.

[0017] According to one embodiment, the image sensor further includes a gain control unit that generates a first image by correcting a first input image using a first gain and generates a fourth image by correcting a second input image using a second gain greater than the first gain, and the first input image may be an image captured during a longer exposure time than the second input image.

[0018] According to one embodiment, the image processing device may further include a brightness correction unit that generates a second image by selectively replacing the fourth image with the first image depending on the magnitude relationship between pixel values ​​at the same position in the first image and the fourth image.

[0019] According to one embodiment, the image synthesis unit generates a third image using the pixel value of the first green pixel when the brightness value of the second green pixel is smaller than the first green threshold; generates a third image by adding the pixel value of the first green pixel and the pixel value of the second green pixel in a predetermined ratio when the brightness value of the second green pixel is greater than or equal to the first green threshold and smaller than the second green threshold; and generates a third image using the pixel value of the second green pixel when the brightness value of the second green pixel is greater than or equal to the second green threshold.

[0020] According to one embodiment, the image synthesis unit generates a third image using the pixel value of the first blue pixel when the brightness value of the second blue pixel is smaller than the first blue threshold; generates a third image by adding the pixel value of the first blue pixel and the pixel value of the second blue pixel in a predetermined ratio when the brightness value of the second blue pixel is greater than or equal to the first blue threshold and less than the second blue threshold; and generates a third image using the pixel value of the second blue pixel when the brightness value of the second blue pixel is greater than or equal to the second blue threshold.

[0021] According to one embodiment, the image synthesis unit generates a third image using the pixel value of the first red pixel when the brightness value of the second red pixel is smaller than the first red threshold; generates a third image by adding the pixel value of the first red pixel and the pixel value of the second red pixel at a predetermined ratio when the brightness value of the second red pixel is equal to or greater than the first red threshold and smaller than the second red threshold; and generates a third image using the pixel value of the second red pixel when the brightness value of the second red pixel is equal to or greater than the second red threshold.

[0022] In one embodiment, a first green pixel may be saturated if the brightness value of the first green pixel is equal to the second green threshold value. According to one embodiment, the first and second green thresholds may be determined at least in part by the spectral sensitivity characteristics of the color filters and a white balance value. According to one embodiment, the brightness value of the first green pixel may correspond to the pixel value of the first green pixel.

[0023] An image processing method according to an exemplary embodiment of the present disclosure may include the steps of: generating a third image by replacing a second pixel value of a second image with a first pixel value according to the magnitude relationship between a first pixel value of the first image and a second pixel value of a second image; and generating a fourth image by combining the first image and the third image using brightness values ​​for each color of the first image or the third image and different thresholds for each color. The above briefly summarized features of the present disclosure are illustrative aspects of the detailed description of the present disclosure that follows and are not intended to limit the scope of the present disclosure. [Effects of the Invention]

[0024] The image processing device according to the exemplary embodiment of the present disclosure can combine images with different brightness levels. The image processing device according to the exemplary embodiment of the present disclosure can synthesize an image using different threshold values ​​for each color and different brightness values ​​for each color.

[0025] An image processing device according to an exemplary embodiment of the present disclosure can correct brightness by selectively replacing an image with another image. The image processing device according to the exemplary embodiment of the present disclosure can synthesize an image using pixel values ​​of other images when pixel values ​​of an image are saturated.

[0026] The effects obtained by the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the following description. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3a] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3b] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 5] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 6a] FIG. 10 is a diagram for explaining an image processing method according to a comparative example. [Figure 6b] FIG. 10 is a diagram for explaining an image processing method according to a comparative example. [Figure 6c] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 7] 2 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, with reference to the accompanying drawings, detailed descriptions of embodiments of the present disclosure will be given so that those skilled in the art can easily implement the present disclosure, however, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0029] In describing embodiments of the present disclosure, if a detailed description of known configurations or functions is deemed to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, in the drawings, parts that are not related to the description of the present disclosure will be omitted, and similar parts will be designated by similar reference numerals.

[0030] In this disclosure, when a component is "coupled," "coupled," or "connected" to another component, it means not only a direct connection, but also an indirect connection where there is another component between them. Furthermore, when a component "includes" or "has" another component, it does not mean that the other component is excluded, and that the component may further include the other component, unless otherwise specified.

[0031] In this disclosure, terms such as first and second are used only to distinguish one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0032] In this disclosure, components that are distinguished from one another are used to clearly describe the characteristics of each component and do not necessarily mean that the components are separate. That is, multiple components may be integrated and consist of a single hardware or software unit, or a single component may be distributed and consist of multiple hardware or software units. Therefore, even if not otherwise specified, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0033] In this disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of this disclosure. Furthermore, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of this disclosure.

[0034] In this disclosure, expressions of positional relationships used in this specification, such as upper, lower, left side, right side, etc., are described for the convenience of explanation, and when the drawings shown in this specification are viewed upside down, the positional relationships described in this specification may be interpreted in reverse.

[0035] In this disclosure, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" may include any one or all possible combinations of the items listed along with the corresponding phrase.

[0036] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to FIGS. FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure.

[0037] Referring to FIG. 1, an image processing device 100 can perform at least one image signal processing on image data IDATA to generate processed image data IDATA_P.

[0038] The image processing device 100 may perform image signal processing on the image data IDATA to reduce noise and improve image quality, such as demosaicing, defective pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction. The image processing device 100 may also compress image data that has undergone image signal processing for improving image quality to generate an image file, or restore image data from the image file. The image compression format may be a lossless format or a lossy format. For example, the Joint Photographic Experts Group (JPEG) format or JPEG2000 format may be used for still images. For moving images, a moving image file may be generated by compressing multiple frames according to the Moving Picture Experts Group (MPEG) standard.

[0039] The image data IDATA may be generated by an image sensing device that captures an optical image of a scene, although the scope of the present invention is not limited thereto. The image sensing device may include a pixel array including a plurality of pixels for sensing light incident from a scene, a control circuit for controlling the pixel array, and a readout circuit that converts analog pixel signals received from the pixel array into digital image data IDATA and outputs the digital image data. In this disclosure, the image data IDATA will be described assuming that it is generated by the image sensing device.

[0040] The pixel array may include a color filter array (CFA) in which color filters are arranged according to a certain pattern (e.g., a Bayer pattern, a quad Bayer pattern, a nona Bayer pattern, an RGBW pattern, etc.) so that each pixel can sense light of a predetermined wavelength band. The pattern of the image data IDATA may be determined according to the type of pattern the CFA has.

[0041] The image processing device 100 may be, but is not limited to, a computing device mounted on a chip separate from the chip on which the image sensing device is mounted. The chip on which the image sensing device is mounted and the chip on which the image processing device 100 is mounted may communicate with each other via a predetermined interface. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the image processing device 100 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), but the scope of the present invention is not limited thereto.

[0042] The image processing device 100 may include a gain control unit 110, a brightness correction unit 120, and / or an image synthesis unit . The gain control unit 110 may control a gain to be applied to an input image. Specifically, the gain control unit 110 may determine a gain based on the exposure time applied when the input image was captured and apply the gain to the input image. For example, the gain control unit 110 may generate a first image by correcting a first input image using a first gain. The gain control unit 110 may also generate a second image by correcting a second input image using a second gain greater than the first gain. In this case, the first input image may be an image captured with a longer exposure time than the second input image. That is, the first input image may be an image brighter than the second input image. If the first input image includes a signal in a saturated region due to a long exposure time, a signal in a region brighter than the saturated brightness may be obtained by applying the second gain to the second input image, which is darker than the first input image. A more detailed description of applying a gain will be provided below.

[0043] The luminance correction unit 120 may correct the luminance of an image. Specifically, the luminance correction unit 120 may correct the luminance of an image by replacing pixel values ​​of one image with pixel values ​​of another image. For example, the luminance correction unit 120 may generate a third image by selectively replacing a second image with the first image depending on the magnitude relationship between pixel values ​​at the same position in a first image and a second image. If pixel values ​​at the same position in a first image and a second image are a first pixel value and a second pixel value, respectively, the luminance correction unit 120 may generate a third image by replacing the second pixel value of the second image with the first pixel value if the first pixel value of the first image is equal to or greater than the second pixel value of the second image. More specific details regarding the luminance correction unit 120 will be described later.

[0044] The image synthesis unit 130 can synthesize input images. Specifically, the image synthesis unit 130 can synthesize images based on the brightness values ​​of the input images. For example, the image synthesis unit 130 can generate a fourth image by synthesizing the first image and the third image using brightness values ​​for each color of the first image or the third image and different thresholds for each color. Specifically, the image synthesis unit 130 can generate a fourth image by synthesizing the pixel value of the first green pixel of the first image with the pixel value of the second green pixel of the third image according to the magnitude relationship between the brightness value of the first green pixel of the first image, the first green threshold, and the second green threshold. Specifically, if the brightness value of the first green pixel of the first image is smaller than the first green threshold, the image synthesis unit 130 can generate the fourth image using the pixel value of the first green pixel. Furthermore, if the brightness value of the first green pixel is equal to or greater than the first green threshold and less than the second green threshold, the image synthesis unit 130 may generate a fourth image by adding the pixel value of the first green pixel and the pixel value of the second green pixel of the third image at a predetermined ratio. Furthermore, if the brightness value of the first green pixel is equal to or greater than the second green threshold, the image synthesis unit 130 may generate a fourth image using the pixel value of the second green pixel.

[0045] The image synthesis unit 130 may generate a fourth image by synthesizing the pixel value of the first red pixel of the first image with the pixel value of the second red pixel of the third image according to the magnitude relationship between the brightness value of the first red pixel of the first image, the first red threshold, and the second red threshold. Specifically, the image synthesis unit 130 may generate the fourth image using the pixel value of the first red pixel when the brightness value of the first red pixel of the first image is smaller than the first red threshold. Furthermore, the image synthesis unit 130 may generate the fourth image by adding the pixel value of the first red pixel and the pixel value of the second red pixel of the third image at a predetermined ratio when the brightness value of the first red pixel is equal to or greater than the first red threshold and smaller than the second red threshold. Furthermore, the image synthesis unit 130 may generate the fourth image using the pixel value of the second red pixel when the brightness value of the first red pixel is equal to or greater than the second red threshold.

[0046] The image synthesis unit 130 may generate a fourth image by synthesizing the pixel value of the first blue pixel of the first image with the pixel value of the second blue pixel of the third image according to the magnitude relationship between the brightness value of the first blue pixel, the first blue threshold, and the second blue threshold. Specifically, if the brightness value of the first blue pixel of the first image is less than the first blue threshold, the image synthesis unit 130 may generate the fourth image using the pixel value of the first blue pixel. Also, if the brightness value of the first blue pixel is equal to or greater than the first blue threshold and less than the second blue threshold, the image synthesis unit 130 may generate the fourth image by adding the pixel value of the first blue pixel and the pixel value of the second blue pixel of the third image at a predetermined ratio. Also, if the brightness value of the first blue pixel is equal to or greater than the second blue threshold, the image synthesis unit 130 may generate the fourth image using the pixel value of the second blue pixel. More detailed information about synthesizing images will be described later.

[0047] FIG. 2 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. FIG. 3a is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. FIG. 3b is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure.

[0048] FIG. 2 will now be described with reference to FIGS. 3a and 3b. Referring to FIG. 2, an image processing method according to an exemplary embodiment of the present disclosure can correct an image by applying a gain to the image, correct the brightness of the image, and combine the images.

[0049] For example, a first image may be generated by applying a first gain to a first input image, and a second image may be generated by applying a second gain to a second input image.

[0050] 3a, the second input image may be darker than the first input image. For example, the second input image may be captured with a shorter exposure time than the first input image. The first gain may be greater than the second gain. By applying the second gain to the second input image, data regarding pixel values ​​greater than saturated pixel values ​​may be obtained through the second image, which is the gain-corrected second input image.

[0051] The second image may be subjected to brightness correction using the first image. For example, the brightness of the second image may be corrected by selectively replacing the second image with the first image depending on the magnitude relationship between pixel values ​​at the same position in the first image and the second image. More details regarding brightness correction will be described later.

[0052] A composite image can be generated by combining the first image and the third image. A combination parameter can be used to generate the composite image. The combination parameter may be a threshold value used in the combination process. Specifically, the combination parameter may be a first threshold Th1 and a second threshold Th2 for brightness values. For example, if the brightness of a pixel at coordinate (i,j) in the first image is smaller than the first threshold Th1, the pixel value of the pixel at coordinate (i,j) in the composite image may be the pixel value of the pixel at coordinate (i,j) in the first image. The brightness value of the pixel at coordinate (i,j) in the first image may correspond to the pixel value of the pixel at coordinate (i,j) in the first image. For example, the brightness value of the pixel at coordinate (i,j) in the first image may be, but is not limited to, the pixel value of the pixel at coordinate (i,j) in the first image. Furthermore, if the brightness of the pixel at coordinate (i,j) in the first image is equal to or greater than the first threshold Th1 and less than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) in the composite image may be (1-α)*(pixel value at coordinate (i,j) in the first image)+α*(pixel value at coordinate (i,j) in the third image). If the brightness of the pixel at coordinate (i,j) in the first image is equal to the second threshold, the pixel value of the pixel may be equal to the saturated pixel value. However, if gain correction is performed, the pixel value at the second threshold brightness may be equal to or greater than the saturated pixel value. Furthermore, if the brightness of the pixel at coordinate (i,j) in the first image is equal to or greater than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) in the composite image may be equal to the pixel value at coordinate (i,j) in the third image. The above content can be simply expressed in mathematical terms as follows:

[0053] TIFF2026025914000002.tif20115

[0054] The composite image i,j may mean the pixel value of the pixel at coordinate (i, j) of the composite image. i,jmay mean the pixel value of the pixel at coordinate (i, j) of the first image. i,j may mean the pixel value of the pixel at coordinate (i,j) of the third image.

[0055] Unlike the above, image synthesis may be performed based on the brightness value of the pixel at coordinate (i,j) in the third image. For example, if the brightness of the pixel at coordinate (i,j) in the third image is smaller than a first threshold Th1, the pixel value of the pixel at coordinate (i,j) in the synthesized image may be the pixel value of coordinate (i,j) in the first image. The brightness value of the pixel at coordinate (i,j) in the third image may correspond to the pixel value of coordinate (i,j) in the third image. For example, the brightness value of the pixel at coordinate (i,j) in the third image may be, but is not limited to, the pixel value of the pixel at coordinate (i,j) in the third image. Furthermore, if the brightness of the pixel at coordinate (i,j) of the third image is equal to or greater than the first threshold Th1 and less than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) of the composite image may be (1-α)*(pixel value at coordinate (i,j) of the first image)+α*(pixel value at coordinate (i,j) of the third image). Furthermore, if the brightness of the pixel at coordinate (i,j) of the third image is equal to or greater than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) of the composite image may be the pixel value at coordinate (i,j) of the third image.

[0056] In an image processing method according to an exemplary embodiment of the present disclosure, the first threshold Th1 and the second threshold Th2 may be set differently for each color. For example, if the pixel at coordinate (i, j) in the first or third image is a green pixel, the first and second green thresholds may be set to the first and second thresholds Th1 and Th2. If the pixel at coordinate (i, j) in the first or third image is a red pixel, the first and second red thresholds may be set to the first and second thresholds Th1 and Th2. If the pixel at coordinate (i, j) in the first or third image is a blue pixel, the first and second blue thresholds may be set to the first and second thresholds Th1 and Th2.

[0057] In this case, the first green threshold, the first red threshold, and the first blue threshold may be different from each other or may be at least partially the same depending on the situation, and the second green threshold, the second red threshold, and the second blue threshold may be different from each other or may be at least partially the same depending on the situation.

[0058] A composite image can be generated by combining the first image and the third image at a predetermined ratio. For example, as described above, ((1-α) × first image i,j +α×Third image i,j ) can determine the pixel value of the pixel at coordinate (i,j) of the composite image. Referring to Figure 3b, the α value can be changed according to the brightness value of the first image or the third image. Specifically, as the brightness value of the first image or the third image increases from Th1 to Th2, the α value can also increase linearly from 0 to 1, but is not limited to this.

[0059] The first green threshold, second green threshold, first red threshold, second red threshold, first blue threshold, and second blue threshold, which can be the first threshold and the second threshold, respectively, can be determined according to the spectral sensitivity characteristics of the color filter, the white balance value in a preset light source, the white balance value at the time of shooting, etc.

[0060] FIG. 4 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. In step S410, the brightness correction unit according to an exemplary embodiment of the present disclosure may obtain a first image and a second image. The first image may be an image generated by applying a first gain to a first input image. The second image may be an image generated by applying a second gain to a second input image.

[0061] In step S420, the luminance correction unit calculates a first image pixel value (i, j) of the first image pixel. i,j is the pixel value of the pixel at coordinate (i,j) in the second image i,j It can be determined whether or not the value is equal to or greater than this.

[0062] In step S430, the brightness correction unit i,j is the second image i,j If greater than or equal to, the first image i,j is the pixel value of the pixel at coordinate (i,j) of the output image. i,j can be decided.

[0063] In step S440, the brightness correction unit i,j is the second image i,j If it is smaller than the second image i,j is the pixel value of the pixel at coordinate (i,j) of the output image. i,j can be decided.

[0064] That is, the brightness correction unit can generate an output image by selectively replacing the second image with the first image depending on the magnitude relationship between pixel values ​​at the same coordinates in the first image and the second image. That is, if the pixel value of the first image is greater than the pixel value of the first image at the same coordinates in the first image and the second image, the brightness correction unit can generate an output image by replacing the pixel value of the second image with the pixel value of the first image.

[0065] FIG. 5 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. Referring to FIG. 5, the image synthesis unit according to an exemplary embodiment of the present disclosure may obtain the first image and the third image from the gain control unit and the brightness correction unit in step S510.

[0066] In step S520, the image synthesis unit can determine the color of the pixel at the (i, j) coordinate of the synthesized image to be generated. In step S530, if the color of the pixel at the (i, j) coordinate of the composite image to be generated is green, the image synthesis unit can perform image synthesis by setting the first green threshold and the second green threshold to the first threshold Th1 and the second threshold Th2.

[0067] In addition, in step S540, if the color of the pixel at the (i, j) coordinate of the composite image to be generated is red, the image synthesis unit can perform image synthesis by setting the first red threshold and the second red threshold to the first threshold Th1 and the second threshold Th2.

[0068] In addition, in step S550, if the color of the pixel at the (i, j) coordinate of the composite image to be generated is blue, the image synthesis unit can perform image synthesis by setting the first blue threshold and the second blue threshold to the first threshold Th1 and the second threshold Th2.

[0069] In step S560, the image synthesis unit generates a synthesized image using the set first threshold value Th1 and second threshold value Th2. i,j As mentioned above, the composite image i,j can be determined by the following formula:

[0070] TIFF2026025914000003.tif18115

[0071] That is, the image synthesis unit can synthesize images by setting different brightness thresholds used for image synthesis depending on the color of pixels of the synthesized image to be generated.

[0072] FIG. 6a is a diagram for explaining an image processing method according to a comparative example. 6a, the image processing method according to the comparative example may set the same threshold for brightness values ​​for each color in relation to image synthesis. That is, the image processing method does not need to distinguish between thresholds for brightness values ​​for each color. Therefore, assuming that pixel values ​​of a first image are brightness values, and generating a synthesized image by comparing the pixel values ​​of the first image with first and second thresholds, since the red and blue pixel values ​​of the first image are smaller than the first threshold, the red and blue pixel values ​​of the synthesized image may be determined by the red and blue pixel values ​​of the first image. Furthermore, since the green pixel value of the first image is equal to the second threshold, the green pixel value of the synthesized image may be determined by the green pixel value of a third image.

[0073] FIG. 6b is a diagram for explaining an image processing method according to a comparative example. Referring to FIG. 6b, the image processing method according to the comparative example may set the same threshold for brightness values ​​for each color in relation to image synthesis. That is, the image processing method does not need to distinguish between different thresholds for brightness values ​​for each color. Therefore, assuming that pixel values ​​of a first image are brightness values, a synthesized image is generated by comparing the pixel values ​​of the first image with first and second thresholds. Since the red and blue pixel values ​​of the first image are smaller than the first threshold, the red and blue pixel values ​​of the synthesized image can be determined by the red and blue pixel values ​​of the first image. Furthermore, since the green pixel value of the first image is equal to the second threshold, the green pixel value of the synthesized image can be determined by the green pixel value of the third image. If brightness fluctuations occur due to a flicker phenomenon or the like, and the pixel value of the green pixel is smaller than the saturation value even after gain application, as in the third image of FIG. 6b, the color of the synthesized image may be distorted. Specifically, the red and blue pixel values ​​of the composite image are determined to be the red and blue pixel values ​​of the first image, but the green pixel value of the composite image is determined to be the green pixel value of the third image, which is smaller than the saturated pixel value, so that the composite image may appear to have stronger red and blue components than the actual image.

[0074] FIG. 6c is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. 6c, an image processing method according to an exemplary embodiment of the present disclosure may set different thresholds for brightness values ​​for different colors in relation to image synthesis. Therefore, assuming that pixel values ​​of a first image are brightness values, a synthesized image is generated by comparing the pixel values ​​of the first image with a first threshold and a second threshold. Since the red and blue pixel values ​​of the first image are greater than or equal to the first red / blue threshold and less than the second red / blue threshold, the red pixel value of the synthesized image is determined by combining the red pixel value of the first image with the red pixel value of the third image. The blue pixel value of the synthesized image is determined by combining the blue pixel value of the first image with the blue pixel value of the third image. Furthermore, since the green pixel value of the first image is equal to the second threshold, the green pixel value of the synthesized image may be determined by the green pixel value of the third image. In the third image of Figure 6c, brightness fluctuations occur due to flickering or the like, and even after gain is applied, the pixel value of the green pixel is less than the saturation value. However, by combining the red / blue pixel values ​​of the first image with the red / blue pixel values ​​of the third image to generate a composite image, the degree of color distortion can be reduced.

[0075] FIG. 7 is a block diagram illustrating an example of a computing device that corresponds to the image processing device of FIG. Referring to FIG. 7, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 100 of FIG.

[0076] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), although the scope of the present invention is not limited thereto.

[0077] Computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0078] The processor 1010 can process data and / or instructions required to perform the operations of the components 200 to 240 of the image processing device 100 described in FIG.

[0079] The memory 1020 can store data and / or instructions necessary to perform the operations of the components 200 to 240 of the image processing device 100, and can be accessed by the processor 1010. For example, the memory 1020 can be realized with volatile memory (e.g., Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), etc.) or non-volatile memory (e.g., Programmable Read Only Memory (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), flash memory, etc.).

[0080] That is, a computer program for performing the operations of the image processing device 100 disclosed in this document is recorded in the memory 1020 and is executed and processed by the processor 1010, thereby realizing the operations of the image processing device 100.

[0081] The input / output interface 1030 may provide an interface that connects an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010, enabling data to be transmitted and received.

[0082] The communication interface 1040 is configured to be able to send and receive various data to and from an external device (for example, an application processor, an external memory, etc.), and may be a device that can support wired or wireless communication.

[0083] The above description merely exemplifies the technical concept of the present disclosure, and various modifications and variations are possible within the scope of the essential characteristics of the present disclosure, provided that such modifications and variations are made by a person skilled in the art to which the present disclosure pertains. Therefore, the embodiments disclosed in the present disclosure are intended to illustrate, rather than limit, the technical concept of the present disclosure, and the scope of the technical concept of the present disclosure is not limited by such embodiments. The scope of protection of the present disclosure should be interpreted by the scope of the claims below, and all technical concepts within the scope equivalent thereto should be interpreted as being within the scope of the present disclosure.

Claims

1. a brightness correction unit that generates a third image by selectively replacing the second image with the first image according to a magnitude relationship between pixel values ​​at the same position in the first image and the second image; an image synthesis unit that generates a fourth image by synthesizing the first image and the third image using brightness values ​​for each color of the first image or the third image and different threshold values ​​for each color; 12. An image processing device comprising:

2. a gain control unit that generates the first image by correcting a first input image using a first gain, and that generates the second image by correcting a second input image using a second gain greater than the first gain; The first input image is 2. The image processing apparatus of claim 1, wherein the second input image is an image taken during a longer exposure time than the second input image.

3. The image synthesis unit If a brightness value of a first green pixel of the first image is less than a first green threshold, generating the fourth image using a pixel value of the first green pixel; If the brightness value of the first green pixel is greater than or equal to the first green threshold and less than a second green threshold, generating the fourth image by adding the pixel value of the first green pixel and the pixel value of the second green pixel of the third image in a predetermined ratio; The image processing device of claim 1 , wherein if the brightness value of the first green pixel is equal to or greater than the second green threshold, the fourth image is generated according to the pixel value of the second green pixel.

4. The image synthesis unit If a brightness value of a first red pixel of the first image is less than a first red threshold, generating the fourth image using a pixel value of the first red pixel; If the brightness value of the first red pixel is equal to or greater than the first red threshold and less than a second red threshold, generating the fourth image by adding the pixel value of the first red pixel and the pixel value of the second red pixel of the third image in a predetermined ratio; The image processing device of claim 1 , wherein if the brightness value of the first red pixel is equal to or greater than the second red threshold, the fourth image is generated using the pixel value of the second red pixel.

5. The image synthesis unit If a brightness value of a first blue pixel of the first image is less than a first blue threshold, generating the fourth image using a pixel value of the first blue pixel; If the brightness value of the first blue pixel is greater than or equal to the first blue threshold and less than a second blue threshold, generating the fourth image by adding the pixel value of the first blue pixel and the pixel value of the second blue pixel of the third image in a predetermined ratio; The image processing device of claim 1 , wherein if the brightness value of the first blue pixel is equal to or greater than the second blue threshold, the fourth image is generated using the pixel value of the second blue pixel.

6. The predetermined ratio is 4. The image processing apparatus of claim 3, wherein the brightness value of said first green pixel changes linearly as it increases from the first green threshold to said second green threshold.

7. The first green pixel is 4. The image processing apparatus of claim 3, wherein the first green pixel is saturated when its brightness value is equal to the second green threshold value.

8. The brightness value of the first green pixel is 4. The image processing apparatus of claim 3, wherein the pixel value corresponds to the pixel value of the first green pixel.

9. The first green threshold and the second green threshold are 4. The image processing device of claim 3, wherein the color filter is determined at least in part by a spectral sensitivity characteristic of a color filter and a white balance value.

10. The luminance correction unit The image processing device of claim 1 , wherein the third image is generated by replacing the second pixel value of the second image with the first pixel value if the first pixel value of the first image is greater than or equal to the second pixel value of the second image.

11. generating a third image by combining a pixel value of a first green pixel of the first image with a pixel value of a second green pixel of the second image according to a magnitude relationship between the brightness value of the first green pixel of the first image, the first green threshold value, and the second green threshold value; generating the third image by combining a pixel value of the first blue pixel of the first image with a pixel value of the second blue pixel of the second image according to a magnitude relationship between a brightness value of the first blue pixel of the first image, a first blue threshold, and a second blue threshold; an image synthesis unit that generates the third image by synthesizing a pixel value of the first red pixel and a pixel value of a second red pixel of the second image according to a magnitude relationship between a brightness value of the first red pixel of the first image, a first red threshold, and a second red threshold.

12. a gain control unit that generates the first image by correcting a first input image using a first gain, and generates a fourth image by correcting the second input image using a second gain greater than the first gain; The first input image is 12. The image processing apparatus of claim 11, wherein the second input image is an image taken during a longer exposure time than the second input image.

13. 13. The image processing device of claim 12, further comprising a brightness correction unit that generates the second image by selectively replacing the fourth image with the first image according to a magnitude relationship between pixel values ​​at the same position in the first image and the fourth image.

14. The image synthesis unit If the brightness value of the second green pixel is less than the first green threshold, generating the third image using the pixel value of the first green pixel; if the brightness value of the second green pixel is greater than or equal to the first green threshold and less than the second green threshold, generating the third image by adding the pixel value of the first green pixel and the pixel value of the second green pixel in a predetermined ratio; The image processing device of claim 11 , wherein if the brightness value of the second green pixel is equal to or greater than the second green threshold, the third image is generated according to the pixel value of the second green pixel.

15. The image synthesis unit If the brightness value of the second blue pixel is less than the first blue threshold, generating the third image using the pixel value of the first blue pixel; If the brightness value of the second blue pixel is greater than or equal to the first blue threshold and less than the second blue threshold, generating the third image by adding the pixel value of the first blue pixel and the pixel value of the second blue pixel in a predetermined ratio; The image processing device of claim 11 , wherein if the brightness value of the second blue pixel is equal to or greater than the second blue threshold, the third image is generated according to the pixel value of the second blue pixel.

16. The image synthesis unit If the brightness value of the second red pixel is less than the first red threshold, generating the third image using the pixel value of the first red pixel; If the brightness value of the second red pixel is equal to or greater than the first red threshold and less than the second red threshold, generating the third image by adding the pixel value of the first red pixel and the pixel value of the second red pixel in a predetermined ratio; The image processing device of claim 11 , wherein if the brightness value of the second red pixel is equal to or greater than the second red threshold, the third image is generated using the pixel value of the second red pixel.

17. The first green pixel is 12. The image processing apparatus of claim 11, wherein the first green pixel is saturated when its brightness value is equal to the second green threshold value.

18. The first green threshold and the second green threshold are 12. The image processing device of claim 11, wherein the color temperature is determined at least in part by the spectral sensitivity characteristics of the color filters and a white balance value.

19. The brightness value of the first green pixel is The image processing apparatus of claim 11 , wherein the pixel value corresponds to the pixel value of the first green pixel.

20. generating a third image by replacing the second pixel value of the second image with the first pixel value according to a magnitude relationship between a first pixel value of the first image and a second pixel value of the second image; generating a fourth image by combining the first image and the third image using brightness values ​​for each color of the first image or the third image and different threshold values ​​for each color; An image processing method comprising:

Images