Image processing apparatus and method

Abstract

To provide an image processing apparatus for synthesizing a plurality of images in which flicker or a moving object occurs.SOLUTION: According to another aspect of the present invention, there is provided an image processing apparatus including: a determination unit configured to determine whether a flicker or a moving object occurs using a plurality of images including a first image captured during a longest first exposure time and a second image captured during a shortest second exposure time; and an image synthesis unit configured to synthesize the plurality of images by setting a first weight corresponding to the first image to a largest value among a plurality of weights corresponding to the plurality of images when it is determined that the flicker occurs, and to set a second weight corresponding to the second image to a largest value among the plurality of weights when it is determined that the moving object occurs.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical concepts of the present disclosure relate to an apparatus and method for processing images. [Background technology]

[0002] Image sensing devices are devices that capture optical images using the properties of photosensitive semiconductor materials that react to light. With the development of industries such as automobiles, medicine, computers, and communications, there is an increasing demand for high-performance image sensing devices in various fields, such as smartphones, digital cameras, game consoles, the Internet of Things, robots, security cameras, and medical microcameras.

[0003] An image captured by an image sensing device may differ depending on exposure time, gain, etc., and image distortion may occur due to a flicker phenomenon, etc. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an image processing device that combines multiple images in which flicker or a moving object occurs. An object of the present disclosure is to provide an image processing device that determines whether flicker or a moving object occurs based on the luminance values ​​of pixels to be composited.

[0005] An object of the present disclosure is to provide an image processing device that changes weights used in image synthesis depending on whether flicker or a moving object has occurred. The technical problem of the present disclosure is to provide an image processing device that corrects the color of an image using a white balance coefficient before the image is synthesized to prevent color distortion in the synthesized image.

[0006] The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the following description. [Means for solving the problem]

[0007] An image processing device according to an exemplary embodiment of the present disclosure may include a determination unit that determines whether flicker or a moving object has occurred using a plurality of images including a first image captured during a first exposure time and a second image captured during a second exposure time shorter than the first exposure time, and an image synthesis unit that, if it is determined that flicker has occurred, synthesizes the plurality of images by setting a first weight for the first image to the largest value among a plurality of weights corresponding to each of the plurality of images, and, if it is determined that a moving object has occurred, synthesizes the plurality of images by setting a second weight for the second image to the largest value among the plurality of weights.

[0008] According to one embodiment, the determination unit may determine that flicker has occurred if the ratio of the number of pixels having a luminance value equal to or less than a threshold luminance value among the pixels to be synthesized that exist at the same position in the plurality of images is smaller than a threshold ratio, and may determine that a moving object has occurred if the ratio is equal to or greater than the threshold ratio.

[0009] According to one embodiment, the determination unit may determine the threshold brightness value based on an average brightness value of pixels having a color similar to that of the first synthesis target pixel in a predetermined area including the first synthesis target pixel to be synthesized in the first image.

[0010] According to one embodiment, the plurality of images includes a third image captured during a third exposure time that is shorter than the first exposure time and longer than the second exposure time, and the determination unit can determine that flicker has occurred if it is determined that a decrease in brightness has occurred in one of the second and third images.

[0011] According to one embodiment, the plurality of images includes a third image captured during a third exposure time that is shorter than the first exposure time and longer than the second exposure time, and the determination unit can determine that a moving object has occurred if it is determined that a decrease in brightness has occurred in the second image and the third image.

[0012] According to an embodiment, the image processing device may further include a color correction unit that corrects the color of the first image by correcting a first red component value of the first image using a first white balance coefficient for the red component and correcting a first blue component value of the first image using a second white balance coefficient for the blue component.

[0013] According to one embodiment, the color correction unit may correct the color of the first image by multiplying the first red component value by a first white balance coefficient to generate a second red component value, dividing the smaller of the second red component value and the green component saturation value by the first white balance coefficient to generate a third red component value, replacing the first red component value of the first image with the third red component value, multiplying the first blue component value by the second white balance coefficient to generate a second blue component value, dividing the smaller of the second blue component value and the green component saturation value by the second white balance coefficient to generate a third blue component value, and replacing the first blue component value of the first image with the third blue component value.

[0014] According to one embodiment, the color correction unit corrects the color of the first image when it is determined that flicker has occurred, and the image synthesis unit can synthesize multiple images using the corrected first image.

[0015] According to one embodiment, the determination unit may determine that flicker and a moving object have not occurred if the difference between the largest pixel value and the smallest pixel value among the pixel values ​​of pixels to be synthesized that exist at the same position in multiple images is smaller than a threshold difference value.

[0016] According to one embodiment, when it is determined that no flicker or moving object has occurred, the image synthesis unit may synthesize multiple images while changing multiple weights according to the brightness value of any one of the multiple images.

[0017] According to one embodiment, the image synthesis unit generates a synthetic image using the pixel value of the first pixel when the brightness value of the first pixel in the first image is smaller than a first threshold brightness value; generates a synthetic image by adding the pixel value of the first pixel and the pixel value of the second pixel located at the same position as the first pixel in the second image at a predetermined ratio when the brightness value of the first pixel is equal to or greater than the first threshold brightness value and smaller than a second threshold brightness value; and generates a synthetic image using the pixel value of the second pixel when the brightness value of the first pixel is equal to or greater than the second threshold brightness value. According to one embodiment, the predetermined ratio may vary linearly as the luminance value of the first pixel increases from the first threshold luminance value to the second threshold luminance value.

[0018] According to one embodiment, the image processing device may further include a gain control unit that generates a first image by correcting a first input image using a first gain, and generates a second image by correcting a second input image using a second gain greater than the first gain. According to an embodiment, the gain control unit may determine the ratio of the first gain to the second gain in inverse proportion to the first exposure time to the second exposure time.

[0019] An image processing device according to an exemplary embodiment of the present disclosure may include a judgment unit that determines whether the ratio of the number of pixels having a brightness value equal to or less than a threshold brightness value among the pixels to be synthesized to the number of pixels to be synthesized that exist at the same position in a plurality of images is smaller than a threshold ratio, and an image synthesis unit that performs image synthesis while changing a plurality of weights for each of the pixels to be synthesized based on the judgment result.

[0020] According to one embodiment, the plurality of images includes a first image captured during the longest exposure time among the plurality of images, and if the ratio is smaller than a threshold ratio, the image synthesis unit may perform image synthesis by setting a first weight for the first image among the plurality of weights to the largest value.

[0021] According to one embodiment, the plurality of images includes a second image captured during the shortest exposure time among the plurality of images, and the image synthesis unit may perform image synthesis by setting a second weight for the second image to the largest value among the plurality of weights if the ratio is greater than or equal to a threshold ratio.

[0022] According to one embodiment, the plurality of images may include a first image captured during the longest exposure time among the plurality of images, and the image processing device may further include a color correction unit that corrects a color of the first image by correcting a first red component value of the first image using a first white balance coefficient for the red component and correcting a first blue component value of the first image using a second white balance coefficient for the blue component.

[0023] According to one embodiment, the color correction unit may correct the color of the first image by multiplying the first red component value by a first white balance coefficient to generate a second red component value, dividing the smaller of the second red component value and the green component saturation value by the first white balance coefficient to generate a third red component value, replacing the first red component value of the first image with the third red component value, multiplying the first blue component value by the second white balance coefficient to generate a second blue component value, dividing the smaller of the second blue component value and the green component saturation value by the second white balance coefficient to generate a third blue component value, and replacing the first blue component value of the first image with the third blue component value.

[0024] An image processing method according to an exemplary embodiment of the present disclosure may include a step of determining whether flicker or a moving object has occurred in a plurality of images based on the luminance values ​​of pixels to be synthesized that exist at the same position in the plurality of images, and a step of performing image synthesis while changing a plurality of weights for each pixel to be synthesized based on the determination result. The above briefly summarized features of the present disclosure are illustrative aspects of the detailed description of the present disclosure that follows and are not intended to limit the scope of the present disclosure. [Effects of the Invention]

[0025] The image processing device according to the exemplary embodiment of the present disclosure can combine multiple images in which flicker or moving objects occur. The image processing device according to the exemplary embodiment of the present disclosure can determine whether flicker or a moving object occurs based on the luminance value of the pixel to be synthesized.

[0026] The image processing device according to the exemplary embodiment of the present disclosure can change the weight used for image synthesis depending on whether flicker or a moving object has occurred. The image processing device according to the exemplary embodiment of the present disclosure can prevent color distortion of the image by correcting the color of the image using the white balance coefficient before the image is combined.

[0027] The effects obtained by the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the following description. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3a] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 3b] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 5a] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 5b] FIG. 10 is a diagram for explaining an image processing method according to a comparative example. [Figure 5c] FIG. 1 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 6] 1 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. [Figure 7] 2 is a block diagram illustrating an example of a computing device corresponding to the image processing device of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, with reference to the accompanying drawings, detailed descriptions of embodiments of the present disclosure will be given so that those skilled in the art can easily implement the present disclosure, however, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0030] In describing embodiments of the present disclosure, if a detailed description of known configurations or functions is deemed to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, in the drawings, parts that are not related to the description of the present disclosure will be omitted, and similar parts will be designated by similar reference numerals.

[0031] In this disclosure, when a component is "coupled," "coupled," or "connected" to another component, it means not only a direct connection, but also an indirect connection where there is another component between them. Furthermore, when a component "includes" or "has" another component, it does not mean that the other component is excluded, and that the component may further include the other component, unless otherwise specified.

[0032] In this disclosure, terms such as first and second are used only to distinguish one component from another, and do not limit the order or importance of the components unless otherwise specified. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0033] In this disclosure, components that are distinguished from one another are used to clearly describe the characteristics of each component and do not necessarily mean that the components are separate. That is, multiple components may be integrated and consist of a single hardware or software unit, or a single component may be distributed and consist of multiple hardware or software units. Therefore, even if not otherwise specified, such integrated or distributed embodiments are also included within the scope of this disclosure.

[0034] In this disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of this disclosure. Furthermore, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of this disclosure.

[0035] In this disclosure, expressions of positional relationships used in this specification, such as upper, lower, left side, right side, etc., are described for the convenience of explanation, and when the drawings shown in this specification are viewed upside down, the positional relationships described in this specification may be interpreted in reverse.

[0036] In this disclosure, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" may include any one or all possible combinations of the items listed along with the corresponding phrase.

[0037] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to FIGS. FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure.

[0038] Referring to FIG. 1, an image processing device 100 can perform at least one image signal processing on image data IDATA to generate processed image data IDATA_P.

[0039] The image processing device 100 may perform image signal processing on the image data IDATA to reduce noise and improve image quality, such as demosaicing, defective pixel correction, gamma correction, color filter array interpolation, color matrix, color correction, color enhancement, and lens distortion correction. The image processing device 100 may also compress image data that has undergone image signal processing for improving image quality to generate an image file, or restore image data from the image file. The image compression format may be a lossless format or a lossy format. For example, the Joint Photographic Experts Group (JPEG) format or JPEG2000 format may be used for still images. For moving images, a moving image file may be generated by compressing multiple frames according to the Moving Picture Experts Group (MPEG) standard.

[0040] The image data IDATA may be generated by an image sensing device that captures an optical image of a scene, although the scope of the present invention is not limited thereto. The image sensing device may include a pixel array including a plurality of pixels for sensing light incident from a scene, a control circuit for controlling the pixel array, and a readout circuit that converts analog pixel signals received from the pixel array into digital image data IDATA and outputs the digital image data. In this disclosure, the image data IDATA will be described assuming that it is generated by the image sensing device.

[0041] The pixel array may include a color filter array (CFA) in which color filters are arranged according to a certain pattern (e.g., a Bayer pattern, a quad Bayer pattern, a nona Bayer pattern, an RGBW pattern, etc.) so that each pixel can sense light of a predetermined wavelength band. The pattern of the image data IDATA may be determined according to the type of pattern the CFA has.

[0042] The image processing device 100 may be, but is not limited to, a computing device mounted on a chip separate from the chip on which the image sensing device is mounted. The chip on which the image sensing device is mounted and the chip on which the image processing device 100 is mounted may communicate with each other via a predetermined interface. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the image processing device 100 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), but the scope of the present invention is not limited thereto.

[0043] The image processing device 100 may include a gain control unit 110 , a determination unit 120 , an image synthesis unit 130 , and / or a color correction unit 140 . The gain control unit 110 can control a gain to be applied to an input image. Specifically, the gain control unit 110 can determine a gain based on an exposure time applied when the input image was captured, and apply the gain to the input image. For example, the gain control unit 110 can generate a first image by correcting a first input image using a first gain. Also, the gain control unit 110 can generate a second image by correcting a second input image using a second gain greater than the first gain. In this case, the first input image may be an image captured during a longer exposure time than the second input image. That is, the first input image may be an image brighter than the second input image. Therefore, the second gain may have a value greater than the first gain. Specifically, when the exposure time of the input image is T (the exposure time of the i-th input image is T), i ), the gain magnification of the i-th input image is max(T) / T i In other words, the gain ratio may be determined in an inverse ratio to the ratio with respect to the exposure time. For example, assuming that the image processing device 100 performs image composition using a first input image captured during a first exposure time, a second input image captured during a second exposure time, and a third input image captured during a third exposure time, and assuming that the second exposure time is ¼ the first exposure time and the third exposure time is ½ the first exposure time, the second gain applied to the second input image may be four times the first gain applied to the first input image, and the third gain applied to the third input image may be twice the first gain.

[0044] When the first input image contains signals in the saturated region due to a long exposure time, the image processing device 100 can obtain signals in regions brighter than the saturated brightness by applying a second gain to a second input image that is darker than the first input image.

[0045] The determination unit 120 may determine whether flicker or a moving object has occurred in the plurality of images using the plurality of images. Specifically, the determination unit 120 may determine whether flicker or a moving object has occurred by calculating a ratio of the number of pixels having a luminance value equal to or less than a threshold luminance value among the pixels to be combined to the number of pixels to be combined that exist at the same position in the plurality of images (e.g., pixels located at coordinates (i, j) in each of the plurality of images) and determining whether the calculated ratio is smaller than a threshold ratio. For example, the determination unit 120 may determine that flicker has occurred if the calculated ratio is smaller than a threshold ratio. Furthermore, the determination unit 120 may determine that a moving object has occurred if the calculated ratio is equal to or greater than the threshold ratio. Furthermore, the determination unit 120 may determine that flicker or a moving object has not occurred if the difference between the largest pixel value and the smallest pixel value among the pixels to be combined is smaller than a threshold difference value. The operation of the determination unit 120 will be described in more detail below.

[0046] The image composition unit 130 may combine multiple images. Specifically, the image composition unit 130 may change a method for combining multiple images depending on whether flicker or a moving object occurs. For example, in order to deal with flicker, it is effective to prioritize images captured during an exposure time in which image distortion due to flicker is less likely to occur. Therefore, when it is determined that flicker has occurred, the image composition unit 130 may combine multiple images by setting the weight for the image captured during the longest exposure time to the largest value among multiple weights corresponding to each of the multiple images. For example, the image composition unit 130 may perform image composition by setting the weight for the image captured during the longest exposure time to 1 and setting the other weights to 0. However, this is not limiting.

[0047] Furthermore, in order to reduce image distortion in an area where a moving object is located in multiple images, it is effective to perform image composition by prioritizing images with short exposure times. This is because images with short exposure times have less subject shaking, are less affected by the moving object, and are less likely to become saturated in high-illumination areas of the image. Therefore, when it is determined that a moving object has occurred, the image composition unit 130 may combine multiple images by setting the weight for the image captured with the shortest exposure time to the largest value among the multiple weights. For example, the image composition unit 130 may perform image composition by setting the weight for the image captured with the shortest exposure time to 1 and setting the other weights to 0, but this is not limiting.

[0048] Furthermore, if it is determined that no flicker or moving object has occurred, the image synthesis unit 130 may synthesize the multiple images by changing the weights corresponding to the multiple images according to the luminance value of any one of the multiple images. More detailed details of synthesizing the images will be described later.

[0049] The color correction unit 140 can correct the colors of the images before image synthesis to prevent color distortion in the synthesized image. For example, the color correction unit 140 can correct the colors of the images by correcting the red component values ​​of the images using a white balance coefficient for the red component and correcting the blue component values ​​of the images using a white balance coefficient for the blue component. The white balance coefficient may be a coefficient used to correct the color temperature difference between the images so that the colors of the images appear natural. More specific details regarding color correction will be described later.

[0050] FIG. 2 is a diagram illustrating an image processing method according to an exemplary embodiment of the present disclosure. FIG. 3a is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. FIG. 3b is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure.

[0051] FIG. 2 will now be described with reference to FIGS. 3a and 3b. Referring to FIG. 2 , an image processing method according to an exemplary embodiment of the present disclosure may perform gain control in step S210. For example, the image processing method may apply a gain to an input image. Specifically, the image processing method may determine a gain based on an exposure time applied when the input image was captured and apply the gain to the input image. The image processing method may generate a first image by correcting a first input image captured during a first exposure time using a first gain. The image processing method may also generate a second image by correcting a second input image captured during a second exposure time using a second gain. The image processing method may also generate a third image by correcting a third input image captured during a third exposure time using a third gain. In this case, the first exposure time may be the longest, the second exposure time may be the shortest, and the third exposure time may be shorter than the first exposure time and longer than the second exposure time.

[0052] In step S220, the image processing method may determine whether the difference between the largest and smallest pixel values ​​of pixels to be synthesized that exist at the same position in multiple images is smaller than a threshold difference value. For example, the pixel value of a pixel located at coordinates (i, j) in the first image may be calculated by LEP i and the pixel value of the pixel located at coordinate (i,j) in the second image is called SEP i If we say, (|LEP i -SEP i |<threshold difference value).

[0053] The image processing method may perform a first synthesis process in step S230 if the difference in pixel values ​​is smaller than a threshold difference value. If the difference in pixel values ​​is smaller than a threshold difference value, it may be considered that no flicker or moving object occurs, but is not limited to this.

[0054] The first synthesis process may refer to a process of synthesizing multiple images while changing multiple weights corresponding to the multiple images according to the luminance value of one of the multiple images. Referring to FIG. 3A, assuming that a first image, which is a gain-corrected image, and a second image, which is a gain-corrected image (the second image may have pixel values ​​greater than a saturation value due to the application of gain), are to be synthesized, a synthesized image can be generated by synthesizing the first and second images. Synthesis parameters can be used to generate the synthesized image through the first synthesis process. The synthesis parameters may be thresholds used in the first synthesis process. Specifically, the synthesis parameters may be a first threshold Th1 and a second threshold Th2 for luminance values. For example, if the brightness of a pixel at coordinate (i, j) in the first image is less than the first threshold Th1, the pixel value of the pixel at coordinate (i, j) in the synthesized image may be the pixel value of coordinate (i, j) in the first image. The luminance value of the pixel at coordinate (i, j) in the first image may correspond to the pixel value of the pixel at coordinate (i, j) in the first image. For example, the luminance value of the pixel at the (i,j) coordinate in the first image may be, but is not limited to, the pixel value of the pixel at the (i,j) coordinate in the first image. Furthermore, if the brightness of a pixel at coordinate (i,j) in the first image is greater than or equal to a first threshold Th1 and less than a second threshold Th2, the pixel value of the pixel at coordinate (i,j) in the composite image may be (1-α*(pixel value at coordinate (i,j) in the first image))+α*(pixel value at coordinate (i,j) in the second image). If the brightness value of the pixel at coordinate (i,j) in the first image is equal to the second threshold, the pixel value of the pixel may be the same as a saturated pixel value. However, if gain correction is performed, the pixel value at the second threshold brightness may be greater than or equal to the saturated pixel value. Furthermore, if the brightness value of the pixel at coordinate (i,j) in the first image is greater than or equal to the second threshold Th2, the pixel value of the pixel at coordinate (i,j) in the composite image may be the pixel value at coordinate (i,j) in the second image. The above content can be simply expressed in mathematical terms as follows:

[0055]

number

[0056] The composite image i,j may mean the pixel value of the pixel at coordinate (i, j) of the composite image. i,j may mean the pixel value of the pixel at coordinate (i, j) of the first image. i,j may mean the pixel value of the pixel at coordinate (i,j) of the second image.

[0057] Unlike the above, image synthesis may be performed based on the brightness value of the pixel at coordinate (i,j) in the second image. For example, if the brightness of the pixel at coordinate (i,j) in the second image is less than a first threshold Th1, the pixel value of the pixel at coordinate (i,j) in the synthesized image may be the pixel value of coordinate (i,j) in the first image. The brightness value of the pixel at coordinate (i,j) in the second image may correspond to the pixel value of coordinate (i,j) in the second image. For example, the brightness value of the pixel at coordinate (i,j) in the second image may be, but is not limited to, the pixel value of the pixel at coordinate (i,j) in the second image. Furthermore, if the brightness of the pixel at coordinate (i,j) of the second image is equal to or greater than the first threshold Th1 and less than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) of the composite image may be (1-α)*(pixel value at coordinate (i,j) of the first image)+α*(pixel value at coordinate (i,j) of the second image). Furthermore, if the brightness of the pixel at coordinate (i,j) of the second image is equal to or greater than the second threshold Th2, the pixel value of the pixel at coordinate (i,j) of the composite image may be the pixel value at coordinate (i,j) of the second image.

[0058] A composite image can be generated by combining the first image and the second image at a predetermined ratio. For example, as described above, ((1-α) × first image i,j +α×2nd imagei,j ) may determine the pixel value of the pixel at coordinate (i, j) of the composite image. Referring to Figure 3b, the α value may be changed according to the luminance value of the first image or the second image. Specifically, as the luminance value of the first image or the second image increases from Th1 to Th2, the α value may also increase linearly from 0 to 1, but is not limited to this.

[0059] If the pixel value difference is equal to or greater than the threshold difference value, the image processing method may determine a desired compositing method in step S240. The second compositing process (step S260) may be a method of compositing multiple images by setting the weight for the image captured during the longest exposure time to the largest value among multiple weights corresponding to each of the multiple images. For example, the image processing method may perform image compositing by setting a first weight for the first image to 1 and setting the other weights to 0, but is not limited to this.

[0060] The third synthesis process (step S270) may be a method of synthesizing multiple images by setting the weight for the image captured during the shortest exposure time to the largest value among the multiple weights. For example, the image processing method may perform image synthesis by setting the second weight for the second image to 1 and setting the other weights to 0, but is not limited thereto.

[0061] Assuming that the image processing method is to combine two images, a first image and a second image, a combination mode value can be set and the combination processing method can be switched according to the combination mode value. For example, an image taken by a surveillance camera can be combined using the second combination processing method, and an image taken by a smartphone can be combined using the third combination processing method. The combination processing method can also be determined by receiving a mode value from a camera control program, but the method for determining the combination processing method is not limited to the above.

[0062] Furthermore, assuming that a composite image is generated by combining three or more images, the image processing method may determine a combining processing method based on the luminance value of a composite target pixel located at the same position in the composite target image. For example, assuming that a first image, a second image, and a third image are to be combined, if it is determined that a luminance drop has occurred in one of the second image and the third image, the image processing method may determine that a flicker has occurred and perform image combining using the second combining processing method. Furthermore, if it is determined that a luminance drop has occurred in the second image and the third image, the image processing method may determine that a moving object has occurred and perform image combining using the third combining processing method.

[0063] The image processing method may also set a predetermined region for a first image, a second image, and a third image, and detect a pixel in which a luminance drop occurs among pixels included in the predetermined region. The image processing method may also determine which of the second image and the third image the pixel in which a luminance drop occurs is included in. The image processing method may also determine whether an image in which a luminance drop occurs based on a first coordinate included in the predetermined region is different from an image in which a luminance drop occurs based on a second coordinate included in the predetermined region, and determine whether flicker or a moving object has occurred in the predetermined region based on the determination result. The image processing method may determine that the predetermined region is a region in which a moving object has occurred if the image in which a luminance drop occurs based on the first coordinate included in the predetermined region is different from the image in which a luminance drop occurs based on the second coordinate included in the predetermined region. The image processing method may also determine that the predetermined region is a region in which a flicker has occurred if the image in which a luminance drop occurs based on the first coordinate included in the predetermined region is identical to the image in which a luminance drop occurs based on the second coordinate included in the predetermined region. However, the image processing method according to an exemplary embodiment of the present disclosure is not limited to the above.

[0064] In addition, the image processing method may perform a second compositing process when a ratio of the number of pixels having a luminance value equal to or less than a threshold luminance value to the number of pixels to be composited that exist at the same position in the plurality of images is smaller than a threshold ratio. In addition, the image processing method may perform a third compositing process when the ratio is equal to or greater than a threshold ratio. More detailed details regarding this will be described later.

[0065] The image processing method may perform color correction in step S250. Specifically, the image processing method may perform color correction of the image before performing the second combining process, if necessary. More specific details of color correction will be described later.

[0066] FIG. 4 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. The image processing method may determine an image synthesis method and perform image synthesis using the determined method. Specifically, the image processing method may calculate, in step S410, a ratio of the number of synthesis target pixels having a luminance value equal to or less than a threshold luminance value to the number of synthesis target pixels located at the same position in multiple images. For example, assuming that a first image, a second image, and a third image are to be synthesized, the number of synthesis target pixels located at (i, j) in the first image, the second image, and the third image having a luminance value equal to or less than a threshold luminance value may be calculated, and the ratio may be calculated using the calculated number.

[0067] The threshold luminance value may be determined based on the luminance value of the first image, which is the image captured with the longest exposure time among the multiple images. For example, a predetermined area including the first composite target pixel (located at coordinates (i, j) of the first image) to be composited in the first image may be determined, and the threshold luminance value may be determined based on the average luminance value of pixels having the same color as the first composite target pixel in the predetermined area. For example, but not limited to, a value corresponding to 10% of the average luminance value of the pixels may be determined as the threshold luminance value. If the first, second, and third images contained in the predetermined area have different numbers of pixels, they may be normalized to have the same number of pixels. This is because a sensor that captures images with different exposure times may generate images with different numbers of pixels depending on the exposure time.

[0068] In step S420, the image processing method may determine whether the calculated ratio is less than a threshold ratio. For example, if the threshold ratio is assumed to be 35%, it may determine whether the ratio of pixels having a luminance value equal to or less than the threshold luminance value among the pixels to be synthesized is less than 35%, but is not limited to this.

[0069] The image processing method may perform a second combining process if the calculated ratio is smaller than a threshold ratio in step S430. For example, the image processing method may determine that flicker has occurred when the calculated ratio is smaller than a threshold ratio and perform a second combining process.

[0070] Furthermore, the image processing method may perform a third synthesis process if the calculated ratio is equal to or greater than a threshold ratio in step S440. For example, the image processing method may determine that a moving object has occurred when the calculated ratio is equal to or greater than a threshold ratio, and perform a third synthesis process.

[0071] FIG. 5a is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. Referring to FIG. 5a, an image processing method according to an exemplary embodiment of the present disclosure can apply a white balance coefficient (WB coefficient) to a composite image and perform clipping on the composite image to which the white balance coefficient has been applied.

[0072] Assuming that a first image with a long exposure time and a second image with a short exposure time are combined, the process of applying the white balance coefficients can be expressed as follows:

[0073]

number

[0074]

number

[0075]

number

[0076] JPEG2026025915000006.jpg42170

[0077] 5a shows a diagram illustrating a method for combining images by prioritizing the second image, which is a single-exposure image. For example, the first and second images can be combined by setting the first weight for the first image to 0 and the second weight for the second image to 1. Therefore, the combined image obtained by combining the first and second images may be identical to the second image, but is not limited to this.

[0078] When a white balance coefficient is applied to a composite image, the pixel values ​​of the R component and the B component of the composite image to which the white balance coefficient is applied may exceed the G saturation value. Therefore, image distortion may occur. Therefore, the image processing method may clip the pixel values ​​of the R component and the B component so that the pixel values ​​of the R component and the B component do not exceed a predetermined value. For example, but not limited to, the image processing method may clip the pixel values ​​of the R component and the B component so that the pixel values ​​of the R component and the B component do not exceed the G saturation value. The image processing method may store a separate table and adjust the pixel values ​​of the R component and the B component by referring to the table.

[0079] FIG. 5b is a diagram for explaining an image processing method according to a comparative example. Referring to FIG. 5b, the image processing method according to the comparative example can combine a first image taken during a long exposure time with a second image taken during a short exposure time, apply a white balance coefficient to the combined image, and perform clipping on the combined image to which the white balance coefficient has been applied.

[0080] Assuming that a luminance drop occurs in the second image due to a flicker phenomenon or the like, and assuming that an image processing method performs image synthesis by setting the weight for the first image to 1 and the weight for the second image to 0, the pixel value of the synthesized image is smaller than the G saturation value. In addition, even if a white balance coefficient is applied to the synthesized image, an image having a pixel value smaller than the G saturation value can be generated. Therefore, even if clipping is performed, the pixel values ​​of the R component and the B component do not change. Because the pixel values ​​of the R component and the G component of the synthesized image to which the white balance coefficient is applied are increased due to the application of the white balance coefficient, the color of the synthesized image after clipping may be distorted.

[0081] FIG. 5c is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. Referring to FIG. 5c, an image processing method according to an exemplary embodiment of the present disclosure may perform color correction on a first image captured using a long exposure time, combine the color-corrected first image with a second image captured using a short exposure time, apply a white balance factor to the combined image, and perform clipping on the combined image to which the white balance factor has been applied.

[0082] The image processing method may perform color correction on the first image using a white balance factor before the image composition occurs. For example, the image processing method may perform color correction on the red component value R of the first image using a white balance factor. L is the white balance coefficient WB for the red component. R and correct the blue component value B of the first image using L is the white balance coefficient WB for the blue component. B The color correction may be performed by reducing the red and blue component values ​​of the first image, as shown in Figure 5c, but is not limited to this. More specific details of the color correction will be described later.

[0083] Assuming that a luminance drop occurs in the second image due to a flicker phenomenon or the like, and assuming that an image processing method performs image synthesis by setting the weight for the first image to 1 and the weight for the second image to 0, the pixel value of the synthesized image is less than the G saturation value. Furthermore, even if a white balance coefficient is applied to the synthesized image, an image having pixel values ​​less than the G saturation value can be generated. Therefore, even if clipping is performed, the pixel values ​​of the R component and the B component remain unchanged. By applying the white balance coefficient, the red and blue component values ​​that were reduced in the color-corrected first image can be increased again. Therefore, color distortion can be prevented according to the image processing method according to an exemplary embodiment of the present disclosure.

[0084] FIG. 6 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure. 6, an image processing method according to an exemplary embodiment of the present disclosure can perform color correction on an image before combining the images. For example, a first image captured using a long exposure time and a second image captured using a short exposure time can be combined, and the color of the first image can be corrected before combining the images. The pixel values ​​for the R and B components of the first image before color correction are respectively defined as a first red component value R. L , the first blue component value B L The image processing method begins in step S610 with a first red component value R L and the first blue component value B L White balance coefficient WB R , W.B. B Multiplying these two values ​​gives the second red component value R L ' and the second blue component value B L ' can be generated.

[0085] The image processing method includes, in step S620, determining a second red component value R L The smaller of the two values, R and G, is the first value L '', and the second blue component value B L The second smaller value B L '' can be determined.

[0086] The image processing method includes, in step S630, determining a first value R L '' is the white balance coefficient WB for the red component R The third red component value is generated by dividing by the first red component value R of the first image. L The red color of the first image can be corrected by replacing the second value B with the third red component value. L '' is the white balance coefficient WB for the blue component B The third blue component value is generated by dividing the first blue component value B of the first image by LThe blue color of the first image can be corrected by replacing with the third blue component value.

[0087] The white balance coefficient used for color correction may be a value related to development, or a value corresponding to a predetermined reference color temperature (for example, D65), but is not limited to this.

[0088] FIG. 7 is a block diagram illustrating an example of a computing device that corresponds to the image processing device of FIG. Referring to FIG. 7, a computing device 1000 may represent one embodiment of a hardware configuration for performing the operations of the image processing device 100 of FIG.

[0089] The computing device 1000 may be mounted on a chip separate from the chip on which the image sensing device is mounted. According to one embodiment, the chip on which the image sensing device is mounted and the chip on which the computing device 1000 is mounted may be implemented in a single package, for example, a multi-chip package (MCP), although the scope of the present invention is not limited thereto.

[0090] Computing device 1000 may include a processor 1010 , a memory 1020 , an input / output interface 1030 , and a communication interface 1040 .

[0091] The processor 1010 is capable of processing data and / or instructions necessary to perform the operations of the image processing device 100 configuration described in FIG.

[0092] The memory 1020 can store data and / or instructions necessary to perform the configuration operations of the image processing device 100 and can be accessed by the processor 1010. For example, the memory 1020 can be implemented as a volatile memory (e.g., a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.) or a non-volatile memory (e.g., a programmable read only memory (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, etc.).

[0093] That is, a computer program for performing the operations of the image processing device 100 disclosed in this document is recorded in the memory 1020 and is executed and processed by the processor 1010, thereby realizing the operations of the image processing device 100.

[0094] The input / output interface 1030 may provide an interface that connects an external input device (e.g., a keyboard, a mouse, a touch panel, etc.) and / or an external output device (e.g., a display) to the processor 1010, enabling data to be transmitted and received.

[0095] The communication interface 1040 is configured to be able to send and receive various data to and from an external device (for example, an application processor, an external memory, etc.), and may be a device that can support wired or wireless communication.

[0096] The above description merely exemplifies the technical concept of the present disclosure, and various modifications and variations are possible within the scope of the essential characteristics of the present disclosure, provided that such modifications and variations are made by a person skilled in the art to which the present disclosure pertains. Therefore, the embodiments disclosed in the present disclosure are intended to illustrate, rather than limit, the technical concept of the present disclosure, and the scope of the technical concept of the present disclosure is not limited by such embodiments. The scope of protection of the present disclosure should be interpreted by the scope of the claims below, and all technical concepts within the scope equivalent thereto should be interpreted as being within the scope of the present disclosure.

Claims

1. a determination unit that determines whether a flicker or a moving object occurs using a plurality of images including a first image captured during a first exposure time and a second image captured during a second exposure time shorter than the first exposure time; If it is determined that the flicker has occurred, the first weight for the first image is set to the largest value among a plurality of weights corresponding to the plurality of images, and the plurality of images are combined; an image synthesis unit that, when it is determined that the moving object has occurred, sets a second weight for the second image to the largest value among the plurality of weights and synthesizes the plurality of images; 12. An image processing device comprising:

2. The determination unit determining that the flicker has occurred when a ratio of the number of pixels having a luminance value equal to or less than a threshold luminance value among the pixels to be synthesized to the number of pixels to be synthesized that are present at the same position in the plurality of images is smaller than a threshold ratio; The image processing device according to claim 1 , wherein it is determined that the moving object has occurred when the ratio is equal to or greater than the threshold ratio.

3. The determination unit 3. The image processing device according to claim 2, wherein the threshold luminance value is determined based on an average value of luminance of pixels having the same color as the first pixel to be synthesized in a predetermined region including the first pixel to be synthesized in the first image.

4. The plurality of images includes: a third image captured during a third exposure time that is shorter than the first exposure time and longer than the second exposure time; The determination unit The image processing device according to claim 1 , wherein the flicker is determined to have occurred when it is determined that a decrease in brightness has occurred in one of the second image and the third image.

5. The plurality of images includes: a third image captured during a third exposure time that is shorter than the first exposure time and longer than the second exposure time; The determination unit The image processing device according to claim 1 , wherein the moving object is determined to have occurred when it is determined that a decrease in brightness has occurred in the second image and the third image.

6. correcting a first red component value of the first image using a first white balance coefficient for the red component; The image processing device of claim 1 , further comprising a color correction unit that corrects a color of the first image by correcting a first blue component value of the first image using a second white balance coefficient for the blue component.

7. The color correction unit generating a second red component value by multiplying the first red component value by the first white balance coefficient, and generating a third red component value by dividing a smaller first value of the second red component value and a green component saturation value by the first white balance coefficient; and replacing the first red component value of the first image with the third red component value; 7. The image processing device of claim 6, wherein a second blue component value is generated by multiplying the first blue component value by the second white balance coefficient, a third blue component value is generated by dividing a second smaller value of the second blue component value and the green component saturation value by the second white balance coefficient, and the first blue component value of the first image is replaced with the third blue component value, thereby correcting the color of the first image.

8. The color correction unit correcting the color of the first image when it is determined that the flicker has occurred; The image synthesis unit The image processing apparatus of claim 6 , wherein the corrected first image is used to combine the multiple images.

9. The determination unit 2. The image processing device according to claim 1, wherein, when a difference between a largest pixel value and a smallest pixel value among pixel values ​​of pixels to be synthesized that exist at the same position in the plurality of images is smaller than a threshold difference value, it is determined that the flicker and the moving object have not occurred.

10. The image synthesis unit The image processing device according to claim 1 , wherein, when it is determined that the flicker and the moving object have not occurred, the image processing device synthesizes the plurality of images while changing the plurality of weights according to a luminance value of any one of the plurality of images.

11. The image synthesis unit If a luminance value of a first pixel of the first image is less than a first threshold luminance value, generating a composite image using the pixel value of the first pixel; If the luminance value of the first pixel is equal to or greater than the first threshold luminance value and is smaller than the second threshold luminance value, the composite image is generated by adding a pixel value of a second pixel that is located at the same position as the first pixel in the second image to the pixel value of the first pixel at a predetermined ratio; The image processing device of claim 10 , further comprising: generating the composite image using a pixel value of the second pixel when the luminance value of the first pixel is equal to or greater than the second threshold luminance value.

12. The predetermined ratio is The image processing apparatus of claim 11 , wherein the luminance value of the first pixel changes linearly as it increases from the first threshold luminance value to the second threshold luminance value.

13. 2. The image processing device of claim 1, further comprising a gain control unit that generates the first image by correcting a first input image using a first gain, and that generates the second image by correcting a second input image using a second gain greater than the first gain.

14. The gain control unit The image processing apparatus of claim 13 , wherein the ratio of the first gain to the second gain is determined to be the inverse ratio of the first exposure time to the second exposure time.

15. a determination unit that determines whether a ratio of the number of pixels having a luminance value equal to or less than a threshold luminance value among the pixels to be combined to the number of pixels to be combined that exist at the same position in a plurality of images is smaller than a threshold ratio; an image synthesis unit that performs image synthesis while changing a plurality of weights for each of the synthesis target pixels based on the determination result; 12. An image processing device comprising:

16. The plurality of images includes: a first image captured during the longest exposure time among the plurality of images; The image synthesis unit The image processing device according to claim 15 , wherein, when the ratio is smaller than the threshold ratio, the image synthesis is performed by setting a first weight for the first image among the plurality of weights to the largest value.

17. The plurality of images includes: a second image captured during the shortest exposure time among the plurality of images; The image synthesis unit The image processing device of claim 15 , wherein, when the ratio is equal to or greater than the threshold ratio, the image synthesis is performed by setting a second weight for the second image among the plurality of weights to the largest value.

18. The plurality of images includes: a first image captured during the longest exposure time among the plurality of images; correcting a first red component value of the first image using a first white balance coefficient for the red component; The image processing device of claim 15, further comprising a color corrector that corrects a color of the first image by correcting a first blue component value of the first image using a second white balance coefficient for a blue component.

19. The color correction unit generating a second red component value by multiplying the first red component value by the first white balance coefficient, and generating a third red component value by dividing a smaller first value of the second red component value and a green component saturation value by the first white balance coefficient; and replacing the first red component value of the first image with the third red component value; 19. The image processing device of claim 18, wherein a color of the first image is corrected by multiplying the first blue component value by the second white balance coefficient to generate a second blue component value, dividing a second value that is smaller between the second blue component value and the green component saturation value by the second white balance coefficient to generate a third blue component value, and replacing the first blue component value of the first image with the third blue component value.

20. determining whether flicker or a moving object has occurred in the plurality of images based on luminance values ​​of pixels to be synthesized that exist at the same position in the plurality of images; performing image synthesis while changing a plurality of weights for each of the synthesis target pixels based on the determination result; An image processing method comprising:

Images