Electronic device and method for operating the same

Abstract

The present disclosure relates to an electronic device and an operating method of the electronic device. The electronic device includes: an image sensor including normal pixels and phase detection pixels; and a processor for detecting a noise region in a flare region included in a normal image generated based on a plurality of pixel values ​​sensed by the normal pixels and the phase detection pixels, and generating an output image including a final pixel value obtained by correcting the pixel value using the peripheral pixel values ​​of each of the pixel values ​​included in the noise region.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Korean Patent Application No. 10-2021-0124510 filed on September 17, 2021, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to an electronic device and an operating method of the electronic device, and more particularly to an electronic device configured to perform image processing and an operating method of the electronic device. Background Art

[0004] In recent years, with the development of electronic technology, the demand for image sensors in various electronic devices including smartphones, digital cameras, game consoles, the Internet of Things, robots, surveillance cameras, medical cameras, self-driving cars, etc. has continued to increase.

[0005] The image sensor can generate an image by converting light into an electrical signal. To this end, the image sensor may include normal pixels for sensing light and phase detection pixels for focusing on an object. Normal pixels may be arranged according to a Bayer format, etc. Phase detection pixels may be arranged at some positions among the arrangement positions of normal pixels instead of some normal pixels. Phase detection pixels perform a focus adjustment function, but may be regarded as a defect in a normal image. To solve this problem, the pixel value of the phase detection pixel may be compensated.

[0006] Meanwhile, when light is received from a light source having a strong light intensity (e.g., the sun or lighting) to the image sensor, the image sensor may generate an image including a flare area. The flare area may include a high brightness area representing strong light and a peripheral area where light from the high brightness area propagates. When the pixel value of the phase detection pixel is compensated in the flare area by using the same method as the normal area instead of the flare area, noise may occur, such as point noise where a point appears at a certain position or grid noise where a dark part appears in a certain area. Therefore, a technology for solving the problem of noise generated in the flare area is needed. Summary of the invention

[0007] According to one aspect of the present disclosure, an electronic device can be provided, comprising: an image sensor including normal pixels and phase detection pixels; and a processor configured to detect a noise area in a flare area included in a normal image generated based on a plurality of pixel values ​​sensed by the normal pixels and the phase detection pixels, and to generate an output image including a final pixel value obtained by correcting the pixel value using the surrounding pixel values ​​of each of the pixel values ​​included in the noise area.

[0008] According to another aspect of the present disclosure, a method for operating an electronic device can be provided, the method comprising: generating a normal image based on multiple pixel values ​​sensed by normal pixels and phase detection pixels; detecting a noise area in a flare area included in the normal image; and generating an output image including a final pixel value, wherein the final pixel value is obtained by correcting the pixel value included in the noise area using the surrounding pixel values ​​of each of the pixel values. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Examples of embodiments will be described more fully hereinafter with reference to the accompanying drawings; however, they may be embodied in different forms and should not be construed as limited to the embodiments set forth herein.

[0010] In the accompanying drawings, dimensions may be exaggerated for clarity of illustration. It should be understood that when an element is referred to as being "between" two elements, it may be the only element between the two elements, or one or more intermediate elements may also be present. The same reference numerals always refer to the same elements.

[0011] Figure 1 is a diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0012] Figure 2 is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0013] Figure 3A , Figure 3B and Figure 3C is a diagram showing pixels included in an image sensor according to an embodiment of the present disclosure.

[0014] Figure 4 is a diagram illustrating a processor according to an embodiment of the present disclosure.

[0015] Figure 5 is a diagram illustrating an image processing operation according to an embodiment of the present disclosure.

[0016] Figure 6 is a diagram showing an image according to an embodiment of the present disclosure.

[0017] Fig. 7A , Figure 7B and Figure 7C is a diagram illustrating detecting a flat area according to an embodiment of the present disclosure.

[0018] Fig. 8A and Figure 8B is a diagram illustrating a detection flare area according to an embodiment of the present disclosure.

[0019] Fig. 9is a diagram illustrating a corrected noise region according to an embodiment of the present disclosure.

[0020] Fig. 10A and Fig. 10B is a diagram illustrating primary correction of a central pixel according to an embodiment of the present disclosure.

[0021] Fig.11A , Fig. 11B and Fig. 11C is a diagram illustrating secondary correction of a central pixel according to an embodiment of the present disclosure.

[0022] Fig. 12A , Fig. 12B , Fig. 12C , Fig.12D and Fig.12E is a diagram illustrating correction of an image having a quad-Bayer format according to an embodiment of the present disclosure.

[0023] Fig.13A , Fig. 13B , Fig. 13C , Fig.13D and Fig.13E is a diagram illustrating correction of an image having a quad-Bayer format according to an embodiment of the present disclosure.

[0024] Fig.14 is a diagram showing a correction result of a noise region according to an embodiment of the present disclosure.

[0025] Fig.15 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure.

[0026] Fig.16 is a diagram illustrating an operating method of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The specific structural or functional description disclosed herein is only for describing the embodiments according to the concept of the present disclosure. The embodiments according to the concept of the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein.

[0028] The embodiment provides an electronic device for removing noise in a flare region and an operating method of the electronic device.

[0029] Figure 1 is a diagram showing an electronic device according to an embodiment of the present invention.

[0030] refer to Figure 1 , the electronic device 100 according to an embodiment of the present disclosure may include an image sensor 110 and a processor 130 .

[0031] The electronic device 100 according to an embodiment of the present disclosure may be an image pickup device, a digital camera, a video camera, a closed-circuit television (CCTV), a webcam, a security camera, an industrial visual camera, a mobile device, a smart phone, a personal computer (PC), a tablet computer, a notebook computer, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a portable multimedia player (PMP), a wearable device, a black box, a robot, an autonomous vehicle, a vehicle-mounted visual camera, a set-top box, a game console, an electronic dictionary, an e-book reader, a desktop computer, a server, an MP3 player, an intelligent medical device, a television, a digital video disc (DVD) player, an audio system, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a smart mirror, a smart window, an electronic key, an electronic photo frame, a digital billboard, a security control panel, etc. The wearable device may be a smart watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, a head-mounted device (HMD), a skin pad, an electronic tattoo, or a biological implantable circuit, etc.

[0032] The electronic device 100 according to an embodiment of the present disclosure may be implemented in the form of a package module, a component, etc. That is, the electronic device 100 may be operated as a component included in a computing system. For example, the computing system may be implemented as various devices, including digital cameras, mobile devices, smart phones, personal computers (PCs), tablet PCs, notebook computers, personal digital assistants (PDAs), enterprise digital assistants (EDAs), portable multimedia players (PMPs), wearable devices, black boxes, robots, self-driving cars, etc.

[0033] The image sensor 110 may acquire an image by sensing light, and may be implemented as a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

[0034] The image sensor 110 may include a plurality of pixels. The plurality of pixels may be arranged along rows and columns. Specifically, the image sensor 110 may include normal pixels and phase detection pixels. The image sensor 110 may acquire an image including pixel values ​​sensed by normal pixels and pixel values ​​sensed by phase detection pixels.

[0035] A normal pixel may be a pixel including a color filter and a sensing circuit disposed at the bottom of one microlens. The color filter is used to sense light of a specific color or a specific wavelength. For example, a green color filter may sense green light. A phase detection pixel may be a pixel including a sensing circuit disposed at the bottom of a portion of a microlens. That is, a phase detection pixel may be a pixel that shares a microlens with another phase detection pixel. For example, a left phase detection pixel may sense light received only through the left portion of the microlens, while a right phase detection pixel may sense light received only through the right portion of the microlens. The left and right phase differences may be detected by comparing the pixel values ​​sensed by the left phase detection pixel and the right phase detection pixel, respectively, and then the left and right phase differences may be detected by adjusting the focus on the object and sensing the light again. Such an operation is repeated until the left and right phase differences are reduced, so that the focus on the object can be automatically adjusted.

[0036] An image may include pixel data. Pixel data may be acquired through a plurality of pixels. One pixel data may include information associated with one pixel for acquiring corresponding pixel data. Specifically, pixel data may include information about pixel value, color, and position. Pixel value may represent the brightness of light sensed by the corresponding pixel. Pixel value may have a value within the range of allocated data bits. For example, when the data bit corresponds to 8 bits, the pixel value may have a natural value within the range of 0 to 255. When the data bit corresponds to 10 bits, the pixel value may have a natural value within the range of 0 to 1023. Position may represent the row and column in which the corresponding pixel is arranged. Color may represent the color of light sensed by the corresponding pixel. For example, the color may be one of red, green, and blue. However, this is only one embodiment, and the color is not limited to the above colors. Meanwhile, pixel data may include information about pixel type, which indicates whether the corresponding pixel is a normal pixel or a phase detection pixel.

[0037] The processor 130 may control the overall operation of the electronic device 100. To this end, the processor 130 may include at least one processing device. For example, the processor 130 may include at least one of an image signal processor (ISP), a digital signal processor (DSP), an application processor (AP), a graphics processing unit (GPU), a central processing unit (CPU), a controller, etc.

[0038] When a user request is received, the processor 130 may control the image sensor 110 to acquire an image. Specifically, the processor 130 may receive the user request through an input interface. For example, the input interface may be implemented as one of a touch sensing panel, a keyboard, a mouse, a button, a microphone, etc. Alternatively, the processor 130 may receive a user request from an external device through a communication interface. The processor 130 may send a command to the image sensor 110 instructing the image sensor 110 to acquire an image. In addition, the processor 130 may receive an image acquired from the image sensor 110. The image may include pixel values ​​sensed by normal pixels and pixel values ​​sensed by phase detection pixels.

[0039] The processor 130 may detect a noise region in a flare region included in a normal image generated based on a plurality of pixel values ​​sensed by normal pixels and phase detection pixels. The processor 130 may generate an output image including a final pixel value obtained by correcting each of the pixel values ​​included in the noise region using the surrounding pixel values ​​of the pixel value.

[0040] A normal image may refer to an image in which pixel values ​​sensed by phase detection pixels are compensated to normal pixel values. A flare region may include: a high brightness region, which represents a light source with strong light intensity (such as the sun or lighting) or an object reflecting strong light; and a peripheral region, in which light is propagated from the high brightness region.

[0041] The noise region may be a region where noise such as point noise or grid noise appears in the flare region or a region where noise is expected to appear in the flare region. According to an embodiment, the noise region may be a region satisfying the condition that the region corresponds to the flare region and the flat region. The flat region is a region other than texture, edge, etc. in the image, and may represent a region with a small brightness difference from the surrounding region. For example, the flat region may represent a low-frequency region in the frequency domain.

[0042] As described above, according to the embodiments of the present disclosure, the electronic device 100 capable of removing noise in a flare region and the method of operating the same can be provided. Hereinafter, the present disclosure will be described with reference to the accompanying drawings.

[0043] Figure 2 is a diagram illustrating an image sensor according to an embodiment of the present disclosure.

[0044] refer to Figure 2 , the image sensor 110 may include an optical lens LS, a pixel array 111 , a row decoder 112 , a timing generator 113 , a signal converter 114 , and an output buffer 115 .

[0045] The optical lens LS may refract incident light. The light refracted by the optical lens LS may be transmitted to the pixel array 111. To this end, the optical lens LS may be an assembly of micro lenses arranged on the pixel array 111. At the same time, the light may include information about the object, such as the size, shape, position, and color of the object. The object may be an object to be photographed, such as a person, object, animal, or plant existing outside.

[0046] The pixel array 111 may include a plurality of pixels. The plurality of pixels may be arranged in a row direction and a column direction. The pixels of the pixel array 111 may correspond to pixel values ​​of the image IMG, respectively. The pixels of the pixel array 111 may be pixels arranged in a physical area, and the pixel values ​​of the image IMG may be pixel values ​​arranged in a digital area. The pixels of the pixel array 111 and the pixel values ​​of the image IMG may have a corresponding relationship relative to the same arrangement position.

[0047] Each pixel may include a color filter and a sensing circuit. The pixels may be arranged in various arrangements such as a Bayer format and a quad Bayer format. This will be referred to below. FIG. 3A to FIG. 3C Give a description.

[0048] The color filter may be disposed between the microlens and the sensing circuit. Light refracted by the microlens may reach the sensing circuit while passing through the color filter. The color filter may allow light of a specific color or a specific wavelength to pass through and block light of other colors or other wavelengths. The color filter may have a color corresponding to one of red, green, and blue. Depending on the color of the color filter included in the pixel, the pixel may be designated as a red pixel, a green pixel, or a blue pixel. However, this is only one embodiment, and the color of the color filter may be changed in various ways.

[0049] The sensing circuit may be disposed at the bottom of the color filter. The sensing circuit may include a light sensing element. The light sensing element may generate an electrical pixel signal from the received light using a photoelectric effect. For example, when light is received by the light sensing element for a period of exposure time, a charge corresponding to the light may be accumulated in the sensing circuit, and a pixel signal corresponding to the charge accumulated in the sensing circuit may be generated. To this end, the light sensing element may be implemented as various semiconductor elements, such as a pn junction diode, a positive-intrinsic-negative (PIN) photodiode, an avalanche photodiode (APD), and a phototransistor.

[0050] The row decoder 112 may select pixels for reading pixel data under the control of the timing generator 113. For example, the row decoder 112 may select pixels located on a row corresponding to an address output from the timing generator 113 from among a plurality of pixels included in the pixel array 111 in response to an address and a control signal.

[0051] The timing generator 113 may control the row decoder 112 and the signal converter 114 to read pixel data from pixels located on a specific row in the pixel array 111. For example, the timing generator 113 may sequentially output row addresses representing pixels located on a specific row to the row decoder 112. The timing generator 113 may output a command for adjusting an exposure time of pixels located on a specific column to the row decoder 112.

[0052] The signal converter 114 may acquire pixel data by using the pixel signal received from the pixel array 111 and the row address received from the timing generator 113. In addition, the signal converter 114 may transfer the pixel data to the output buffer 115.

[0053] Specifically, the signal converter 114 can be connected to the pixels of the pixel array 111 through a plurality of column lines. The number of pixels is proportional to the number of rows connected to one column line. The signal converter 114 can receive pixel signals about pixels located on one row through each column line. That is, the signal converter 114 can receive pixel signals in units of rows. The signal converter 114 can obtain pixel values ​​corresponding to the levels of the pixel signals received from the pixel array 111. For example, the pixel signal can be a voltage. To this end, the signal converter 114 can include an analog-to-digital converter. In addition, based on the color information of the pixels mapped to each row address, the signal converter 114 can obtain information related to the color of the pixels located on the row corresponding to the row address received from the timing generator 113.

[0054] The output buffer 115 may be connected to the signal converter 114 through a plurality of column lines. The output buffer 115 may sequentially store pixel values ​​received from the signal converter 114 in row units. When pixel values ​​regarding all rows are stored, the output buffer 115 may output one image IMG including the stored pixel values.

[0055] FIG. 3A to FIG. 3C is a diagram showing pixels included in an image sensor according to an embodiment of the present disclosure.

[0056] refer to FIG. 3A to FIG. 3C, each of the image sensors 310 and 330 according to the embodiments of the present disclosure may include a plurality of pixels. A plurality of pixels may be arranged in a fixed number along the row and column directions. A plurality of pixels may include normal pixels and phase detection pixels. According to one embodiment, the normal pixels may include Gr pixels, R pixels, B pixels, and Gb pixels. The Gr pixel (or the first green pixel) and the Gb pixel (or the second green pixel) may sense light of green (Gr and Gb) colors, the R pixel (or the red pixel) may sense light of red (or R) colors, and the B pixel (or the blue pixel) may sense light of blue (B) colors. However, this is only one embodiment, and the normal pixels may be modified to pixels that sense light of various colors. In the following, for the convenience of description, it is assumed and described that the normal pixels include Gr pixels, R pixels, B pixels, and Gb pixels.

[0057] refer to Figure 3A , normal pixels included in the image sensor 310 according to an embodiment of the present disclosure may be regularly arranged according to the Bayer format.

[0058] For example, the image sensor 310 may include a plurality of Bayer regions 311 arranged in row and column directions. Each Bayer region 311 may include a unit region arranged in 2×2. Each unit region may include a 1×1 pixel. For example, each Bayer region 311 may include a Gr pixel, an R pixel, a B pixel, and a Gb pixel, and the Gr pixel, the R pixel, the B pixel, and the Gb pixel may be arranged in 2×2.

[0059] The phase detection pixels 315 may be arranged at the positions of some normal pixels among the normal pixels. The phase detection pixels 315 may be arranged at positions adjacent to each other to share one microlens. In one embodiment, the phase detection pixels 315 may include phase detection pixels arranged in 1×2. For example, the phase detection pixel 315 may include a left phase detection pixel LP and a right phase detection pixel RP that share one microlens. However, this is only one embodiment, and the phase detection pixel 315 may be implemented in various forms in which the phase detection pixel 315 is arranged in 2×1, 2×2, etc. At the same time, the peripheral normal pixels 317 of the phase detection pixel 315 may include the most adjacent pixels in the row direction, column direction, and diagonal direction of the row and column relative to the position of the phase detection pixel 315. Alternatively, the peripheral normal pixels 317 of the phase detection pixel 315 may include pixels located within a predetermined distance in the row direction, column direction, and diagonal direction of the row and column relative to the position of the phase detection pixel 315. The pixel value sensed by the peripheral normal pixel 317 may be affected by crosstalk such as due to the phase detection pixel 315 adjacent to the peripheral normal pixel 317. According to an embodiment of the present disclosure, the electronic device 100 can correct the pixel values ​​of the pixels included in the noise area among the phase detection pixels 315 and the surrounding normal pixels 317. In the present disclosure, for the convenience of description, it is assumed and described that the phase detection pixel 315 includes a left phase detection pixel LP and a right phase detection pixel RP that share a microlens. The word "predetermined" as used herein for a parameter, such as a predetermined distance, number, minimum limit, maximum limit or numerical value, means that the value of the parameter is determined before the parameter is used in a process or algorithm. For some embodiments, the value of the parameter is determined before the process or algorithm starts. In other embodiments, the value of the parameter is determined during the process or algorithm but before the parameter is used in the process or algorithm.

[0060] In one embodiment, the phase detection pixels 315 may be regularly arranged for each set unit area 313. The unit area 313 may have a size determined according to the arrangement form of the pixels (e.g., Bayer format, quad Bayer format, etc.). For example, the unit area 313 may be an area including 6×6 pixels. However, this is only one embodiment, and the size of the unit area 313 may be changed differently. In another embodiment, the phase detection pixels 315 may be arranged at random positions.

[0061] refer to Figure 3B, showing a cross-sectional structure 315V of a phase detection pixel 315 and a cross-sectional structure 317V of a peripheral normal pixel 317. In one embodiment, the pixel array 111 may include a microlens layer, a color filter layer, and a sensing circuit layer. The microlens layer may include a plurality of microlenses. The color filter layer may include a plurality of color filters B, Gb, CF. The sensing circuit layer may include a plurality of sensing circuits PD.

[0062] The phase detection pixels 315 may share one microlens 315M. The left phase detection pixel LP may include a color filter 315LF and a sensing circuit 315LP disposed at the bottom left side of one microlens 315M. The right phase detection pixel RP may include a color filter 315RF and a sensing circuit 315RP disposed at the bottom right side of one microlens 315M. In one embodiment, the color filter 315LF of the left phase detection pixel LP and the color filter 315RF of the right phase detection pixel RP may have the same color. For example, the color filter 315LF of the left phase detection pixel LP and the color filter 315RF of the right phase detection pixel RP may have a green color. As the difference between the pixel value sensed in the left phase detection pixel LP and the pixel value sensed in the right phase detection pixel RP becomes larger, the phase difference may become larger (i.e., a state where the focus is not adjusted). However, this is only one embodiment, and the color filter 315LF of the left phase detection pixel LP and the color filter 315RF of the right phase detection pixel RP may have different colors.

[0063] The peripheral normal pixel located on the left side of the phase detection pixel 315 may include a color filter 317F and a sensing circuit 317P disposed at the bottom of one microlens 317M. The peripheral normal pixel located on the right side of the phase detection pixel 315 may include a color filter 318F and a sensing circuit 318P disposed at the bottom of one microlens 318M. The microlenses 317M and 318M of the peripheral normal pixel may have a different size or curvature from the microlens 315M of the phase detection pixel 315.

[0064] refer to Figure 3C , normal pixels included in the image sensor 330 according to an embodiment of the present disclosure may be regularly arranged according to the quad Bayer format.

[0065] For example, the image sensor 330 may include a plurality of four Bayer regions 331 arranged in row and column directions. Each of the four Bayer regions 331 may include unit regions arranged in 2×2. Each unit region may include pixels arranged in 2×2. Each unit region may include pixels of the same color. For example, the first unit region may include Gr pixels arranged in 2×2. The second unit region may include R pixels arranged in 2×2. The third unit region may include Gb pixels arranged in 2×2. The fourth unit region may include B pixels arranged in 2×2.

[0066] The phase detection pixel 335 may be arranged at the position of some normal pixels among the normal pixels. The phase detection pixels 335 may be arranged at positions adjacent to each other to share one microlens. At the same time, the peripheral normal pixels 337 of the phase detection pixel 335 may include pixels located within a predetermined distance in the row direction, the column direction, and the diagonal direction of the row and column relative to the position of the phase detection pixel 335. According to an embodiment of the present disclosure, the electronic device 100 may correct the pixel values ​​of the pixels included in the noise area among the phase detection pixel 335 and the peripheral normal pixels 337.

[0067] In one embodiment, the phase detection pixels 335 may be regularly arranged for each set unit area 333. The unit area 333 may have a size determined according to the arrangement form of the pixels (e.g., Bayer format, quad Bayer format, etc.). For example, the unit area 333 may be an area including 8×8 pixels. However, this is only one embodiment, and the size of the unit area 333 may be changed differently. In another embodiment, the phase detection pixels 335 may be arranged at random positions.

[0068] Figure 4 is a diagram illustrating a processor according to an embodiment of the present disclosure.

[0069] refer to Figure 4 According to an embodiment of the present disclosure, the processor 130 may include a normal image generator 131, a noise detector 133, and a pixel corrector 135. In one embodiment, each of the normal image generator 131, the noise detector 133, and the pixel corrector 135 may be implemented as a software module including instructions, programs, etc. executed by the processor 130. The processor 130 or the electronic device 100 may also include a memory storing the software module. In one embodiment, each of the normal image generator 131, the noise detector 133, and the pixel corrector 135 may be implemented as hardware, such as a logic circuit, a component, an ISP, a DSP, an AP, a CPU, or a GPU.

[0070] The normal image generator 131 may receive an image IMG including pixel values ​​sensed by normal pixels and pixel values ​​sensed by phase detection pixels from the image sensor 110. The normal image generator 131 may acquire a compensated pixel value obtained by correcting a pixel value sensed by a phase detection pixel using a pixel value sensed by a peripheral normal pixel of each phase detection pixel among the normal pixels. The normal image generator 131 may generate a normal image nIMG including a compensated pixel value and a pixel value sensed by the normal pixel. A pixel value included in a noise region among the compensated pixel values ​​may be corrected to a final pixel value by the pixel corrector 135.

[0071] The noise detector 133 may determine whether each of a plurality of regions included in the normal image nIMG received from the normal image generator 131 is a flat region. Specifically, the noise detector 133 may determine whether a selected region is a flat region based on a pixel value included in a selected region among the plurality of regions of the normal image nIMG.

[0072] According to one embodiment, the noise detector 133 may include a flat area detector 133 a .

[0073] The flat area detector 133a may align the first kernel on a selected area among a plurality of areas of the normal image. When a standard deviation value based on a difference between a central pixel value overlapping with a central area of ​​a plurality of unit areas among a plurality of pixel values ​​included in the selected area and an average value of pixel values ​​overlapping with the first kernel and having the same color as the central pixel is less than a first threshold value, the flat area detector 133a may determine that the selected area is a flat area. The first kernel may include a plurality of unit areas corresponding to an arrangement form of pixel values ​​included in the image.

[0074] According to one embodiment, the flat area detector 133a may adjust the first threshold according to a period to which the average value of the pixel values ​​included in the selected area belongs among a plurality of periods, and the plurality of periods may include a period in which the first threshold increases as the average value of the pixel values ​​included in the selected area increases.

[0075] The noise detector 133 may determine whether each of a plurality of regions included in the image IMG received from the image sensor 110 is a flare region. Specifically, the noise detector 133 may determine whether the selected region is a flare region based on pixel values ​​corresponding to the selected region sensed by the left phase detection pixel LP and the right phase detection pixel RP among the phase detection pixels.

[0076] According to one embodiment, the noise detector 133 may include a flare region detector 133 b .

[0077] When the ratio of the pixel value sensed by the left phase detection pixel LP to the pixel value sensed by the right phase detection pixel RP exceeds a second threshold corresponding to the selected area, the flare area detector 133b may determine that the selected area is a flare area. The left phase detection pixel LP and the right phase detection pixel RP may be pixels arranged at positions adjacent to each other.

[0078] When the selected area is a flare area, the noise detector 133 may generate mapping data MDATA indicating that the selected area is a noise area.

[0079] The pixel corrector 135 may determine a noise region among a plurality of regions included in the normal image nIMG based on the normal image nIMG and the mapping data MDATA. The pixel corrector 135 may obtain a final pixel value obtained by correcting the pixel value included in the noise region using the surrounding pixel value of each pixel value included in the noise region. The pixel corrector 135 may generate and output an output image oIMG including the final pixel value. This is because noise occurs in a region satisfying the flat region and the flare region, or it is highly likely that noise occurs in a region satisfying the flat region and the flare region.

[0080] The pixel corrector 135 can calculate a correction value based on a first average value of a first peripheral pixel value included in a unit area that is first closest to the unit area including the central pixel value among pixel values ​​having the same color as the central pixel value included in the noise area, a second average value of a second peripheral pixel value included in a unit area that is second closest to the unit area including the central pixel value among pixel values ​​having the same color as the central pixel value included in the noise area, and a correction rate.

[0081] The correction rate is a value multiplied by the difference between the first average value and the second average value, and can be adjusted according to the dynamic range.

[0082] In one embodiment, when the correction value is less than the limit value, the pixel corrector 135 may acquire a final pixel value obtained by correcting the central pixel value based on the sum of the central pixel value and the correction value.

[0083] In one embodiment, when the correction value is equal to or greater than the limit value, the pixel corrector 135 may acquire a final pixel value obtained by correcting the central pixel value based on the sum of the central pixel value and the limit value.

[0084] In one embodiment, the central pixel value may be a compensated pixel value obtained by compensating a pixel value sensed via a phase detection pixel. In another embodiment, the central pixel value may correspond to a pixel value sensed via a peripheral normal pixel located around the phase detection pixel. That is, the pixel values ​​sensed via the phase detection pixel or the peripheral normal pixel among the pixel values ​​included in the normal area may be corrected. However, this is merely one embodiment, and the present disclosure is not limited thereto. The pixel value to be corrected may be changed differently.

[0085] In one embodiment, the pixel corrector 135 can adjust the correction rate. Specifically, when the difference between the maximum value and the minimum value of the pixel value included in the predetermined number of peripheral areas of the unit area including the central pixel value among the plurality of unit areas is equal to or greater than the cutoff value, the pixel corrector 135 can reduce the correction rate as the difference increases. The predetermined number of peripheral areas can be areas overlapping with the second core. The difference between the maximum value and the minimum value of the pixel value can be specified as the dynamic range. The cutoff value is a predetermined value and can be a value less than the maximum value of the bit data.

[0086] In one embodiment, the pixel corrector 135 may adjust the limit value. Specifically, when the central pixel value is equal to or greater than the first reference value and less than the second reference value as a value greater than the first reference value, the pixel corrector 135 may increase the limit value as the central pixel value increases. When the central pixel value is equal to or greater than the second reference value and less than the third reference value as a value greater than the second reference value, the pixel corrector 135 decreases the limit value as the central pixel value increases. In one embodiment, when the central pixel value is less than the first reference value or equal to or greater than the third reference value, the pixel corrector 135 may change the limit value to a predetermined minimum limit value. When the limit value exceeds a predetermined maximum limit value, the pixel corrector 135 may change the limit value to a maximum limit value.

[0087] In one embodiment, the pixel corrector 135 may select a unit area including any one of the first peripheral pixel value and the second peripheral pixel value from among the plurality of unit areas as the central area. The pixel corrector 135 may obtain the first peripheral correction pixel value and the second peripheral correction pixel value, which are obtained by correcting the first peripheral pixel value and the second peripheral pixel value using the average pixel value of the pixel values ​​included in the first peripheral area corresponding to each central area and the average pixel value of the pixel values ​​included in the second peripheral area corresponding to each central area. When the correction value is less than the limit value, the pixel corrector 135 may obtain a value according to the sum of the central pixel value and the correction value and the weighted sum of the first peripheral correction pixel value and the second peripheral correction pixel value as the final pixel value.

[0088] In one embodiment, when each unit area among multiple unit areas includes a pixel value of 1×1, the first peripheral area can be a first unit area close to the center area among the multiple unit areas, and the second peripheral area can be a second unit area close to the center area among the multiple unit areas.

[0089] In one embodiment, when each of the plurality of unit areas includes 2×2 pixel values, the first peripheral areas may be central areas, and the second peripheral areas may be unit areas first close to the central areas among the plurality of unit areas.

[0090] Meanwhile, the electronic device 100 according to an embodiment of the present disclosure may include a memory. The memory may be included in the processor 130 or located outside the processor 130. The memory may store data required to drive the electronic device 100 or the processor 130. The memory may be implemented as a non-volatile storage element. For example, the memory may be configured as various non-volatile storage elements, such as a read-only memory (ROM) that can only read data, a one-time programmable (OTP) memory that can only write data once, an erasable programmable ROM (EPROM) that can erase and write stored data, a NAND flash memory, and a NOR flash memory.

[0091] Figure 5 is a diagram illustrating an image processing operation according to an embodiment of the present disclosure.

[0092] refer to Figure 5 , the processor 130 may acquire a normal image 510 from the image 500. The image 500 may include pixel values ​​sensed by normal pixels and phase detection pixels LP and RP of the image sensor 110. The processor 130 may acquire compensated pixel values ​​obtained by correcting pixel values ​​sensed by phase detection pixels LP and RP using pixel values ​​sensed by peripheral normal pixels of phase detection pixels LP and RP among the normal pixels. That is, the normal image 510 may be an image in which pixel values ​​sensed by phase detection pixels LP and RP among the pixels included in the image 500 are replaced by compensated pixel values. For example, the normal image 510 may include pixel values ​​sensed by normal pixels and compensated pixel values. This will be referred to below Figure 6 Give a description.

[0093] The processor 130 may detect a flat area among a plurality of areas of the normal image 510 by using an average value and a standard deviation of pixel values ​​included in each of the plurality of areas, and acquire information 530 about the flat area. This will be referred to below. FIG. 7A to FIG. 7C Give a description.

[0094] The processor 130 may detect a flare region among a plurality of regions included in the image 500 by using pixel values ​​sensed by the phase detection pixels LP and RP, and acquire information 550 about the flare region. This will be described below with reference to Fig. 8A and Figure 8B Give a description.

[0095] The processor 130 may obtain information 560 about the noise region by using the information 530 about the flat region and the information 550 about the flare region. The noise region may be a region among a plurality of regions that satisfies the following conditions: the region is a flat region and is a flare region. The information 560 about the noise region may be implemented in various forms, including mapping data, tables, etc.

[0096] Multiple regions of each of the image 500 and the normal image 510 may correspond to each other. Each of the multiple regions may be a region including pixel values ​​sensed by the phase detection pixels LP and RP or compensated pixel values. Each of the multiple regions may be a region defined as having a size corresponding to the arrangement form of the pixel values. For example, in the case of the Bayer format, each of the multiple regions may be a region including pixel values ​​arranged in 6×6. In the case of the quad Bayer format, each of the multiple regions may be a region including pixel values ​​arranged in 8×8. However, this is merely one embodiment, and the multiple regions may be modified and embodied as regions of various sizes.

[0097] The processor 130 may determine a noise region among a plurality of regions included in the normal image 510 based on the information 560 about the noise region. The processor 130 may acquire a final pixel value C obtained by correcting pixel values ​​included in the noise region of the normal image 510. The processor 130 may generate and output an output image 570 including the final pixel value C. That is, the output image 570 may be an image in which pixel values ​​included in the noise region among pixel values ​​included in the normal image 510 are replaced with the final pixel value C. The pixel values ​​included in the noise region may include pixel values ​​sensed by the phase detection sensing pixels LP and RP or pixel values ​​sensed by the peripheral normal pixels of the phase detection pixels LP and RP. This will be referred to below. Figures 9 to 13E Give a description.

[0098] Figure 6 is a diagram showing an image according to an embodiment of the present disclosure.

[0099] refer to Figure 6 , the image 600 acquired from the image sensor 110 may be converted into a normal image 610 .

[0100] The image 600 may include a plurality of pixel values. The plurality of pixel values ​​may be arranged in the same arrangement form as the plurality of pixels included in the image sensor 110. The arrangement form may be a Bayer format, a quad-Bayer format, etc. In the following, for ease of explanation, it is assumed that the plurality of pixel values ​​have an arrangement form in the Bayer format and described.

[0101] The plurality of pixel values ​​may include pixel values ​​sensed by normal pixels and pixel values ​​sensed by phase detection pixels. The pixel values ​​sensed by normal pixels may include green Gr pixel values, green Gb pixel values, red R pixel values, and blue B pixel values ​​601T, 601B, 601L, and 601R. The pixel values ​​sensed by phase detection pixels may include a pixel value 601C representing a left phase and a pixel value representing a right phase.

[0102] According to the Bayer format, the pixel value 601C representing the left phase may be the position where the B pixel value is arranged, and the pixel value representing the right phase may be the position where the Gb pixel value is arranged.

[0103] The processor 130 may acquire a B-compensated pixel value 611C obtained by correcting the pixel value 601C indicating the left phase according to arrangement information indicating that the image 600 is in the Bayer format.

[0104] In one embodiment, the processor 130 may obtain a value corresponding to the pixel value 601C representing the left phase from calibration information pre-stored in the memory as the compensation pixel value 611C. The calibration information may include a value mapped to a pixel value of a phase detection pixel. In addition, another calibration information may be used according to the color or position of the pixel value converted from the pixel value of the phase detection pixel. The calibration information may be generated by using a reference image acquired when the image sensor 100 captures a light source emitting a constant amount of light.

[0105] In one embodiment, the processor 130 may obtain an average value of the B pixel values ​​601T, 601B, 601L, and 601R closest to the pixel value 601C representing the left phase as the compensation pixel value 611C. In one embodiment, the processor 130 may obtain a weighted sum of the average value of the B pixel values ​​601T, 601B, 601L, and 601R closest to the pixel value 601C representing the left phase and the pixel value 601C representing the left phase as the compensation pixel value 611C.

[0106] In one embodiment, the processor 130 may obtain a weighted sum of a value corresponding to the pixel value 601C representing the left phase and an average value of the B pixel values ​​601T, 601B, 601L, and 601R closest to the pixel value 601C representing the left phase from the calibration information as the compensation pixel value 611C.

[0107] In a similar manner to this, the processor 130 can acquire a Gb-compensated pixel value obtained by correcting a pixel value representing the right phase.

[0108] FIG. 7A to FIG. 7C is a diagram illustrating detecting a flat area according to an embodiment of the present disclosure.

[0109] refer to Fig. 7A (1), the normal image 710 may include multiple regions. The normal image 710 may include multiple pixel values. The multiple pixel values ​​may be arranged in multiple regions. Fig. 7A , a case where each of the plurality of regions includes 6×6 pixel values ​​is shown, but this is merely one embodiment, and each of the plurality of regions may be modified and implemented differently.

[0110] refer to Fig. 7A According to (1) and (2), the processor 130 may select any one of the plurality of regions. In the following, it is assumed and described that the first region 711 is selected from among the plurality of regions.

[0111] The processor 130 may calculate the standard deviation STD about the selected first region 721 by using the pixel values ​​included in the selected first region 711 among the plurality of regions included in the normal image 710. For example, the processor 130 may calculate the standard deviation STD according to Fig. 7A The formula shown in (2) calculates the standard deviation STD with respect to the selected first region 721 by using the pixel values ​​included in the first region 711 .

[0112] The gain gain and the offset offset may be predetermined constant values. The center pixel value PV_c may be a pixel value located at the center in the row and column directions among the pixel values ​​included in the selected first area 711 or the selected partial area of ​​the first area 711. The average pixel value PV_avg may be an average value of pixel values ​​of the same color as the center pixel value PV_c among the pixel values ​​included in the selected first area 711 or the selected partial area of ​​the first area 711.

[0113] The central pixel value PV_c may be a green Gr pixel value or a Gb pixel value. In one embodiment, when the central pixel value PV_c is a Gr pixel value, the average pixel value PV_avg may be an average value of Gr pixel values ​​among pixel values ​​included in the selected first region 711 or a partial region of the selected first region 711. In another embodiment, when the central pixel value PV_c is a Gr pixel value, the average pixel value PV_avg may be an average value of Gr pixel values ​​and Gb pixel values ​​among pixel values ​​included in the selected first region 711 or a partial region of the selected first region 711. When the central pixel value PV_c is a Gb pixel value, a similar approach may be applied.

[0114] The selected partial region of the first region 711 may be a region overlapping with the first core.

[0115] For example, refer to Figure 7B , the first core 715 may include a plurality of unit regions. The plurality of unit regions may include a central region 715C. The central region 715C may be a unit region located at the center in the row and column directions.

[0116] like Figure 7B As shown in (1) to (4), the processor 130 may align the first cores 715 on some regions 711K to 714K of the selected first region 711 along the row direction and the column direction.

[0117] like Figure 7B As shown in (1) and (4), the processor 130 can obtain the green pixel value 711C or 714C overlapping with the central area 715C of the first core 715 as the above-mentioned central pixel value PV_c. In addition, the processor 130 can obtain the average value of the green pixel values ​​included in some areas 711K and 714K overlapping with multiple unit areas of the first core 715 as the above-mentioned average pixel value PV_avg.

[0118] At the same time, if Figure 7B As shown in (2) and (3) of FIG. 1 , when the color of the pixel value overlapping the central area 715C is not green, the processor 130 may obtain the average value of the green pixel values ​​included in the areas 712C and 713C overlapping the peripheral area of ​​the central area 715C of the first core 715 as the above-mentioned central pixel value PV_c. In addition, the processor 130 may obtain the average value of the green pixel values ​​included in some areas 712K and 713K overlapping the plurality of unit areas of the first core 715 as the above-mentioned average pixel value PV_avg.

[0119] Return to reference Fig. 7A ,like Fig. 7A As shown in (2) and (3) of FIG. 1 , the processor 130 may determine whether each of the plurality of regions is a flat region by comparing a standard deviation STD with respect to each of the plurality of regions with a first threshold value.

[0120] For example, when the standard deviation STD about the first area 721 is less than the first threshold, the processor 130 may generate information 730 indicating that the first area 731 is a flat area. When the standard deviation STD about the first area 721 is equal to or greater than the first threshold, the processor 130 may generate information 730 indicating that the first area 731 is an edge area. A flat area is an area where the pixel value varies less with the surrounding area, and may correspond to a low-frequency area in the frequency domain. An edge area is an area where the pixel value varies more with the surrounding area, and may correspond to a high-frequency area in the frequency domain.

[0121] refer to Fig. 7A and Figure 7C, the processor 130 may adjust the first threshold TH1 according to the average pixel value PV_avg of the pixel values ​​included in the selected first area 711 .

[0122] For example, Figure 7C The graph showing the mapping information between the average pixel value PV_avg and the reference value TH_STD shown in (1) of FIG. 1 may include a period in which the reference value TH_STD is maintained as the average pixel value PV_avg increases and a period in which the reference value TH_STD increases as the average pixel value PV_avg increases. The mapping information may be stored in a memory included in the electronic device 100 or the processor 130. The mapping information may be as follows: Figure 7C The predetermined graph or function shown in (1).

[0123] like Figure 7C As shown in (1) and (2) of FIG. 1 , the processor 130 may obtain the reference value TH_STD corresponding to the average pixel value PV_avg from the mapping information between the average pixel value PV_avg and the reference value TH_STD. In addition, the processor 130 may obtain a value obtained by multiplying the reference value TH_STD by a coefficient A and adding a constant B to the product as the first threshold TH1. At the same time, Figure 7C The graph shown in (1) is only one embodiment and various changes can be made. For example, when the data bit corresponds to 10 bits instead of 8 bits, the maximum value of the average pixel value PV_avg in the figure can be changed to 1023. In addition, the reference value TH_STD in the figure can also be changed.

[0124] For example, when the average pixel value PV_avg has a value belonging to a first period in which the average pixel value PV_avg is equal to or greater than 0 and less than 16, the reference value TH_STD may remain the same value. When the average pixel value PV_avg has a value belonging to a second period in which the average pixel value PV_avg is equal to or greater than 16 and less than 48, the reference value TH_STD may have a value proportional to the average pixel value PV_avg. When the average pixel value PV_avg has a value belonging to a third period in which the average pixel value PV_avg is equal to or greater than 48 and less than 192, the reference value TH_STD may remain the same value. When the average pixel value PV_avg has a value belonging to a third period in which the average pixel value PV_avg is equal to or greater than 192 and less than 255, the reference value TH_STD may have a value proportional to the average pixel value PV_avg. In addition, the processor 130 may acquire a value obtained by multiplying the reference value TH_STD by a coefficient A and adding a constant B to the product as the first threshold value TH1.

[0125] Fig. 8A and Figure 8B is a diagram illustrating a detection flare area according to an embodiment of the present disclosure.

[0126] refer to Fig. 8A and Figure 8B , the image 800 may include a plurality of regions. The image 800 may include a plurality of pixel values. The plurality of pixel values ​​may be arranged in a plurality of regions. The plurality of pixel values ​​may be arranged in the same arrangement form as the plurality of pixels included in the image sensor 110. The arrangement form may be a Bayer format, a quad-Bayer format, etc. In the following, for ease of explanation, it is assumed that the plurality of pixel values ​​have an arrangement form in a Bayer format and is explained.

[0127] The plurality of pixel values ​​may include pixel values ​​sensed by normal pixels and pixel values ​​sensed by phase detection pixels. For example, the pixel values ​​sensed by normal pixels may include R pixel values ​​sensed by R pixels, B pixel values ​​sensed by B pixels, Gb pixel values ​​sensed by Gb pixels, and Gr pixel values ​​sensed by Gr pixels. The pixel values ​​sensed by phase detection pixels may include left-phase pixel values ​​indicating left phases sensed by left phase detection pixels LP and right-phase pixel values ​​indicating right phases sensed by right phase detection pixels RP.

[0128] refer to Fig. 8A (1) and (2), the processor 130 may select any one of the multiple regions. For example, the processor 130 may select a region including a left-phase pixel value and a right-phase pixel value from among the multiple regions. In the following, it is assumed and described that the first region 801 including a left-phase pixel value and a right-phase pixel value from among the multiple regions included in the image 800 is selected.

[0129] The processor 130 may obtain a ratio LR of a left phase pixel value and a right phase pixel value about a first region 841 by using a left phase pixel value and a right phase pixel value included in a selected first region 801 among a plurality of regions included in the image 800. The processor 130 may obtain phase difference information 840 about each region including a left phase pixel value and a right phase pixel value among a plurality of regions. The phase difference information 840 may include a ratio LR for each region. The ratio LR may represent a phase difference.

[0130] refer to Fig. 8A (2) and (3), the processor 130 may determine whether the first region 841 is a flare region by comparing the ratio LR with respect to the first region 841 with the second threshold TH2.

[0131] For example, when the ratio LR about the first area 841 exceeds the second threshold TH2, the processor 130 may determine that the first area 851 is a flare area, and generate information 850 indicating that the first area 851 is a flare area. When the ratio LR about the first area 841 is equal to or less than the second threshold TH2, the processor 130 may determine that the first area 851 is a normal area, and generate information 850 indicating that the first area 851 is a normal area. The flare area may include: a high brightness area, which represents a light source with strong light intensity (such as the sun or lighting) or an object reflecting strong light; and a peripheral area, where light is transmitted from the high brightness area. The normal area may be an area representing a general object such as a person, an animal, an object, or a background, rather than a flare area.

[0132] Meanwhile, the second threshold TH2 of the present disclosure is a predetermined value and can be stored in a memory. Figure 8B The second threshold value TH2 is described.

[0133] For example, Figure 8B As shown in (1), the image sensor 110 can obtain a reference image 810R in a reference environment. The reference environment can be an environment where other light sources are blocked except for light sources with the same brightness and output color temperature of 3100K, 5100K, etc. The ratio LR of the left phase pixel value to the right phase pixel value included in each of the multiple regions included in the reference image 810R can be expressed as Figure 8B The phase difference image 840R shown in (2) of . The second threshold TH2 may be a value obtained by performing addition or subtraction on the ratio LR included in each of the plurality of regions of the phase difference image 840R and a predetermined value. The predetermined value may be a fixed constant value or a percentage value relative to each ratio LR. Meanwhile, although the image sensor 110 receives light having the same brightness in the reference environment, the ratio LR having the left phase pixel value and the right phase pixel value may have unequal values ​​depending on the region. Therefore, the second threshold TH2 may be set to different values ​​depending on the region.

[0134] For example, Figure 8B (3) shows an image 810F obtained by sensing a light source with strong light. The image 810F may include a flare region. The ratio LR of the left phase pixel value and the right phase pixel value included in each of the multiple regions included in the image 810F can be expressed as Figure 8B The processor 130 can detect the flare region in the plurality of regions by comparing the ratio LR included in the currently acquired phase difference image 840F with the second threshold TH2 included in the previously acquired phase difference image 840R for each of the same regions.

[0135] Meanwhile, according to one embodiment, the second threshold TH2 for each of the plurality of regions may be stored in a memory. In another embodiment, some of the plurality of regions are used as representative regions, and the second threshold TH2 for the representative regions may be stored in a memory.

[0136] Fig. 9 is a diagram illustrating a corrected noise region according to an embodiment of the present disclosure.

[0137] refer to Fig. 9 According to (1) and (2), the processor 130 according to an embodiment of the present disclosure can detect a noise region 911 among a plurality of regions included in the normal image 910. The noise region 911 can be a region among a plurality of regions that satisfies the conditions that the region is a flat region and a flare region. This is because noise such as point noise or grid noise is likely to appear in a region that satisfies the corresponding conditions.

[0138] The processor 130 may generate an output image 970 including a correction region 971 obtained by correcting the noise region 911. The noise region 911 may include compensated pixel values ​​of the corrected phase detection pixels and pixel values ​​of normal pixels. The correction region 971 may include final pixel values ​​obtained by correcting selected pixel values ​​among the pixel values ​​included in the noise region 911. The selected pixel values ​​may be compensated pixel values ​​or peripheral pixel values ​​located within a predetermined distance from each compensated pixel value.

[0139] In one embodiment, the processor 130 may correct the noise region 911 in kernel units.

[0140] Specifically, the processor 130 may align the second core 960 on the noise region 911. The second core 960 may include a plurality of unit regions. The plurality of unit regions may be arranged in row and column directions. The plurality of unit regions may include a central region. The central region may be a unit region located at the center in the row and column directions. For example, when the plurality of unit regions are arranged in 5×5, the central region may be a unit region arranged at (3,3). In another example, when the plurality of unit regions are arranged in 9×9, the central region may be a unit region arranged at (5,5). At least one pixel value among the pixel values ​​included in the noise region 911 may overlap with each unit region included in the second core 960.

[0141] The processor 130 may correct the pixel values ​​overlapping with the central area included in the second core 960 by using the pixel values ​​overlapping with the peripheral area of ​​the central area included in the second core 960. The peripheral area may be an area overlapping with pixel values ​​having the same color as the pixel values ​​overlapping with the central area. The peripheral area may include a first peripheral area close to the central area and a second peripheral area close to the central area.

[0142] For example, when Fig. 10A As shown, when the Bayer format is assumed, the central area included in the second core 960 may overlap with the pixel value arranged at (5,5). The first peripheral area included in the second core 960 may overlap with the pixel values ​​arranged at (4,4), (4,6), (6,4), and (6,6), respectively. The second peripheral area included in the second core 960 may overlap with the pixel values ​​arranged at (3,5), (5,3), (5,7), and (7,7), respectively.

[0143] For example, when Fig. 12A and 13C As shown, when assuming a quad-Bayer format, the central area included in the second core 960 may overlap with an area including pixel values ​​arranged at (5,5) to (6,6). The first peripheral area included in the second core 960 may overlap with an area including pixel values ​​arranged at (3,3) to (4,4), an area including pixel values ​​arranged at (3,7) to (4,8), an area including pixel values ​​arranged at (7,3) to (8,4), and an area including pixel values ​​arranged at (7,7) to (8,8). The second peripheral area included in the second core 960 may overlap with an area including pixel values ​​arranged at (1,5) to (2,6), an area including pixel values ​​arranged at (5,1) to (6,2), an area including pixel values ​​arranged at (5,9) to (6,10), and an area including pixel values ​​arranged at (9,5) to (10,6).

[0144] In one embodiment, the processor 130 may correct the pixel values ​​overlapping with the central area by using the pixel values ​​overlapping with the first peripheral area and the pixel values ​​overlapping with the second peripheral area. In one embodiment, the processor 130 may correct the pixel values ​​overlapping with the first peripheral area by using the pixel values ​​included in the peripheral area of ​​the first peripheral area. In addition, the processor 130 may correct the pixel values ​​overlapping with the second peripheral area by using the pixel values ​​included in the peripheral area of ​​the second peripheral area. The processor 130 may obtain the final pixel value of the pixel value overlapping with the central area using the corrected pixel values ​​obtained by correcting the pixel values ​​overlapping with the first peripheral area and the second peripheral area and the corrected pixel values ​​obtained by correcting the pixel values ​​overlapping with the central area.

[0145] Subsequently, the processor 130 may realign the position of the second core 960 and obtain the final pixel value of the pixel value overlapping with the central area of ​​the second core 960 in the above manner. By repeating the above operation, the processor 130 may generate an output image 970 including a correction area 971 obtained by correcting the noise area 911. FIG. 10A to FIG. 13E Describe the calibration method.

[0146] Fig. 10A and Fig. 10B is a diagram illustrating primary correction of a central pixel according to an embodiment of the present disclosure.

[0147] refer to Fig. 10A (1), the processor 130 may select a pixel value included in any one of the plurality of unit areas included in the noise area 1061 as the central pixel value PX0. For example, the processor 130 may align a kernel including a plurality of unit areas on the noise area 1061, and select a unit area corresponding to the central area of ​​the kernel as the area including the central pixel value PX0. Assume that the central pixel value PX0 is the pixel value P55 arranged at (5,5).

[0148] In one embodiment, FIG. 10A to FIG. 11C As shown, each of the plurality of unit regions may include a 1×1 pixel value. In another embodiment, as FIG. 12A to FIG. 13E As shown, each of the plurality of unit regions may include 2×2 pixel values.

[0149] The pixel value selected as the center pixel value PX0 may be a Gr pixel value or a Gb pixel value of green. However, this is merely one embodiment, and pixel values ​​of other colors may be selected.

[0150] refer to Fig. 10A In (2), the processor 130 may acquire a center-corrected pixel value cPX0 obtained by correcting the center pixel value PX0. The center-corrected pixel value cPX0 may be a final pixel value or a value used to calculate a final pixel value.

[0151] Specifically, when the correction value A is less than the limit value Limit, the processor 130 may obtain the center correction pixel value cPX0 based on the sum of the center pixel value PX0 and the correction value A. Alternatively, when the correction value A is equal to or greater than the limit value Limit, the processor 130 may obtain the center correction pixel value cPX0 based on the sum of the center pixel value PX0 and the limit value Limit.

[0152] The processor 130 can calculate the correction value A based on the first average value AVG(PX1) of the first peripheral pixel values ​​PX1 included in the unit area that is first closest to the unit area including the central pixel value PX0 among the pixel values ​​having the same color as the central pixel value PX0 included in the noise area 1061, the second average value AVG(PX2) of the second peripheral pixel values ​​PX2 included in the unit area that is second closest to the unit area including the central pixel value PX0 among the pixel values ​​having the same color as the central pixel value PX0 included in the noise area 1061, and the correction rate rate.

[0153] For example, the first peripheral pixel value PX1 may be a pixel value having the same color as the central pixel value PX0 and being the first closest to the central pixel value PX0. The second peripheral pixel value PX2 may be a pixel value having the same color as the central pixel value PX0 and being the second closest to the central pixel value PX0. As the priority order including the first level, the second level, etc. becomes lower, the pixel value may represent a pixel value closer to the central pixel value PX0.

[0154] In one embodiment, the processor 130 may obtain a value obtained by multiplying the difference between the first average value AVG(PX1) and the second average value AVG(PX2) by the correction rate rate and the gain value gain as the correction value A. The gain value may be a predetermined value, such as 1, 1 / 2, or 1 / 4. For example, the correction rate rate may be a value within a range of more than 0 and less than 1. However, this is merely one embodiment, and the correction rate rate may be variously modified and implemented.

[0155] refer to Fig. 10B (1), the processor 130 according to an embodiment of the present disclosure may adjust the correction rate rate according to the dynamic range DR. The dynamic range DR may represent the difference between the maximum value and the minimum value of the pixel values ​​included in a predetermined number of peripheral areas relative to a unit area including a central pixel value PX. For example, the dynamic range DR may represent the difference between the maximum value and the minimum value among 9×9 pixel values ​​based on the central pixel value PX. The pixel value may be a green pixel value. However, this is merely one embodiment, and the color of the pixel value applied to the dynamic range DR may be changed in various ways.

[0156] When the dynamic range DR is equal to or greater than the cut-off value cut-off, the processor 130 may reduce the correction rate as the dynamic range DR increases. When the dynamic range DR is less than the cut-off value cut-off, the processor 130 may apply a constant value. Since the area represents a flat area as the dynamic range DR decreases, the processor 130 may apply a correction value A that becomes larger. Since the area represents an edge area as the dynamic range DR increases, the processor 130 may apply a correction value A that becomes smaller. The cut-off value cut-off is a value that becomes a reference for distinguishing a flat area and an edge area from each other, and may be predetermined and stored in a memory. The maximum value of the dynamic range DR may be a maximum bit value max bit. For example, in the case of 8 bits, the maximum bit value max bit may be 255.

[0157] As described above, the processor 130 may adjust the correction rate rate to a variable value.

[0158] refer to Fig. 10B (2), according to an embodiment of the present disclosure, the processor 130 may adjust the limit value Limit according to the central pixel value PX0.

[0159] For example, when the central pixel value PX0 is equal to or greater than the first reference value and less than the second reference value, the processor 130 may increase the limit value Limit as the central pixel value PX0 increases. The second reference value may be a value greater than the first reference value. For example, the second reference value may be a median bit. For example, in the case of 8 bits, the median bit may be 128.

[0160] When the central pixel value PX0 is equal to or greater than the second reference value and less than the third reference value, the processor 130 may decrease the limit value Limit as the central pixel value PX0 increases. The third reference value may be a value greater than the second reference value.

[0161] When the central pixel value PX0 is less than the first reference value or equal to or greater than the third reference value, the processor 130 may change the limit value Limit to a predetermined minimum limit value Limit min.

[0162] A pixel value less than the first reference value may represent a dark portion, and a pixel value greater than the third reference value may represent a bright portion. Since the deviation between the Gb pixel value and the Gr pixel value appears smaller in the dark portion, the limit value Limit is set to be smaller. Since the bright portion cannot be well recognized due to human visual characteristics, the limit value Limit is set to be smaller.

[0163] When the limit value Limit exceeds a predetermined maximum limit value Limit max, the processor 130 may change the limit value Limit to the maximum limit value Limit max. The maximum limit value Limit max may be a value equal to or greater than a maximum bit value max bit.

[0164] As described above, the processor 130 may adjust the limit value Limit to a variable value.

[0165] Fig.11A and Fig. 11B is a diagram illustrating secondary correction of a central pixel according to an embodiment of the present disclosure.

[0166] Fig.11A It is a diagram showing a method of obtaining peripheral corrected pixel values ​​cPX0_p obtained by correcting first peripheral pixel values ​​PX1 (for example, pixel values ​​P44, P46, P64 and P66 located first close to pixel value P55 and having the same color as pixel value P55) located around a central pixel value PX0 (for example, pixel value P55) in a noise region 1162. Fig. 11B It is a diagram showing a method of obtaining peripheral corrected pixel values ​​cPX0_p by correcting second peripheral pixel values ​​PX2 (for example, pixel values ​​P35, P53, P57 and P75 located second close to pixel value P55 and having the same color as pixel value P55) located around a central pixel value PX0 (for example, pixel value P55) in a noise region 1163. Fig. 11C 1163 is a diagram showing a method of acquiring a final pixel value fPX0 by using the center correction pixel value cPX0 and the peripheral correction pixel value cPX0_p in the noise region 1163 .

[0167] refer to FIG. 11A to FIG. 11C According to an embodiment of the present disclosure, the processor 130 can obtain a final pixel value fPX0 regarding the central pixel value cPX0, and the final pixel value fPX0 is obtained by correcting the central pixel value PX0 using the central correction pixel value cPX0 and the peripheral correction pixel value cPX0_p obtained by correcting the peripheral pixel value PX0_p.

[0168] refer to Fig. 10A and Fig.11A The processor 130 may select a unit area including any one of the first peripheral pixel value PX1 and the second peripheral pixel value PX2 among the plurality of unit areas as the central area.

[0169] like Fig.11AAs shown in (1), the first peripheral pixel value P44 located at (4,4) among the first peripheral pixel values ​​PX1 will be mainly described. The processor 130 can select the unit area including the first peripheral pixel value P44 as the central area PX0_p, and select the first peripheral area PX1_p that is first close to the central area PX0_p and the second peripheral area PX2_p that is second close to the central area PX0_p.

[0170] In addition, if Fig.11A As shown in (2), the processor 130 may acquire a first peripheral correction pixel value cP44, which is obtained by correcting the first peripheral pixel value P44 using the average pixel value AVG(PX1_p) of the pixel values ​​included in the first peripheral area PX1_p and the average pixel value AVG(PX2_p) of the pixel values ​​included in the second peripheral area PX2_p. For example, the processor 130 may acquire a value obtained by adding the first peripheral pixel value P44 and a value obtained by multiplying the difference between the average pixel value AVG(PX1_p) of the pixel values ​​included in the first peripheral area PX1_p and the average pixel value AVG(PX2_p) of the pixel values ​​included in the second peripheral area PX2_p by a gain value gain as the first peripheral correction pixel value cP44. The gain value gain is a predetermined value such as 1, 1 / 2, or 1 / 4, and may be stored in a memory.

[0171] In the same manner as this, the processor 130 can obtain other first peripheral correction pixel values ​​cP46, cP64, and cP66 with respect to other first peripheral pixel values ​​P46, P64, and P66.

[0172] refer to Fig. 11B , the second peripheral pixel value P35 located at (3,5) among the second peripheral pixel values ​​PX2 will be mainly described. The processor 130 may select the unit area including the second peripheral pixel value P35 as the central area PX0_p, and select the first peripheral area PX1_p that is first close to the central area PX0_p and the second peripheral area PX2_p that is second close to the central area PX0_p.

[0173] In addition, the processor 130 may acquire a second peripheral correction pixel value cP35 obtained by correcting the second peripheral pixel value P35 using the average pixel value AVG(PX1_p) of the pixel values ​​included in the first peripheral area PX1_ and the average pixel value AVG(PX2_p) of the pixel values ​​included in the second peripheral area PX2_p. For example, the processor 130 may acquire a value obtained by adding the second peripheral pixel value P35 and a value obtained by multiplying the difference between the average pixel value AVG(PX1_p) of the pixel values ​​included in the first peripheral area PX1_p and the average pixel value AVG(PX2_p) of the pixel values ​​included in the second peripheral area PX2_p by a gain value gain as the second peripheral correction pixel value cP35. The gain value gain is a predetermined value such as 1, 1 / 2, or 1 / 4, and may be stored in a memory.

[0174] In the same manner as this, the processor 130 can obtain other first peripheral correction pixel values ​​cP53, cP57, and cP75 with respect to other first peripheral pixel values ​​P53, P57, and P75.

[0175] refer to Fig. 11C The processor 130 may obtain a final pixel value fPX0 regarding the central pixel value PX0 by using the central correction pixel value cPX0 regarding the central pixel value PX0 and the peripheral correction pixel values ​​cPX0_p regarding the peripheral pixel values ​​PX1 and PX2.

[0176] The processor 130 can obtain the final pixel value fPX0 about the central pixel value PX0 through the above operation by selecting the pixel value included in another unit area among the multiple unit areas included in the noise area 1061 as the central pixel value PX0. As described above, by repeating the above operation, the processor 130 can obtain the final pixel value fPX0 about each central pixel value PX0. The processor 130 can generate an output image including the final pixel value fPX0 obtained by correcting the central pixel value PX0.

[0177] Meanwhile, according to an embodiment of the present disclosure, the pixel value selected as the central pixel value PX0 may be the pixel value of the phase detection pixel or the compensation pixel value. In addition, the pixel value selected as the central pixel value PX0 may be the pixel value of the surrounding normal pixel of the phase detection pixel.

[0178] FIG. 12A to FIG. 12E is a diagram illustrating correction of an image having a quad-Bayer format according to an embodiment of the present disclosure.

[0179] refer to Fig. 12A , Fig. 12A(1) shows the arrangement of pixel values ​​included in the noise region 1261, and Fig. 12A (2) shows a unit area included in the noise area 1261.

[0180] The normal image may include a noise region 1261. The noise region 1261 may be divided into a plurality of unit regions 1262. It is assumed and described that each of the plurality of unit regions 1262 includes 2×2 pixel values ​​having the same color. This may indicate that the pixel values ​​are arranged according to the quad Bayer format.

[0181] refer to Fig. 12A (1) and (2), the processor 130 may select any one of the multiple unit areas 1262 included in the noise area 1261 as the central area. For example, the processor 130 may align a kernel including multiple unit areas on the noise area 1261, and select a unit area corresponding to the central area of ​​the kernel as the central area. Assume that the unit area A33 arranged at (3,3) is selected as the central area. The processor 130 may select the pixel value included in the central area A33 as the central pixel value A0.

[0182] refer to Fig. 12B , the processor 130 may acquire a center-corrected pixel value cA0 obtained by correcting the center pixel value A0 included in the center area A33. The center-corrected pixel value cA0 may be a final pixel value or a value used to calculate the final pixel value.

[0183] Specifically, when the correction value A is less than the limit value Limit, the processor 130 may obtain the center correction pixel value cA0 based on the sum of the center pixel value A0 and the correction value A. Alternatively, when the correction value A is equal to or greater than the limit value Limit, the processor 130 may obtain the center correction pixel value cA0 based on the sum of the center pixel value A0 and the limit value Limit.

[0184] The processor 130 can calculate the correction value A based on the first average value AVG(A1) of the first peripheral pixel values ​​A1 included in the unit areas A22, A24, A42 and A44 which are first close to the unit area A33 including the central pixel value A0 among the pixel values ​​having the same color as the central pixel value A0 included in the noise area 1261, the second average value AVG(A2) of the second peripheral pixel values ​​A2 included in the unit areas A13, A31, A35 and A53 which are second close to the unit area A33 including the central pixel value A0 among the pixel values ​​having the same color as the central pixel value A0 included in the noise area 1261, and the correction rate rate.

[0185] The first average value AVG(A1) may be an average value of the following values: an average value of pixel values ​​included in the first unit area A22, an average value of pixel values ​​included in the second unit area A24, an average value of pixel values ​​included in the third unit area A42, and an average value of pixel values ​​included in the fourth unit area A44.

[0186] The second average value AVG(A2) may be the average value of the following values: the average value of the pixel values ​​included in the fifth unit area A13, the average value of the pixel values ​​included in the sixth unit area A31, the average value of the pixel values ​​included in the seventh unit area A35, and the average value of the pixel values ​​included in the eighth unit area A53.

[0187] The processor 130 according to an embodiment of the present disclosure may adjust the correction rate according to the dynamic range DR. The processor 130 according to an embodiment of the present disclosure may adjust the limit value Limit according to the average value of the central pixel value A0. Since this has been described above, a detailed description will be omitted.

[0188] Fig. 12C and Fig.12D Used to describe a unit area for correcting pixel values ​​in a plurality of unit areas 1263 , 1264 , 1265 , and 1266 .

[0189] refer to Fig. 12A and Fig. 12C , the processor 130 may correct the first peripheral pixel value A1 by using the peripheral pixel values ​​of the first peripheral pixel value A1. The first peripheral pixel value A1 may be a pixel value included in unit areas A22, A24, A42, and A44 that are first closest to the central area A33 including the central pixel value A0, among the pixel values ​​having the same color as the central pixel value A0. The processor 130 may select any one of the unit areas A22, A24, A42, and A44 including the first peripheral pixel value A1 as the new central area A0_p. Assume that, as Fig. 12C As shown, the first unit area A22 may be selected as the central area A0_p, and the unit areas A11, A13, A31, and A33 may be selected as the peripheral areas A1_p closest to the central area A0_p.

[0190] The processor 130 can obtain the corrected pixel value cA22 regarding the first unit area A22 based on the pixel values ​​included in the first unit area A22, the average value AVG(A22) of the pixel values ​​included in the first unit area A22, and the average pixel value AVG(AVG(A11), AVG(A13), AVG(A31) and AVG(A33)) of the pixel values ​​included in the surrounding areas A11, A13, A31 and A33 that are first closest to the first unit area A22. For example, the processor 130 may obtain a value obtained by adding a value obtained by multiplying a difference between an average value AVG(A22) of pixel values ​​included in the first unit area A22 and an average pixel value AVG(AVG(A11), AVG(A13), AVG(A31), and AVG(A33)) of pixel values ​​included in the first peripheral areas A11, A13, A31, and A33 close to the first unit area A22 by a gain value gain as the peripheral correction pixel value cA22_p with respect to the first unit area A22. The gain value gain is a predetermined value such as 1, 1 / 2, or 1 / 4 and may be stored in a memory.

[0191] In the same way, Fig.12D As shown, the processor 130 may obtain the peripheral-corrected pixel values ​​cA24, cA42, and cA44 for the other unit areas A24, A42, and A44.

[0192] Fig.12E (1) is used to describe a unit area for correcting a pixel value among the plurality of unit areas 1267, and Fig.12E (2) is used to describe the pixel values ​​used to correct the pixel values ​​in the noise region 1268. Fig.12E (3) is used to describe the equation for correcting pixel values.

[0193] refer to Fig.12E , the processor 130 can obtain final pixel values ​​fP55 to fP66 by using the center correction pixel value cA0 and the peripheral correction pixel values ​​cA22, cA24, cA42 and cA44.

[0194] For example, in Fig. 12A Among the plurality of unit regions 1262 included in the noise region 1261 shown, pixel values ​​included in some regions A33, A22, A24, A42, and A44 among the plurality of unit regions 1262 may be corrected. The corrected some regions may include a corrected central region cA33 and corrected peripheral regions cA22, cA24, cA42, and cA44.

[0195] The processor 130 can select a central correction pixel value cP55 from among the central correction pixel values ​​cP55, cP56, cP65 and cP66 included in the corrected central area cA33, and select peripheral correction pixel values ​​cP33, cP37, cP73 and cP77 from among the peripheral correction pixel values ​​included in the corrected peripheral areas cA22, cA24, cA42 and cA44 that are located at the same arrangement position as the selected central correction pixel value cP55. The processor 130 can obtain a value based on the weighted sum of the selected central correction pixel value cP55 and the average value of the selected peripheral correction pixel values ​​cP33, cP37, cP73 and cP77 as the final pixel value fP55 regarding the central pixel value P55. The weight W can be a value from 0 to 1. By Fig.12E In the manner shown in (3), the processor 130 can obtain the final pixel values ​​fP56, fP65 and fP66 regarding the central pixel values ​​P56, P65 and P66.

[0196] The processor 130 can repeatedly acquire final pixel values ​​fP55, fP56, fP65, and fP66 about the central pixel values ​​P55, P56, P65, and P66 by selecting another unit area among the plurality of unit areas 1262 included in the noise area 1261 as the central area through the above operation. The processor 130 can generate an output image including the acquired final pixel values ​​fP55, fP56, fP65, and fP66.

[0197] FIG. 13A to FIG. 13E is a diagram illustrating correction of an image having a quad-Bayer format according to an embodiment of the present disclosure.

[0198] refer to FIG. 13A to FIG. 13E , the normal image may include a noise region 1361. The noise region 1361 may be divided into a plurality of unit regions 1362. It is assumed and described that each of the plurality of unit regions 1362 includes 2×2 pixel values ​​having the same color. This may mean that the pixel values ​​are arranged according to a quad Bayer format.

[0199] refer to Fig.13A (1) and (2), the processor 130 may select any one of the plurality of unit regions 1362 included in the noise region 1361 as the central region. For example, the processor 130 may align a kernel including a plurality of unit regions on the noise region 1361, and select a unit region corresponding to the central region of the kernel as the central region. It is assumed that a unit region including the central pixel value G0 is selected as the central region.

[0200] like Fig. 13BAs shown, when the correction value A is less than the limit value Limit, the processor 130 can obtain the center correction pixel value cG0 based on the sum of the center pixel value G0 and the correction value A. Alternatively, when the correction value A is equal to or greater than the limit value Limit, the processor 130 can obtain the correction pixel value cG0 based on the sum of the center pixel value G0 and the limit value Limit.

[0201] The processor 130 may calculate the correction value A based on the first average value AVG(G1), the second average value AVG(G2) and the correction rate rate. For example, the processor 130 may calculate the correction value A by multiplying the difference between the first average value AVG(G1) and the second average value AVG(G2) by the correction rate rate and the gain value gain.

[0202] The processor 130 can calculate: a first average value AVG(G1) of the average value of the pixel value G11 at the first position in the first unit area that is first close to the unit area including the central pixel value G0, the average value of the pixel value G12 at the second position in the first unit area, the average value of the pixel value G13 at the third position in the first unit area, and the average value of the pixel value G14 at the fourth position in the first unit area.

[0203] The processor 130 can calculate: a second average value AVG(G2) of the average value of the pixel value G21 at the first position in the second unit area arranged second closest to the unit area including the central pixel value G0, the average value of the pixel value G22 at the second position in the second unit area, the average value of the pixel value G23 at the third position in the second unit area, and the average value of the pixel value G24 at the fourth position in the second unit area.

[0204] The processor 130 according to an embodiment of the present disclosure may adjust the correction rate rate according to the dynamic range DR. The processor 130 according to an embodiment of the present disclosure may adjust the limit value Limit according to the average value of the central pixel value G0.

[0205] Fig. 13C and Fig.13D It is shown which unit regions are used to correct pixel values ​​among the plurality of unit regions 1363 , 1364 , 1365 , and 1366 .

[0206] refer to Fig. 13C and Fig.13D, the processor 130 may correct the first peripheral pixel values ​​G11 to G14 by using the peripheral pixel values ​​of the first peripheral pixel values ​​G11 to G14. The processor 130 may select any one of the unit areas including the first peripheral pixel values ​​G11 to G14 as the new central area G0_p. Assume that the unit area including the pixel values ​​arranged at (3,3) to (4,4) is selected as the new central area G0_p.

[0207] The processor 130 may acquire a correction pixel value cG0_p regarding the new central area G0_p based on the pixel values ​​included in the new central area G0_p, the average value AVG(G0_p) of the pixel values ​​included in the new central area G0_p, and the average pixel value AVG(AVG(G11), AVG(G12), AVG(G13), and AVG(G14)) of the pixel values ​​included in the first peripheral area that is first closest to the new central area G0_p. Since the acquired correction pixel value cG0_p is a pixel value of the peripheral area regarding the previous central area G0_p, in order to prevent confusion, the acquired correction pixel value cG0_p is designated as the peripheral correction pixel value cG0_p.

[0208] The average pixel value AVG (AVG(G11), AVG(G12), AVG(G13) and AVG(G14)) of the pixel values ​​included in the first peripheral area can be the average of the following values: the average of the pixel value G11 arranged at the first position in the first unit area, the average of the pixel value G12 arranged at the second position in the first unit area, the average of the pixel value G13 arranged at the third position in the first unit area, and the average of the pixel value G14 arranged at the fourth position in the first unit area.

[0209] In the same way, Fig.13D As shown, the processor 130 may acquire peripheral-corrected pixel values ​​regarding the pixel values ​​G01 , G02 , G03 , and G04 included in the new central area G0_p.

[0210] Fig.13E (1) is used to describe a unit area for correcting a pixel value among the plurality of unit areas 1367, and Fig.13E (2) is used to describe the pixel values ​​used to correct the pixel values ​​in the noise region 1368. Fig.13E (3) is used to describe the equation for correcting pixel values.

[0211] refer to Fig.13E, the processor 130 can obtain the final pixel value fP55 by using the central correction pixel value cP55 and the peripheral correction pixel values ​​cP33, cP37, cP73 and cP77. For example, the processor 130 can determine the peripheral correction pixel values ​​included in the peripheral area that is most adjacent to the unit area including the central correction pixel value cP55 among the pixel values ​​having the same color as the central correction pixel value cP55, and select the peripheral correction pixel values ​​cP33, cP37, cP73 and cP77 at the same position as the central correction pixel value cP55 among the peripheral correction pixel values ​​included in the peripheral area. In addition, the processor 130 can obtain a value based on the weighted sum of the average value of the peripheral correction pixel values ​​cP33, cP37, cP73 and cP77 at the same position as the central correction pixel value cP55 and the central correction pixel value cP55 as the final pixel value fP55. The weight W can be a value from 0 to 1. By comparing with Fig.13E In the same manner as (3), the processor 130 can obtain the final pixel values ​​fP56, fP65 and fP66 regarding the central pixel values ​​P56, P65 and P66.

[0212] The processor 130 can repeatedly acquire final pixel values ​​fP55, fP56, fP65, and fP66 about the central pixel values ​​P55, P56, P65, and P66 by selecting another unit area among the plurality of unit areas 1362 included in the noise area 1361 as the central area through the above operation. The processor 130 can generate an output image including the acquired final pixel values ​​fP55, fP56, fP65, and fP66.

[0213] Fig.14 is a diagram showing a correction result of a noise region according to an embodiment of the present disclosure.

[0214] refer to Fig.14 (1), the normal image 1410 acquired by the electronic device 100 may include a flare region. When the noise region 1415, which is a flat region and a flare region among the plurality of regions included in the normal image 1410, is enlarged, this may appear as Fig.14 As shown in (2). Fig.14 In the expanded noise region 1425 shown in (2), point noise may appear depending on the position. The point noise represents a pixel value whose difference from the surrounding pixel values ​​is equal to or greater than a constant value.

[0215] According to an embodiment of the present disclosure, the electronic device 100 may correct the pixel values ​​included in the noise region 1415 among the plurality of regions included in the normal image 1410. When the correction region obtained by correcting the noise region 1415 is enlarged, this may appear as follows: Fig.14 As shown in (3). Fig.14 The pixel values ​​in the expanded correction region 1475 shown in (3) can be uniformly presented. As described above, the electronic device 100 according to the embodiment of the present disclosure can remove the noise of the noise region 1425.

[0216] Fig.15 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure.

[0217] refer to Fig.15 , the electronic device 100 may be implemented as a computing system 2000. The computing system 2000 may include an image sensor 2010, a processor 2020, a storage device 2030, a memory device 2040, an input / output (I / O) device 2050, and a display device 2060. Fig.15 Although not shown, the computing system 2000 may further include a port capable of communicating with the storage device 2030 , the memory device 2040 , the I / O device 2050 , the display device 2060 , etc. or communicating with an external device.

[0218] The image sensor 2010 may acquire an image. The image sensor 2010 may be connected to the processor 2020 through an address bus, a control bus, and a data bus, or a communication line different therefrom, to perform communication.

[0219] The image sensor 2010 may be implemented in various types of packages. For example, at least some components of the image sensor 2010 may be implemented using packages such as package on package (PoP), ball grid array (BGA), chip scale package (CSP), plastic lead chip carrier (PLCC), plastic dual in-line package (PDIP), die in a waffle package, die in wafer form, chip on board (COB), ceramic dual in-line package (CERDIP), plastic metric quad flat package (MQFP), thin quad flat package (TQFP), small outline integrated circuit (SOIC), shrink small outline package (SSOP), thin small outline package (TSOP), system in package (SIP), multi-chip package (MCP), wafer-level manufacturing package (WFP), or wafer-level processing stack package (WSP). In some embodiments, the image sensor 2010 may be integrated with the processor 2020 in one chip, or the image sensor 2010 and the processor 2020 may be integrated in different chips.

[0220] The processor 2020 may include at least one of a central processing unit (CPU), an application processing unit (APU), a graphics processing unit (GPU), and the like.

[0221] The processor 2020 may be connected to the storage device 2030, the memory device 2040, and the I / O device 2050 through an address bus, a control bus, and a data bus to perform communication. According to an embodiment of the present disclosure, the processor 2020 may also be connected to an expansion bus such as a peripheral component interconnect (PCI) bus.

[0222] The storage device 2030 may store data including images, motion maps, etc. The data stored in the storage device 2030 may be saved not only when the computing system 2000 is driven but also when the computing system 2000 is not driven. For example, the storage device 2030 may be configured with at least one of all types of nonvolatile memory devices such as a flash memory device, a solid state drive (SSD), a hard disk drive (HDD), and an optical disk.

[0223] The storage device 2040 may store data including images, motion pictures, etc. The storage device 2040 may temporarily store data to be processed by the processor 2020 or temporarily store data processed by the processor 2020. The data stored in the storage device 2040 may be saved only when the computing system 2000 is driven. Alternatively, the data stored in the storage device 2040 may be saved not only when the computing system 2000 is driven, but also when the computing system 2000 is not driven. For example, the storage device 2040 may include a volatile memory device (such as a dynamic random access memory (DRAM) and a static random access memory (SRAM)) and a non-volatile memory device (such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory device).

[0224] The I / O device 2050 may include an input device and an output device. The input device is a device capable of inputting a user command through interaction, and may be implemented as, for example, a keyboard, a keypad, a mouse, a microphone, etc. The output device is a device capable of outputting data, and may be implemented as a printer, a speaker, etc.

[0225] The display device 2060 is a device for visually outputting an image. To this end, the display device 2060 can be implemented with various types of displays, such as a liquid crystal display (LCD) that uses a separate backlight unit (e.g., a light emitting diode (LED), etc.) as a light source to control the arrangement of liquid crystal molecules to adjust the degree of light emitted from the backlight unit passing through the liquid crystal (brightness of light or intensity of light), and a display using a light source, a self-luminous element (e.g., a mini LED with a size of 100 to 200 μm, a micro LED with a size of 100 μm or less, an organic LED (OLED), a quantum dot LED (QLED), etc.). The display device 2060 can emit red, green, and blue light corresponding to the output image to the outside.

[0226] Fig.16 FIG. 1 is a diagram illustrating an operating method of an electronic device according to an embodiment of the present invention.

[0227] refer to Fig.16 According to an embodiment of the present disclosure, the operating method of the electronic device may include: step S1610, generating a normal image based on a plurality of pixel values ​​sensed by normal pixels and phase detection pixels; step S1620, detecting a noise area in a flare area included in the normal image; and step S1630, generating an output image including a final pixel value obtained by correcting each pixel value included in the noise area by using surrounding pixel values ​​of the pixel value.

[0228] Specifically, the electronic device 100 may generate a normal image based on a plurality of pixel values ​​sensed by normal pixels and phase detection pixels ( S1610 ).

[0229] In one embodiment, the electronic device 100 may obtain a compensated pixel value obtained by correcting a pixel value sensed by a phase detection pixel using a pixel value sensed by a peripheral normal pixel of each phase detection pixel among normal pixels. The electronic device 100 may generate a normal image including the compensated pixel value and the pixel value sensed by the normal pixel.

[0230] Furthermore, the electronic device 100 may detect a noise region in the flare region included in the normal image (S1620).

[0231] In one embodiment, the electronic device 100 may determine whether the selected area is a flat area based on the pixel values ​​included in the selected area among the multiple areas of the normal image. The electronic device 100 may determine whether the selected area is a flare area based on the pixel values ​​corresponding to the selected area sensed by the left phase detection pixel and the right phase detection pixel among the phase detection pixels. When the selected area is a flat area and a flare area, the electronic device 100 may detect that the selected area is a noise area.

[0232] Furthermore, the electronic device 100 may generate an output image including a final pixel value obtained by correcting a pixel value using a peripheral pixel value of each pixel included in the noise region ( S1630 ).

[0233] In one embodiment, the electronic device 100 may select a pixel value included in any one of a plurality of unit areas included in the noise region as the central pixel value. The electronic device 100 may calculate the correction value based on a first average value of a first peripheral pixel value included in a unit area that is first closest to a unit area including the central pixel value included in the noise region among pixel values ​​having the same color as the central pixel value, a second average value of a second peripheral pixel value included in a unit area that is second closest to a unit area including the central pixel value included in the noise region among pixel values ​​having the same color as the central pixel value, and a correction rate. When the correction value is less than the limit value, the electronic device 100 may obtain a final pixel value obtained by correcting the central pixel value based on the sum of the central pixel value and the correction value.

[0234] According to the present disclosure, an electronic device for removing noise in a flare region and a method of operating the electronic device can be provided.

[0235] Although the present disclosure has been shown and described with reference to certain examples of its embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the examples of the above-mentioned embodiments, but should be determined not only by the appended claims but also by their equivalents.

[0236] In the above-described embodiments, all steps may be selectively performed or some steps may be performed, and some steps may be omitted. In each embodiment, these steps are not necessarily performed in the order described and may be rearranged. The embodiments disclosed in this specification and the accompanying drawings are only examples for facilitating understanding of the present disclosure, and the present disclosure is not limited thereto. That is, it should be clear to those skilled in the art that various modifications may be made based on the technical scope of the present disclosure.

[0237] Meanwhile, examples of embodiments of the present disclosure have been described in the drawings and the specification. Although specific terms are used herein, these terms are only used to explain the embodiments of the present disclosure. Therefore, the present disclosure is not limited to the above-described embodiments, and many changes can be made within the spirit and scope of the present disclosure. It should be clear to those skilled in the art that, in addition to the embodiments disclosed herein, various modifications can also be made based on the technical scope of the present disclosure.

Claims

1. An electronic device, comprising: an image sensor including normal pixels and phase detection pixels; as well as A processor that: detecting a noise region in a flare region included in a normal image generated based on a plurality of pixel values ​​sensed by the normal pixels and the phase detection pixels; and generating an output image including a final pixel value obtained by correcting each of the pixel values ​​included in the noise region by using the surrounding pixel values ​​of the pixel values, The processor includes a noise detector, which determines that a flat area in the flare area among multiple areas of the normal image is the noise area.

2. The electronic device according to claim 1, wherein: The processor comprises a normal image generator, wherein: acquiring a compensated pixel value obtained by correcting a pixel value sensed by the phase detection pixel using a pixel value sensed by a peripheral normal pixel of each of the phase detection pixels among the normal pixels; as well as The normal image including the compensated pixel values ​​and the pixel values ​​sensed by the normal pixels is generated.

3. The electronic device according to claim 2, wherein: Each of the pixel values ​​included in the noise region is any one of the compensation pixel values.

4. The electronic device according to claim 1, wherein: The noise detector further: determining whether the selected area is the flat area based on pixel values ​​included in the selected area among the plurality of areas of the normal image; determining whether the selected area is the flare area based on pixel values ​​corresponding to the selected area sensed by left and right phase detection pixels among the phase detection pixels; as well as When the selected area is the flat area and is a flare area, map data indicating that the selected area is the noise area is generated.

5. The electronic device according to claim 4, wherein: The noise detector comprises a flat area detector, wherein: aligning a kernel including a plurality of unit regions on the selected region among a plurality of regions of the normal image; as well as When the standard deviation value of the difference between a central pixel value overlapping with the central area of ​​the multiple unit areas and an average value of pixel values ​​overlapping with the core and having the same color as the central pixel value among multiple pixel values ​​included in the selected area is less than a first threshold, the selected area is determined to be the flat area.

6. The electronic device according to claim 5, wherein: The flat area detector adjusts the first threshold value according to a period to which an average value of the pixel values ​​included in the selected area among a plurality of periods belongs, and The multiple time periods include a time period in which the first threshold increases as an average value of the pixel values ​​included in the selected area increases.

7. The electronic device according to claim 4, wherein: The noise detector includes a flare area detector, which determines the selected area as the flare area when the ratio of the pixel value sensed by the left phase detection pixel to the pixel value sensed by the right phase detection pixel exceeds a second threshold corresponding to the selected area.

8. The electronic device according to claim 1, wherein: The processor includes a pixel corrector, the pixel corrector: selecting a pixel value included in any one unit area among a plurality of unit areas included in the noise area as a central pixel value; calculating a correction value based on a first average value of first peripheral pixel values ​​included in a unit area firstly close to the unit area including the central pixel value included in the noise area, among pixel values ​​having the same color as the central pixel value, a second average value of second peripheral pixel values ​​included in a unit area secondly close to the unit area including the central pixel value, among the pixel values ​​having the same color as the central pixel value, and a correction rate; as well as When the correction value is smaller than a limit value, a final pixel value obtained by correcting the central pixel value based on a sum of the central pixel value and the correction value is acquired.

9. The electronic device according to claim 8, wherein: Each of the plurality of unit areas includes 1×1 pixel values ​​or 2×2 pixel values.

10. The electronic device according to claim 8, wherein: When the correction value is equal to or greater than the limit value, the pixel corrector acquires the final pixel value obtained by correcting the center pixel value based on a sum of the center pixel value and the limit value.

11. The electronic device according to claim 8, wherein: When the difference between the maximum and minimum values ​​of pixel values ​​included in a predetermined number of peripheral areas relative to the unit area including the central pixel value among the multiple unit areas is equal to or greater than a cutoff value, the pixel corrector reduces the correction rate as the difference increases.

12. The electronic device according to claim 8, wherein: The pixel corrector: When the central pixel value is equal to or greater than a first reference value and less than a second reference value that is a value greater than the first reference value, increasing the limit value as the central pixel value increases; and When the central pixel value is equal to or greater than the second reference value and less than a third reference value that is a value greater than the second reference value, the limit value is decreased as the central pixel value increases.

13. The electronic device according to claim 12, wherein: The pixel corrector: When the central pixel value is less than the first reference value or equal to or greater than the third reference value, changing the limit value to a predetermined minimum limit value; as well as When the limit value exceeds a predetermined maximum limit value, the limit value is changed to the maximum limit value.

14. The electronic device according to claim 8, wherein: The pixel corrector: Selecting a unit area including any one pixel value of the first surrounding pixel value and the second surrounding pixel value among the plurality of unit areas as a central area; acquiring a first peripheral correction pixel value and a second peripheral correction pixel value, the first peripheral correction pixel value and the second peripheral correction pixel value being obtained by correcting the first peripheral pixel value and correcting the second peripheral pixel value using an average pixel value of pixel values ​​included in a first peripheral area corresponding to each of the central areas and an average pixel value of pixel values ​​included in a second peripheral area corresponding to each of the central areas; as well as When the correction value is less than the limit value, a value according to the sum of the central pixel value and the correction value and the weighted sum of the first peripheral correction pixel value and the second peripheral correction pixel value is obtained as the final pixel value.

15. The electronic device according to claim 14, wherein: When each of the plurality of unit areas includes a pixel value of 1×1, each of the first peripheral areas corresponds to a unit area that is first closest to each of the central areas among the plurality of unit areas, and When each of the plurality of unit areas includes a pixel value of 1×1, each of the second peripheral areas corresponds to a unit area second closest to each of the central areas among the plurality of unit areas.

16. The electronic device according to claim 14, wherein: When each of the plurality of unit areas includes 2×2 pixel values, each of the first peripheral areas corresponds to each of the central areas, and When each of the plurality of unit areas includes 2×2 pixel values, each of the second peripheral areas corresponds to a unit area that is first closest to each of the central areas among the plurality of unit areas.

17. A method of operating an electronic device, the method comprising: generating a normal image based on a plurality of pixel values ​​sensed by the normal pixels and the phase detection pixels; detecting a noise region in a flare region included in the normal image; as well as generating an output image including a final pixel value obtained by correcting each of the pixel values ​​included in the noise region using a peripheral pixel value of the pixel value, Wherein, detecting the noise area includes determining that a flat area in the flare area among multiple areas of the normal image is the noise area.

18. The method of claim 17, wherein: Generating the normal image comprises: acquiring a compensated pixel value obtained by correcting a pixel value sensed by the phase detection pixel using a pixel value sensed via a peripheral normal pixel of each of the phase detection pixels among the normal pixels; and The normal image including the compensated pixel values ​​and the pixel values ​​sensed by the normal pixels is generated.

19. The method of claim 17, wherein: Detecting the flat area comprises: determining whether the selected area is the flat area based on pixel values ​​included in the selected area among the plurality of areas of the normal image; determining whether the selected area is the flare area based on pixel values ​​corresponding to the selected area sensed by left and right phase detection pixels among the phase detection pixels; and When the selected region is the flat region and the flare region, the selected region is detected as the noise region.

20. The method of claim 17, wherein: Generating the output image comprises: selecting a pixel value included in any one unit area among a plurality of unit areas included in the noise area as a central pixel value; calculating a correction value based on a first average value of first peripheral pixel values ​​included in a unit area firstly close to a unit area including the central pixel value included in the noise area, among pixel values ​​having the same color as the central pixel value, a second average value of second peripheral pixel values ​​included in a unit area secondly close to the unit area including the central pixel value, among the pixel values ​​having the same color as the central pixel value, and a correction rate; and When the correction value is smaller than a limit value, a final pixel value obtained by correcting the central pixel value based on a sum of the central pixel value and the correction value is acquired.

Images