Electronic device and method for adjusting color of image data by using infrared sensor

Abstract

Provided are an electronic device and method for adjusting the color of image data using an infrared sensor. The electronic device includes a lens, an infrared filter, an image sensor, the infrared sensor, and at least one processor operatively coupled to the image sensor and the infrared sensor. The at least one processor receives image data based on external light from the image sensor, wherein the external light passes through the lens and the infrared filter and reaches the image sensor, identifies the intensity of infrared light included in the external light based on at least sensor data from the infrared sensor, and adjusts the color of at least a portion of the image data based on at least the intensity of the infrared light in response to identifying the intensity of the infrared light.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for adjusting color of image data by using an infrared sensor. Background Art

[0002] With the recent development of digital technology, various electronic devices, such as mobile communication terminals, smartphones, tablet personal computers (PCs), electronic organizers, personal digital assistants (PDAs), and wearable devices, have become widely used. These electronic devices may include one or more cameras for capturing images of a subject. External light reaching the camera may be electronically processed by the electronic device's image sensor and / or its image signal processor (ISP).

[0003] The above information is presented as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the Invention

[0004] Solution to the problem

[0005] Aspects of the present disclosure are to address at least the above-mentioned problems and / or disadvantages and to provide at least the advantages described below. Accordingly, one aspect of the present disclosure is to provide an electronic device and method for adjusting the color of image data by using an infrared sensor.

[0006] External light reaching the image sensor of a camera in an electronic device may be distorted by the lens and / or infrared filter through which the external light passes. The electronic device may compensate for this distortion based on the brightness and color temperature of the external light. To more accurately compensate for the distortion of the external light caused by the lens and / or infrared filter, the electronic device may require a method for compensating for the distortion based on the intensity of infrared light included in the external light.

[0007] The technical solutions that this document seeks to implement are not limited to the above-mentioned technical solutions, and those skilled in the art will be able to clearly understand other technical solutions not mentioned above from the following statements.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0009] According to one aspect of the present disclosure, an electronic device is provided. The electronic device includes: a lens; an infrared filter; an image sensor; the infrared sensor; and at least one processor operably coupled to the image sensor and the infrared sensor. The at least one processor receives image data from the image sensor based on external light that passes through the lens and the infrared filter and reaches the image sensor, identifies the intensity of infrared light included in the external light based on at least sensor data from the infrared sensor, and, in response to identifying the intensity of the infrared light, adjusts the color of at least a portion of the image data based on at least the intensity of the infrared light.

[0010] According to another aspect of the present disclosure, a method for an electronic device is provided. The method includes receiving image data based on external light that passes through a lens and an infrared filter of the electronic device and reaches the image sensor of the electronic device from an image sensor, identifying the intensity of infrared light included in the external light based on at least sensor data of the infrared sensor of the electronic device, and adjusting the color of at least a portion of the image data based on at least the intensity of the infrared light in response to the identification of the intensity of the infrared light.

[0011] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes an image sensor, an infrared sensor, and at least one processor operably coupled to the image sensor and the infrared sensor. The at least one processor receives first image data from the image sensor, and in response to receiving the first image data, identifies brightness and color temperature of the first image data, identifies the intensity of infrared light included in external light from the infrared sensor associated with the first image data, and adjusts the color of at least a portion of second image data received from the image sensor after receiving the first image data based on at least one of the identified brightness, color temperature, or intensity of infrared light.

[0012] The electronic device and method thereof of various embodiments may provide quality-enhanced image data by more accurately compensating or correcting distortion of external light caused by a lens and / or an infrared filter based on the intensity of infrared light included in the external light.

[0013] Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a block diagram of an electronic device within a network environment according to an embodiment of the present disclosure.

[0016] Figure 2 is a block diagram illustrating a camera module according to an embodiment of the present disclosure.

[0017] Figure 3A and Figure 3B is a diagram illustrating hardware components included in an electronic device according to various embodiments of the present disclosure.

[0018] Figure 4 is a block diagram illustrating operations performed by a processor of an electronic device according to an embodiment of the present disclosure.

[0019] Figure 5A and Figure 5B is a diagram illustrating lens shading provided by an image sensor of an electronic device according to various embodiments of the present disclosure.

[0020] Figure 6 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.

[0021] Figure 7 is a flowchart illustrating operations performed by an electronic device in order to adjust the color of image data according to an embodiment of the present disclosure.

[0022] Figure 8A and Figure 8B is a diagram illustrating lens shading correction (LSC) information used by an electronic device according to various embodiments of the present disclosure.

[0023] Figure 9A and Figure 9B is a diagram illustrating an operation of an electronic device selecting LSC information based on brightness of external light according to various embodiments of the present disclosure.

[0024] Figure 10 is a diagram illustrating an operation of an electronic device selecting LSC information based on the color temperature of external light according to an embodiment of the present disclosure.

[0025] Figure 11 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.

[0026] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0027] The following description, with reference to the accompanying drawings, is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these details are to be considered as exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0028] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0029] It will be understood that singular forms include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0030] In the document, expressions “have”, “may have”, “include”, “may include”, etc. indicate the existence of corresponding features (for example, constituent elements such as values, functions, operations, parts, etc.) and do not exclude the existence of additional features.

[0031] In this document, expressions such as "A or B," "at least one of A or / and B," "one or more of A or / and B," and the like may include all available combinations of the items listed together. For example, "A or B," "at least one of A and B," or "at least one of A or B" may mean all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0032] The expressions "first," "second," "first," "second," etc. may be used with various components, regardless of order and / or importance, and are used only to distinguish a component from another component, and do not limit the corresponding component. For example, a first user device and a second user device may represent mutually different user devices, regardless of order or importance. For example, a first component may be named a second component without departing from the scope of the rights mentioned in this document. It is possible that even the second component may be named interchangeably with the first component.

[0033] When it is mentioned that some components (for example, a first component) are “(operably or communicatively) coupled / coupled to” or “connected to” another component (for example, a second component), it must be understood that some components may be directly coupled to another component or coupled to another component through another component (for example, a third component). On the other hand, when it is mentioned that some components (for example, a first component) are “directly coupled to” or “directly connected to” another component (for example, the second component), it can be understood that there is no other component (for example, the third component) between the some components and the other component.

[0034] Depending on the context, the expression "configured (or set) to ~" used in the document may be used interchangeably with, for example, "suitable for ~", "capable of ~", "designed to ~", "suitable for ~", "manufactured to ~" or "capable of ~". The term "configured (or set) to" may not necessarily mean only "specially designed to" in hardware. On the contrary, in some contexts, the expression "a device configured to ~" may mean that the device, together with other devices or parts, is "capable of ~". For example, the phrase "a processor configured (or set) to perform A, B and C" may represent a dedicated processor for performing the corresponding operations (e.g., an embedded processor), or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

[0035] The terms used in this document are only used to explain specific embodiments and are not intended to limit the scope of other embodiments. The terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those skilled in the art mentioned herein. Among the terms used in this document, terms defined in general dictionaries may be interpreted as having the same or similar meaning as the contextual meaning of the relevant technology, and shall not be interpreted as having an ideal or overly formal meaning unless explicitly defined in the document. Depending on the circumstances, even terms defined in the document cannot be interpreted as excluding embodiments of the document.

[0036] The electronic device of various embodiments of the document may include, for example, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an electronic book (e-book) reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a mobile medical device, a camera, or a wearable device. According to various embodiments of the present disclosure, the wearable device may include at least one of an accessory type (e.g., a watch, a ring, a wristband, an anklet, a necklace, glasses, contact lenses, a head-mounted device (HMD), etc.), a fabric or clothing integrated type (e.g., electronic clothing), a body-mounted type (e.g., a skin pad or tattoo), or a bio-implant type (e.g., an implantable circuit).

[0037] In some embodiments of the present disclosure, the electronic device may be a home appliance. The home appliance may include, for example, a television (TV), a digital video disc (DVD) player, an audio system, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync), a home automation system, ... TM , Apple TV TM or Google TV TM ), game consoles (e.g., Xbox TM , PlayStation TM ), at least one of an electronic dictionary, an electronic locking system, a camera, or an electronic photo frame.

[0038] In another embodiment of the present disclosure, the electronic device may include at least one of various medical devices (e.g., various portable medical measuring devices (i.e., blood glucose measuring devices, heart rate measuring devices, blood pressure measuring devices, body temperature measuring devices, etc.), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), cameras, ultrasound machines, etc.), navigation devices, global navigation satellite systems (GNSS), event data recorders (EDRs), flight data recorders (FDRs), in-vehicle infotainment devices, etc.), marine electronic devices (e.g., marine navigation devices, gyrocompasses, etc.), avionics equipment, security devices, car head units, industrial or home robots, automatic teller machines (ATMs) of financial institutions, point of sale (POS) of stores, or Internet of Things (IoT) devices (e.g., light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.).

[0039] According to some embodiments of the present disclosure, an electronic device may include at least one of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various metering devices (e.g., water, electricity, gas, or radio wave metering devices, etc.). In various embodiments of the present disclosure, the electronic device may be a combination of one or more of the various devices described above. The electronic device of some embodiments may be a flexible electronic device or a foldable electronic device. In addition, the electronic devices of the embodiments of this document are not limited to the aforementioned devices and may include new electronic devices based on technological developments.

[0040] In this document, the term "user" may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) that uses an electronic device.

[0041] Various embodiments are described below with reference to the accompanying drawings. However, for ease of description, the sizes of the components in the drawings may be exaggerated or reduced. For example, the size and thickness of each component shown in the drawings are arbitrarily illustrated for ease of description, and therefore, the present disclosure is not necessarily limited to the drawings shown.

[0042] Figure 1 is a block diagram illustrating electronic devices within a network environment according to an embodiment of the present disclosure.

[0043] Reference Figure 1 , an electronic device 101 in a network environment 100 can communicate with an electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or communicate with an electronic device 104 or a server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment of the present disclosure, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an embodiment of the present disclosure, the electronic device 101 may include a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments of the present disclosure, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments of the present disclosure, some components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in the display device 160 (eg, a display).

[0044] The processor 120 may execute, for example, software (e.g., program 140) to control at least one other component (e.g., a hardware component or a software component) of the electronic device 101 coupled to the processor 120, and may perform various data processing or calculations. According to one embodiment of the present disclosure, as at least part of the data processing or calculation, the processor 120 may load commands or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment of the present disclosure, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or to be adapted for a specified function. The auxiliary processor 123 may be implemented as a separate processor from the main processor 121 or as part of the main processor 121.

[0045] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (instead of the main processor 121) may control at least some of the functions or states related to at least one of the components of the electronic device 101 (e.g., the display device 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one of the components of the electronic device 101 (e.g., the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment of the present disclosure, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) that is functionally related to the auxiliary processor 123.

[0046] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0047] The program 140 may be stored as software in the memory 130 and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0048] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by another component of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0049] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment of the present disclosure, the receiver can be implemented separately from the speaker or as part of the speaker.

[0050] The display device 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment of the present disclosure, the display device 160 may include a touch circuit adapted to detect a touch, or a sensor circuit adapted to measure the strength of a force caused by a touch (e.g., a pressure sensor).

[0051] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment of the present disclosure, the audio module 170 can obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly coupled to the electronic device 101.

[0052] The sensor module 176 can detect the operating state of the electronic device 101 (for example, power or temperature) or the environmental state outside the electronic device 101 (for example, the state of the user), and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment of the present disclosure, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0053] The interface 177 may support one or more specific protocols for directly (e.g., wired) or wirelessly coupling the electronic device 101 to an external electronic device (e.g., the electronic device 102). According to an embodiment of the present disclosure, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0054] The connection end 178 may include a connector, through which the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102). According to an embodiment of the present disclosure, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0055] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment of the present disclosure, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0056] The camera module 180 may capture still images or moving images. According to an embodiment of the present disclosure, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0057] The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment of the present disclosure, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0058] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment of the present disclosure, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0059] The communication module 190 can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors that can operate independently of the processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment of the present disclosure, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate via a first network 198 (e.g., such as Bluetooth TM The wireless communication module 192 may communicate with an external electronic device via a wireless communication module 192 (e.g., a short-range communication network such as a wireless cellular network, a Wi-Fi Direct connection, or an infrared data association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip) or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196 to identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199).

[0060] Antenna module 197 can transmit or receive signals or power to or from an external device (e.g., an external electronic device) outside of electronic device 101. According to embodiments of the present disclosure, antenna module 197 may include an antenna having a radiating element comprising a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to embodiments of the present disclosure, antenna module 197 may include multiple antennas. In this case, at least one antenna suitable for the communication scheme used in a communication network (e.g., first network 198 or second network 199) may be selected from the multiple antennas by communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. According to embodiments of the present disclosure, another component in addition to the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be formed as part of antenna module 197.

[0061] At least some of the above components may be coupled to each other via a peripheral communication scheme (e.g., a bus, general-purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and transmit signals (e.g., commands or data) therebetween.

[0062] According to an embodiment of the present disclosure, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 coupled to second network 199. Each of electronic device 102 and electronic device 104 can be of the same or different type as electronic device 101. According to an embodiment of the present disclosure, all or some operations intended for electronic device 101 can be executed at one or more of external electronic devices 102, 104, or server 108. For example, if electronic device 101 is to automatically execute a function or service, or execute a function or service in response to a request from a user or another device, electronic device 101 can request one or more external electronic devices to execute at least part of the function or service instead of executing the function or service itself, or in addition to executing the function or service itself. The one or more external electronic devices that receive the request can execute at least part of the requested function or service, or execute another function or service related to the request, and transmit the results of the execution to electronic device 101. The electronic device 101 may provide the result as at least part of a reply to the request with or without further processing the result. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0063] Figure 2 is a block diagram illustrating a camera module according to an embodiment of the present disclosure.

[0064] Reference Figure 1 and Figure 2The camera module 180 may include a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., a buffer memory), or an image signal processor 260. The lens assembly 210 may collect light emitted or reflected from an object whose image is to be captured. The lens assembly 210 may include one or more lenses. According to an embodiment of the present disclosure, the camera module 180 may include multiple lens assemblies 210. In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the multiple lens assemblies 210 may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that differ from those of another lens assembly. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

[0065] The flash 220 can emit light used to enhance light reflected from an object. According to an embodiment of the present disclosure, the flash 220 may include one or more light-emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white light LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor 230 can obtain an image corresponding to the object by converting light emitted from or reflected from the object and transmitted through the lens assembly 210 into an electrical signal. According to an embodiment of the present disclosure, the image sensor 230 may include one sensor selected from image sensors with different properties (such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), multiple image sensors with the same properties, or multiple image sensors with different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0066] Image stabilizer 240 can move image sensor 230 or at least one lens included in lens assembly 210 in a specific direction, or control the operating properties of image sensor 230 (e.g., adjust readout timing) in response to movement of camera module 180 or electronic device 101 including camera module 180. This allows for compensating for at least a portion of the negative effects (e.g., image blur) caused by movement on the image being captured. According to an embodiment of the present disclosure, image stabilizer 240 can use a gyroscope sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside camera module 180 to detect such movement of camera module 180 or electronic device 101. According to an embodiment of the present disclosure, image stabilizer 240 can be implemented as, for example, an optical image stabilizer.

[0067] Memory 250 can, at least temporarily, store at least a portion of an image obtained via image sensor 230 for subsequent image processing tasks. For example, if image capture is delayed due to shutter lag or multiple images are captured rapidly, the obtained original image (e.g., a Bayer-patterned image, a high-resolution image) can be stored in memory 250, and its corresponding copy image (e.g., a low-resolution image) can be previewed via display device 160. Thereafter, if a specified condition is met (e.g., through user input or system command), at least a portion of the original image stored in memory 250 can be obtained and processed, for example, by image signal processor 260. According to an embodiment of the present disclosure, memory 250 can be configured as at least a portion of memory 130, or as a separate memory that operates independently of memory 130.

[0068] The image signal processor 260 may perform one or more image processing operations on images obtained via the image sensor 230 or stored in the memory 250. The one or more image processing operations may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 260 may control at least one of the components included in the camera module 180 (e.g., image sensor 230) (e.g., exposure time control or readout timing control). Images processed by the image signal processor 260 may be stored back in the memory 250 for further processing or provided to an external component outside the camera module 180 (e.g., memory 130, display device 160, electronic device 102, electronic device 104, or server 108). According to embodiments of the present disclosure, the image signal processor 260 may be configured as at least a portion of the processor 120, or as a separate processor operating independently of the processor 120. If the image signal processor 260 is configured as a separate processor from the processor 120 , the at least one image processed by the image signal processor 260 may be displayed by the processor 120 via the display device 160 as it is or after being further processed.

[0069] According to an embodiment of the present disclosure, the electronic device 101 may include multiple camera modules 180 having different properties or functions. In this case, at least one of the multiple camera modules 180 may form, for example, a wide-angle camera, and at least another of the multiple camera modules 180 may form a telephoto camera. Similarly, at least one of the multiple camera modules 180 may form, for example, a front-facing camera, and at least another of the multiple camera modules 180 may form a rear-facing camera.

[0070] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to embodiments of the present disclosure, the electronic device is not limited to the aforementioned ones.

[0071] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and include various changes, equivalents, or alternative forms for the corresponding embodiments. With respect to the description of the drawings, similar figure numerals may be used to refer to similar or related elements. As used herein, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as "first" and "second" or "first" and "second" may be used to simply distinguish a corresponding component from another component and do not limit the components in other respects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., a second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the term “operably” or “communicatively”, it should be understood that the element may be directly (e.g., wired) coupled to the other element, wirelessly coupled to the other element, or coupled to the other element via a third element.

[0072] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "portion," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or portion of a single integrated component. For example, according to an embodiment of the present disclosure, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0073] The various embodiments described herein can be implemented as software (e.g., program 140) comprising one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that is readable by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This allows the machine to operate to perform at least one function in accordance with the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but does not distinguish between data stored semi-permanently and data stored temporarily in the storage medium.

[0074] According to an embodiment of the present disclosure, the methods according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or via an application store (e.g., PlayStore). TM ) or distributed directly between two user devices (e.g., smartphones). If distributed online, at least parts of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a relay server).

[0075] According to various embodiments of the present disclosure, each component of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. According to various embodiments of the present disclosure, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments of the present disclosure, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as performed by the corresponding components of the multiple components before integration. According to various embodiments of the present disclosure, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or may be run in a different order or one or more operations may be omitted, or one or more other operations may be added.

[0076] Figures 3A to 3B is a diagram illustrating hardware components included in an electronic device according to various embodiments of the present disclosure.

[0077] Reference Figures 3A to 3B , Figures 3A to 3B The electronic device 101 may correspond to Figure 1 or Figure 2 The electronic device 101 may be a smartphone, a PDA, a tablet PC such as a smart tablet, a desktop PC, and / or a laptop PC. According to various embodiments of the present disclosure, the electronic device 101 may be an embedded PC that may be included as part of another electronic device, or may be a wearable device such as a smart watch.

[0078] The electronic device 101 may include a camera module 180 including a lens assembly 210, an infrared filter 310, an image sensor 230, and a processor 260. The camera module 180 may be operatively and / or electrically connected to other hardware components of the electronic device 101 (e.g., the second processor 120 and the infrared sensor 320). The second processor 120 may correspond to Figures 1 to 2 The camera module 180, the second processor 120, and the infrared sensor 320 may be connected, for example, through an electrical interface such as a communication bus (not shown).

[0079] The lens assembly 210 and the image sensor 230 included in the camera module 180 may correspond to Figure 2 The lens assembly 210 and the image sensor 230. The processor 260 may correspond to Figure 2 ISP 260. Figure 3A 1 , one camera module 180 is shown, but the number of camera modules 180 included in the electronic device 101 may vary depending on the embodiment. Hereinafter, the lens assembly 210 and the camera module 180 may be referred to as a lens and a camera, respectively. Hereinafter, the ISP may be referred to as the processor 260.

[0080] In an embodiment of the present disclosure, at least a portion of the lens assembly 210 may be exposed to the outside through the housing of the electronic device 101. External light emitted from the main body toward the lens assembly 210 may be transmitted through the lens assembly 210 to the electronic device 101 and / or the camera module 180. The external light that passes through the lens assembly 210 may pass through the infrared filter 310 and then reach the image sensor 230. The infrared filter 310 may reduce and / or block the infrared band component of the external light that reaches the image sensor 230.

[0081] Image sensor 230 may include pixels arranged in two dimensions. Image sensor 230 may convert an image formed by a lens into an electrical signal per pixel based on the photoelectric effect. Each pixel may include multiple photodiodes (PDs). These PDs may convert optical signals into electrical signals based on the photoelectric effect. Each of the multiple PDs within any pixel of image sensor 230 may receive light of different wavelengths (e.g., red, blue, and green light) and output an electrical signal based on the intensity of the received light. Each of the multiple PDs within any pixel of image sensor 230 may be arranged based on a specified pattern, such as a Bayer pattern.

[0082] The image sensor 230 may output image data to the processor 260 according to a specified time point and / or cycle. The image data may include data corresponding to an electrical signal of each of a plurality of pixels included in the image sensor 230 and an electrical signal of each of a plurality of PDs included in the pixel. For example, the data corresponding to the electrical signal of each of the plurality of PDs may be arranged based on the arrangement of the plurality of PDs in a Bayer pattern within the image data and the arrangement of the plurality of pixels within the image sensor 230.

[0083] According to various embodiments of the present disclosure, the processor 260 may perform image processing related to the image data. Image processing may include, for example, automatic exposure (AE), automatic white balance (AWB), demosaicing, color interpolation (CI), gamma correction, edge enhancement (EE), noise reduction (NR), etc. Figure 4 The sequence of image processing performed by the processor 260 is described.

[0084] According to various embodiments of the present disclosure, the processor 260 may perform image processing to compensate for distortion of the image data provided by the lens assembly 210 and the infrared filter 310. For example, the distortion may include lens shading and / or chromatic aberration. For example, the processor 260 may compensate for the distortion of the image data provided by the lens assembly 210 and the infrared filter 310 based on lens shading correction (LSC). Figures 5A to 5B The distortion of image data provided by lens assembly 210 and infrared filter 310 is described.

[0085] In an embodiment of the present disclosure, the processor 260 may measure the intensity of infrared light included in the external light using the infrared sensor 320. The infrared sensor 320 may output a digital electrical signal indicating the intensity of the infrared light. By performing LSC based on the measured infrared light intensity, the processor 260 may compensate for distortion of the image data provided by the lens assembly 210 and the infrared filter 310. The processor 260 may transmit the image data to which the image processing has been applied to the second processor 120.

[0086] Reference Figure 3B, shows the arrangement of lens assemblies 210-1, 210-2, and 210-3 and infrared sensor 320-1 of electronic device 101 of an embodiment. Electronic device 101 may include multiple cameras, and lens assemblies 210-1, 210-2, and 210-3 of the respective multiple cameras may be exposed to the outside through mutually different portions of the housing of electronic device 101. Infrared sensor 320-1 may correspond to, for example, at least one of a proximity sensor, a heart rate monitor (HRM) sensor, an IR-specific sensor, or a time-of-flight (TOF) sensor.

[0087] Reference Figure 3B , the infrared sensor 320-1 may be arranged adjacent to a portion of the housing of the electronic device 101 that exposes the lens assemblies 210-1, 210-2, and 210-3. The infrared sensor 320-1 may transmit sensor data indicating the intensity of infrared light included in the external light to at least one of the plurality of cameras corresponding to the lens assemblies 210-1, 210-2, and 210-3. The sensor data may be used for LSC performed by at least one of the plurality of cameras. Figure 4 Operations performed by the camera of the electronic device 101 using sensor data of the infrared sensor 320 - 1 are described.

[0088] Figure 4 is a block diagram illustrating operations performed by a processor of an electronic device according to an embodiment of the present disclosure.

[0089] Figure 4 The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101. Figure 4 The processor 260 may correspond to Figures 3A to 3B The processor 260 and / or Figure 2 ISP 260.

[0090] Reference Figure 4 An example of the format of image data 410 output from image sensor 230 is shown. Image sensor 230 may output image data 410 to processor 260 at a specified period (e.g., a frame per second (FPS) period adjusted between 24 Hz, 60 Hz, or 120 Hz). The plurality of pixels included in image sensor 230 may be arranged in a two-dimensional array. Within portion 415 of image data 410, the colors measured in each of the plurality of PDs included in image sensor 230 may be arranged based on the arrangement of the plurality of PDs (e.g., a Bayer pattern).

[0091] Reference Figure 4The processor 260 of various embodiments may include an LSC processing unit 420, an AE processing unit 430, an AWB processing unit 440, a WB adjustment unit 445, a Bayer pattern conversion unit 450, a color conversion unit 460, and / or a gamma conversion unit 470 for performing image processing related to the image data 410. Each of the LSC processing unit 420, the AE processing unit 430, the AWB processing unit 440, the WB adjustment unit 445, the Bayer pattern conversion unit 450, the color conversion unit 460, and the gamma conversion unit 470 may correspond to a plurality of instructions and / or threads executed in the processor 260, or correspond to at least a portion of a circuit for signal processing included in the processor 260.

[0092] In response to receiving the image data 410 from the image sensor 230, the processor 260 of an embodiment may compensate for distortions due to lens shading and / or chromatic aberration included in the image data 410 based on the LSC processing unit 420. For example, the processor 260 may adjust the color of each of the plurality of pixels included in the image data 410 using a specified value (e.g., gain) based on the positions of the plurality of pixels. The specified value may be associated with specified information related to LSC (e.g., a lens shading correction coefficient and / or an LSC table).

[0093] In an embodiment, the processor 260 may process the image data 410 processed by the LSC processing unit 420 based on the AE processing unit 430. The processor 260 may identify the brightness of the image data 410 based on the AE algorithm of the AE processing unit 430. For example, the processor 260 may adjust the brightness of the image data 410 (e.g., the average brightness of a plurality of pixels included in the image data 410) to within a specified range based on the AE processing unit 430. The processor 260 may identify the brightness of the image data 410 based on the AE processing unit 430. Based on the identified brightness of the image data 410, the processor 260 may control the LSC processing unit 420 to perform LSC that takes into account the brightness identified by the AE processing unit 430. The processor 260 may identify and / or adjust the time for which the image sensor 230 is exposed to external light and the gain of the image sensor 230 based on the AE processing unit 430.

[0094] The processor 260 of this embodiment may process the image data 410 processed by the AE processing unit 430 based on the AWB processing unit 440 and / or the WB adjustment unit 445. The processor 260 may identify the color temperature of the image data 410 based on the AWB algorithm of the AWB processing unit 440. For example, the processor 260 may adjust the color temperature of the image data 410 based on the AWB processing unit 440. The processor 260 may identify the color temperature of the image data 410 based on the Kelvin light temperature (K) based on the AWB processing unit 440. Color temperature can be expressed as a Kelvin light temperature (or K value) based on the relationship between the temperature of a black body and the color emitted by the black body. For example, the lower the K value, the more external light includes red light, and the higher the K value, the more external light includes blue light (or violet light).

[0095] The color temperature of image data 410 may vary depending on the type of light source emitting external light reaching image sensor 230. Processor 260 may identify the type of light source based on AWB processing unit 440. Using WB adjustment unit 445, processor 260 may adjust the white balance of image data 410 based on the identified light source type. For example, processor 260 may control WB adjustment unit 445 based on the white balance gain (WB gain) determined using AWB processing unit 440 to adjust the white balance of image data 410. In an embodiment of the present disclosure, processor 260 may adjust the white balance of image data 410 based on the brightness identified using AE processing unit 430 and the color temperature identified using AWB processing unit 440.

[0096] When adjusting the color temperature and / or white balance of the image data 410, the processor 260 of various embodiments may use the intensity of infrared light measured by the infrared sensor 320. For example, the processor 260 may identify the type of light source associated with the external light reaching the image sensor 230 based on the brightness identified using the AE processing unit 430, the color temperature identified using the AWB processing unit 440, and the intensity of infrared light measured by the infrared sensor 320. For example, the processor 260 may identify whether infrared light is relatively more included in the external light. Information related to the type of light source identified by the processor 260 using the AWB processing unit 440 (e.g., light source information) may be used in the LSC processing unit 420 to compensate for lens shading. Figure 4 , processor 260 may compensate for lens shading included in image data 410 based on the brightness, color temperature of image data 410, and the intensity of infrared light measured from infrared sensor 320. For example, processor 260 may compensate for lens shading included in image data received from image sensor 230 after image data 410 based on the brightness, color temperature of image data 410, and the intensity of infrared light measured from infrared sensor 320.

[0097] The processor 260 of this embodiment may process the image data 410 processed by the AWB processing unit 440 and / or the WB adjustment unit 445 based on the Bayer pattern conversion unit 450. For example, the processor 260 may change the image data 410 displaying the colors measured by the image sensor 230 according to the Bayer pattern, wherein each of the plurality of pixels of the image data 410 includes a plurality of color components (e.g., three components of red (R), green (G), and blue (B) and / or three components of Y, Cb, and Cr) based on demosaicing. For example, by interpolating the values ​​of the image data 410 based on the Bayer pattern, the processor 260 may obtain the image data 410 based on R, G, and B.

[0098] In an embodiment of the present disclosure, before the processor 260 processes the image data 410 based on the Bayer pattern conversion unit 450, the image data 410 may include a plurality of signals output from a plurality of photodiodes (PDs) included in the image sensor 230. The plurality of signals may be arranged within the image data 410 based on a specified pattern (e.g., a Bayer pattern) in which the plurality of photodiodes are arranged within the image sensor 230. After the processor 260 processes the image data 410 based on the Bayer pattern conversion unit 450, the image data 410 may indicate the color of each of the plurality of pixels based on a plurality of specified color components (e.g., three components of red (R), green (G), and blue (B) and / or three components of Y, Cb, and Cr).

[0099] The processor 260 of the embodiment may process the image data 410 processed by the Bayer pattern conversion unit 450 based on the color conversion unit 460. The processor 260 may adjust the color of the image data 410 according to the characteristics of the image sensor 230. For example, the processor 260 may adjust the color that is not accurately expressed according to the characteristics of the image sensor 230.

[0100] The processor 260 of the embodiment may process the image data 410 processed by the color conversion unit 460 based on the gamma conversion unit 470. The processor 260 may nonlinearly adjust the color of the image data 410 according to a nonlinear gamma characteristic (eg, a gamma characteristic of a display of an electronic device).

[0101] In various embodiments of the present disclosure, the processor 260 may output the image data 410 to which various image processing schemes are applied. For example, the processor 260 may output the image data 410 to a computer connected to the processor 260 and / or a memory (e.g., Figure 1 memory 130) of a different second processor (eg, the second processor 120, such as Figure 3AThe image data 410 may be stored as at least a portion of a photo and / or video captured by the electronic device.

[0102] The processor 260 of the electronic device of various embodiments may compensate for lens shading and / or chromatic aberration included in the image data 410 based on the LSC algorithm of the LSC processing unit 420. The LSC algorithm may be performed based on the brightness identified using the AE processing unit 430, the color temperature identified using the AWB processing unit 440, and the intensity of infrared light identified using the infrared sensor 320. For example, the processor 260 may select any one of designated information corresponding to mutually different intensities of infrared light based on the brightness, the color temperature, and the intensity of infrared light. The information may have a format of a table including coefficients (e.g., color gain for red) corresponding to respective mutually different pixels of the image data 410 and multiplied by the color of the pixel (e.g., red). By adjusting the colors of a plurality of pixels included in the image data 410 based on the selected information, the processor 260 may compensate for lens shading and / or chromatic aberration included in the image data 410. Reference Figures 8A to 8B An example of designated information used by the processor 260 is described.

[0103] Figures 5A to 5B is a diagram illustrating lens shading provided by an image sensor of an electronic device according to various embodiments of the present disclosure.

[0104] Reference Figures 5A to 5B , Figures 5A to 5B The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B The electronic device 101 of various embodiments can be used by using an infrared sensor (e.g., Figures 3A to 3B The infrared sensor 320 is used to identify the intensity of the infrared light included in the light source, and compensate for the lens shading based on the identified intensity.

[0105] Reference Figure 5A , shows an example graph 500 illustrating infrared filters (eg, Figure 3A The characteristics of the infrared filter 310 are shown in Figure 3. Lens shading is a type of distortion in which, as the intensity of light passing through the camera lens decreases further from the center to the periphery of the lens, the brightness of a portion of the image data corresponding to the periphery of the lens becomes lower than the brightness of another portion of the image data corresponding to the center of the lens. Chromatic aberration is a distortion in which the color of the portion of the image data corresponding to the center of the lens differs from the color of the portion of the image data corresponding to the periphery of the lens.

[0106] The transmission characteristics of an infrared filter can vary depending on the angle of incidence of light reaching the infrared filter. Figure 5A , shows a graph 500 showing the amount of light passing through the infrared filter based on the incident angle and wavelength. The y-axis refers to the amount of light passing through the infrared filter, and means that the larger the value, the more light passes through the infrared filter. Figure 5A , curve 510 indicates that when the incident angle is The amount of light at each wavelength when the lens is perpendicular to the infrared filter (e.g., when the light reaches the infrared filter vertically). External light passing through the lens at the center of the lens may reach the infrared filter vertically. In this case, the infrared filter may filter the external light similarly to curve 510.

[0107] Comparing curves 510, 520, 530, 540, and 550, the greater the increase in the angle of incidence, the more the infrared filter blocks shorter wavelengths of light. The greater the increase in the angle of incidence, the more the wavelength of light blocked by the infrared filter decreases. The further from the center of the lens to its periphery, the greater the angle of incidence of light reaching the infrared filter due to the lens's curved surface increases. The amount of external light passing through the lens at the periphery may be less than that passing through the lens at the center. Referring to graph 500, the further from the center of the lens to its periphery, the greater the infrared filter blocks shorter wavelengths of light (e.g., light in the red band adjacent to the infrared band). This variation in the transmission characteristics of the infrared filter depending on the angle of incidence may cause chromatic aberration.

[0108] Reference Figure 5B , shows lens assembly 210, infrared filter 310, and image sensor 230 aligned along designated axis 560 within a camera of an electronic device according to various embodiments. External light passing through lens assembly 210 and infrared filter 310 may reach image sensor 230. Designated axis 560 may coincide with a central axis of lens assembly 210.

[0109] As the lens shading becomes farther from designated axis 560, the intensity of light reaching image sensor 230 may decrease. In infrared filter 310, based on transmission characteristics such as those shown in graph 500, the farther from designated axis 560, the more shorter wavelength light (for example, light in the red band, which is adjacent to the infrared band and has a shorter wavelength than the infrared band) is blocked. The farther from designated axis 560, the more red wavelength light is blocked by infrared filter 310. Therefore, the farther from designated axis 560, the more the color of external light reaching image sensor 230 shifts toward the green band (chromatic aberration). The more the intensity of infrared light and / or the infrared wavelength component included in the external light increases, the more the color of the external light reaching the portion of image sensor 230 corresponding to the outer periphery of lens assembly 210 shifts toward green wavelengths.

[0110] By adjusting the colors of a plurality of pixels included in the image data received from the image sensor 230 based on the intensity of the infrared light included in the external light, the electronic device of various embodiments can compensate for color distortion caused by the transmittance characteristics of the infrared filter 310 and the infrared light included in the external light. Figure 6 The operation of an electronic device adjusting the color of image data based on the intensity of infrared light is described.

[0111] Figure 6 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure. Figure 6 The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101. Figure 6 The operation of the electronic device can be Figure 2 ISP 260 and / or Figure 3A Executed by processor 260.

[0112] Reference Figure 6 In operation 610, the electronic device of various embodiments may receive image data based on external light from the image sensor. The electronic device receiving the image data based on operation 610 may be performed in response to a user input for activating a camera included in the electronic device (eg, a user input for running a camera application). The image sensor may correspond to Figure 2 and Figure 3A The image data may correspond to the image sensor 230. Figure 4 For example, the image data received by the electronic device may be data in which color values ​​measured in a plurality of PDs are arranged based on an arrangement of the plurality of PDs within the image sensor (eg, an arrangement of the plurality of PDs based on a Bayer pattern).

[0113] The image data received by the electronic device may be based on a signal transmitted through a lens included in a camera (e.g., Figure 2 and Figure 3A lens assembly 210) and an infrared filter (e.g., Figure 3A The external light may be caused by the infrared filter 310) and reach the image sensor. Figures 5A to 5B Distortion is caused by the transmission characteristics of the infrared filter explained in [1]. Distortion caused by the transmission characteristics of the infrared filter can be associated with, for example, chromatic aberration, a phenomenon in which the color of a portion of image data corresponding to the peripheral portion of the lens relatively far from the central axis of the lens shifts to the green band.

[0114] Reference Figure 6In operation 620, the electronic device of various embodiments may recognize the intensity of infrared light included in the external light based on at least the sensor data of the infrared sensor. The infrared sensor may correspond to, for example, Figures 3A to 3B Infrared sensor 320. In an embodiment of the present disclosure, the electronic device may obtain sensor data from the infrared sensor at the time of receiving the image data based on operation 610. In an embodiment of the present disclosure, the electronic device may identify the type of light source related to the external light based on the sensor data obtained from the infrared sensor. In an embodiment of the present disclosure, the electronic device may identify color difference based on the intensity of the infrared light. The electronic device may identify a change in color of a portion of the image data corresponding to the peripheral portion of the lens based on the intensity of the infrared light.

[0115] Reference Figure 6 , in operation 630, the electronic device of various embodiments may adjust the color of at least a portion of the image data based at least on the intensity of the infrared light. In an embodiment of the present disclosure, in response to identifying the intensity of the infrared light based on operation 620, the electronic device may adjust the color of at least a portion of the image data. For example, in order to compensate for distortion (e.g., chromatic aberration) of the image data caused by a lens and an infrared filter, the electronic device may adjust the color of at least a portion of the image data based on the intensity of the infrared light. The electronic device may adjust the color of each of the plurality of pixels by applying a specified coefficient or gain to the color of each of the plurality of pixels included in the image data based on the LSC algorithm.

[0116] In response to receiving the image data according to operation 610, the electronic device of the embodiment may identify the brightness of the external light associated with the image data. The electronic device may use the identified brightness to adjust the brightness of the image data based on the AE algorithm. In an embodiment of the present disclosure, the electronic device may adjust the color of at least a portion of the image data based on the intensity of the infrared light and the brightness of the external light identified according to operation 620.

[0117] In response to receiving the image data according to operation 610, the electronic device of the embodiment may identify the color temperature of the external light associated with the image data. The electronic device may use the identified color temperature to adjust the color temperature of the image data based on the AWB algorithm. The electronic device may adjust the color of at least a portion of the image data based on the intensity of the infrared light and the color temperature of the external light identified according to operation 620.

[0118] In an embodiment of the present disclosure, in response to the identification of the intensity of infrared light based on operation 620, the electronic device may adjust the color of at least a portion of the image data corresponding to another portion of the lens different from the portion including the center of the lens. The color adjusted by the electronic device may be a color (e.g., red light) included in a band (e.g., a red band) adjacent to the infrared band. For example, the more the intensity of infrared light included in external light increases, the more the electronic device may highlight the red light of at least a portion of the image data corresponding to the edge portion of the lens. For example, the farther away from the portion of the image data corresponding to the center portion of the lens, and / or the greater the intensity of infrared light included in external light, the more the electronic device highlights the red light. In an embodiment of the present disclosure, the electronic device may enhance the red light by amplifying the value associated with the red light within the image data.

[0119] In another embodiment of the present disclosure, the electronic device may emphasize red light by decreasing the increment of a value associated with another color (e.g., at least one of green light or blue light) within the image data other than red light. For example, the electronic device may decrease the increment of at least one of green light or blue light in at least a portion of the image data corresponding to the peripheral portion of the lens more as the intensity of infrared light included in the external light increases.

[0120] In response to identifying the intensity of infrared light in operation 620, the electronic device of various embodiments may obtain information including color gains corresponding to different pixels of the image data and corresponding to the intensity of the infrared light. Based on the obtained information, the electronic device may adjust the colors of the plurality of pixels included in the image data. This information may be used to execute an LSC algorithm to compensate for lens shading and / or chromatic aberration.

[0121] Figure 7 is a flowchart illustrating operations performed by an electronic device in order to adjust the color of image data according to an embodiment of the present disclosure. Figure 7 The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101. Figure 7 The operation of the electronic device can be Figure 2 ISP 260 and / or Figure 3A Executed by processor 260. Figure 7 The operation can be done with Figure 6 At least one of the operations (e.g., Figure 6 Operation 630 is related. Figure 7 The operation may for example be at least partially related to Figure 4 The LSC processing unit 420 and / or the LSC algorithm are related.

[0122] Reference Figure 7In operation 710, the electronic device of various embodiments may obtain a threshold value of at least one infrared light based on the brightness of the external light. In an embodiment of the present disclosure, the electronic device may, for example, obtain a threshold value of at least one infrared light based on the brightness of the external light. Figure 4 The AE processing unit 430 and / or the AE algorithm of the electronic device is used to identify the brightness of the external light related to the image data. The brightness of the external light is a value representing the brightness of multiple pixels included in the image data, and may be, for example, at least one of the average value, median value, maximum value, or minimum value of the brightness of the multiple pixels. The threshold value recognized by the electronic device may be used to determine whether infrared light is relatively more included in the external light. Figure 9A Describes the relationship between a threshold value and brightness obtained by an electronic device.

[0123] Reference Figure 7 In operation 720, the electronic device of various embodiments may obtain a plurality of LSC information based on the color temperature of the external light. In an embodiment of the present disclosure, the electronic device may, for example, obtain a plurality of LSC information based on the color temperature of the external light. Figure 4 The AWB processing unit 440 and / or AWB algorithm is used to identify the color temperature of the external light associated with the image data. The LSC information is information for compensating for lens shading and / or chromatic aberration, and may represent a lens shading correction coefficient and / or an LSC table. The LSC table may represent a data set including a color gain that is to be applied to each different pixel of the image data. A plurality of LSC information may respectively correspond to different intensities of infrared light. In an embodiment of the present disclosure, the electronic device may obtain at least two of the plurality of LSC information based on the color temperature of the external light. Referring to Figures 8A to 8B An example of LSC information acquired by an electronic device is described.

[0124] Reference Figure 7 In operation 730, the electronic device of various embodiments may select any one of the plurality of LSC information based on the intensity of the infrared light and the threshold value acquired in operation 710. For example, Figure 3A The infrared sensor 320 is used to identify the intensity of the infrared light. In response to obtaining at least two pieces of LSC information among the multiple LSC information based on the color temperature of the external light, the electronic device may select any one of the at least two LSC information based on the intensity of the identified infrared light. In an embodiment of the present disclosure, based on at least one of the first LSC information corresponding to the specified first intensity of the infrared light or the second LSD information corresponding to the specified second intensity, the electronic device may identify at least one color gain corresponding to the intensity of the identified infrared light. The color gain may indicate the degree to which at least one of red light, green light, or blue light is highlighted or enhanced. Figure 9B The operation of the electronic device selecting LSC information based on the intensity and threshold of infrared light is described.

[0125] Reference Figure 7In operation 740, the electronic device of various embodiments may adjust the color of at least a portion of the image data based on the selected LSC information. In an embodiment of the present disclosure, the electronic device may adjust the color of at least a portion of the image data based on at least one color gain included in the selected LSC information. In an embodiment of the present disclosure, the color adjusted by the electronic device may be the color of red light included in the image data. The more the color moves from the center of the image data toward its periphery and / or the more the intensity of infrared light increases, the more the electronic device may adjust (e.g., increase) the color of red light. In an embodiment of the present disclosure, the color adjusted by the electronic device may be the color of at least one of blue light or green light included in the image data. The more the color moves from the center of the image data toward its periphery and / or the more the intensity of infrared light increases, the more the electronic device may adjust (e.g., decrease) the color of at least one of blue light or green light included in the image data.

[0126] Figures 8A to 8B is a diagram illustrating LSC information used by an electronic device according to various embodiments of the present disclosure. Figures 8A to 8B The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101. Figures 8A to 8B The LSC information can be included in Figure 7 In the LSC information of operations 720, 730 and 740.

[0127] The LSC information recognized by the electronic device of various embodiments may include a plurality of color gains. The plurality of color gains included in the LSC information may be used to adjust the colors of pixels at different positions in the image data. Figures 8A to 8B , the designated first LSC information 810 and the designated second LSC information 820 are shown in table form. For example, the LSC information may include a (17×13) color gain. The color gain may indicate the degree to which the color of a pixel of the image data is emphasized or enhanced. The electronic device may, for example, multiply the color values ​​of the image data based on the Bayer pattern by the color gain to adjust the color of the image data. The color gain corresponding to each of the multiple pixels of the image data may be determined by interpolating the color gains included in the LSC information.

[0128] The position of the color gain in the table may indicate the position of the pixel in the image data to which the color gain is applied. For example, the color gain (1.000) in the center portion of the table (9th row and 7th column) may be applied to the center portion of the image data. In an embodiment of the present disclosure, the further one moves from the center portion of the image data toward its peripheral portion, the more the electronic device may increase the degree to which the color of the image data is adjusted based on the LSC information. Figure 8A, the further you move from the center portion of the table to its outer periphery, the more the color gain can be increased.

[0129] Electronic devices in various embodiments may select LSC information that emphasizes red light in image data, based on the greater intensity of infrared light included in the external light, from among multiple LSC information, including first LSC information 810 and second LSC information 820. Comparing the gain of the peripheral portion of first LSC information 810 (e.g., the gain of row 1 and column 1 (2.957)) with the gain of the peripheral portion of second LSC information 820 (e.g., the gain of row 1 and column 1 (3.139)), the gain of the peripheral portion of second LSC information 829 may be greater. In response to the intensity of infrared light included in the external light exceeding a specified threshold, the electronic device may select second LSC information 820, from among the first LSC information 810 and second LSC information 820, that includes a relatively greater color gain. The multiple color gains included in the selected second LSC information 820 may be used to adjust the colors of different pixels in the image data.

[0130] The electronic device of this embodiment may select LSC information that reduces the increment of at least one of blue light or green light in the image data, based on the intensity of infrared light included in the external light being greater, from among a plurality of LSC information including first LSC information 810 and second LSC information 820. As the increment of blue light or green light decreases, red light may be relatively emphasized. For example, in response to the intensity of infrared light included in the external light exceeding a specified threshold, the electronic device may adjust at least one of blue light or green light in the image data based on the first LSC information 810 including a relatively small color gain.

[0131] Figure 9A and Figure 9B is a diagram illustrating an operation of an electronic device selecting LSC information based on brightness of external light according to various embodiments of the present disclosure. Figures 9A to 9B The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101.

[0132] Reference Figure 9A , shows an example of a lookup table (LUT) 910 used by an electronic device to determine a threshold value for comparison with the intensity of infrared light. In an embodiment of the present disclosure, the electronic device may use the lookup table 910 to obtain Figure 7 The lookup table 910 may be a table indicating the relationship between the brightness value (BV) representing the brightness of the external light and the threshold value. For example, Figure 4 The AE processing unit 430 and / or AE algorithm of the embodiment can be used to identify the brightness of the external light. The brightness value can be based on a logarithmic scale with a base of 2.

[0133] The amount of infrared light included in the light source may vary depending on the brightness of the light source. The electronic device of the embodiment may identify at least one threshold value for determining whether relatively more infrared light is included in the external light based on the brightness of the external light. For example, in response to identifying a brightness value of -2, the electronic device may determine the extent to which infrared light is included in the external light based on threshold values ​​(60, 100). For example, in response to identifying a brightness value of 0, the electronic device may determine the extent to which infrared light is included in the external light based on threshold values ​​(300, 450). In response to identifying a brightness value not included in the lookup table 910, the electronic device may identify at least one threshold value by interpolating the threshold values ​​included in the lookup table 910 based on the brightness values ​​included in the lookup table 910.

[0134] Reference Figure 9B , Figure 9B 9 is a graph 920 illustrating an operation of acquiring LSC information for adjusting the color of image data based on a threshold value identified by the electronic device from the lookup table 910. The x-axis of the graph 920 may indicate the intensity of infrared light identified from the infrared sensor. Figure 9A The lookup table 910 is used to identify the thresholds for the graph 920 (ie, the IR low threshold and the IR high threshold).

[0135] For example, when the brightness value is recognized as -2, each threshold value of the graph 920 may be 60 or 100. When the brightness value is recognized as -2, the electronic device may determine that relatively little infrared light is included in the external light in response to the intensity of the infrared light recognized from the infrared sensor included in the duration 922 being less than 60. When the brightness value is recognized as -2, the electronic device may determine that relatively more infrared light is included in the external light in response to the intensity of the infrared light recognized from the infrared sensor included in the duration 924 being greater than 100.

[0136] For example, when a brightness value of 3 is recognized, each threshold value of graph 920 may be (1200, 1400). When a brightness value of 3 is recognized, the electronic device may determine that relatively little infrared light is included in the external light in response to the intensity of infrared light recognized from the infrared sensor during duration 922 being less than 1200. When a brightness value of 3 is recognized, the electronic device may determine that relatively more infrared light is included in the external light in response to the intensity of infrared light recognized from the infrared sensor during duration 924 exceeding 1400.

[0137] For example, in response to the electronic device based on Figure 8A and Figure 8BThe electronic device may identify the LSC information to be applied to the image data based on the first LSC information 810 and the second LSC information 820, select the first LSC information 810 in response to identifying the intensity of the infrared light included in the duration 922, and select the second LSC information 820 in response to identifying the intensity of the infrared light included in the duration 924. Because the color gain of the second LSC information 820 is a relatively high value, the electronic device may select the second LSC information 820 of the duration 924 corresponding to the intensity of relatively more infrared light.

[0138] The duration 926 between the durations 922 and 924 is a duration of the LSC information identified based on the interpolation. In response to identifying the intensity of the infrared light included in the duration 926, the electronic device may interpolate the first LSC information 810 and the second LSC information 820 (e.g., based on Figure 9B The linear interpolation coefficients (often called linear interpolation coefficients) are used to obtain the LSC information that will be applied to the image data.

[0139] LSC information corresponding to at least one of the durations 922, 924, and 926 may be selected from previously stored designated LSC information. In an embodiment of the present disclosure, the electronic device may select LSC information corresponding to at least one of the durations 922, 924, and 926 based on the color temperature of the external light.

[0140] Figure 10 is a diagram illustrating an operation of an electronic device selecting LSC information based on the color temperature of external light according to an embodiment of the present disclosure. Figure 10 The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101.

[0141] According to various embodiments of the present disclosure, the electronic device may select any one LSC information among a plurality of LSC information (eg, LSC tables 1 to 4) that differ from each other according to the intensity of infrared light based on the brightness, color temperature, and intensity of infrared light of external light. Figure 10 Table 1000 shows the type of LSC table selected by the electronic device based on the brightness, color temperature, and intensity of infrared light of the external light. Each of LSC tables 1 to 4 may correspond to each of a specified first intensity to a specified fourth intensity of infrared light. The intensity of infrared light may increase more as the intensity increases from the first intensity to the fourth intensity. LSC tables 1 to 4 may include a plurality of color gains corresponding to different positions of image data, such as Figures 8A to 8B shown.

[0142] Reference Figure 10Based on the brightness of external light, the electronic device can determine whether the image data was captured indoors or outdoors. For example, in response to identifying external light of a specified threshold or more, the electronic device can determine that the image data was captured outdoors. In an embodiment of the present disclosure, the electronic device can determine whether the image data was captured indoors or outdoors based on a global positioning system (GPS) sensor and / or user input.

[0143] In response to determining that the image data is captured outdoors, the electronic device of various embodiments may select any one of the designated LSC tables (e.g., LSC table 2 and LSC table 4) regardless of the color temperature of the external light. For example, in response to the intensity of infrared light being equal to or greater than a designated threshold, the electronic device may adjust the color of the image data based on LSC table 4. For example, in response to the intensity of infrared light being less than a designated threshold, the electronic device may adjust the color of the image data based on LSC table 2. For example, the electronic device may adjust the color of the image data based on LSC table 4. Figure 9A The threshold value is determined by using a lookup table 910.

[0144] In response to image data being captured indoors, the electronic device of various embodiments may select at least two of the designated LSC tables based on the color temperature of the external light. The selection of any of the LSC tables may be based on the intensity of infrared light measured by the infrared sensor. For example, in response to image data being captured indoors and having a color temperature of 4000K or higher, the electronic device may select either LSC Table 1 or LSC Table 3. For example, in response to image data being captured indoors and having a color temperature of 3000K or lower, the electronic device may select either LSC Table 2 or LSC Table 3.

[0145] The electronic device may select any one of the LSC tables based on a result of comparing the intensity of infrared light with a threshold value determined based on the brightness of external light. For example, in response to the image data having been taken indoors and the brightness value of the external light being -2, the electronic device may select any one of the LSC tables based on Figure 9A Lookup table 910 compares the intensity of infrared light with thresholds (60, 100). In response to the color temperature being 3000K or lower and the intensity of infrared light being less than 60K, the electronic device may select LSC Table 2 based on Table 1000 to perform LSC. In response to the color temperature being 3000K or lower and the intensity of infrared light exceeding 100K, the electronic device may select LSC from Table 3 to perform LSC. In response to the color temperature being 3000K or lower and the intensity of infrared light being equal to or greater than 60 and less than 100, the electronic device may identify the LSC table to be applied to the image data by interpolating LSC Table 2 and LSC Table 3.

[0146] As another example, in response to the image data having been captured indoors, and the brightness value of the external light being 3, the electronic device may Figure 9ALookup table 910 compares the intensity of infrared light with thresholds (1200, 1400). In response to the color temperature being equal to or greater than 3000K and less than 4000K, and the intensity of infrared light exceeding 1400K, the electronic device may select LSC table 3 to compensate for color difference. In response to the color temperature being equal to or greater than 3000K and less than 4000K, and the intensity of infrared light being less than 1200K, the electronic device may select LSC table 1 to compensate for color difference. In response to the color temperature being equal to or greater than 3000K and less than 4000K, and the intensity of infrared light being equal to or greater than 1200 and less than 1400K, the electronic device may determine the LSC table for compensating for color difference by interpolating LSC table 1 and LSC table 3.

[0147] By comparing thresholds based not only on the color temperature of the external light but also on the intensity and brightness of the infrared light, electronic devices in various embodiments can select any of a number of LSC tables. This allows the electronic device to compensate for color difference based on the actual intensity of infrared light included in the external light. For example, in a room exposed to sunlight, the color temperature identified by the AWB algorithm may exceed 5000K. Without considering the intensity of infrared light, color difference compensation can be performed based on indoor light sources exceeding 5000K (e.g., fluorescent lights and / or LED lights). In this case, due to the infrared light included in the sunlight, the central portion of the image data may be represented relatively red, while the peripheral portions may be represented relatively green. In this case, by compensating for color difference based on the infrared sensor's consideration of the intensity of infrared light, the electronic device in various embodiments can relatively emphasize the red light in the peripheral portions of the image data or relatively deemphasize the blue or green light in the peripheral portions.

[0148] Figure 11 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure. Figure 11 The electronic device may correspond to Figures 1 to 2 and Figures 3A to 3B electronic device 101. Figure 11 The operation of the electronic device can be Figure 2 ISP260 and / or Figure 3A Executed by processor 260.

[0149] Reference Figure 11 In operation 1110, the electronic device of various embodiments may receive first image data from an image sensor. The first image data may be arranged according to a pattern (e.g., a Bayer pattern) in which a plurality of PDs are arranged in the image sensor, and may include color values ​​measured in the corresponding plurality of PDs included in the image sensor.

[0150] Reference Figure 11In operation 1120, the electronic device of various embodiments may identify the brightness and color temperature of the first image data. For example, Figure 4 The AE processing unit 430 and / or AE algorithm of the first image data can identify the brightness of the first image data. The brightness of the first image data can represent the brightness of the external light related to the first image data. For example, Figure 4 The AWB processing unit 440 and / or the AWB algorithm may be used to identify the color temperature of the first image data. The color temperature of the first image data may represent the color temperature of external light associated with the first image data.

[0151] In an embodiment of the present disclosure, in response to identifying the brightness of other image data received from an image sensor prior to the first image data, the electronic device may adjust the brightness of the first image data. In an embodiment of the present disclosure, in response to identifying the color temperature of other image data received from an image sensor prior to the first image data, the electronic device may adjust the white balance of the first image data. In an embodiment of the present disclosure, compensation for color differences included in the first image data may be performed based on the brightness, color temperature, and intensity of infrared light included in the external light associated with the other image data.

[0152] Reference Figure 11 In operation 1130, the electronic device of various embodiments may identify the intensity of infrared light included in the external light associated with the first image data from the infrared sensor. The brightness and color temperature identified in operation 1120 and the intensity of infrared light identified in operation 1130 may be used to compensate for color differences in other image data received after the first image data.

[0153] Reference Figure 11 , in operation 1140, the electronic device of various embodiments may receive second image data from the image sensor. For example, after receiving the first image data, the electronic device may receive the second image data from the image sensor. In response to receiving the second image data, in operation 1150, the electronic device of various embodiments may adjust the color of at least a portion of the second image data based on at least one of the identified brightness, color temperature, and intensity of infrared light. In an embodiment of the present disclosure, based on the brightness and color temperature of the first image data identified in operation 1120 and the intensity of infrared light included in the external light associated with the first image data identified in operation 1130, the electronic device may adjust the color of at least a portion of the second image data.

[0154] The electronic device may adjust the color of at least a portion of the second image data to compensate for the color difference of the second image data. The operation performed by the electronic device to compensate for the color difference of the second image data may be based on Figures 9A to 9B and Figure 10 Various embodiments described in. Figure 11 In an embodiment, in response to the image sensor outputting first image data and second image data in a time sequence, the electronic device can compensate for lens shading and / or chromatic aberration of the second image data following the first image data based on brightness, color temperature, and intensity of infrared light associated with the first image data.

[0155] Electronic devices according to various embodiments may use an infrared sensor included in the electronic device to compensate for lens shading and / or chromatic aberration included in image data and depending on the structure of the lens and infrared filter. The electronic device may determine the degree to which the color of the image data should be adjusted based on the intensity of infrared light included in external light measured using the infrared sensor.

[0156] An electronic device according to various embodiments may include a lens, an infrared filter, an image sensor, the infrared sensor, and a processor operatively coupled to the image sensor and the infrared sensor. The processor may receive image data from the image sensor based on external light that has passed through the lens and the infrared filter and reached the image sensor, identify the intensity of infrared light included in the external light based at least on sensor data from the infrared sensor, and, in response to identifying the intensity of the infrared light, adjust the color of at least a portion of the image data based at least on the intensity of the infrared light.

[0157] In an embodiment of the present disclosure, the processor may adjust the color of at least a portion of the image data based on the intensity of the infrared light so as to compensate for distortion of the image data caused by the lens and the infrared filter.

[0158] In an embodiment of the present disclosure, the processor can obtain information including color gains corresponding to different pixels of the image data and corresponding to the intensity of the infrared light in response to identifying the intensity of the infrared light, and adjust the colors of multiple pixels included in the image data based on the obtained information.

[0159] In an embodiment of the present disclosure, the processor may acquire information including color gain associated with red light based on the intensity of infrared light.

[0160] In an embodiment of the present disclosure, the processor may, in response to receiving image data, identify the brightness and color temperature of external light associated with the image data, and adjust the color of at least a portion of the image data based on the brightness, color temperature, and intensity of infrared light.

[0161] In an embodiment of the present disclosure, a processor may receive image data from an image sensor and, after adjusting the color of at least a portion of the image data based at least on the intensity of infrared light, change the image data based on a specified pattern, wherein, in the image data, multiple signals output from multiple photodiodes within the image sensor are arranged based on a specified pattern of the multiple photodiodes, and wherein the image data indicates the color of each of multiple pixels based on multiple specified color components.

[0162] In an embodiment of the present disclosure, the processor may adjust the color of at least a portion of the image data corresponding to another portion of the lens in response to identifying the intensity of infrared light, wherein the other portion of the lens is different from the portion including the center of the lens.

[0163] In an embodiment of the present disclosure, the processor can identify at least one color gain corresponding to the intensity of the identified infrared light based on at least one of first information corresponding to the specified first intensity of the infrared light or second information corresponding to the specified second intensity, and adjust the color of at least a portion of the image data based on the at least one color gain.

[0164] The method of the electronic device of various embodiments may include: receiving image data based on external light that passes through a lens and an infrared filter of the electronic device and reaches the image sensor of the electronic device from an image sensor, and identifying the intensity of infrared light included in the external light based at least on sensor data of the infrared sensor of the electronic device, and in response to identifying the intensity of the infrared light, adjusting the color of at least a portion of the image data based at least on the intensity of the infrared light.

[0165] According to an embodiment of the present disclosure, the adjusting may include adjusting a color of at least a portion of the image data based on the intensity of the infrared light so as to compensate for distortion of the image data caused by the lens and the infrared filter.

[0166] According to an embodiment of the present disclosure, the adjustment may include: in response to identifying the intensity of infrared light, obtaining information including color gain, wherein the color gain corresponds to each different pixel of the image data and corresponds to the intensity of the infrared light; and based on the obtained information, adjusting the colors of multiple pixels included in the image data.

[0167] According to an embodiment of the present disclosure, the method may further include: in response to receiving image data, identifying the brightness and color temperature of external light related to the image data, and the adjustment may include adjusting the color of at least a portion of the image data based on the brightness, color temperature and intensity of infrared light.

[0168] According to an embodiment of the present disclosure, receiving may include receiving image data from an image sensor, wherein in the image data, a plurality of signals output from a plurality of photodiodes within the image sensor are arranged based on a specified pattern of the plurality of photodiodes, and the method may further include changing the image data based on the specified pattern after adjusting a color of at least a portion of the image data based at least on an intensity of infrared light, wherein the image data indicates a color of each of a plurality of pixels based on a plurality of specified color components.

[0169] According to an embodiment of the present disclosure, the adjusting may include adjusting a color of at least a portion of the image data corresponding to another portion of the lens in response to identifying the intensity of the infrared light, wherein the another portion of the lens is different from a portion including the center of the lens.

[0170] According to an embodiment of the present disclosure, the adjustment may include: identifying at least one color gain corresponding to the intensity of the identified infrared light based on at least one of first information corresponding to a specified first intensity of the infrared light or second information corresponding to a specified second intensity, and adjusting the color of at least a portion of the image data based on the at least one color gain.

[0171] An electronic device according to various embodiments may include an image sensor, an infrared sensor, and a processor operably coupled to the image sensor and the infrared sensor. The processor may receive first image data from the image sensor, and in response to receiving the first image data, identify brightness and color temperature of the first image data, identify intensity of infrared light included in external light from the infrared sensor associated with the first image data, and adjust the color of at least a portion of second image data received from the image sensor after receiving the first image data based on at least one of the identified brightness, color temperature, and intensity of infrared light.

[0172] According to an embodiment of the present disclosure, the processor may adjust the color of at least a portion of the second image data based on the intensity of infrared light so as to compensate for distortion of the second image data caused by the lens and infrared filter through which external light reaching the image sensor passes.

[0173] According to an embodiment of the present disclosure, the processor can obtain information including color gains corresponding to different pixels of the second image data in response to identifying the intensity of infrared light, and adjust the colors of multiple pixels included in the second image data based on the obtained information.

[0174] According to an embodiment of the present disclosure, the processor may select information corresponding to brightness, color temperature, and intensity of infrared light from among a plurality of designated information including color gain, and adjust the colors of a plurality of pixels included in the second image data based on the selected information.

[0175] According to an embodiment of the present disclosure, the processor may adjust the color of at least a portion of the second image data corresponding to another portion of the lens based on the selected information, wherein the another portion of the lens is different from the portion including the center of the lens associated with the image sensor.

[0176] The methods of the embodiments mentioned in the claims of the present disclosure or in the specification thereof may be implemented in the form of hardware, software, or a combination of hardware and software.

[0177] In response to software implementation, a computer-readable storage medium storing one or more programs (i.e., software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured to be executed by one or more processors of an electronic device. The one or more programs include instructions for enabling the electronic device to execute the methods of the embodiments described in the claims of this disclosure or in the specification thereof.

[0178] These programs (i.e., software modules and / or software) may be stored in random access memory (RAM), non-volatile memory including flash memory, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), magnetic disk storage, compact disk-ROM (CD-ROM), digital versatile disk (DVD), another form of optical storage, and / or magnetic tape cassettes. Alternatively, the programs may be stored in a memory configured to include some or all of these. Furthermore, each configured memory may include multiple memories.

[0179] In addition, the program may be stored in an attachable storage device that can be accessed via a communication network such as the Internet, an intranet, a local area network (LAN), a wireless LAN (WLAN), or a storage area network (SAN), or a combination thereof. The storage device can access the apparatus for executing the embodiments of the present disclosure via an external port. Alternatively, a separate storage device on the communication network can also access the apparatus for executing the embodiments of the present disclosure.

[0180] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device comprising: lens; infrared filter; Image sensor; Infrared sensor; as well as at least one processor operatively coupled to the image sensor and the infrared sensor, Wherein, the at least one processor is configured to: receiving image data based on external light from the image sensor, wherein the external light passes through the lens and the infrared filter and reaches the image sensor, identifying the intensity of infrared light included in the external light based on at least sensor data of the infrared sensor, and In response to identifying the intensity of the infrared light, adjusting a color of at least a portion of the image data based at least on the intensity of the infrared light, Wherein, adjusting the color of at least a portion of the image data includes: In response to identifying that the intensity of the infrared light exceeds a threshold, enhancing the red light compared to the green light or the blue light of the image data, and Here, the threshold value is determined based on the brightness of the external light, and the threshold value has a larger value as the brightness of the external light increases.

2. The electronic device according to claim 1, wherein The at least one processor is further configured to: adjusting the color of the at least a portion of the image data based on the intensity of the infrared light, and Compensates for distortion of image data caused by lenses and infrared filters.

3. The electronic device according to claim 1, wherein: The at least one processor is further configured to: In response to identifying the intensity of the infrared light, acquiring information including color gains, wherein the color gains correspond to respective mutually different pixels of the image data and correspond to the intensity of the infrared light, and Based on the acquired information, colors of a plurality of pixels included in the image data are adjusted.

4. The electronic device according to claim 3, wherein: The at least one processor is further configured to obtain information including a color gain associated with red light based on the intensity of the infrared light.

5. The electronic device according to claim 1, wherein The at least one processor is further configured to: In response to receiving the image data, identifying the brightness and color temperature of external light associated with the image data, and The color of the at least a portion of the image data is adjusted based on the brightness, the color temperature, and the intensity of the infrared light.

6. The electronic device according to claim 1, wherein The at least one processor is further configured to: receiving image data from an image sensor, wherein, in the image data, a plurality of signals output from a plurality of photodiodes within the image sensor are arranged based on a specified pattern of the plurality of photodiodes, and After adjusting the color of the at least a portion of the image data based at least on the intensity of the infrared light, the image data based on the designated pattern is changed, wherein the image data indicates a color of each of a plurality of pixels based on a plurality of designated color components.

7. The electronic device according to claim 1, wherein: In response to identifying the intensity of infrared light, the at least one processor is further configured to adjust a color of at least a portion of the image data corresponding to another portion of the lens, wherein the other portion of the lens is different from a portion including a center of the lens.

8. The electronic device according to claim 1, wherein: The at least one processor is further configured to: identifying at least one color gain corresponding to the identified intensity of the infrared light based on at least one of first information corresponding to the specified first intensity of the infrared light or second information corresponding to the specified second intensity, and The color of the at least a portion of the image data is adjusted based on the at least one color gain.

9. A method for an electronic device, the method comprising: receiving image data based on external light from the image sensor, wherein the external light passes through a lens and an infrared filter of the electronic device and reaches the image sensor of the electronic device; identifying the intensity of infrared light included in external light based on at least sensor data of an infrared sensor of the electronic device; and In response to identifying the intensity of the infrared light, adjusting a color of at least a portion of the image data based at least on the intensity of the infrared light, Wherein, adjusting the color of at least a portion of the image data includes: In response to identifying that the intensity of the infrared light exceeds a threshold, enhancing the red light compared to the green light or the blue light of the image data, and Here, the threshold value is determined based on the brightness of the external light, and the threshold value has a larger value as the brightness of the external light increases.

10. The method according to claim 9, wherein: Adjusting the color of at least a portion of the image data includes adjusting the color of the at least a portion of the image data based on the intensity of the infrared light and compensating for distortion of the image data caused by the lens and the infrared filter.

11. The method according to claim 9, wherein The step of adjusting the color of at least a portion of the image data comprises: In response to identifying the intensity of the infrared light, acquiring information including color gains, wherein the color gains correspond to respective mutually different pixels of the image data and correspond to the intensity of the infrared light; and Based on the acquired information, colors of a plurality of pixels included in the image data are adjusted.

12. The method according to claim 9, further comprising: In response to receiving the image data, identifying the brightness and color temperature of external light associated with the image data, and The adjusting includes adjusting the color of the at least part of the image data based on brightness, color temperature and intensity of infrared light.

13. The method according to claim 9, wherein: The step of receiving the image data includes receiving the image data from the image sensor, wherein the image data includes a plurality of signals output from a plurality of photodiodes within the image sensor arranged based on a specified pattern of the plurality of photodiodes, and The method further includes changing the image data based on a specified pattern after adjusting the color of the at least a portion of the image data based at least on the intensity of the infrared light, wherein the image data indicates a color of each of a plurality of pixels based on a plurality of specified color components.

14. The method according to claim 9, wherein Adjusting the color of at least a portion of the image data includes adjusting the color of at least a portion of the image data corresponding to another portion of the lens different from a portion including a center of the lens in response to identifying the intensity of the infrared light.

15. The method according to claim 9, wherein The step of adjusting the color of at least a portion of the image data comprises: identifying at least one color gain corresponding to the identified intensity of the infrared light based on at least one of first information corresponding to the specified first intensity of the infrared light or second information corresponding to the specified second intensity; and The color of the at least a portion of the image data is adjusted based on the at least one color gain.

Images