Electronic device comprising image sensor, and operating method thereof

Abstract

An electronic device according to one embodiment comprises a camera module including: a lens assembly including at least one lens and a sensor that outputs a signal for acquiring motion data corresponding to motion of the electronic device; an image sensor; and an optical image stabilizer (OIS) for moving the lens assembly and / or the image sensor in order to perform an optical image stabilization operation on the basis of the motion data. The electronic device can perform an autofocus function on the basis of autofocus data corrected on the basis of parameters for the optical image stabilization operation.

Description

Electronic device including image sensor and method of operating same

[0001] The present disclosure relates to a technology for performing an AF (auto focus) function in an electronic device including an image sensor.

[0002] As high-resolution modes become increasingly necessary, solutions are being proposed to improve AF performance using image sensors. Recently, due to camera layout constraints, image sensors are being developed with reduced pixel size and increased pixel count. In particular, with the demand for image sensors that not only support high-resolution modes but also offer high AF performance, pixel structures capable of full-pixel phase detection are being proposed.

[0003] An electronic device that adopts the AF method includes an image sensor in which two PDs (photodiodes) having a 2x1 array are arranged under one micro lens, and the AF function is performed by classifying the AF data output from each PD into left data and right data and focusing using the phase difference between the two. However, as the size of the PD decreases and the demand for a structure that can have high resolution increases, a structure in which four PDs having a 2x2 array rather than a 2x1 array share one micro lens is proposed. In this case, in addition to the advantage of having high resolution, it has the advantage of further improving the AF performance because the phase differences between the left, right, up, and down can be identified.

[0004] Meanwhile, the output of the PD pixel has a very large sensitivity difference depending on the CRA (chief ray angle) of the incident light, and if the value of the PD pixel is used as is to generate actual image data, deviation in the pixel value may occur depending on the physical location of the lens and pixel.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] An electronic device according to one embodiment may include a sensor, a camera module, a processor, and a memory. The sensor may output a signal for obtaining motion data regarding a movement of the electronic device. The camera module may include a lens assembly, an image sensor, and an optical image stabilizer (OIS). The lens assembly may include at least one lens. The optical image stabilizer may be configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization operation based on the motion data. The memory may store instructions executable by the processor. The instructions may be executed by the processor to cause the electronic device to obtain first AF data from the image sensor. The instructions may be executed by the processor to cause the electronic device to correct the first AF data based on parameters for an OIS operation to generate second AF data. The instructions may be executed by the processor to cause the electronic device to perform an AF (auto focus) function based on the second AF data.

[0007] An operating method of an electronic device according to one embodiment may include an operation of outputting a signal for obtaining motion data corresponding to a movement of the electronic device. The operating method may include an operation of controlling the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data. The operating method may include an operation of obtaining first AF data from the image sensor and an operation of generating second AF data by correcting the first AF data based on parameters for the OIS operation. The operating method may include an operation of performing an AF (auto focus) function based on the second AF data.

[0008] A computer-readable non-transitory recording medium according to one embodiment of the present disclosure may store at least one command and / or instructions that, when executed, cause an electronic device to perform the method or operation of the electronic device described above.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 3 is a diagram showing the main hardware configuration of an electronic device according to one embodiment.

[0012] FIG. 4 is a cross-sectional view of a pixel array of an image sensor according to one embodiment.

[0013] FIG. 5 is a drawing for explaining the effect of shading on an image sensor according to one embodiment.

[0014] Figure 6 illustrates an example of a shadow pattern detected according to the operation of an optical image stabilizer.

[0015] FIG. 7 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0016] FIGS. 8 and 9 illustrate the OIS operation of an electronic device according to one embodiment.

[0017] Figure 10 illustrates examples of determining whether to perform shading correction depending on the OIS operation direction when detecting a horizontal phase difference.

[0018] FIG. 11 illustrates an example of weights determined based on OIS positions according to one embodiment.

[0019] Hereinafter, embodiments will be described in detail with reference to the attached drawings. The advantages and features, and methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the attached drawings. The embodiments disclosed in this document are not limited to the embodiments disclosed below and may be implemented in various different forms, and are defined solely by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their commonly understood meanings by those of ordinary skill in the art. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terminology used herein is for the purpose of describing embodiments and is not intended to limit the embodiments. In this specification, singular forms also include plural forms, unless specifically stated otherwise.

[0021] The terms "comprises" and / or "comprising" as used in the specification do not exclude the presence or addition of one or more other components, operations, operations and / or elements.

[0022] In one embodiment, an electronic device and a method of operating the same can be provided that can correct a pattern that occurs due to a mismatch between an angle of incidence of light incident on a lens (lens angle of incidence) and an angle of incidence of light incident on a sensor (sensor angle of incidence).

[0023] In one embodiment, an electronic device and an operating method thereof can be provided that can improve accuracy when operating an autofocus function by correcting a pattern that occurs due to a mismatch between a lens incident angle and a sensor incident angle.

[0024] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains.

[0025] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0026] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0027] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0028] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0029] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0033] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0034] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0038] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0040] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0043] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0045] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0048] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0049] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0050] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0051] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0052] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0053] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0054] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0055] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0056] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0057] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0058] FIG. 3 illustrates the main hardware configuration of an electronic device according to one embodiment. In the description of FIG. 3, the configurations described in FIGS. 1 and 2 may be briefly described or omitted.

[0059] Referring to FIG. 3, in one embodiment, an electronic device (101) (e.g., the electronic device (101) of FIG. 1) may include a lens assembly (210) (e.g., the lens assembly (210) of FIG. 2), an image sensor (230) (e.g., the image sensor (230) of FIG. 2), an image signal processor (260) (e.g., the image signal processor (260) of FIG. 2), a processor (120) (e.g., the processor (120) of FIG. 1), a display (160) (e.g., the display module (160) of FIG. 1), and a memory (250) (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2).

[0060] According to one embodiment, the lens assembly (210) may have different numbers, arrangements, types, etc. of lenses depending on the front camera and the rear camera. Depending on the type of lens assembly (210), the front camera and the rear camera may have different characteristics (e.g., focal length, maximum magnification, etc.).

[0061] In one embodiment, if the image signal processor (260) and the image sensor (230) are physically separate, there may be a sensor interface that conforms to the standard.

[0062] According to one embodiment, the image signal processor (260) can perform image processing on electrically converted image data. The process in the image signal processor (260) can be divided into pre-ISP (hereinafter, pre-processing) and ISP chain (hereinafter, post-processing). The pre-processing process can include 3A processing, lens shading correction, edge enhancement, dead pixel correction, knee correction, etc. The 3A can include at least one of AWB (auto white balance), AE (auto exposure), and AF (auto focusing). The post-processing process can include at least one of changing a sensor index value, changing a tuning parameter, and adjusting an aspect ratio. The post-processing process can include a process of processing image data output from the image sensor (230) or image data output from a scaler. The image signal processor (260) can adjust the contrast, sharpness, saturation, dithering, etc. of the image through a post-processing process. Here, the contrast, sharpness, and saturation adjustment procedures can be performed in the YUV color space, and the dithering procedure can be performed in the RGB (Red Green Blue) color space. The image signal processor (260) can transmit the image data obtained after the post-processing process to a memory (e.g., a display buffer) (250). The display (160) can display the image data stored in the memory (e.g., a display buffer) (220) on a display screen under the control of the processor (120).

[0063] According to one embodiment, the processor (120) can execute / control various functions supported by the electronic device (101). For example, the processor (120) can execute an application and control various hardware by executing a code written in a programming language stored in the memory (250). For example, the processor (120) can execute an application supporting a photographing function stored in the memory (250). In addition, the processor (120) can execute a camera module (e.g., the camera module (180) of FIG. 1) and set and support an appropriate photographing mode so that the camera module (180) can perform an operation intended by the user.

[0064] According to one embodiment, memory (250) may store instructions executable by processor (120). Memory (250) may be understood as a concept that includes a component such as RAM (random access memory) in which data is temporarily stored and / or a component such as SSD (solid state drive) in which data is permanently stored. For example, processor (120) may call instructions stored in SSD to implement a software module in RAM space. In various embodiments, memory (250) may include various types, and an appropriate type may be adopted according to the purpose of the device. According to one embodiment, memory (250) may include one or more storage media (or one or more storage devices). For example, memory (250) may include a memory collection including one or more storage media. For example, the one or more storage media may include a hard drive, flash memory, permanent memory such as read-only memory (ROM), semi-permanent memory such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (250) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (101). As a non-limiting example, the cache memory may be included within the processor (120).The memory (250) may be fixedly embedded within the electronic device (101) or incorporated into one or more suitably types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that can be repeatedly inserted into and removed from the electronic device (101).

[0065] According to one embodiment, the memory (250) may store an application associated with the camera module (180). For example, the memory (250) may store a camera application. The camera application may support various shooting functions, such as photo shooting, video shooting, panoramic shooting, and slow motion shooting.

[0066] According to one embodiment, the memory (250) may store one or more software applications, such as an operating system (or system software application), a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by at least one processor (120). For example, the memory (250) may store instructions callable by an application programming interface (API). For example, the memory (250) may store instructions within a library.

[0067] According to one embodiment, the processor (120) may display, on the display (160), an execution screen of an application executed by the processor (120) or contents such as images and / or videos stored in the memory (250). In addition, the processor (120) may display image data acquired through the camera module (180) on the display (160).

[0068] According to one embodiment, the electronic device (101) may include a gyro sensor (330) and a hall sensor (320). For example, the gyro sensor (330) may measure a rotation angle or inclination of the electronic device (101) about three axes (e.g., X-axis, Y-axis, or Z-axis). However, the gyro sensor is exemplary, and the electronic device (101) may include various sensors (e.g., motion sensors) such as an acceleration sensor or a hall sensor (e.g., sensor module (176) of FIG. 1).

[0069] According to one embodiment, the OIS module (310) can obtain motion data from the gyro sensor (330). The OIS module (310) can obtain information on the degree of shaking of the electronic device (101) by performing an integral operation on the motion data. The OIS module (310) can control the OIS function of the camera module (180) based on the information on the motion data. In FIG. 3, the OIS module (310) is illustrated as controlling the lens assembly (210), but is not limited thereto. For example, the OIS module (310) may also be configured to control the position of the image sensor (230).

[0070] In one embodiment, the AF (auto focus) module (315) can control the AF function of the camera module (180) based on information obtained from the processor. For example, the AF module (315) can control the position of the lens assembly (210) (e.g., to adjust the distance between the lens assembly (210) and the image sensor (230)) so that the image sensor (230) is focused. In FIG. 3, the AF module (315) is illustrated as controlling the lens assembly (210), but is not limited thereto. For example, the AF module (315) may also be configured to control the position of the image sensor (230).

[0071] According to one embodiment, when the camera module (180) is of a lens shift type that moves the lens assembly (210), the processor (120) can detect the position to which the lens assembly (210) has moved through the hall sensor (320). According to one embodiment, when the camera module (180) is of a sensor shift type that moves the image sensor (230), the processor (120) can detect the position to which the image sensor (230) has moved through the hall sensor (320).

[0072] FIG. 4 is a cross-sectional view of a pixel array of an image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) according to one embodiment.

[0073] Referring to FIG. 4, the image sensor (230) may include a plurality of unit pixels (410). According to one embodiment, each unit pixel (410) may include four or more photo diodes (PD) (413). According to one embodiment, the plurality of unit pixels (410) may be positioned on a plane substantially perpendicular to a Z-axis corresponding to a direction in which light is incident. According to one embodiment, a first direction (e.g., a direction of the X-axis) of the plurality of unit pixels (410) may be substantially perpendicular to a second direction (e.g., a direction of the Y-axis) of the unit pixels (410). According to one embodiment, the first direction (e.g., a direction of the X-axis) and the second direction (e.g., a direction of the Y-axis) may be substantially perpendicular to a direction of the Z-axis.

[0074] According to one embodiment, each unit pixel (410) may include a micro lens (411), a color filter (412), and a plurality of photodiodes (413), or a combination thereof. According to one embodiment, each of the plurality of photodiodes (413) may also be referred to as a light-receiving element. According to one embodiment, the plurality of photodiodes (413) may also be referred to as a multi-photodiode. The color filter (412) of FIG. 4 may be arranged with color filters of the same color channel, but color filters including different color channels may also be arranged.

[0075] According to one embodiment, the micro lens (411) can focus light incident on the micro lens (411). According to one embodiment, the micro lens (411) can adjust the path of light incident on the micro lens (411) so that the light reaches each of the plurality of photodiodes (413).

[0076] According to one embodiment, the color filter (412) can pass light of a pre-specified color (or color channel). According to one embodiment, the color filter (412) of each of the plurality of photodiodes (413) can pass light of one color (e.g., red) among the pre-specified colors (e.g., red, blue, or green) according to a pre-specified pattern (e.g., Bayer pattern). According to one embodiment, the color filter (412) can block light of a color other than the pre-specified color (or color channel).

[0077] According to one embodiment, the number of photodiodes (413) arranged at a position corresponding to one micro lens (411) may be four or more. However, the present invention is not limited thereto. For example, two photodiodes may be arranged at a position corresponding to one micro lens (411). In the present disclosure, the phrase "a photodiode is arranged at a position corresponding to a micro lens" may mean that it is arranged at a position to receive light passing through the micro lens. The plurality of photodiodes (413) may be arranged symmetrically, but the present invention is not limited thereto. In the present disclosure, an image sensor having a plurality of photodiodes (413) corresponding to one micro lens (411) may be referred to as a multi-photodiode image sensor. According to one embodiment, each of the plurality of photodiodes (413) may output a value corresponding to incident light. According to one embodiment, each of the plurality of photodiodes (413) may output a value corresponding to incident light based on the photoelectric effect. According to one embodiment, each of the plurality of photodiodes (413) can output a value corresponding to the intensity (or illuminance) of incident light based on the photoelectric effect.

[0078] According to one embodiment, each of the plurality of photodiodes (413) can generate a charge according to the intensity (or illuminance) of incident light based on the photoelectric effect. According to one embodiment, each of the plurality of photodiodes (413) can output a current according to the amount of generated charge.

[0079] FIG. 5 is a drawing for explaining the effect of shading on an image sensor according to one embodiment.

[0080] The unit pixel (410) illustrated in FIGS. 4 and 5 may have a different amount of light reaching the photodiode (413) included in the unit pixel (410) depending on the angle at which light is incident. Therefore, the unit pixel (410) may detect light that enters within a certain range of angles with respect to the micro lens (411) or the photodiode (413). The light incident on the image sensor may include a chief ray and a marginal ray (or, marginal ray). The chief ray may refer to the main ray among the light rays incident on the image sensor. The marginal ray may refer to the light rays surrounding the chief ray. Among the light rays that reach the photodiode (413) of the image sensor, the chief ray is the strongest, and the intensity of the light becomes weaker as it goes toward the marginal ray. Outside of the marginal ray, light other than light generated by flare or internal reflection may not be incident on the image sensor. The range of angles at which the principal light for detecting light by the unit pixel (410) is incident on the image sensor (e.g., microlens (411) or photodiode (413)) may be referred to as an incidence angle range. Among the plurality of photodiodes (413) (e.g., two photodiodes, four photodiodes) that receive light passing through the microlens (411), the principal light may be incident on only some of the photodiodes depending on the incidence angle of the principal light. Bright light may be detected on the photodiode to which the principal light is incident, and dark light may be detected on the photodiode to which the principal light is not incident. This may cause a shading phenomenon.

[0081] Additionally, a camera module (180) equipped with an OIS function may have the principal ray angle of the optical system shifted according to the OIS operation. Shading may occur in an image sensor having multiple photodiodes arranged to correspond to a single micro lens according to the OIS operation.

[0082] (a), (b), and (c) of FIG. 5 illustrate the same unit pixel included in the image sensor. Referring to (b) of FIG. 5, when the operating position of the optical image stabilizer (e.g., the OIS module (310) of FIG. 3) is at a basic position (e.g., the center of the OIS operating range), the chief ray angle (512) of the image sensor may substantially match the chief ray angle (522) of the lens. The chief ray angle of the image sensor may refer to an angle at which light is incident so as to detect light without a shading effect within the unit pixel. The chief ray angle of the lens may refer to an angle at which light passing through the lens assembly (e.g., the lens assembly (210) of FIG. 2, the lens assembly (210) of FIG. 3) is incident on the unit pixel (or the light-receiving surface of the image sensor). The operation of the optical image stabilizer may cause a lens assembly (e.g., a lens assembly (210) of FIG. 2, a lens assembly (210) of FIG. 3) or an image sensor (e.g., an image sensor (230) of FIG. 2, an image sensor (230) of FIG. 3) of a camera module (e.g., a camera module (180) of FIG. 1, a camera module (180) of FIG. 2) to move or tilt. The movement of the lens assembly or the image sensor may cause a shift in the chief ray. The tilting of the lens assembly or the image sensor may cause a change in the angle at which the lens assembly and the image sensor are aligned with respect to each other. Accordingly, the chief ray angle (521, 522, 523) of the lens with respect to a unit pixel (e.g., a unit pixel (410) of FIG. 4)) and the chief ray angle (511, 512, 513) of the image sensor viewed by the unit pixel may change. This may cause a mismatch in the chief ray angles of the image sensor and the lens due to the operation of the optical image stabilizer. Fig. 5 (a) illustrates the chief ray angle (511) of the image sensor and the chief ray angle (521) of the lens when the optical image stabilizer is operating in the first direction.FIG. 5 (c) illustrates the chief ray angle (513) of the image sensor and the chief ray angle (523) of the lens when the optical image stabilizer operates in a second direction different from the first direction. Referring to FIG. 5 (a) and (c), among the photodiodes (413-1, 413-2), the chief ray of the lens is incident at a different angle from the chief ray of the image sensor, and one photodiode absorbs stronger light than the other photodiode, so that shadows may occur in the light detected through the photodiodes (413-1, 413-2). The amount of light detected by the photodiodes (413-1, 413-2) may vary depending on the angle at which the chief ray is incident. When detecting a phase difference in a state where shadows generated by OIS operation are included in the values ​​detected through the photodiodes (413-1, 413-2), it may not be possible to distinguish whether the phase difference determined through a correlation operation on the detected values ​​is generated due to the shadows or due to the phase difference for the actual subject. As a result, when an auto-focus function based on phase differences is performed while the optical image stabilizer is operating, the accuracy of the auto-focus function may be reduced. Alternatively, when an image is composed using each of the data detected from the first photodiode (413-1) and the second photodiode (413-2) as a single pixel value (e.g., when an image is composed without processing that merges data (e.g., binning operation)), the image quality of the acquired image may be low.

[0083] Figure 6 illustrates an example of a shadow pattern detected according to the operation of an optical image stabilizer.

[0084] FIG. 6 illustrates a shading pattern detected in a plurality of unit pixels (311, 312, 313, 314), each of which includes one microlens and four photodiodes corresponding to the microlens.

[0085] FIG. 6 illustrates shadow patterns detected in four operating states (601, 602, 603, 604) of the optical image stabilizer (e.g., states in which the optical image stabilizer is controlled to different positions). In FIG. 6, XOIS may mean an X-axis coordinate value indicating an operating state (or operating position) of the image stabilizer. Alternatively, XOIS may mean a first displacement value in the X-axis direction of the center of the lens assembly defined with the center of the image sensor as a reference point. In FIG. 6, YOIS may mean a Y-axis coordinate value indicating an operating state (or operating position) of the image stabilizer. Alternatively, YOIS may mean a second displacement value in the Y-axis direction of the center of the lens assembly defined with the center of the image sensor as a reference point. The first state (601) illustrates a case in which the OIS position in the first direction is about -3 degrees (XOIS = 8), and the OIS position in the second direction different from the first direction is about -3 degrees (YOIS = 8). The second state (602) illustrates a case where the OIS position in the first direction is about 3 degrees (XOIS = 248) and the OIS position in the second direction is about -3 degrees (YOIS = 8). The third state (603) illustrates a case where the OIS position in the first direction is about -3 degrees (XOIS = 8) and the OIS position in the second direction is about 3 degrees (YOIS 248). The fourth state (604) illustrates a case where the OIS position in the first direction is about 3 degrees (XOIS = 248) and the OIS position in the second direction is about 3 degrees (YOIS = 248).

[0086] In the first state (601), among the detection values ​​(611, 612, 613, 614) of the photodiodes included in the first unit pixel (311), the gain value (612) detected from the photodiode in the (+x, +y) direction can be detected as the highest. Conversely, the gain value (613) detected from the photodiode in the (-x, -y) direction can be detected as the lowest.

[0087] In the second state (602), among the detection values ​​(621, 622, 623, 624) of the photodiodes included in the first unit pixel (311), the gain value (621) detected from the photodiode in the (-x, +y) direction can be detected as the highest. Conversely, the gain value (624) detected from the photodiode in the (+x, -y) direction can be detected as the lowest.

[0088] In the third state (603), among the detection values ​​(631, 632, 633, 634) of the photodiodes included in the first unit pixel (311), the gain value (634) detected from the photodiode in the (+x, -y) direction can be detected as the highest. Conversely, the gain value (631) detected from the photodiode in the (-x, +y) direction can be detected as the highest.

[0089] In the fourth state (604), among the detection values ​​(631, 632, 633, 634) of the photodiodes included in the first unit pixel (6311), the gain value (643) detected from the photodiode in the (-x, -y) direction can be detected as the highest. Conversely, the gain value (642) detected from the photodiode in the (+x, +y) direction can be detected as the lowest.

[0090] Due to the shading pattern, the gain value of each photodiode can vary from approximately -80% to approximately +80%.

[0091] When the image of 4PD is binning, the luminance difference due to shading may not be a problem because the average value is used, but when performing AF or remosaic process, each gain value becomes meaningful data, and the luminance difference unrelated to the subject may become a problem. The remosaic process may mean an operation of converting the color order of data detected by the image sensor so that a demosaicing operation that converts the image data into a color image can be performed. In one embodiment, the electronic device may correct the luminance difference due to shading before performing AF or remosaic process.

[0092] The present disclosure can provide an electronic device and an operating method thereof that can efficiently correct shading caused by a difference between the chief ray angle of a lens and the chief ray angle of an image sensor in an image sensor having unit pixels such as 2PD or 4PD.

[0093] FIG. 7 is a flowchart (700) illustrating an operation method of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment.

[0094] In the present disclosure, the operation of the electronic device may be understood as being performed by a processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (120) of FIG. 3) executing instructions stored in a memory (e.g., the memory (130) of FIG. 1) to perform operations or control components of the electronic device. In one embodiment, FIG. 7 may be performed while the electronic device activates a camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2) to capture an image. For example, the operations illustrated in FIG. 7 may be performed while the electronic device executes a camera application and displays a preview screen including an image acquired through the camera.

[0095] According to one embodiment, in operation 701, the electronic device may obtain first AF data from an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (230) of FIG. 3). For example, a unit pixel according to FIG. 4 (e.g., the unit pixel (410) of FIG. 4) may include a multi-photodiode structure in which the phases of light incident on PDs (e.g., the photodiode (413) of FIG. 4) corresponding to one microlens (e.g., the microlens (411) of FIG. 4) are separated. The electronic device may obtain first AF data including phase difference information for at least one phase difference in at least one direction.

[0096] According to one embodiment, in operation 703, the electronic device may obtain at least one of motion data and OIS position information. For example, the electronic device may obtain motion data that detects the movement of the electronic device through a sensor (e.g., the sensor module (176) of FIG. 1). The sensor may include, for example, a gyro sensor, but the type of sensor is not limited thereto. For example, the electronic device may control the OIS position of the optical image stabilizer to compensate for the movement of the electronic device based on the motion data. The electronic device may obtain OIS position information indicating the OIS position of the optical image stabilizer. For example, the electronic device may obtain OIS position information from information detected by a Hall sensor (e.g., the Hall sensor (320) of FIG. 3)) disposed in the optical image stabilizer.

[0097] According to one embodiment, in operation 705, the electronic device may determine whether to perform shading compensation on the first AF data. The electronic device may be configured to determine whether to perform shading compensation based on whether at least one specified condition is satisfied. For example, the electronic device may determine whether a first condition indicating whether a direction corresponding to at least one of the motion data or the OIS position information (e.g., the direction of the motion data or the direction in which the OIS moved) corresponds to a phase difference direction of the first AF data is satisfied. The electronic device may determine whether to perform shading compensation based on the satisfaction of the first condition. A more specific description of the first condition will be described in more detail later with reference to FIG. 10. For example, the electronic device may determine whether to perform shading compensation based on the satisfaction of a second condition indicating whether an amount of movement of the electronic device during a specified time interval is greater than or equal to a threshold. For example, the electronic device may determine that the second condition is satisfied based on the integration of motion data detected during the specified time interval being greater than or equal to the threshold. For example, the electronic device may be configured to determine whether shading compensation is to be performed based on the satisfaction of both the first condition and the second condition. The at least one specified condition may be configured as a condition for determining whether shading compensation is required, and may include at least one condition other than the conditions described above.

[0098] According to one embodiment, based on the determination that shading correction is not performed in operation 705, the electronic device may perform an AF operation based on the first AF data acquired in operation 701. The AF operation may include an operation of controlling an AF actuator included in the camera (e.g., an actuator of the AF module (315) of FIG. 3) according to an AF position determined based on the first AF data.

[0099] According to one embodiment, based on the determination that shading correction is to be performed in operation 705, the electronic device may perform shading correction on the first AF data in operation 707. In one embodiment, the electronic device may perform shading correction on the first AF data based on a look-up table (LUT) stored in a memory. The look-up table may include a gain value for correcting the first AF data according to OIS position information based on a shading pattern (e.g., the shading pattern illustrated in FIG. 6). The gain value may include a value for correcting a luminance value of a pixel included in the first AF data. The electronic device may obtain a gain value corresponding to the OIS position information through the look-up table, and obtain corrected second AF data by applying the obtained gain value to the first AF data. In one embodiment, the electronic device may also adjust the gain value based on the OIS position. For example, the electronic device may determine a weight based on the OIS position information, and adjust the gain value based on the determined weight. In operation 709, the electronic device can perform an AF operation based on the second AF data corrected in operation 707. The AF operation can include an operation of controlling an AF actuator included in the camera according to an AF position determined based on the second AF data.

[0100] According to one embodiment, in operation 711, the electronic device may determine whether to terminate the AF operation. For example, the electronic device may determine to terminate the AF operation if the focus is on a region of interest (ROI) based on the AF operation performed in operation 709. For example, the electronic device may determine to terminate the AF operation if the camera operation is determined to be inactive or terminated (e.g., if the execution of the camera application is terminated). If the AF operation is determined not to be terminated, the electronic device may execute the operations illustrated in the flowchart (700) from operation 701.

[0101] In one embodiment, the electronic device may be configured to omit operation 705 and perform operation 707, which performs shading compensation.

[0102] In one embodiment, the electronic device can change the direction in which the phase detection is performed based on the direction of the motion data or the direction in which the OIS position is moved. For example, a 4PD image sensor in which PDs are arranged in a 2 x 2 array to correspond to one micro lens can detect phase difference data in a first direction (e.g., horizontal direction) or phase difference data in a second direction (e.g., vertical direction). When performing horizontal phase detection, if the horizontal value among the motion data has a continuous positive or negative value (or the integral value for a specified section is a positive or negative value), it means that a horizontal movement is occurring, and therefore the electronic device can change the direction in which the phase detection is performed to a vertical direction. When performing vertical phase detection, if the vertical value among the motion data has a continuous positive or negative value (or the integral value for a specified section is a positive or negative value), it means that a vertical movement is occurring, and therefore the electronic device can change the direction in which the phase detection is performed to a horizontal direction. In one embodiment, the electronic device may be configured to omit operation 707 of compensating for shading when the direction in which the phase detection is performed is changed and there is little movement of the electronic device corresponding to that direction.

[0103] FIGS. 8 and 9 illustrate the OIS operation of an electronic device according to one embodiment.

[0104] A gyro sensor (330) according to one embodiment can detect acceleration acting on three axes (e.g., X-axis, Y-axis, or Z-axis) of an electronic device (101), and the OIS operation can move on two axes (e.g., X-axis or Y-axis). FIGS. 8 and 9 show a position where a lens assembly (210) is placed by performing an OIS operation based on the X-axis component and the Y-axis component of the gyro sensor (330).

[0105] According to one embodiment, when the value of the motion data acquired from the gyro sensor is 0, the electronic device (101) may perform an OIS operation to control the OIS position (e.g., the x-axis / y-axis direction position of the lens assembly) so that the center of the lens is located at a first position (e.g., 800 of FIG. 8, 900 of FIG. 9) which is the center of the aperture of the lens.

[0106] According to one embodiment, when motion data acquired from a gyro sensor includes an x-axis direction acceleration value (Gx) of a first magnitude, the electronic device (101) can perform an OIS operation to control an OIS position (e.g., an x-axis direction position of a lens assembly) so that the center of the lens is located at a second position (801) moved from the center of the aperture of the lens.

[0107] According to one embodiment, when motion data acquired from the gyro sensor includes an x-axis direction acceleration value (2Gx) of a second magnitude greater than the first magnitude, the electronic device (101) may perform an OIS operation to control the OIS position (e.g., the x-axis direction position of the lens assembly) so that the center of the lens is located at a third position (802) that is moved by a greater amount than the second position (801) from the center of the aperture of the lens.

[0108] According to one embodiment, when motion data acquired from a gyro sensor includes an acceleration value (Gy) in the y-axis direction, the electronic device (101) can perform an OIS operation to control the OIS position (e.g., the y-axis direction position of the lens assembly) so that the center of the lens is located at a fourth position (901) moved in the y-axis direction from the center of the aperture of the lens.

[0109] According to one embodiment, when motion data acquired from a gyro sensor includes an acceleration value (-Gy) in the -y-axis direction, the electronic device (101) may perform an OIS operation to control the OIS position (e.g., the y-axis direction position of the lens assembly) so that the center of the lens is located at a fifth position (902) moved in the -y-axis direction from the center of the aperture of the lens.

[0110] According to one embodiment, since the electronic device performs the OIS operation based on the motion data, the electronic device may be configured to perform an operation (e.g., operation 705 of FIG. 7) of determining whether to perform shading compensation based on the motion data. In one embodiment, the electronic device may also be configured to perform an operation (e.g., operation 705 of FIG. 7) of determining whether to perform shading compensation based on a parameter (e.g., a parameter indicating an OIS position) obtained through a Hall sensor (e.g., a Hall sensor (320) of FIG. 3) disposed in the optical image stabilizer.

[0111] An electronic device according to one embodiment may determine whether shading compensation is required by determining the OIS position based on at least one of the OIS movement direction or parameters. The electronic device may compare the direction of the OIS with the direction of AF detection (phase difference direction). If the phase difference direction does not match the direction of OIS eccentricity and is perpendicular, the electronic device may not perform shading compensation even if the lens assembly is in a specific direction. This is because shading may exist in the vertical direction of the phase difference, but shading in the horizontal direction may be small or non-existent.

[0112] Figure 10 illustrates examples of determining whether to perform shading correction according to the OIS operation direction when detecting a phase difference in the horizontal direction (x-axis direction).

[0113] In one embodiment, data acquired from an image sensor may include a repeating pattern as illustrated in FIG. 10. For example, the data illustrated in FIG. 10 may represent data acquired from an image sensor having a designated pattern. The designated pattern may include a pattern in which a 2 X 2 array of photodiodes arranged at positions corresponding to one microlens is repeated. The designated pattern may include a pattern in which a 2 X 2 array of microlenses arranged to correspond to the same color pattern is repeated. Accordingly, the designated pattern may be a 4 X 4 array of photodiodes arranged to correspond to the same color pattern. Referring to FIG. 10, a pattern may be formed in units of 4 X 4 arrays within the data. The pattern illustrated in FIG. 10 may represent a pattern of pixel values ​​included in the acquired data.

[0114] In the first data (910) configured based on information acquired from the photodiodes of the image sensor while the OIS operation is being performed in the Y-axis direction, the shading included in the values ​​acquired from the photodiodes arranged in the horizontal direction (911) may be small or non-existent (i.e., shading in the horizontal direction may be small or non-existent). Accordingly, since the influence of shading on the phase difference detection in the horizontal direction (x-axis direction) is low, the electronic device can omit an operation for performing shading compensation and perform an AF operation.

[0115] In a state where an OIS operation is performed in the X-axis direction and the Y-axis direction (for example, in a state where the camera module moves in a diagonal direction with respect to the X-axis and the Y-axis), the second data (920) configured based on information acquired from the photodiodes of the image sensor may include shading that occurs in the values ​​acquired from the photodiodes arranged in the vertical and horizontal directions (921) (i.e., shading in the horizontal / vertical direction may occur). Accordingly, since the shading may affect the phase difference detection in the horizontal direction (x-axis direction), the electronic device may perform the AF operation based on the AF data for which shading compensation has been performed.

[0116] Since no shading occurs due to the operation of the OIS in the third data (930) configured based on information acquired from the photodiodes of the image sensor in a state where the OIS position is at the center (e.g., the first position (800, 900) of FIG. 8 or FIG. 9), the electronic device can perform the AF operation without performing the shading correction operation.

[0117] In the fourth data (940) configured based on information acquired from photodiodes of the image sensor while the OIS operation is being performed in the X-axis direction, shading may be included in the values ​​acquired from photodiodes arranged in the horizontal direction (941) (i.e., shading in the horizontal direction may occur). Accordingly, since shading may affect phase difference detection in the horizontal direction (x-axis direction), the electronic device may perform the AF operation based on the AF data for which shading compensation has been performed.

[0118] In one embodiment, when the phase difference direction and the OIS direction are aligned, whether or not to perform shading compensation and the amount of compensation can be adjusted. However, an electronic device including a 4PD sensor can also adjust the phase difference calculation direction according to the direction requiring compensation only when the phase difference direction is not important. Accordingly, shading compensation can be omitted by changing the calculation direction.

[0119] FIG. 11 illustrates an example of weights determined based on OIS positions according to one embodiment.

[0120] The M x N pixel-by-pixel luminance correction values ​​for performing shading correction may differ depending on the OIS position. The M x N pixel-by-pixel luminance correction values ​​are assigned to a look-up table (LUT), and each correction value can be determined by considering the CRA characteristics of the microlens provided for each unit pixel.

[0121] The coordinate values ​​shown in Fig. 11 may indicate the OIS location. Within the OIS range shown in Fig. 11, a higher brightness may indicate a higher weight value.

[0122] Fig. 11 illustrates the weight distribution applied when the operating direction of the OIS is the X-axis. For example, when the eccentricity in the X-axis direction is severe due to the OIS operation (section C), the correction value assigned to the LUT can be increased. Conversely, when the eccentricity in the X-axis direction is weak due to the OIS operation (section B), the correction value assigned to the LUT can be decreased. In addition, when the eccentricity in the X-axis direction due to the OIS operation is intermediate (section B), the correction values ​​used in sections A and C can be considered and an average value can be applied.

[0123] By determining correction values ​​for performing shading correction using the weights illustrated in FIG. 11, an electronic device can perform shading correction without having to store LUTs for all OIS positions. Therefore, the amount of memory storage capacity can be saved.

[0124] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0125] An electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may include a sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device. An electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may include a lens assembly including at least one lens (e.g., the lens assembly (210) of FIG. 2 and the lens assembly (210) of FIG. 3), an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (230) of FIG. 3), and a camera module including an optical image stabilizer (OIS) configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization operation based on motion data (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2). An electronic device (e.g., the electronic device 101 of FIG. 1, the electronic device 101 of FIG. 3) according to one embodiment may include a processor (e.g., the processor 120 of FIG. 1, the processor 120 of FIG. 3) and a memory that stores instructions executable by the processor. The instructions according to one embodiment may be configured to cause the electronic device to obtain first AF data from an image sensor. The instructions according to one embodiment may be configured to cause the electronic device to correct the first AF data based on parameters for an OIS operation to generate second AF data. The instructions according to one embodiment may be configured to cause the electronic device to perform an AF (auto focus) function based on the second AF data.

[0126] According to one embodiment, the parameters may include a first displacement value and a second displacement value, which are coordinate values ​​of the center of a lens assembly (e.g., 210 of FIG. 2 and 210 of FIG. 3) defined with the center of the image sensor (e.g., 230 of FIG. 2 and 230 of FIG. 3) as a reference point.

[0127] Instructions according to one embodiment may obtain a look up table (LUT) corresponding to at least one parameter or a Chief Ray Angle (CRA) of at least one lens, and correct first AF data based on the LUT.

[0128] Instructions according to one embodiment can identify a phase difference direction of first AF data and determine whether to perform correction based on a relationship between the phase difference direction and the parameter.

[0129] An instruction according to one embodiment may omit correction if the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

[0130] Instructions according to one embodiment can adjust the weights of a LUT for correction when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship.

[0131] Instructions according to one embodiment can change the direction of phase detection based on first AF data when the direction of movement of at least one lens due to OIS operation is maintained for a defined period of time or longer.

[0132] An instruction according to one embodiment may perform an AF (auto focus) function based on a first phase difference and a second phase difference included in the second AF data having a changed luminance value. The first phase difference may indicate a phase difference in a first direction, and the second phase difference may indicate a phase difference in a second direction different from the first direction.

[0133] An image sensor according to one embodiment (e.g., 230 of FIG. 2, 230 of FIG. 3) includes a plurality of unit pixels, each unit pixel including four or more PDs (photo diodes), the PDs being arranged adjacent to each other in a first direction and a second direction different from the first direction, and each of the unit pixels may include at least one color filter formed on the four or more PDs (photo diodes) included in the unit pixel and at least one micro lens formed on the at least one color filter.

[0134] In one embodiment, the first direction may be perpendicular to the second direction.

[0135] An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of outputting a signal for obtaining motion data corresponding to a movement of the electronic device. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of controlling a lens assembly (e.g., 210 of FIG. 2 and 210 of FIG. 3) or an image sensor (e.g., 230 of FIG. 2, 230 of FIG. 3) to perform an optical image stabilization (OIS) operation based on the motion data. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to an embodiment may include an operation of obtaining first AF data from an image sensor (e.g., 230 of FIG. 2, 230 of FIG. 3). An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to one embodiment may include an operation of generating second AF data by correcting first AF data based on parameters for OIS operation. An operating method of an electronic device (e.g., 101 of FIG. 1, 101 of FIG. 3) according to one embodiment may include an operation of performing an AF (auto focus) function based on the second AF data.

[0136] According to one embodiment, the parameters may include a first displacement value and a second displacement value, which are coordinate values ​​of the center of the lens assembly defined with the center of the image sensor as a reference point.

[0137] An operating method of an electronic device according to one embodiment may further include an operation of obtaining a look up table (LUT) corresponding to at least one of the parameters or a Chief Ray Angle (CRA) of the at least one lens; and an operation of correcting the first AF data based on the LUT.

[0138] An operating method of an electronic device according to one embodiment may further include an operation of confirming a phase difference direction of the first AF data; and an operation of determining whether to perform the correction based on a relationship between the phase difference direction and the parameter.

[0139] According to one embodiment, the method of operating an electronic device may omit an operation of generating the second AF data when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

[0140] The operating method of the electronic device according to one embodiment may further include an operation of adjusting a weight of the LUT for the correction when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship.

[0141] An operating method of an electronic device according to one embodiment may further include an operation of changing a direction of phase difference detection based on the first AF data when the direction of movement of the at least one lens by the OIS operation is maintained for a defined period of time or longer.

[0142] An operating method of an electronic device according to one embodiment may further include performing an AF (auto focus) function based on a first phase difference and a second phase difference included in the second AF data having a changed luminance value. The first phase difference may indicate a phase difference in a first direction, and the second phase difference may indicate a phase difference in a second direction different from the first direction.

[0143] An image sensor according to one embodiment includes a plurality of unit pixels, each unit pixel including four or more photo diodes (PDs) arranged adjacent to each other in a first direction and a second direction different from the first direction, and each of the unit pixels may include at least one color filter formed on the four or more photo diodes (PDs) included in the unit pixel and at least one micro lens formed on the at least one color filter.

[0144] In one embodiment, the first direction may be perpendicular to the second direction.

[0145] An electronic device and an operating method thereof according to embodiments of the present disclosure can improve the accuracy of a focus adjustment function performed by the electronic device to capture an image.

[0146] Electronic devices and their operating methods according to embodiments of the present disclosure can improve the accuracy of their autofocus functions by correcting autofocus data based on the situation in which the electronic device captures an image. For example, the accuracy of the autofocus function can be improved in situations in which an image is captured by tracking a subject, in situations in which an image is captured while panning a camera, in situations in which a super-resolution image is captured using an optical image stabilizer, or in situations in which a panoramic image is captured.

[0147] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

Claims

1. In electronic devices, A sensor that outputs a signal for obtaining motion data corresponding to the movement of the electronic device; A camera module comprising a lens assembly including at least one lens, an image sensor, and an optical image stabilizer (OIS) configured to move at least one of the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data; processor; a memory storing instructions executable by the processor; The above instructions cause the electronic device to: Acquire first AF data from the image sensor, Generate second AF data by correcting the first AF data based on the parameters for the above OIS operation, An electronic device that performs an AF (auto focus) function based on the second AF data.

2. In paragraph 1, The above parameters are, An electronic device including a first displacement value and a second displacement value, which are coordinate values of the center of the lens assembly defined with the center of the image sensor as a reference point.

3. In paragraph 2, At least one of the above instructions causes the electronic device to: Obtain a look up table (LUT) corresponding to at least one of the above parameters or the CRA (Chief Ray Angle) of the at least one lens, An electronic device for correcting the first AF data based on the LUT.

4. In paragraph 2, At least one of the above instructions causes the electronic device to: Check the phase difference direction of the above first AF data, An electronic device that determines whether to perform the correction based on the relationship between the phase difference direction and the parameters.

5. In paragraph 4, At least one of the above instructions causes the electronic device to: An electronic device that allows the correction to be omitted when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

6. In paragraph 4, At least one of the above instructions causes the electronic device to: An electronic device that adjusts the weight of the LUT for the correction when the relationship between the phase difference direction and the parameter corresponds to a horizontal relationship.

7. In paragraph 1, At least one of the above instructions causes the electronic device to: An electronic device that changes the direction of phase difference detection based on the first AF data when the direction of movement of at least one lens by the OIS operation is maintained for a defined period of time or longer.

8. In paragraph 1, At least one of the above instructions causes the electronic device to: The AF (auto focus) function is performed based on the first phase difference and the second phase difference included in the second AF data whose brightness value has been changed. An electronic device wherein the first phase difference indicates a phase difference in a first direction, and the second phase difference indicates a phase difference in a second direction different from the first direction.

9. In paragraph 1, The above image sensor, A plurality of unit pixels, each unit pixel including four or more PDs (photo diodes), the PDs being arranged adjacent to each other in a first direction and a second direction different from the first direction, An electronic device wherein each of the unit pixels includes at least one color filter formed on four or more photo diodes (PDs) included in the unit pixel and at least one micro lens formed on the at least one color filter.

10. In paragraph 9, An electronic device, wherein the first direction is perpendicular to the second direction.

11. In the method of operating an electronic device, An action of outputting a signal for obtaining motion data corresponding to the movement of the electronic device; An operation of controlling the lens assembly or the image sensor to perform an optical image stabilization (OIS) operation based on the motion data; An operation of acquiring first AF data from the image sensor; An operation of generating second AF data by correcting the first AF data based on the parameters for the OIS operation; and An operating method of an electronic device including an operation of performing an AF (auto focus) function based on the second AF data.

12. In paragraph 11, The above parameters are, An operating method of an electronic device including a first displacement value and a second displacement value, which are coordinate values of the center of the lens assembly defined with the center of the image sensor as a reference point.

13. In paragraph 12, An operation of obtaining a look up table (LUT) corresponding to at least one of the above parameters or the CRA (Chief Ray Angle) of the at least one lens; and An operating method of an electronic device further comprising an operation of correcting the first AF data based on the LUT.

14. In paragraph 12, An operation for checking the phase difference direction of the above first AF data; and A method of operating an electronic device further comprising an operation of determining whether to perform the correction based on the relationship between the phase difference direction and the parameters.

15. In paragraph 14, An operating method of an electronic device, wherein an operation of generating the second AF data is omitted when the relationship between the phase difference direction and the parameter corresponds to a vertical relationship.

Images