Method for stabilization in shooting and electronic device thereof

Abstract

At least one processor included in the electronic device can acquire at least one image frame through an image sensor, determine a first position of a lens based on a first light intensity acquired through a first photodiode and a second light intensity acquired through a second photodiode, determine the amount of movement of the lens moved by an OIS actuator based on the reference position of the lens and the first position of the lens, and perform digital image stabilization (DIS) on the at least one image frame based on the amount of movement of the lens. In addition, various embodiments identified through the specification are possible.

Description

Technology Field

[0001] The embodiments disclosed in this document relate to an electronic device and a method for shake correction during camera shooting. Background Technology

[0002] In camera functions, image stabilization is an essential and important feature for obtaining sharp photos. Generally, image stabilization methods include optical image stabilization (OIS) and digital image stabilization (DIS). Optical image stabilization reduces shake by moving the lens or sensor, while digital image stabilization, a method adopted by portable devices, reduces shake through digital processing. Typically, to ensure smooth and effective stabilization, portable devices operate both optical and digital image stabilization simultaneously to perform precise stabilization over a wide correction range. The problem to be solved

[0003] To perform smooth image stabilization, synchronization must be achieved between optical image stabilization (OIS) and digital image stabilization (DIS). However, to achieve such synchronization, the portable terminal must employ an image sensor capable of transmitting image frames and motion data associated with the image frames (e.g., OIS data and gyroscope data) together to the processor.

[0004] Outputting shake information (e.g., OIS data and gyroscope data) synchronized with the captured image frame complicates the hardware configuration and requires significant costs. Additionally, while the aforementioned shake information can be output to the processor in accordance with the image frame's vertical synchronization signal (Vsync), this also requires a separate hardware component—an OIS MCU (microcontroller unit)—to transmit both the image frame and the synchronized shake information.

[0005] Various embodiments of the present disclosure may provide a method for obtaining OIS information synchronized with an image by calculating the amount of lens movement through an image sensor composed of multiple photodiodes (i.e., a multi-PD image sensor) and an electronic device thereof. means of solving the problem

[0006] An electronic device according to one embodiment disclosed in this document may include a lens assembly including a lens, an OIS actuator that moves the lens assembly in a direction perpendicular to an optical axis, a micro lens, an image sensor including at least two photodiodes corresponding to the micro lens, at least one motion sensor, a memory, and at least one processor operatively connected to the lens assembly, the OIS actuator, the image sensor, the at least one motion sensor, and the memory. The at least one processor may acquire at least one image frame through the image sensor, identify a first position of the lens based on a first light intensity acquired through a first photodiode and a second light intensity acquired through a second photodiode, identify the amount of movement of the lens moved by the OIS actuator based on the reference position of the lens and the first position of the lens, and perform digital image stabilization (DIS) on the at least one image frame based on the amount of movement of the lens.

[0007] Additionally, a method of operation of an electronic device according to an embodiment disclosed in this document may include the operation of acquiring at least one image frame through an image sensor, the operation of identifying a first position of a lens based on a first light intensity acquired through a first photodiode and a second light intensity acquired through a second photodiode, the operation of identifying the amount of movement of the lens moved by an OIS actuator based on a reference position of the lens and the first position of the lens, and the operation of performing digital image stabilization (DIS) on the at least one image frame based on the amount of movement of the lens.

[0008] Additionally, an electronic device according to one embodiment disclosed in this document may include a lens assembly including a lens, an OIS actuator for moving the lens assembly in a direction perpendicular to an optical axis, a micro lens, an image sensor including at least two photodiodes corresponding to the micro lens, at least one motion sensor, a memory, and at least one processor operatively connected to the lens assembly, the OIS actuator, the image sensor, the at least one motion sensor, and the memory. The above at least one processor acquires at least one image frame through the image sensor, acquires data regarding a first difference in light intensity based on a first light intensity acquired through a first photodiode and a second light intensity acquired through a second photodiode, acquires data regarding a second difference in light intensity based on a third light intensity acquired through the first photodiode and a fourth light intensity acquired through the second photodiode in response to shaking of the electronic device, determines the amount of movement of the lens moved by the OIS actuator based on the data regarding the first difference in light intensity and the data regarding the second difference in light intensity, and can perform shake correction on the at least one image frame based on the determined amount of movement of the lens and the movement of the electronic device acquired through the at least one motion sensor. Effects of the invention

[0009] According to the various embodiments disclosed in this document, OIS driving information can be obtained through an input image.

[0010] In addition, according to various embodiments, a separate hardware configuration for synchronizing OIS and VDIS is not required, which can reduce costs.

[0011] In addition, according to various embodiments, since the OIS movement amount is directly determined from the captured image, errors caused by external factors such as temperature changes and the passage of time that occur when using separate hardware can be prevented.

[0012] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram illustrating a camera module according to various embodiments. FIG. 3 is a block diagram illustrating a camera module according to one embodiment. FIG. 4 is a diagram illustrating photodiodes included in a pixel of an image sensor according to one embodiment. FIG. 5 is a diagram illustrating the arrangement of photodiodes corresponding to one pixel according to one embodiment. FIG. 6 is a diagram illustrating the amount of light incident on pixels according to one embodiment. FIG. 7 is a two-dimensional diagram showing how the optical center changes due to OIS operating to compensate for the effects of shaking of the electronic device in an electronic device according to one embodiment. FIG. 8 is a three-dimensional diagram showing how the optical center changes due to shaking of the electronic device in an electronic device according to one embodiment. FIG. 9a is a flowchart illustrating the process of analyzing the optical characteristics of a lens using multiple photodiodes and performing shake correction based thereon in an electronic device according to one embodiment. FIG. 9b illustrates a graph showing the average value of the luminance value and the normalized shading difference (NSD) obtained pixel by pixel in an image obtained by photographing a non-patterned subject in an electronic device according to one embodiment, and a graph showing the average value of the luminance value and the NSD obtained pixel by pixel in an image obtained by photographing a patterned subject. Figure 10a shows an image obtained when a subject without a pattern is photographed. Figure 10b shows an image obtained when a patterned subject is photographed. FIG. 11a is a graph showing luminance values ​​obtained per pixel when the image of FIG. 10a is captured in an electronic device according to one embodiment. FIG. 11b is a graph showing luminance values ​​obtained per pixel when the image of FIG. 10b is captured in an electronic device according to one embodiment. FIG. 12a is a graph showing the average value of the normalized shading difference (NSD) when the image of FIG. 10a is taken in an electronic device according to one embodiment. FIG. 12b is a graph showing the average value of the normalized shading difference (NSD) when the image of FIG. 10b is taken in an electronic device according to one embodiment. FIG. 13a is a graph showing the amount of OIS movement detected in an electronic device according to one embodiment. FIG. 13b is a graph showing data for an initial NSD obtained from an electronic device according to one embodiment. FIG. 13c is a graph showing data for NSD calculated by correcting the OIS movement amount in an electronic device according to one embodiment. FIG. 14 is a diagram illustrating an operation to correct shaking based on first motion information obtained from a motion sensor and second motion information generated by OIS driving in an electronic device according to one embodiment. FIG. 15 is a drawing illustrating the operation of correcting an image frame line by line in an electronic device according to one embodiment. Specific details for implementing the invention

[0014] Hereinafter, various embodiments of the present document are described with reference to the accompanying drawings. However, this is not intended to limit the present document to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0015] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0016] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0017] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0018] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0019] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0020] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0021] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0022] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0023] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0024] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0025] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0026] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0027] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0028] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0029] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0030] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0031] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0032] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0033] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0034] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0035] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0036] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0037] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0038] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments.

[0039] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 350-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.

[0040] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0041] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented, for example, as an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0042] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be composed of at least a part of the processor (120), or It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) either as is or after additional image processing by the processor (120).

[0043] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0044] FIG. 3 is a block diagram illustrating a camera module (300) according to one embodiment. In one embodiment, the camera module (300) may correspond to the camera module (180) of FIG. 1 and FIG. 2.

[0045] In one embodiment, the camera module (300) may include an image acquisition module (301), an image signal processor (ISP) (303), a memory (250), or a combination thereof. In one embodiment, the image signal processor (303) may correspond to the image signal processor (260) of FIG. 2. In one embodiment, the memory (250) may store a plurality of reference luminance ratios corresponding to different optical characteristics (e.g., F-value, position of the lens (320), or degree of tilting of the lens (320)). In one embodiment, the reference luminance ratio may be the ratio of luminance values ​​between at least two photodiodes to a reference image acquired in advance to have arbitrary optical characteristics. For example, the reference luminance ratio may be understood as the luminance ratio corresponding to the second image frame when the electronic device (101) acquires a first image frame and a second image frame acquired prior to the first image frame.

[0046] In one embodiment, the image acquisition module (301) may include an image sensor (230), an aperture (310), a lens (320), an actuator (330), or a combination thereof. Referring to FIG. 3, the aperture (310) is illustrated as being located in front of the lens (320) (e.g., in a direction relatively adjacent to light), but this is merely an example and the position of the aperture may not be limited thereto. For example, the aperture (310) may be located between the lens (320) and the image sensor (230). In one embodiment, the lens (320) may be composed of one or more lenses. In one embodiment, the image sensor (230) may correspond to the image sensor (230) of FIG. 2. In one embodiment, the lens (320) may be included in the lens assembly (210) of FIG. 2.

[0047] In one embodiment, the image signal processor (303) may include an image processing unit (340), a preprocessing unit (350), noise reduction units (361, 362, 363, 364, 366, and 367), a ratio calculation unit (371, 373), a movement amount detection unit (381), or a combination thereof. In one embodiment, the image processing unit (340), the preprocessing unit (350), the noise reduction units (361, 362, 363, 364, 366, and 367), the ratio calculation unit (371, 373), the movement amount detection unit (381), or a combination thereof may be implemented through a hardware circuit. In one embodiment, an image processing unit (340), a preprocessing unit (350), noise reduction units (361, 362, 363, 364, 366, and 367), a ratio calculation unit (371, 373), a movement amount detection unit (381), or a combination thereof may be implemented as software capable of driving an image signal processor (303). In one embodiment, at least one operation performed by the image signal processor (303) may be performed by an application processor (AP). For example, data output from the image processing unit (340), a motion sensor, and a movement amount detection unit (381) (e.g., image data, motion data, and / or lens movement amount by OIS) may be transmitted to an application processor to perform shake correction (e.g., VDIS, video digital image stabilization).

[0048] In one embodiment, the aperture (310) can adjust the size of the entrance pupil under the control of the processor (120) or the image signal processor (303). In one embodiment, if the size of the entrance pupil of the aperture (310) increases, the amount of light reaching the image sensor (230) may increase. In one embodiment, if the size of the entrance pupil of the aperture (310) decreases, the amount of light reaching the image sensor (230) may decrease.

[0049] In one embodiment, the lens (320) can collect light incident through the aperture (310). In one embodiment, the actuator (330) can move the lens (320) to one of different positions (331 to 339) under the control of the image signal processor (303).

[0050] In one embodiment, the image sensor (230) can detect a signal corresponding to light passing through the aperture (310) and the lens (320). Light information of the subject incident through the lens assembly (210) can be converted into an electrical signal by the image sensor (230) and input to the image signal processor (303).

[0051] In one embodiment, the image sensor (230) may be a multi-PD sensor. That is, the image sensor (230) may include a plurality of pixels, and each of the plurality of pixels may include at least two photodiodes. Depending on the number of photodiodes included in each pixel, the image sensor (230) may be understood or referred to as a multi-PD sensor, such as a dual photodiode sensor or a triple photodiode sensor. For example, a first pixel included in the image sensor (230) (e.g., the first pixel (410) of FIG. 4) may include a first photodiode (e.g., the first photodiode (415L) of FIG. 4) and a second photodiode (e.g., the second photodiode (415R) of FIG. 4). For example, the first pixel (410) included in the image sensor (230) may include a first photodiode (e.g., the first photodiode (555L) of FIG. 5), a second photodiode (e.g., the second photodiode (555M) of FIG. 5), and a third photodiode (e.g., the third photodiode (555R) of FIG. 5).

[0052] In one embodiment, the electronic device (101) can control the movement of the lens (320) and / or lens assembly (210) through an actuator (330). The actuator (330) may include an auto focus (AF) actuator and / or an optical image stabilization (OIS) actuator. In one embodiment, when the electronic device (101) shakes, at least one processor (e.g., processor (120) of FIG. 1) may generate an OIS control value corresponding to the shake to compensate for the shake of the electronic device (101) (e.g., shake caused by hand tremor). The electronic device (101) may perform shake correction by transmitting an electrical signal corresponding to the generated OIS control value to at least one coil of the OIS actuator. When taking a picture with a camera, at least one processor (e.g., processor (120) of FIG. 1) can generate an AF control value to adjust the focus position of the subject and the camera, and the electronic device (101) can implement AF by transmitting an electrical signal corresponding to the AF control value to at least one coil of the AF actuator.

[0053] In an embodiment of the actuator, the preprocessing unit (350) may provide a first pixel value and a second pixel value to the noise reduction units (361, 362, 363, or 364). The first pixel value may include the output values ​​of the first photodiode and the second photodiode. The second pixel value may include the output values ​​of the third photodiode and the fourth photodiode. In one embodiment, the preprocessing unit (350) may provide the first pixel value to the noise reduction units (361, 362). In one embodiment, the preprocessing unit (350) may provide the second pixel value to the noise reduction units (363, 364).

[0054] In one embodiment, the number of noise reduction units may be determined by the number or combination of photodiodes included in the pixel. For example, if there are two photodiodes included in one pixel, there are two photodiode output values, so there may be two noise reduction units corresponding thereto. For example, if there are three photodiodes included in one pixel, there are three photodiode output values, so there may be three noise reduction units corresponding thereto.

[0055] In one embodiment, each of the noise reduction units (361, 362, 363, and 364) can reduce noise in input data (e.g., a first pixel value and / or a second pixel value). In one embodiment, each of the noise reduction units (361, 362, 363, and 364) can provide the noise-reduced data to the ratio calculation unit (371, 373).

[0056] In one embodiment, the noise reduction units (361, 362, 363, and 364) can reduce noise by attenuating high-frequency components of the input data using binning, averaging, interpolation, low-pass filtering, or a combination thereof. The noise reduction units (361, 362, 363, and 364) can also reduce phase-dependent noise by correcting the phase difference of the input data. For example, the noise reduction units (361, 362) can reduce phase difference noise of light detected by the first photodiode and the second photodiode forming a multi-photodiode structure with the first photodiode. Noise reduction units (363, 364) can reduce phase difference noise of light detected by the third photodiode and the fourth photodiode forming a multi-photodiode structure with the third photodiode.

[0057] In one embodiment, the ratio calculation unit (371, 373) can calculate the luminance ratio of the noise (e.g., phase difference noise) reduced data (e.g., luminance value of a noise-reduced pixel and / or luminance value of a noise-reduced photodiode) from the noise reduction units (361, 362, 363 and 364). For example, the first ratio calculation unit (371) can calculate the first luminance ratio based on the noise (e.g., phase difference noise) reduced data from the first noise reduction unit (361) and the second noise reduction unit (362). For example, the second ratio calculation unit (373) can calculate the second luminance ratio based on the noise (e.g., phase difference noise) reduced data from the third noise reduction unit (363) and the fourth noise reduction unit (364).

[0058] In one embodiment, the ratio calculation unit (371, 373) may transmit the first luminance ratio and the second luminance ratio to the noise reduction unit (366, 367). The luminance ratio may be understood or referred to as NSD (normalized shading difference) or ΔS below. In one embodiment, the noise reduction unit (366, 367) may reduce the noise of the NSD obtained from the ratio calculation unit (371, 373).

[0059] In one embodiment, the movement amount detection unit (381) can obtain data for detecting the movement amount of the lens (320) by OIS driving from the noise reduction unit (366, 367) and the memory (250). The movement amount detection unit (381) can obtain information regarding the reference luminance ratio, reference NSD value, and the position of the reference lens (320) that are stored in advance from the memory (250). The reference luminance ratio, reference NSD value, and the position of the reference lens (320) may be data corresponding to an image frame acquired prior to the image frame currently being acquired, or data corresponding to the time of the previously acquired image frame. In one embodiment, the movement amount detection unit (381) can detect the OIS movement amount corresponding to the image frame currently being acquired based on the data corresponding to the image frame currently being acquired (e.g., lens movement amount, NSD value, luminance ratio) and the data acquired from the memory. The OIS movement amount can be understood as the movement amount of the lens (320) moved by the OIS drive. In one embodiment, the movement amount detection unit (381) can transmit the detected OIS movement amount to the VDIS module (391).

[0060] In one embodiment, the motion sensor (383) may include an acceleration sensor, a gyroscope sensor, or a magnetic sensor. However, these sensors are exemplary, and the motion sensor may include at least one other type of sensor. The acceleration sensor is a sensor configured to measure acceleration acting on the three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101), and can measure, estimate, and / or detect the force applied to the electronic device (101) using the measured acceleration. The gyroscope sensor is a sensor configured to measure angular velocity acting on the three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101), and can measure and / or detect the amount of rotation for each axis of the electronic device (101) using the measured angular velocity information for each axis. Referring to FIG. 3, the motion sensor (383) is shown as being included in the block of the image signal processor (303), but it may be separate hardware separated from the image signal processor (303). In one embodiment, the motion sensor (383) can transmit the acquired motion data to the VDIS module (390).

[0061] In one embodiment, the VDIS module (390) can determine a correction value to be additionally corrected through data obtained from the movement amount detection unit (381) and the motion sensor (383). The VDIS module (390) can perform VDIS by applying the correction value to an image frame obtained from the image processing unit (340).

[0062] FIG. 4 is a drawing illustrating photodiodes included in a pixel (410) of an image sensor (230) according to one embodiment.

[0063] Referring to FIG. 4, the image sensor (230) may include a pixel array (401). The pixel array (401) may be referred to or understood as an array of photodiodes. In one embodiment, the pixel array (401) may include a plurality of pixels. For example, the pixel array (401) may be composed of an array of individual pixels such as a first pixel (410) (e.g., an mxn array, where m and n are natural numbers). In one embodiment, the pixel array (401) may be located in a plane perpendicular to the Z-axis (e.g., the optical axis) corresponding to the direction in which light is incident. In one embodiment, the first direction of the pixel array (401) (e.g., the direction of the X-axis) may be perpendicular to the second direction of the pixel array (401) (e.g., the direction of the Y-axis). In one embodiment, the first direction (e.g., the direction of the X-axis) and the second direction (e.g., the direction of the Y-axis) may be perpendicular to the direction of the Z-axis.

[0064] In one embodiment, the pixel array (401) may include at least one microlens, an infrared cut filter (IR cut filter, infra-red cut filter), a color filter array (CFA), and at least one anti-reflection film. For example, the first pixel (410) may include a microlens (411), an infrared cut filter (412), a color filter (413), an anti-reflection film (414), at least two photodiodes (415L, 415R), or a combination thereof.

[0065] In one embodiment, the photodiode may also be referred to as a light-receiving element. In one embodiment, a configuration in which at least two photodiodes (e.g., 415L, 415R) are included in one pixel (e.g., the first pixel (410)) may be referred to as a multi-photodiode (multi PD).

[0066] In one embodiment, the micro lens (411) can concentrate light incident on the micro lens (411). In one embodiment, the micro lens (411) can adjust the path of light incident on the micro lens (411) so that light reaches the first photodiode (415L) and the second photodiode (415R).

[0067] In one embodiment, the infrared blocking filter (412) can block at least some of the infrared light incident through the micro lens (411). According to various embodiments, the infrared blocking filter (412) may be placed between the micro lens (411) and the color filter (413), but is not limited thereto and may be placed at various locations.

[0068] In one embodiment, the color filter (413) may allow light of a predetermined color (or color channel) to pass through. In one embodiment, the color filter (413) of each of the plurality of pixels (e.g., the first pixel (410) of FIG. 4) may allow light of one of the predetermined colors (e.g., red, blue, or green) to pass through according to a predetermined pattern (e.g., Bayer pattern). In one embodiment, the color filter (413) may block light of a color other than the predetermined color (or color channel).

[0069] In one embodiment, the anti-reflection film (414) can prevent light incident through the micro lens (411) from being reflected outward.

[0070] In one embodiment, the first sub-photodiode (415L) and the second sub-photodiode (415R) can output a value corresponding to the incident light. In one embodiment, the first sub-photodiode (415L) and the second sub-photodiode (415R) can output a value corresponding to the incident light based on the photoelectric effect. In one embodiment, the first sub-photodiode (415L) and the second sub-photodiode (415R) can output a value corresponding to the intensity (or illuminance) of the incident light based on the photoelectric effect.

[0071] In one embodiment, the first sub-photodiode (415L) and the second sub-photodiode (415R) can generate a charge based on the intensity (or illuminance) of incident light based on the photoelectric effect. In one embodiment, the first sub-photodiode (415L) and the second sub-photodiode (415R) can output a current based on the amount of generated charge. In one embodiment, the value output by the first sub-photodiode (415L) may also be referred to as the first detection value. In one embodiment, the value output by the second sub-photodiode (415R) may also be referred to as the second detection value.

[0072] Referring to FIG. 4, two sub-photodiodes (415L, 415R) are exemplified as being placed on the left and right sides of a single pixel (410), but this is merely an example. A single pixel (e.g., the first pixel (410)) may contain two or more sub-photodiodes at different locations.

[0073] In one embodiment, the image sensor (230) may provide a value corresponding to the intensity of the detected light to the image signal processor (303). In one embodiment, the image sensor (230) may provide the image signal processor (303) with a first detection value output by the first photodiode (415L) of each pixel (e.g., the first pixel (410) of FIG. 4) and a second detection value output by the second photodiode (415R). In one embodiment, the image sensor (230) may provide the image signal processor (303) with the sum of the first detection value output by the first photodiode (415L) of each pixel (e.g., the first pixel (410) of FIG. 4) and the second detection value output by the second photodiode (415R). In one embodiment, when the image sensor (230) provides a sum value to the image signal processor (303), it may also provide a low-resolution luminance signal to the image signal processor (303). In one embodiment, the low-resolution luminance signal may correspond to the sum of the outputs of adjacent pixels among the pixels (e.g., the first pixel (410) of FIG. 4). In one embodiment, the adjacent pixels may be pixels among the surrounding pixels that are assigned different colors.

[0074] In one embodiment, the preprocessing unit (350) may output a luminance signal based on data from the image sensor (230) (e.g., a first detection value and a second detection value, or a sum value and a low-resolution luminance signal). In one embodiment, the preprocessing unit (350) may output a first luminance signal corresponding to a first detection value output by the first photodiode (415L) of the pixel (410). In one embodiment, the preprocessing unit (350) may output a second luminance signal corresponding to a second detection value output by the second photodiode (415R) of the pixel (410).

[0075] FIG. 5 is a diagram illustrating the arrangement of photodiodes corresponding to one pixel according to one embodiment.

[0076] Referring to FIG. 5, the pixel (510) may include a first photodiode (515L) positioned on the left (e.g., -x direction) and a second photodiode (515R) positioned on the right (e.g., +x direction). In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident in the horizontal direction on the photodiodes (515L, 515R) positioned on the left and right.

[0077] Referring to FIG. 5, the pixel (520) may include a first photodiode (525T) positioned on the upper side (e.g., +y direction) and a second photodiode (525B) positioned on the lower side (e.g., -y direction). In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident in the vertical direction on the sub-photodiodes (525T, 525B) positioned vertically.

[0078] Referring to FIG. 5, the pixel (530) may include a first photodiode (535T) positioned on the upper side (e.g., +y direction), a second photodiode (535B) positioned on the lower side (e.g., -y direction), a third photodiode (535L) positioned on the left side (e.g., -x direction), and a fourth photodiode (535R) positioned on the right side (e.g., +x direction). Light having different phases and / or different intensities (or illuminance) may be incident on the four photodiodes (535T, 535B, 535L, 535R) in the horizontal and / or vertical directions. In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident on one pair of photodiodes (535T, 535L) and another pair of photodiodes (535B, 535R) among the four photodiodes (535T, 535B, 535R) in a first diagonal direction (e.g., 135 degrees clockwise from the x-axis). In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident on a pair of photodiodes (535B, 535L) and another pair of photodiodes (535T, 535R) among the four photodiodes (535T, 535B, 535L) in a second diagonal direction (e.g., 45 degrees counterclockwise from the x-axis). In one embodiment, the first diagonal direction and the second diagonal direction may be perpendicular to each other.

[0079] Referring to FIG. 5, the pixel (540) may include a first photodiode (545LT) positioned in the upper left (e.g., 135 degrees clockwise from the x-axis), a second photodiode (545RT) positioned in the upper right (e.g., 45 degrees clockwise from the x-axis), a third photodiode (545LB) positioned in the lower left (e.g., 135 degrees counterclockwise from the x-axis), and a fourth photodiode (545RB) positioned in the lower right (e.g., 45 degrees counterclockwise from the x-axis). Light having different phases and / or different intensities (or illuminance) may be incident on the four photodiodes (545LT, 545RT, 545LB, 545RB) in the horizontal and / or vertical directions. In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident in a third diagonal direction on the photodiodes (545LT, 545RB) positioned diagonally among the four photodiodes (545LT, 545RT, 545LB, 545RB). In one embodiment, light having different phases and / or different intensities (or illuminance) may be incident in a fourth diagonal direction on the photodiodes (545RT, 545LB) positioned diagonally among the four photodiodes (545LT, 545RT, 545LB, 545RB). In one embodiment, the third diagonal direction and the fourth diagonal direction may be perpendicular to each other.

[0080] According to various embodiments, the number of NSDs can be set differently depending on the number of photodiodes included in a single pixel and the summing method. For example, if a single pixel includes 4 photodiodes, 4 NSDs can be set, and if the number of NSDs is calculated by summing 2 photodiodes, 3 NSDs can be set.

[0081] FIG. 6 is a drawing illustrating the amount of light incident on pixels (610, 620, and 630) according to one embodiment.

[0082] Referring to FIG. 6, light can pass through a lens (320) and then be incident on pixels (610, 620, and 630). Referring to FIG. 6, it can be seen that the intensity (or illuminance) of the light incident on the pixels (610, 620, and 630) varies depending on the position in the lens (320) through which the light passes. Referring to FIG. 6, one or more pixels (641) can be placed between the pixels (610, and 620). Referring to FIG. 6, one or more pixels (645) can be placed between the pixels (620, and 630).

[0083] Referring to FIG. 6, different amounts of light may be incident on the photodiodes (611, and 615) of the pixel (610). For example, referring to FIG. 6, the area of ​​light passing through the right portion of the lens (320) that is incident on the photodiode (615) of the pixel (610) may be smaller than the area of ​​light passing on the photodiode (611) of the pixel (610). Also, for example, referring to FIG. 6, the area of ​​light passing through the left portion of the lens (320) that is incident on the photodiode (631) of the pixel (630) may be smaller than the area of ​​light passing on the photodiode (635) of the pixel (630).

[0084] As illustrated in FIG. 6, depending on the location of the pixels (610, 620, and 630), the intensity of incident light may differ between the photodiodes (611, 615, 621, 625, 631, and 635) of each of the pixels (610, 620, and 630). In one embodiment, as the intensity of incident light differs between the photodiodes (611, 615) of the pixel (610), the output of each of the photodiodes (611, 615) may differ. In one embodiment, as the intensity of incident light differs between the photodiodes (621, 625) of the pixel (620), the output of each of the photodiodes (621, 625) may differ. In one embodiment, as the intensity of incident light between the photodiodes (631, 635) of the pixel (630) is different, each of the photodiodes (631, 635) may have a different output.

[0085] FIG. 7 is a two-dimensional diagram showing the change in the optical center due to OIS operating to correct the effect of shaking of the electronic device (101) in an electronic device (101) according to one embodiment. FIG. 8 is a three-dimensional diagram showing the change in the optical center due to shaking of the electronic device (101) in an electronic device (101) according to one embodiment.

[0086] Referring to FIGS. 7 and FIGS. 8, it can be seen that the optical center changes due to shaking of the electronic device (101) (e.g., shaking caused by hand tremor). The change in the optical center can be understood as the position of the lens (320) within the camera module (180) changing relatively due to the shaking, thereby changing the focal position of the lens (320) or the center position of the lens (320).

[0087] Referring to FIGS. 7 and FIGS. 8, i ois the center of the first optical axis (O) with respect to the optical axis (e.g., z-axis). o The i-th pixel (P) in ) i It is the angle up to ), and i c is the center of the second optical axis (O) based on the optical axis (e.g., z-axis). c The i-th pixel (P) in ) i It is the angle up to ). j o is the center of the first optical axis (O) with respect to the optical axis (e.g., z-axis). o The j-th pixel (P) in ) j It is the angle of ). j c is the center of the second optical axis (O) based on the optical axis (e.g., z-axis). c The j-th pixel (P) in ) j It is the angle of ). k o is the center of the first optical axis (O) with respect to the optical axis (e.g., z-axis). o The k-th pixel (P) in ) k It is the angle of ). k c is the center of the second optical axis (O) based on the optical axis (e.g., z-axis). c The k-th pixel (P) in ) k It is the angle of ).

[0088] In one embodiment, the first optical axis center (O o ) can be understood as the optical axis center corresponding to the initial optical axis. The second optical axis center (O c ) can be understood as the center of the optical axis corresponding to the current optical axis. The above initial optical axis may refer to the optical axis at a point in time prior to the above current optical axis.

[0089]

[0090] Referring to [Equation 1], ΔS may be the normalized shading difference (NSD). ΔS may be referred to as the luminance ratio or light intensity ratio of the first photodiode (e.g., the first photodiode (415L) of FIG. 4) and the second photodiode (e.g., the second photodiode (415R) of FIG. 4). Referring to [Equation 1], Y1 and Y2 may be luminance values ​​(or brightness values ​​or light intensity) obtained through the multiple photodiodes. In other words, Y1 may be the luminance value of the first photodiode (e.g., 415L) constituting the multiple photodiodes, and Y2 may be the luminance value of the second photodiode (e.g., 415R) constituting the multiple photodiodes.

[0091] In one embodiment, patch W k Optical axis shift using internal NSD values ​​( O 0 → O c NSD values ​​can be calculated based on ). When the optical axis is shifted, the CRA (chief ray angle) of each pixel can be determined to correspond to the new position. For example, when the optical axis is shifted, P1, P k and P j For the angle determined corresponding to two pixels, the relationship as shown in [Equation 2] below may hold.

[0092]

[0093] In one embodiment, a relatively small patch W k In cases where the NSD value is related only to the change in CRA, a relationship like [Equation 3] can be established.

[0094]

[0095] ΔS i o is the center of the first optical axis (O o It can represent the NSD value calculated through the i-th pixel in ). ΔS k ois the center of the first optical axis (O o It can represent the NSD value calculated through the k-th pixel in ). ΔS i c is the center of the second optical axis (O c It can represent the NSD value calculated through the i-th pixel in ). ΔS k c is the center of the second optical axis (O c The NSD value calculated through the k-th pixel in ) can be represented.

[0096] In one embodiment, the first optical axis center (O o NSD(ΔS at ) o ) and change in angle Δ , Δ As a new second optical axis center (O c NSD(ΔS calculated from ) e ) has a relationship like [Equation 4].

[0097]

[0098] In one embodiment, assuming that the movement of Δx and Δy due to OIS at a fixed position on the optical axis (z-axis) is a relatively small amount, [Equation 4] can be calculated as a translation in the x and y-axis directions as shown in [Equation 5] below. ΔS o is the reference NSD, and ΔS e may be a newly calculated NSD.

[0099]

[0100] In one embodiment, the electronic device (101) is a reference NSD (ΔS o Referring to ), the current NSD(ΔS e The OIS displacement can be detected by calculating the amount of parallel displacement of ).

[0101] FIG. 9a is a flowchart illustrating a process of analyzing the optical characteristics of a lens using multiple photodiodes and performing shake correction based thereon in an electronic device according to one embodiment. Referring to FIG. 9a, although it is described that the operation is performed by an electronic device (e.g., the electronic device (101) of FIG. 1), the entity performing the operation in the flowchart illustrated in FIG. 9a can also be understood as a processor (e.g., the processor (120) of FIG. 1) or an image signal processor (e.g., the image signal processor (260) of FIG. 2 or the image signal processor (303) of FIG. 3).

[0102] In operation 910 according to one embodiment, the electronic device (101) can acquire at least one image frame through an image sensor (e.g., image sensor (230) of FIG. 2). The electronic device (101) can drive a camera (e.g., camera module (180) of FIG. 2) in response to an input executing a camera application. The electronic device (101) can sequentially acquire a plurality of image frames output from the image sensor (230) in response to the camera being driven.

[0103] In one embodiment, shaking (e.g., shaking caused by hand tremor) may occur in the electronic device (101) while the camera is operating. In one embodiment, when shaking (e.g., shaking caused by hand tremor) occurs in the electronic device (101), a plurality of image frames reflecting the shaking of the electronic device (101) can be acquired. In other words, the electronic device (101) can acquire image frames having a field of view (FOV) changed by the shaking.

[0104] In one embodiment, the electronic device (101) can primarily correct the shaking. In other words, the electronic device (101) can optically correct the shaking through an OIS actuator. The electronic device (101) can acquire at least one image frame in which the shaking is optically corrected through OIS driving.

[0105] In operation 920 according to one embodiment, the electronic device (101) can determine a first position of a lens (e.g., lens (320) of FIG. 3) based on a first light amount obtained through a first photodiode (e.g., first photodiode (415L) of FIG. 4) and a second light amount obtained through a second photodiode (e.g., second photodiode (415R) of FIG. 4). The first position may be a position of the lens (320) corresponding to a first time point in time when a first image frame is obtained using the first photodiode and the second photodiode.

[0106] In one embodiment, the electronic device (101) can determine a ratio of a first light amount obtained through a first photodiode (415L) and a second light amount obtained through a second photodiode (415R). In other words, the electronic device (101) can determine a luminance ratio based on a first luminance value obtained through the first photodiode (415L) and a second luminance value obtained through the second photodiode (415R).

[0107] In one embodiment, the electronic device (101) can calculate a normalized shading difference (NSD) value based on the first light quantity and the second light quantity. The NSD may be a value normalized by luminance based on the difference in light quantity entering each photodiode. More specifically, the electronic device (101) can remove phase difference noise from the first luminance data obtained through the first photodiode (415L) and the second luminance data obtained through the second photodiode (415R). The electronic device (101) can determine the luminance ratio based on the luminance data from which the phase difference noise has been removed and calculate the NSD value. In one embodiment, the electronic device (101) can identify the position of the lens corresponding to the calculated NSD value.

[0108] In operation 930 according to one embodiment, the electronic device (101) can determine the amount of movement of the lens (320) moved by the OIS actuator (e.g., actuator (330) of FIG. 3) based on the reference position of the lens (320) and the first position of the lens (320). The electronic device (101) can calculate an NSD value corresponding to the reference position of the lens (320) and an NSD value corresponding to the first position of the lens (320). The reference position may be the position of the lens (320) corresponding to a second time point in which a second image frame, which is a frame prior to the first image frame, is acquired. The reference position may be the position of the lens (320) in an unobstructed state that is stored in memory in advance. The operation of determining the amount of movement of the lens (320) is described in FIG. 7 and FIG. 8, so it will be omitted here.

[0109] In operation 940 according to one embodiment, the electronic device (101) can perform digital image stabilization (DIS) on at least one image frame based on the amount of movement of the lens (320). In other words, the electronic device (101) can perform optical image stabilization (OIS) prior to performing digital image stabilization (DIS) to correct the shaking of the electronic device (101) (e.g., shaking caused by hand tremor). Digital image stabilization can be performed on at least one image frame based on the amount of OIS correction of the electronic device (101).

[0110] In one embodiment, the electronic device (101) can obtain an image frame obtained through an image sensor (230), motion data obtained from a motion sensor (383), and a movement amount of the lens (320) detected through a movement amount detection unit (e.g., movement amount detection unit (381) of FIG. 3). The electronic device (101) can perform shake correction (e.g., VDIS, video digital image stabilization) on the image frame by considering the motion data and the movement amount of the lens (320). For example, a VDIS module (e.g., VDIS module (390) of FIG. 3) can obtain the movement amount of the lens (320) moved by OIS from the movement amount detection unit (381). The VDIS module (390) can obtain an image frame from an image processing unit (e.g., image processing unit (340) of FIG. 3). The VDIS module (390) can acquire motion data (e.g., gyroscope data and / or acceleration data) from the motion sensor (383). The VDIS module (390) can perform VDIS based on the acquired image frames and data. Further description of VDIS may be provided below in FIGS. 14 and FIGS. 15.

[0111] FIG. 9b illustrates a graph showing the average value of the luminance value and the normalized shading difference (NSD) obtained pixel by pixel in an image obtained by photographing a non-patterned subject in an electronic device according to one embodiment, and a graph showing the average value of the luminance value and the NSD obtained pixel by pixel in an image obtained by photographing a patterned subject.

[0112] Referring to FIG. 9b (a), an electronic device (101) according to one embodiment can obtain a first image (945) by photographing a subject that has no pattern and uniform brightness. Referring to FIG. 9b (b), an electronic device (101) according to one embodiment can obtain a second image (950) by photographing a subject that has a pattern.

[0113] FIG. 9b(c) is a graph showing the luminance distribution according to position on the first image (945). FIG. 9b(d) is a graph showing the luminance distribution according to position on the second image (950). In the case of the second image (950), it may be difficult to detect shading characteristics using the luminance distribution due to the influence of the subject. FIG. 9b(e) and (f) are graphs showing the average value of the normalized shading difference (NSD) when the first image (945) and the second image (950) are each captured in an electronic device (101) according to one embodiment. When using NSD, the influence of the subject can be removed through normalization, so a constant characteristic can always be detected regardless of the captured subject. According to one embodiment, even when two photodiodes are included in a single pixel, the NSD exhibits different shapes in the Y-axis direction as well as the X-axis direction, so it is possible to detect the amount of movement in the Y-axis direction by using this.

[0114] FIG. 10a shows an image (1010) obtained when a subject without a pattern is captured. FIG. 10b shows an image (1020) obtained when a subject with a pattern is captured. FIG. 11a is a graph showing luminance values ​​obtained per pixel when the image (1010) of FIG. 10a is captured in an electronic device (101) according to one embodiment. FIG. 11b is a graph showing luminance values ​​obtained per pixel when the image (1020) of FIG. 10b is captured in an electronic device (101) according to one embodiment. The horizontal axis of FIG. 11a and FIG. 11b represents the x-axis (e.g., the x-axis of FIG. 4), and the vertical axis may represent normalized illuminance. The normalized illuminance may be understood as the luminance value obtained or output by each pixel and / or each photodiode.

[0115] Referring to FIGS. 11a and 11b, data (1111, 1121) for luminance values ​​obtained through a first photodiode (e.g., the first photodiode (415L) of FIG. 4) of a plurality of pixels (e.g., the first pixel (410) of FIG. 4), data (1113, 1123) for luminance values ​​obtained through a second photodiode (e.g., the second photodiode (415R) of FIG. 4) of the plurality of pixels, and data (1115, 1125) for luminance values ​​obtained through the first photodiode and the second photodiode of the plurality of pixels are shown in the graph.

[0116] In one embodiment, referring to FIG. 11a, the optical characteristics (e.g., shading characteristics) of the lens (320) can be determined. For example, since the lens (320) has curvature, the amount of light reaching the image sensor (230) is greatest closer to the center depending on the curvature, and the amount of light reaching the image sensor (230) may decrease as it moves away from the center. It can be seen that the luminance value obtained by the pixel or photodiode decreases as it moves closer to the edge of the image sensor (230) (e.g., 1 field) from the center of the image sensor (230) (e.g., 0 field). Referring to FIG. 11b, when a patterned subject is photographed, it can be seen that the luminance values ​​obtained by each photodiode are inconsistent, making it difficult to determine the optical characteristics (e.g., shading characteristics of the lens) of the lens (320).

[0117] FIG. 12a is a graph showing the average value of the normalized shading difference (NSD) in the vertical direction when the image (1010) of FIG. 10a is captured in an electronic device (101) according to one embodiment. FIG. 12b is a graph showing the average value of the normalized shading difference (NSD) when the image (1020) of FIG. 10b is captured in an electronic device (101) according to one embodiment. The horizontal axis of FIG. 12a and FIG. 12b represents the x-axis (e.g., the x-axis of FIG. 4), and the vertical axis represents the average value of the NSD as a line in the vertical direction (e.g., the y-axis direction of FIG. 4).

[0118] Referring to FIGS. 12a and 12b, unlike the luminance values ​​in FIGS. 11a and 11b, optical characteristics that are not affected by the subject can be obtained, and here, data regarding the mean of ΔS according to the lens focal position is shown. Referring to FIGS. 12a and 12b, Near represents the NSD characteristics at the lens position corresponding to when the subject is close, Far represents when the subject is far, and Center represents when the subject is located between Near and Far.

[0119] According to one embodiment, data for a focal position close (1211, 1221), data for a focal position in the middle (1213, 1223), and data for a focal position far (1215, 1225) can be identified. The closer the focal position, the greater the variation of the mean of ΔS value corresponding to the x-axis.

[0120] FIG. 13a is a graph showing the amount of OIS movement detected in an electronic device (101) according to one embodiment. The horizontal axis of FIG. 13a may represent the x-axis (e.g., the x-axis of FIG. 4), and the vertical axis may represent the y-axis (e.g., the y-axis of FIG. 4). FIG. 13b is a graph showing data for the initial NSD obtained in an electronic device (101) according to one embodiment. FIG. 13c is a graph showing data for the NSD calculated by correcting the amount of OIS movement in an electronic device (101) according to one embodiment. The horizontal axis of FIG. 13b and FIG. 13c may represent the x-axis (e.g., the x-axis of FIG. 4), and the vertical axis may represent the NSD (normalized shading difference) value. The amount of OIS movement can be understood as the amount of movement of the lens (320) moved by OIS driving. The above OIS movement amount may be referred to or understood as the movement amount of the lens (320) and / or the OIS correction amount.

[0121] Referring to FIG. 13a, the path along which the lens (320) moves by driving the OIS on the x-axis-y-axis plane in response to shaking of the electronic device (101) (e.g., shaking caused by hand tremor) can be seen. FIG. 13b shows a graph of NSD calculated based on multiple shaking, obtained by acquiring multiple sample data through multiple shaking. FIG. 13c shows the NSD calculated by correcting the OIS movement amount detected in FIG. 13a. Compared to the graph in FIG. 13b, the graph in FIG. 13c shows a smaller range of variation and the sample data matches, indicating that detecting the OIS movement amount through multiple photodiode sensors is effective.

[0122] FIG. 14 is a diagram illustrating an operation to correct shaking based on first motion information obtained from a motion sensor (383) and second motion information generated by OIS driving in an electronic device (101) according to one embodiment.

[0123] FIG. 14 illustrates an example explaining an operation to correct an image based on first motion information and second motion information according to various embodiments.

[0124] Referring to FIG. 14, the electronic device (101) may be shaken in the direction of reference number 1410 due to various causes, such as hand tremors of the user. The processor (120) can determine first motion information for the direction of reference number 1410 by detecting the movement of the electronic device (101) through a motion sensor (e.g., a gyroscope sensor) (383).

[0125] According to one embodiment, in response to the electronic device (101) shaking in the direction of reference number 1410, an actuator (e.g., an OIS actuator) (330) can control the lens assembly (210) and / or image sensor (230) to move in the direction of reference number 1420, which is opposite to the direction of reference number 1410. Due to physical limitations, such as the range of angles at which the lens assembly (210) and / or image sensor (230) can move, the length of reference number 1420 may be shorter than the length of reference number 1410.

[0126] According to one embodiment, the processor (120) may obtain an image frame in which OIS correction has been performed from the camera module (180). The image frame in which OIS correction has been performed may have residual shake of reference number 1430. The processor (120) may estimate second motion information for the direction of reference number 1420 from first motion information for the direction of reference number 1410, and determine third motion information in which the second motion information is removed from the first motion information. The third motion information may refer to motion information for reference number 1430. The processor (120) may determine the third motion information based on a formula such as "second motion information = first motion information + second motion information".

[0127] FIG. 15 is a drawing for illustrating an operation of correcting an image frame in line units in an electronic device (101) according to one embodiment.

[0128] Referring to FIG. 15, an image sensor (230) can read out light sequentially from the first line (1520) to the last line (1523) for an image frame (1510). The operation of the electronic device (101) reading out light line by line may be referred to as a rolling shutter operation. Since the timing at which light reflected from the subject enters the image sensor (230) varies for each line, distortion may occur within the image due to the time difference of the light. In this document, the distortion caused by the rolling shutter operation may be referred to as rolling shutter distortion or jello effect.

[0129] In one embodiment, the frequency of motion data acquired through the motion sensor (383) may be higher than the frequency of image frames acquired through the camera module (180) during the same period. For example, the electronic device (101) may acquire an image frame (1510) every 30ms and acquire motion data every 10ms. The electronic device (101) may correct the image frame (1510) on a line-by-line basis using two or more motion data.

[0130] In one embodiment, referring to reference numeral 1530, the electronic device (101) can correct rolling shutter distortion in an image frame (1510) on a line-by-line basis based on two or more motion data (e.g., gyro data). In one embodiment, two or more first motion information can be obtained for each line (e.g., 1520, 1521, 1522, 1523) constituting the image frame (1510). In one embodiment, the electronic device (101) can obtain motion data for at least two of the lines. For example, the electronic device (101) can obtain motion data for a first line (1520) and motion data for a second line (1521) within the image frame (1510). By using the two obtained motion data and interpolation, the electronic device (101) can estimate motion data for the remaining lines for which motion data was not obtained. The electronic device (101) can generate motion data for every line within an image frame (1510) based on acquired motion data and estimated motion data, and correct rolling shutter distortion on a line-by-line basis.

[0131] As described above, an electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1) comprises a lens assembly (e.g., lens assembly (210) of FIG. 2) including a lens, an OIS actuator (e.g., actuator (330) of FIG. 3) for moving the lens assembly in a direction perpendicular to the optical axis, a micro lens (e.g., micro lens (411) of FIG. 4), and an image sensor (e.g., image sensor (230) of FIG. 2) including at least two photodiodes corresponding to the micro lens, wherein the at least two photodiodes include a first photodiode and a second photodiode, and comprises at least one motion sensor (e.g., motion sensor (383) of FIG. 3), a memory (e.g., memory (250) of FIG. 2), and at least one processor operatively connected to the lens assembly, the OIS actuator, the image sensor, the at least one motion sensor, and the memory, wherein the at least one processor At least one image frame is acquired through the image sensor, and a first position of the lens is identified based on a first light intensity acquired through the first photodiode and a second light intensity acquired through the second photodiode, and the amount of movement of the lens moved by the OIS actuator is identified based on the reference position of the lens and the first position of the lens, and digital image stabilization (DIS) can be performed on the at least one image frame based on the amount of movement of the lens.

[0132] According to one embodiment, the at least one processor can calculate a first luminance ratio based on the first light amount and the second light amount, and determine the amount of movement of the lens based on the first luminance ratio and the second luminance ratio stored in advance in the memory.

[0133] According to one embodiment, the at least one processor can calculate a first light quantity difference based on the first light quantity and the second light quantity, and calculate a first normalized shading difference (NSD) value by normalizing the first light quantity difference to luminance.

[0134] According to one embodiment, the amount of movement of the lens moved by the OIS actuator can be determined by comparing the first NSD value with a reference NSD value previously stored in the memory.

[0135] According to one embodiment, the first position of the lens may be a position corresponding to the first NSD value, and the reference position may be a position corresponding to the reference NSD.

[0136] According to one embodiment, the at least one processor can determine a second position of the lens assembly based on a third light intensity obtained through the first photodiode and a fourth light intensity obtained through the second photodiode in response to shaking of the electronic device, determine the amount of movement of the lens moved by the OIS actuator based on the first position and the second position, and perform digital image stabilization (DIS) on the at least one image frame based on the determined amount of movement of the lens.

[0137] According to one embodiment, the at least one processor can calculate a first light quantity difference based on the first light quantity and the second light quantity, calculate a first normalized shading difference (NSD) value normalized by luminance of the first light quantity difference, calculate a second light quantity difference based on the third light quantity and the fourth light quantity, and calculate a second NSD value normalized by luminance of the second light quantity difference.

[0138] According to one embodiment, the first position of the lens corresponds to a first time point when acquiring a first image frame, and the reference position may correspond to a second time point which is a time point prior to the first time point.

[0139] According to one embodiment, the image sensor includes a first photodiode, a second photodiode, and a third photodiode corresponding to the microlens, and the at least one processor calculates a plurality of normalized shading difference (NSD) values ​​based on the output value of the first photodiode, the output value of the second photodiode, and the output value of the third photodiode, and can determine the first position of the lens based on the plurality of NSD values.

[0140] According to one embodiment, the at least one processor can perform digital image stabilization (DIS) on the at least one image frame based on the identified amount of movement of the lens and the movement of the electronic device obtained through the at least one motion sensor.

[0141] As described above, the method of operation of an electronic device according to one embodiment may include the operation of acquiring at least one image frame through an image sensor, the operation of identifying a first position of a lens based on a first light intensity acquired through a first photodiode and a second light intensity acquired through a second photodiode, the operation of identifying the amount of movement of the lens moved by an OIS actuator based on a reference position of the lens and the first position of the lens, and the operation of performing digital image stabilization (DIS) on the at least one image frame based on the amount of movement of the lens.

[0142] According to one embodiment, the method of operating an electronic device may include the operation of calculating a first light quantity difference based on the first light quantity and the second light quantity, and the operation of calculating a first NSD (normalized shading difference) value normalized by luminance of the first light quantity difference.

[0143] According to one embodiment, the method of operating the electronic device may include comparing the first NSD value with a reference NSD value stored in memory to determine the amount of movement of the lens moved by the OIS actuator.

[0144] According to one embodiment, the method of operating an electronic device may include: an operation of determining a second position of the lens based on a third light quantity obtained through the first photodiode and a fourth light quantity obtained through the second photodiode in response to shaking of the electronic device; an operation of determining the amount of movement of the lens moved by the OIS actuator based on the first position and the second position; and an operation of performing digital image stabilization (DIS) on the at least one image frame based on the determined amount of movement of the lens.

[0145] According to one embodiment, the method of operating an electronic device may include the operation of calculating a first light quantity difference based on the first light quantity and the second light quantity, the operation of calculating a first normalized shading difference (NSD) value normalized by luminance of the first light quantity difference, the operation of calculating a second light quantity difference based on the third light quantity and the fourth light quantity, and the operation of calculating a second NSD value normalized by luminance of the second light quantity difference.

[0146] As described above, an electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1) comprises a lens assembly (e.g., lens assembly (210) of FIG. 2) including a lens, an OIS actuator (e.g., actuator (330) of FIG. 3) for moving the lens assembly in a direction perpendicular to the optical axis, a micro lens (e.g., micro lens (411) of FIG. 4), and an image sensor (e.g., image sensor (230) of FIG. 2) including at least two photodiodes corresponding to the micro lens, the at least two photodiodes including a first photodiode and a second photodiode, at least one motion sensor (e.g., motion sensor (383) of FIG. 3), a memory (e.g., memory (250) of FIG. 2), and at least one processor operatively connected to the lens assembly, the OIS actuator, the image sensor, the at least one motion sensor, and the memory, wherein the at least one processor At least one image frame is acquired through the image sensor, data regarding a first difference in light intensity based on a first light intensity acquired through the first photodiode and a second light intensity acquired through the second photodiode is acquired, and data regarding a second difference in light intensity based on a third light intensity acquired through the first photodiode and a fourth light intensity acquired through the second photodiode is acquired in response to shaking of the electronic device, and the amount of movement of the lens moved by the OIS actuator is determined based on the data regarding the first difference in light intensity and the data regarding the second difference in light intensity, and shake correction can be performed on the at least one image frame based on the determined amount of movement of the lens and the movement of the electronic device acquired through the at least one motion sensor.

[0147] According to one embodiment, the at least one processor can calculate a first normalized shading difference (NSD) value by normalizing data for the first light difference into luminance, and calculate a second normalized shading difference (NSD) value by normalizing data for the second light difference into luminance.

[0148] According to one embodiment, the at least one processor can determine a first position of the lens corresponding to the first NSD value, determine a second position of the lens corresponding to the second NSD value, and determine the amount of movement of the lens based on the first position and the second position.

[0149] According to one embodiment, the first position of the lens corresponds to a first time point for acquiring a first image frame, and the second position may correspond to a second time point for acquiring a second image frame acquired after the first image frame.

[0150] According to one embodiment, the at least one processor can transmit the identified amount of movement of the lens and the movement of the electronic device obtained through the at least one motion sensor to a VDIS (video digital image stabilization) module to perform shake correction on the at least one image frame.

[0151] The electronic device according to the various embodiments disclosed in this document may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0152] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any component (e.g., the first) is referred to as "coupled" or "connected" to another component (e.g., the second), with or without the terms "functionally" or "communicationally," it means that said component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0153] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0154] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0155] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0156] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 In an electronic device, a lens assembly including a lens; an OIS actuator for moving the lens assembly in a direction perpendicular to an optical axis; an image sensor including a micro lens and at least two photodiodes corresponding to the micro lens, wherein the at least two photodiodes include a first photodiode and a second photodiode; at least one motion sensor; a memory; An electronic device comprising at least one processor, wherein instructions are executed by said at least one processor, so that the electronic device: acquires at least one image frame through said image sensor, identifies a first position of said lens based on the result of comparing a first light intensity acquired through said first photodiode and a second light intensity acquired through said second photodiode, said first position is a position within an area movable by said OIS actuator, identifies the amount of movement of said lens moved by said OIS actuator based on the reference position of said lens and said first position of said lens, and performs digital image stabilization (DIS) on said at least one image frame based on the amount of movement of said lens. Claim 2 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, so that the electronic device: calculates a first luminance ratio based on the first light amount and the second light amount, and determines the amount of movement of the lens based on the first luminance ratio and the second luminance ratio stored in advance in the memory. Claim 3 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor such that the electronic device: calculates a first light quantity difference based on the first light quantity and the second light quantity, and calculates a first NSD (normalized shading difference) value normalized by luminance of the first light quantity difference. Claim 4 An electronic device according to claim 3, which determines the amount of movement of the lens moved by the OIS actuator by comparing the first NSD value with a reference NSD value previously stored in the memory. Claim 5 An electronic device according to claim 4, wherein the first position of the lens is a position corresponding to the first NSD value, and the reference position is a position corresponding to the reference NSD. Claim 6 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, so that the electronic device: determines a second position of the lens assembly based on a third light quantity obtained through the first photodiode and a fourth light quantity obtained through the second photodiode in response to shaking of the electronic device, determines the amount of movement of the lens moved by the OIS actuator based on the first position and the second position, and performs digital image stabilization (DIS) on the at least one image frame based on the determined amount of movement of the lens. Claim 7 The electronic device of claim 6, wherein the instructions are executed by the at least one processor, the electronic device: calculates a first light intensity difference based on the first light intensity and the second light intensity, calculates a first normalized shading difference (NSD) value normalized by luminance of the first light intensity difference, calculates a second light intensity difference based on the third light intensity and the fourth light intensity, and calculates a second NSD value normalized by luminance of the second light intensity difference. Claim 8 An electronic device according to claim 1, wherein the first position of the lens corresponds to a first time point for acquiring a first image frame, and the reference position corresponds to a second time point prior to the first time point. Claim 9 The electronic device according to claim 1, wherein the image sensor comprises the first photodiode, the second photodiode, and the third photodiode corresponding to the microlens, and the instructions are executed by the at least one processor, so that the electronic device: calculates a plurality of normalized shading difference (NSD) values ​​based on the output value of the first photodiode, the output value of the second photodiode, and the output value of the third photodiode, and identifies the first position of the lens based on the plurality of NSD values. Claim 10 An electronic device according to claim 1, wherein the instructions are executed by the at least one processor, so that the electronic device performs digital image stabilization (DIS) on the at least one image frame based on the identified amount of movement of the lens and the movement of the electronic device obtained through the at least one motion sensor. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images