Electronic device that synchronizes lens driving information and images

Abstract

An electronic device according to one embodiment of the present document may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, a first interface for the at least one processor to provide the first signal to the actuator, and a second interface for the at least one processor to provide the first signal to the image sensor.

Description

Technology Field

[0001] Various embodiments of the present disclosure relate to a technology for synchronizing lens driving information and image data. Background Technology

[0002] With the recent diversification of mobile device functions, there is a growing demand for improvements in image capture capabilities. Accordingly, image processing technologies are advancing to correct electronic device shake during capture and reduce noise contained in images.

[0003] An electronic device can perform VDIS (video digital image stabilization) on multiple image frames. VDIS is a method of reducing video shake through digital processing, and the processor can correct multiple image frames through VDIS.

[0004] An electronic device can perform TNR (temporal noise reduction) on multiple image frames. TNR is a method that removes temporally occurring noise between multiple image frames based on the current image frame and the previous image frame. The problem to be solved

[0005] According to conventional technology, the actuator for driving the lens and the image sensor for acquiring image data are configured independently, making it difficult to synchronize the lens driving information controlling the actuator with the image data. When an electronic device performs image processing without information related to OIS (optical image stabilization) or AF (auto focus) included in the lens driving information, there is a problem that accurate image processing is difficult. means of solving the problem

[0006] An electronic device according to one embodiment of the present document may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, a first interface for the at least one processor to provide the first signal to the actuator, and a second interface for the at least one processor to provide the first signal to the image sensor.

[0007] An electronic device according to one embodiment of the present document may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, a first interface for the at least one processor to provide the first signal to the actuator, and a second interface for the image sensor to obtain lens position information corresponding to the position of the lens from the actuator.

[0008] An electronic device according to one embodiment of the present document may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, and at least one processor connected to the image sensor and the actuator via an interface. The at least one processor may provide a first signal containing lens driving information for controlling the actuator to drive the lens to the actuator through at least a first path of the interface, provide the first signal to the image sensor through at least a second path of the interface, and obtain an image frame including the image data and the lens driving information from the image sensor. Effects of the invention

[0009] According to various embodiments disclosed in this document, VDIS, TNR, chromatic aberration correction, lens distortion correction, etc., can be performed on image data based on lens driving information synchronized with image data. The electronic device can perform precise image processing by utilizing motion data acquired through a motion sensor and lens driving information synchronized with image data together.

[0010] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly understood by those skilled in the art to which the present disclosure belongs from the description below. Brief explanation of the drawing

[0011] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram illustrating a camera module according to various embodiments. FIG. 3a is a block diagram showing the hardware configuration of an electronic device including an interface according to one embodiment. FIG. 3b is a block diagram showing the hardware configuration of an electronic device including an interface according to one embodiment. FIG. 3c is a block diagram showing the hardware configuration of an electronic device including an interface according to one embodiment. FIG. 4 is a block diagram showing the hardware configuration of an electronic device including an interface according to one embodiment. FIG. 5 is a flowchart illustrating the operation of a processor acquiring an image frame containing lens driving information according to one embodiment. FIG. 6 is a flowchart illustrating the operation of an image sensor, a processor, and an actuator according to one embodiment. FIG. 7 illustrates an example of lens driving information according to one embodiment. FIG. 8 illustrates an example of an image frame including lens driving information according to one embodiment. FIG. 9 illustrates an example of an image frame including lens driving information according to one embodiment. Specific details for implementing the invention

[0012] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0013] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0014] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0015] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0016] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0017] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0018] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0019] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0020] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0021] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0022] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0023] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0024] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0025] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0026] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0027] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0028] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0029] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0030] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0031] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0032] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0033] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0034] The electronic device according to the various embodiments disclosed in this document may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0035] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0036] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0037] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0038] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0039] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0040] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.

[0041] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0042] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented, for example, as an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0043] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be composed of at least a part of the processor (120), or It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) either as is or after additional image processing by the processor (120).

[0044] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0045] FIG. 3a is a block diagram showing the hardware configuration of an electronic device (301) including an interface according to one embodiment. FIG. 3b is a block diagram showing the hardware configuration of an electronic device (302) including an interface according to one embodiment. FIG. 3c is a block diagram showing the hardware configuration of an electronic device (303) including an interface according to one embodiment.

[0046] Referring to FIGS. 3a, 3b, and 3c, the electronic device (301, 302, 303) may include a lens (310), an image sensor (320, 321), an actuator (330), and a processor (340). In one embodiment, the electronic device (301, 302, 303) may be understood as the electronic device (101) shown in FIG. 1. In one embodiment, the lens (310) may be understood as the lens assembly (210) shown in FIG. 2. In one embodiment, the processor (340) may be understood as including at least one of the processor (120) shown in FIG. 1 or the image signal processor (260) shown in FIG. 2.

[0047] In one embodiment, the lens (310) may be understood as a lens assembly comprising at least one lens (e.g., the lens assembly (210) of FIG. 2). In one embodiment, the lens (310) may move back and forth along the optical axis to change the focal length or operate so that the subject object can be clearly captured.

[0048] In one embodiment, the lens (310) may be understood as a liquid lens. In one embodiment, the interior of the lens (310) (e.g., liquid lens) may be contained in liquid oil, and the outer edge of the lens (310) may be formed of a thin membrane. In one embodiment, the shape of the lens (310) (e.g., liquid lens) may be deformed. The processor (340) may perform an autofocus (AF) function by controlling the actuator (330) to deform the shape of the lens (310). For example, the actuator (330) may apply an external force to the lens (310) while moving in the direction of the optical axis. The outer edge of the lens (310) may be bent in a direction parallel to the optical axis according to the direction of the external force applied by the actuator (330). In one embodiment, the processor (340) may perform an optical image stabilization (OIS) function by controlling the actuator (330) to deform the shape of the lens (310).

[0049] In one embodiment, the image sensor (320, 321) may be a CMOS (complementary metal oxide semiconductor) sensor or a CCD (charged coupled device) sensor. A plurality of individual pixels are integrated in the image sensor (320, 321), and each individual pixel may include a micro lens, a color filter, and a photodiode. Each individual pixel can convert input light into an electrical signal as a type of light detector. A light detector generally cannot detect the wavelength of the captured light itself and cannot determine color information. The light detector may include a photodiode. For example, the image sensor (320, 321) can amplify the current generated by the photoelectric effect of the light receiving element through the light received through the lens (310). For example, each individual pixel may include a photoelectric transformation element (or position sensitive detector; PSD) and a plurality of transistors (e.g., reset transistor, transfer transistor, select transistor, driver transistor).

[0050] In one embodiment, the image sensor (320, 321) can generate image data from incident light incident through the lens (310). The image data may include various color values ​​obtained through a color filter array. For example, the color filter array may include at least one color filter array among an RGB (red, green, blue) pattern, an RGBE (red, green, blue, emerald) pattern, a CYM (cyan, yellow, magenta) pattern, a CYGM (cyan, yellow, green, magenta) pattern, or an RGBW (red, green, blue, white) pattern.

[0051] In one embodiment, the actuator (330) can drive the lens (310). In one embodiment, the actuator (330) can perform AF and OIS functions under the control of the processor (340). In one embodiment, the processor (340) can perform AF functions by moving the lens (310) in the direction of the optical axis through the actuator (330) so that the subject can be clearly captured. For example, the processor (340) can perform AF functions by controlling the actuator (330) to deform the shape of the lens (310) (e.g., liquid lens). In one embodiment, the processor (340) can perform OIS functions to correct shaking of the electronic devices (301, 302, 303) by moving the lens (310) through the actuator (330). For example, the processor (340) can control the OIS function by controlling the actuator (330) to move the position of the lens (310) in the opposite direction to the direction in which the electronic device (301, 302, 303) moves. As another example, the processor (340) can perform the OIS function by controlling the actuator (330) to deform the shape of the lens (310) (e.g., liquid lens).

[0052] In one embodiment, the processor (340) may include at least one of an OIS control circuit, an application processor (AP), and an image signal processor (e.g., the image signal processor (260) of FIG. 2). In one embodiment, the processor (340) may be referred to as at least one processor.

[0053] In one embodiment, the processor (340) may output a first signal containing lens driving information for controlling the actuator (330) to drive the lens (310). For example, the processor (340) may provide the first signal to the actuator (330). The actuator (330) may move the position of the lens (310) or deform the shape of the lens (310) (e.g., liquid lens) based on the lens driving information included in the first signal. In another example, the processor (340) may provide the first signal to an image sensor (320). The image sensor (320) may generate an image frame containing the lens driving information included in the first signal.

[0054] In one embodiment, the image sensor (320) can generate an image frame based on lens driving information included in a first signal obtained from the processor (340). For example, the image sensor (320) can generate image data from incident light incident through a lens (310) that has been moved (or whose shape has been changed) based on the lens driving information. The image sensor (320) can obtain a first signal from the processor (340). The image sensor (320) can generate an image frame including the image data and the lens driving information included in the first signal. The image data and the lens driving information can be understood as being synchronized with each other. The image sensor (320) can provide the image frame to the processor (340).

[0055] In one embodiment, the processor (340) may acquire an image frame containing lens driving information from an image sensor (320). In one embodiment, the processor (340) may perform image processing on the image data based on the lens driving information included in the image frame. For example, the processor (340) may perform at least one of video digital image stabilization (VDIS), temporal noise reduction (TNR), chromatic aberration correction, or lens distortion correction on the image data based on the lens driving information.

[0056] Referring to FIG. 3a, the electronic device (301) may include a first interface (351a) connecting a processor (340) and an actuator (330), and a second interface (352a) connecting a processor (340) and an image sensor (320). As illustrated, the first interface (351a) and the second interface (352a) may include a certain common section.

[0057] In one embodiment, the processor (340) may provide the first signal to the actuator (330) through the first interface (351a). The actuator (330) may drive the lens (310) based on lens driving information included in the first signal obtained through the first interface (351a). In one embodiment, the processor (340) may provide the first signal to the image sensor (320) through the second interface (352a). The image sensor (320) may generate an image frame based on lens driving information included in the first signal obtained through the second interface (352a).

[0058] Referring to FIG. 3b, the electronic device (302) may include a first interface (351b) and a second interface (352b). In one embodiment, the description of the first interface (351a) and the second interface (352a) described in relation to FIG. 3a may also apply to the first interface (351b) and the second interface (352b) shown in FIG. 3b, except for the description related to the arrangement.

[0059] Referring to FIG. 3b, the first interface (351b) and the second interface (352b) may be implemented as separate interfaces without including a common section.

[0060] Referring to FIG. 3c, the electronic device (303) may include a first interface (351c) and a second interface (352c). In one embodiment, unlike FIG. 3a and FIG. 3b, the second interface (352c) of FIG. 3c may be an interface connecting an actuator (330) and an image sensor (321). In one embodiment, a processor (340) may provide a first signal to the actuator (330) through the first interface (351c). In one embodiment, the image sensor (321) may obtain lens position information from the actuator (330) through the second interface (352c). For example, the actuator (330) may include a Hall sensor (not shown) capable of detecting the position of the lens (310). The image sensor (321) may request lens position information from the actuator (330) through the second interface (352c). In response to the request, the image sensor (321) may receive lens position information obtained by the Hall sensor from the actuator (330) through the second interface (352c). In one embodiment, the image sensor (321) may generate an image frame based on the lens position information obtained from the actuator (330). For example, the image data included in the image frame and the lens position information may be understood as being synchronized with each other.

[0061] Referring to FIGS. 3a, 3b, and 3c, the electronic device (301, 302, 303) may further include a third interface (353) connecting an image sensor (320, 321) and a processor (340). In one embodiment, the image sensor (320, 321) may provide an image frame to the processor (340) through the third interface (353). For example, the image sensor (320) may generate an image frame including image data and lens driving information obtained from the processor (340) and provide the image frame to the processor (340) through the third interface (353). In another example, the image sensor (321) may generate an image frame including image data and lens position information obtained from an actuator (330) and provide the image frame to the processor (340) through the third interface (353). In one embodiment, the processor (340) that receives an image frame from an image sensor (320, 321) through a third interface (353) may be an image signal processor.

[0062] In one embodiment, the first interface (351a, 351b, 351c) and the second interface (352a, 352b, 352c) may be at least one of I2C (inter-integrated circuit), I3C (improved inter-integrated circuit), or SPI (serial peripheral interface). For example, at least one of the first interface (351a, 351b, 351c) or the second interface (352a, 352b, 352c) may be an interface that supports bidirectional communication. As another example, at least one of the first interface (351a, 351b, 351c) or the second interface (352a, 352b, 352c) may be an interface that supports unidirectional communication.

[0063] In one embodiment, the third interface (353) may be a MIPI (mobile industry processor interface). For example, the third interface (353) may be an interface that supports unidirectional communication from the image sensor (320, 321) to the processor (340). As another example, the third interface (353) may be an interface that supports bidirectional communication between the image sensor (320, 321) and the processor (340).

[0064] FIG. 4 is a block diagram showing the hardware configuration of an electronic device (400) including an interface according to one embodiment.

[0065] Referring to FIG. 4, the electronic device (400) may include a lens (410), an image sensor (420), an AF module (432), an OIS module (434), an OIS control circuit (442), an AP (444), and a motion sensor (490). Among the configurations shown in FIG. 4, the configurations described in FIG. 3a through 3c may be briefly described or omitted.

[0066] In one embodiment, the electronic device (400) may include a motion sensor (490). A processor (e.g., OIS control circuit (442), AP (444)) may detect the movement of the electronic device (400) through the motion sensor (490). In one embodiment, the motion sensor (490) may provide motion data corresponding to the movement of the electronic device (400) to the OIS control circuit (442). For example, the motion sensor (490) may provide motion data directly to the OIS control circuit (442). In another embodiment, the AP (444) may obtain information about the movement of the electronic device (400) detected by the motion sensor (490) and provide this information to the OIS control circuit (442).

[0067] In one embodiment, the motion sensor (490) may include at least one of an accelerometer, a gyroscope, a magnetic sensor, or a Hall sensor. For example, the accelerometer may measure acceleration acting on three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (400). However, the above sensors are exemplary, and the motion sensor (490) may further include at least one other type of sensor.

[0068] In one embodiment, the electronic device (400) may include an AF module (432) and an OIS module (434). In one embodiment, the AF module (432) may perform an AF function by driving the lens (410). For example, the AF module (432) may perform an AF function by deforming the shape of the lens (410) (e.g., liquid lens). In another example, the AF module (432) may perform an AF function by moving the position of the lens (410). In one embodiment, the OIS module (434) may perform an OIS function by driving the lens (410). For example, the OIS module (434) may perform an OIS function by moving the position of the lens (410). In another example, the OIS module (434) may perform an OIS function by deforming the shape of the lens (410) (e.g., liquid lens).

[0069] In one embodiment, the electronic device (400) may include an OIS control circuit (442) and an AP (444). In one embodiment, the AP (444) may be understood to include an image signal processor (e.g., the image signal processor (260) of FIG. 2). In one embodiment, the OIS control circuit (442) and the AP (444) may be arranged in separate configurations as shown in FIG. 4, and in another embodiment, the OIS control circuit (442) may be placed inside the AP (444). For example, the electronic device (400) may include a system on chip (SoC) in which processing units such as the AP (444), the OIS control circuit (442), and the image signal processor are integrated.

[0070] In one embodiment, the OIS control circuit (442) can perform the OIS function by controlling the OIS module (434) to move the position of the lens (410). The OIS control circuit (442) can also perform the OIS function by controlling the OIS module (434) to deform the shape of the lens (410) (e.g., liquid lens). In one embodiment, the OIS control circuit (442) can acquire motion data corresponding to the movement of the electronic device (400) from the motion sensor (490). The OIS control circuit (442) can control the OIS module (434) to move the position of the lens (410) based on the motion data.

[0071] In one embodiment, the OIS control circuit (442) may provide a first-1 signal, which includes lens driving information for moving the position of the lens (410), to the OIS module (434) and the image sensor (420). For example, the lens driving information may be OIS position information for the OIS control circuit (442) to move the position of the lens (410). In one embodiment, the OIS control circuit (442) may provide the first-1 signal to the OIS module (434) through a first-1 interface (451-1). In one embodiment, the OIS control circuit (442) may provide the first-1 signal to the image sensor (420) through a second-1 interface (452-1). In one embodiment, the first-1 interface (451-1) and the second-1 interface (452-1) may have a common section. In one embodiment, the first-1 interface (451-1) and the second-1 interface (452-1) may be I2C, I3C, or SPI.

[0072] In one embodiment, the AP (444) can perform an AF function by controlling the AF module (432) to drive the lens (410). In one embodiment, the AP (444) can provide a first-2 signal containing lens driving information for driving the lens (410) to the AF module (432) and the image sensor (420). For example, the lens driving information may be AF position information for the AP (444) to change the shape of the lens (410) (e.g., liquid lens). In one embodiment, the AP (444) can provide the first-2 signal to the AF module (432) through a first-2 interface (451-2). In one embodiment, the AP (444) can provide the first-2 signal to the image sensor (420) through a second-2 interface (452-2). In one embodiment, the first-2 interface (451-2) and the second-2 interface (452-2) may have a common section. In one embodiment, the first-2 interface (451-2) and the second-2 interface (452-2) may be I2C, I3C, or SPI.

[0073] In one embodiment, the electronic device (400) may include a third interface (453). In one embodiment, the image sensor (420) may generate an image frame including lens driving information included in a first-1 signal obtained from an OIS control circuit (442) and lens driving information included in a first-2 signal obtained from an AP (444). The image sensor (420) may provide the image frame to the AP (444) through the third interface (453). For example, the image sensor (420) may provide the image frame to an image signal processor included in the AP (444). In one embodiment, the third interface (453) may be MIPI.

[0074] In one embodiment, the electronic device (400) may include a fourth interface (454). In one embodiment, the AP (444) may generate a second signal for controlling the image sensor (420). For example, the second signal may include at least some information regarding the exposure time of the image sensor (420) or the time for reading out image data. In one embodiment, the AP (444) may provide the second signal to the image sensor (420) through the fourth interface (454). In one embodiment, the fourth interface (454) may be I2C, I3C, or SPI.

[0075] Referring to FIGS. 3a to 3c and FIG. 4, the lens (410) of FIG. 4 may correspond to the lens (310) of FIG. 3a to 3c, and the image sensor (420) of FIG. 4 may correspond to the image sensor (320) of FIG. 3a to 3b. In one embodiment, the AF module (432) and OIS module (434) of FIG. 4 may be included in the actuator (330) of FIG. 3a to 3c. In one embodiment, the OIS control circuit (442) and AP (444) of FIG. 4 may be included in the processor (340) of FIG. 3a to 3c. In one embodiment, the first-1 signal or the first-2 signal described in relation to FIG. 4 may correspond to the first signal described in relation to FIG. 3a to 3c. In one embodiment, the first-1 interface (451-1) or the first-2 interface (451-2) may correspond to the first interface (351a) of FIG. 3a. In one embodiment, the second-1 interface (452-1) or the second-2 interface (452-2) may correspond to the second interface (352a) of FIG. 3a. In one embodiment, the third interface (453) of FIG. 4 may correspond to the third interface (353) of FIG. 3a through FIG. 3c.

[0076] FIG. 5 is a flowchart illustrating the operation of a processor (340) acquiring an image frame containing lens driving information according to one embodiment. The operation described in FIG. 5 can be performed by the processor (340) illustrated in FIG. 3a to 3c. In one embodiment, the processor (340) can be understood to include the OIS control circuit (442) and AP (444) illustrated in FIG. 4.

[0077] According to one embodiment, in operation 501, the processor (340) may provide the actuator (330) with a first signal containing lens driving information for controlling the actuator (330) to drive the lens (310).

[0078] In one embodiment, the processor (340) can provide the first signal to the actuator (330) through at least a first path.

[0079] In one embodiment, the processor (340) may provide the first signal to the actuator (330) through the first interface (351a) of FIG. 3a, the first interface (351b) of FIG. 3b, or the first interface (351c) of FIG. 3c. In one embodiment, the section of the first interface (351a) of FIG. 3a excluding the common section with the second interface (352a) may be referenced as the first path. In another embodiment, the first interface (351b) of FIG. 3b and the first interface (351c) of FIG. 3c may be referenced as the first path.

[0080] In one embodiment, the OIS control circuit (442) can provide a first-1 signal to the OIS module (434) through the first-1 interface (451-1) of FIG. 4. In one embodiment, the AP (444) can provide a first-2 signal to the AF module (432) through the first-2 interface (451-2) of FIG. 4. In one embodiment, the section of the first-1 interface (451-1) of FIG. 4 excluding the common section with the second-1 interface (452-1) can be referenced as a first path. In one embodiment, the section of the first-2 interface (451-2) of FIG. 4 excluding the common section with the second-2 interface (452-2) can be referenced as a first path.

[0081] According to one embodiment, in operation 503, the processor (340) can provide the first signal to the image sensor (320).

[0082] In one embodiment, the processor (340) can provide the first signal to the image sensor (320) through at least a second path.

[0083] In one embodiment, the processor (340) may provide the first signal to the actuator (330) through the second interface (352a) of FIG. 3a or the second interface (352b) of FIG. 3b. In one embodiment, the portion of the second interface (352a) of FIG. 3a excluding the portion common with the first interface (351a) may be referenced as the second path. In another embodiment, the second interface (352b) of FIG. 3b may be referenced as the second path.

[0084] In one embodiment, the image sensor (321) may receive lens position information corresponding to lens driving information included in the first signal through the second interface (352c) of FIG. 3c. The second interface (352c) of FIG. 3c may be referenced as a second path.

[0085] In one embodiment, the OIS control circuit (442) can provide a first-1 signal to the OIS module (434) through the second-1 interface (452-1) of FIG. 4. In one embodiment, the AP (444) can provide a first-2 signal to the AF module (432) through the second-2 interface (452-2) of FIG. 4. In one embodiment, the section of the second-1 interface (452-1) of FIG. 4 excluding the common section with the first-1 interface (451-1) can be referenced as a second path. In one embodiment, the section of the second-2 interface (452-2) of FIG. 4 excluding the common section with the first-2 interface (451-2) can be referenced as a second path.

[0086] According to one embodiment, in operation 505, the processor (340) may acquire an image frame including image data and lens driving information from an image sensor (320, 321). According to one embodiment, the image data may be understood as image data acquired by the image sensor (320, 321) to correspond to the first signal (or lens driving information). For example, the image data and lens driving information included in the image frame acquired by the processor (340) in operation 505 may be understood as being synchronized with each other.

[0087] In one embodiment, the processor (340) (or AP (444)) can acquire an image frame from an image sensor (320, 321) through a third interface (353).

[0088] FIG. 6 is a flowchart illustrating the operation of an image sensor (320), a processor (340), and an actuator (330) according to one embodiment. The operations described in FIG. 6 can be performed by the image sensor (320), the processor (340), and the actuator (330) illustrated in FIG. 3a and FIG. 3b.

[0089] According to one embodiment, in operation 601, the image sensor (320) can generate image data from incident light.

[0090] According to one embodiment, in operation 603, the processor (340) may provide a first signal to the actuator (330) and provide a first signal to the image sensor (320). Operation 603 may correspond to operations 501 and 503 described in FIG. 5.

[0091] According to one embodiment, in operation 605, the actuator (330) can drive the lens (310) based on the first signal. For example, the actuator (330) can move the position of the lens (310) or deform the shape of the lens (310) based on the lens driving information included in the first signal. With reference to FIG. 6, operations 603 and 605 are shown as being performed after operation 601, but this is for convenience of explanation and does not determine the order of operations. For example, image data generated in operation 601 may be obtained at the position of the lens (310) driven by the actuator (330) in operation 605.

[0092] According to one embodiment, in operation 607, the image sensor (320) can obtain lens driving information from the processor (340). The image sensor (320) can obtain lens driving information included in the first signal received from the processor (340).

[0093] According to one embodiment, in operation 609, the image sensor (320) can generate an image frame including image data and lens driving information.

[0094] According to one embodiment, in operation 611, the image sensor (320) can provide the image frame to the processor (340).

[0095] According to one embodiment, in operation 613, the processor (340) can acquire an image frame from the image sensor (320).

[0096] According to one embodiment, in operation 615, the processor (340) can perform at least one of VDIS, TNR, chromatic aberration correction (CAC), or lens distortion correction (LDC) on the image data based on lens driving information included in the image frame.

[0097] In one embodiment, the processor (340) can perform VDIS or EIS (electric image stabilization) on image data based on lens driving information. In one embodiment, the processor (340) can acquire motion data corresponding to the movement of the electronic device (400) through a motion sensor (e.g., the motion sensor (490) of FIG. 4). The processor (340) can perform an OIS function by controlling the actuator (330) to move the lens (310) based on the motion data. To perform VDIS on image frames in which OIS has been performed, the processor (340) can use a motion compensation value corresponding to the value of the motion data compensated through the OIS function together with the motion data. Since the lens driving information included in the image frame includes information corresponding to the motion compensation value, the processor (340) can perform VDIS on image data based on the lens driving information.

[0098] In one embodiment, the processor (340) can perform TNR on a plurality of image frames acquired from the image sensor (320). For example, the processor (340) can reduce noise contained in the plurality of image frames by applying a filter (e.g., a Gaussian filter, an average filter) to the plurality of image frames. In one embodiment, the processor (340) can obtain a motion compensation value corresponding to a value that compensates for motion data through an OIS function using lens driving information included in the image frames. The processor (340) can perform TNR on the plurality of image frames based on the motion data and the motion compensation value corresponding to the lens driving information.

[0099] In one embodiment, the position where an image is formed may vary depending on the color (e.g., R, G, B) of the incident light passing through the lens (310) (e.g., liquid lens), that is, depending on the wavelength of the incident light. For example, while the processor (340) is performing an AF function or an OIS function by deforming the shape of the lens (310), the angle of incidence of the incident light passing through the lens (310) may change. As the shape of the lens (310) is deformed, an image may be formed at a different position depending on the color of the incident light, and chromatic aberration may appear. In one embodiment, the image frame received by the processor (340) may include information regarding the deformed shape (e.g., angle) of the lens (310). The processor (340) may correct the chromatic aberration of the image data using lens driving information. The processor (340) may perform chromatic aberration correction on the image data using lens driving information synchronized with the image data.

[0100] In one embodiment, the processor (340) can perform lens distortion correction on image data acquired from the image sensor (320). If the lens (310) is an optically zoomable lens, the degree of barrel distortion of the lens (310) may vary depending on the focal length. In one embodiment, the processor (340) acquires lens driving information synchronized with the image data, so it can correct the barrel distortion contained in the image data based on the lens driving information.

[0101] FIG. 7 illustrates an example of lens driving information according to one embodiment.

[0102] In one embodiment, the image sensor (320) may provide an image frame containing lens driving information along with image data to the processor (340). For example, the image sensor (320) may provide N-frame information (710) along with N-frame image data to the processor (340), and the N-frame information (710) may include lens driving information.

[0103] Referring to FIG. 7, the shutter (790) indicates the point at which the exposure of the image sensor (320) begins, and the readout (795) indicates the point at which image data is read out from the image sensor (320). For example, in FIG. 7, the Nth exposure (719) may correspond to the interval between the Nth shutter (790) and the Nth readout (795). The Nth exposure (719), the N+1st exposure (729), and the N+2nd exposure (739) may correspond to different exposure times. The exposure times may differ depending on the time interval between the shutter (790) and the readout (795).

[0104] Referring to FIG. 7, N-frame information (710), N+1-frame information (720), and N+2-frame information (730) may represent lens driving information included in the Nth image frame, the N+1-th image frame, and the N+2-th image frame, respectively. In the description related to FIG. 7, N, N+1, and N+2 represent the order of image frames according to the flow of time and do not represent information of the next frame (e.g., NFI (next frame information)).

[0105] In one embodiment, the lens driving information may include at least one of AF position information (701) for the processor (340) to perform an AF function through the actuator (330) or OIS position information (702) for performing an OIS function. For example, the image sensor (320) may include information for controlling an AF module (e.g., AF module (432) of FIG. 4) among the lens driving information included in the first signal obtained from the processor (340) as AF position information (701) in the image frame. In another example, the image sensor (320) may include information for controlling an OIS module (e.g., OIS module (434) of FIG. 4) among the lens driving information included in the first signal obtained from the processor (340) as OIS position information (702) in the image frame.

[0106] In one embodiment, the N-frame information (710) may include first lens drive information (711) obtained from the processor (340) before the Nth exposure (719) of the image sensor (320), and second lens drive information (712) obtained from the processor (340) during the Nth exposure (719) of the image sensor (320). In one embodiment, the first lens drive information (711) and the second lens drive information (712) may each include AF position information (701) and OIS position information (702). For example, the first lens drive information (711) of the N-frame information (710) may include 'AF Pos X, OIS Pos X'. The 'AF Pos X, OIS Pos X' may refer to lens drive information obtained from the processor (340) before the Nth shutter (790), which is a point in time prior to the time shown in FIG. 7. As another example, since the image sensor (320) has acquired AF position information (701) named 'AF Pos1' and OIS position information (702) named 'OIS Pos1' after the Nth shutter (790) and before the Nth readout (795), the second lens drive information (712) of the N frame information (710) may include 'AF Pos1, OIS Pos1'.

[0107] In one embodiment, the N+1 frame information (720) may include first lens driving information (721) obtained from the processor (340) before the N+1th exposure (729) of the image sensor (320), and second lens driving information (722) obtained from the processor (340) during the N+1th exposure (729) of the image sensor (320). For example, the first lens driving information (721) may include lens driving information obtained before the N+1th shutter (790) corresponding to the N+1th exposure (729), and the second lens driving information (722) may include lens driving information obtained after the N+1th shutter (790) and before the N+1th readout (795).

[0108] In one embodiment, since the image sensor (320) has acquired AF position information named 'AF Pos2' from the processor (340) prior to the N+1th shutter (790), the first lens drive information (721) of the N+1 frame information (720) may include 'AF Pos2, OIS Pos1'. If the image sensor (320) has not acquired new OIS position information, it may generate an image frame containing previous OIS position information. The processor (340) may perform image processing on the image data included in the image frame based on the OIS position information included in each image frame. For example, the processor (340) may perform filter processing (e.g., interpolation) on the N+1th image data based on the movement distance and / or movement direction of the lens (310) corresponding to the OIS position information included in the N+1 frame information (720). Accordingly, the image sensor (320) can generate an image frame containing previously acquired OIS position information even when it has not acquired new OIS position information from the processor (340).

[0109] In one embodiment, since the image sensor (320) has acquired OIS position information 'OIS Pos2' after the N+1th shutter (790) and before the N+1th readout (795), the second lens drive information (722) of the N+1 frame information (720) may include 'AF None, OIS Pos2'. If the image sensor (320) has not acquired new AF information during exposure, it may generate an image frame containing 'AF None', unlike the OIS position information. Since the electronic device (301, 302, 303) minimizes AF driving during the exposure of the image sensor (320), the image sensor (320) may generate an image frame that does not include AF position information if there is no AF position information acquired after the shutter (790) and before the readout (795).

[0110] Referring to FIG. 7, the number of each AF position information or OIS position information shown in FIG. 7 (e.g., AF Pos1 to AF Pos4, OIS Pos1 to OIS Pos4) represents the number of lens drive information acquired by the image sensor (320), and does not describe whether the lens drive information is included in which image frame. In one embodiment, the processor (340) may or may not provide the lens drive information to the image sensor (320) regardless of the number of image frames output by the image sensor (320).

[0111] In one embodiment, the lens driving information may be lens driving information included in the first signal obtained by the image sensor (320) from the processor (340) in FIG. 3a and 3b, but may be replaced with lens position information obtained by the image sensor (321) from the actuator (330) in FIG. 3c.

[0112] FIG. 8 illustrates an example of an image frame (800) including lens driving information (811) according to one embodiment.

[0113] In one embodiment, the image frame (800) provided by the image sensor (320) to the processor (340) may include an embedded header (810) and image data (820). In one embodiment, lens driving information (811) may be included in the embedded header (810) of the image frame (800). In another embodiment, lens driving information (811) may be included in the embedded footer (not shown) of the image frame (800).

[0114] In one embodiment, the lens driving information (811) may include the first lens driving information (710) and the second lens driving information (720) described in FIG. 7. In one embodiment, the lens driving information (811) may include the AF position information (701) and the OIS position information (702) described in FIG. 7.

[0115] In one embodiment, the processor (340) can perform corrections (e.g., VDIS, OIS, chromatic aberration correction, lens distortion correction) on image data (820) using first lens driving information (710) obtained before the exposure of the image sensor (320) and second lens driving information (720) obtained after the exposure of the image sensor and before the readout of image data. For example, the processor (340) can obtain lens driving information that changed during exposure using lens driving information (811) obtained before and after the exposure of the image sensor (320). The processor (340) can perform corrections on image data (820) based on the position change (or shape change) of the lens (310) driven by the actuator (330) before, during, and after the exposure of the image sensor (320).

[0116] FIG. 9 illustrates an example of an image frame (900) including lens driving information (920) according to one embodiment.

[0117] In one embodiment, the image frame (900) provided by the image sensor (320) to the processor (340) may include image data (910) corresponding to a plurality of lines. For example, the image sensor (320) may read out pixel data in units of at least one line and may provide data in units of at least one line where the readout is performed. In one embodiment, the image sensor (320) may include lens driving information (920) in the packet header or packet footer area of ​​each of the plurality of lines of the image data (910). For example, the image sensor (320) may include lens driving information (920) in the packet footer of each of the plurality of lines and provide it to the processor (340).

[0118] In one embodiment, the image sensor (320) can acquire lens driving information before and after exposure of the image sensor (320) as described in relation to FIGS. 7 and 8, as well as acquire a larger amount of lens driving information (920) while acquiring image data. For example, the image sensor (320) can transmit lens driving information of 1 kHz or more to the processor (340). In one embodiment, the image sensor (320) can include lens driving information (920) in the packet header or packet footer of each of the multiple lines of image data (910).

[0119] In one embodiment, when the processor (340) provides a first signal containing lens driving information to the actuator (330), there may be a certain delay until the actuator (330) drives the lens (310) based on the lens driving information. Accordingly, the image sensor (320) can match the n-th lens driving information with the n+m-th image data. For example, m can be understood as corresponding to the delay required for the operation of the actuator (330).

[0120] In one embodiment, the lens driving information may be lens driving information included in the first signal obtained by the image sensor (320) from the processor (340) in FIG. 3a and 3b, but may be replaced with lens position information obtained by the image sensor (321) from the actuator (330) in FIG. 3c. In one embodiment, when the lens driving information is replaced with lens position information, m may be changed to a different value.

[0121] An electronic device according to one embodiment may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, a first interface for the at least one processor to provide the first signal to the actuator, and a second interface for the at least one processor to provide the first signal to the image sensor.

[0122] In an electronic device according to one embodiment, the actuator may include an AF (auto focus) module and an OIS (optical image stabilization) module.

[0123] In an electronic device according to one embodiment, the lens is a liquid lens, and the at least one processor can perform an AF function by controlling the AF module to deform the shape of the liquid lens, and perform an OIS function by controlling the OIS module to deform the shape of the liquid lens.

[0124] An electronic device according to one embodiment may include a motion sensor capable of detecting the movement of the electronic device. The at least one processor may acquire motion data corresponding to the movement of the electronic device from the motion sensor and control the OIS module to move the position of the lens based on the motion data to perform an OIS function.

[0125] In an electronic device according to one embodiment, the first interface and the second interface may have a common section.

[0126] In an electronic device according to one embodiment, the first interface and the second interface may be at least one of I2C (inter-integrated circuit), I3C (improved inter-integrated circuit), or SPI (serial peripheral interface).

[0127] In an electronic device according to one embodiment, the image sensor can generate image data from the incident light, acquire the first signal from the at least one processor through the second interface, and generate an image frame including the image data and the lens driving information.

[0128] An electronic device according to one embodiment may further include a third interface for the image sensor to provide the image frame to the at least one processor.

[0129] In an electronic device according to one embodiment, the third interface may be a mobile industry processor interface (MIPI).

[0130] In an electronic device according to one embodiment, the lens driving information includes first lens driving information obtained from the at least one processor before the image sensor is exposed, and second lens driving information obtained from the at least one processor after the image sensor is exposed and before the image data is read out, and the lens driving information may be included in an embedded header or an embedded footer of the image frame.

[0131] In an electronic device according to one embodiment, the image frame includes image data corresponding to a plurality of lines, and the lens driving information may be included in the packet header or packet footer of each of the plurality of lines.

[0132] In an electronic device according to one embodiment, the actuator includes an AF module and an OIS module, and the at least one processor includes an application processor (AP) and an OIS control circuit, the AP can control the AF module to perform an AF function, and the OIS control circuit can control the OIS module to perform an OIS function.

[0133] In an electronic device according to one embodiment, the AP can generate a second signal for controlling the image sensor and provide the second signal to the image sensor through a fourth interface.

[0134] An electronic device according to one embodiment may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, a first interface for the at least one processor to provide the first signal to the actuator, and a second interface for the image sensor to obtain lens position information corresponding to the position of the lens from the actuator.

[0135] In an electronic device according to one embodiment, the actuator includes a Hall sensor capable of detecting the position of the lens, and the image sensor requests the lens position information from the actuator through the second interface and receives the lens position information obtained by the Hall sensor from the actuator through the second interface in response to the request.

[0136] An electronic device according to one embodiment may include a lens, an image sensor that generates image data from incident light incident through the lens, an actuator capable of driving the lens, and at least one processor connected to the image sensor and the actuator via an interface. The at least one processor may provide a first signal containing lens driving information for controlling the actuator to drive the lens to the actuator through at least a first path of the interface, provide the first signal to the image sensor through at least a second path of the interface, and obtain an image frame including the image data and the lens driving information from the image sensor.

[0137] In an electronic device according to one embodiment, the at least one processor can perform at least one of video digital image stabilization (VDIS), temporal noise reduction (TNR), chromatic aberration correction, or lens distortion correction on the image data based on the lens driving information.

[0138] In an electronic device according to one embodiment, the lens driving information may include at least one of AF position information for the at least one processor to perform an AF function through the actuator, or OIS position information for the at least one processor to perform an OIS function through the actuator.

[0139] In an electronic device according to one embodiment, the lens driving information includes first lens driving information obtained from the at least one processor before the image sensor is exposed, and second lens driving information obtained from the at least one processor after the image sensor is exposed and before the image data is read out, and the lens driving information may be included in an embedded header or an embedded footer of the image frame.

[0140] In an electronic device according to one embodiment, the image frame includes image data corresponding to a plurality of lines, and the lens driving information may be included in the packet header or packet footer area of ​​each of the plurality of lines.

Claims

Claim 1 An electronic device comprising: a lens; an image sensor that generates image data from incident light incident through the lens; an actuator capable of driving the lens; at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens, and receives an image frame including the lens driving information along with the image data from the image sensor; a first interface for the at least one processor to provide the first signal to the actuator; and a second interface for the at least one processor to provide the first signal to the image sensor. Claim 2 An electronic device according to claim 1, wherein the actuator comprises an AF (auto focus) module and an OIS (optical image stabilization) module. Claim 3 An electronic device according to claim 2, wherein the lens is a liquid lens, and the at least one processor: controls the AF module to deform the shape of the liquid lens to perform an AF function, and controls the OIS module to deform the shape of the liquid lens to perform an OIS function. Claim 4 An electronic device according to claim 2, comprising a motion sensor capable of detecting movement of the electronic device, wherein the at least one processor: acquires motion data corresponding to the movement of the electronic device from the motion sensor, and controls the OIS module to move the position of the lens based on the motion data to perform an OIS function. Claim 5 An electronic device according to claim 1, wherein the first interface and the second interface have a common section. Claim 6 An electronic device according to claim 1, wherein the first interface and the second interface are at least one of I2C (inter-integrated circuit), I3C (improved inter-integrated circuit), or SPI (serial peripheral interface). Claim 7 An electronic device according to claim 1, wherein the image sensor: generates image data from the incident light, obtains the first signal from the at least one processor through the second interface, and generates an image frame including the image data and the lens driving information. Claim 8 An electronic device according to claim 7, further comprising a third interface for the image sensor to provide the image frame to the at least one processor. Claim 9 An electronic device according to claim 8, wherein the third interface is a MIPI (mobile industry processor interface). Claim 10 An electronic device according to claim 7, wherein the lens driving information comprises a first lens driving information obtained by the image sensor from the at least one processor prior to the exposure of the image sensor, and a second lens driving information obtained from the at least one processor prior to the readout of the image data after the exposure of the image sensor, and wherein the lens driving information is included in an embedded header or embedded footer of the image frame. Claim 11 An electronic device according to claim 7, wherein the image frame includes image data corresponding to a plurality of lines, and the lens driving information is included in the packet header or packet footer of each of the plurality of lines. Claim 12 An electronic device according to claim 1, wherein the actuator includes an AF module and an OIS module, and the at least one processor includes an application processor (AP) and an OIS control circuit, wherein the AP controls the AF module to perform an AF function, and the OIS control circuit controls the OIS module to perform an OIS function. Claim 13 An electronic device according to claim 12, wherein the AP: generates a second signal for controlling the image sensor and provides the second signal to the image sensor through a fourth interface. Claim 14 An electronic device comprising: a lens; an image sensor that generates image data from incident light incident through the lens; an actuator capable of driving the lens; at least one processor that outputs a first signal including lens driving information for controlling the actuator to drive the lens; a first interface for the at least one processor to provide the first signal to the actuator; and a second interface for the image sensor to obtain lens position information corresponding to the position of the lens from the actuator, wherein the at least one processor is configured to obtain an image frame including the image data and the lens driving information from the image sensor. Claim 15 An electronic device according to claim 14, wherein the actuator includes a Hall sensor capable of detecting the position of the lens, and the image sensor requests the lens position information from the actuator through the second interface, and receives the lens position information obtained by the Hall sensor from the actuator through the second interface in response to the request. Claim 16 An electronic device comprising: a lens; an image sensor that generates image data from incident light incident through the lens; an actuator capable of driving the lens; and at least one processor connected to the image sensor and the actuator via an interface, wherein the at least one processor: provides a first signal including lens driving information for controlling the actuator to drive the lens to the actuator through at least a first path of the interface, provides the first signal to the image sensor through at least a second path of the interface, and obtains an image frame including the image data and the lens driving information from the image sensor. Claim 17 An electronic device according to claim 16, wherein the at least one processor performs at least one of video digital image stabilization (VDIS), temporal noise reduction (TNR), chromatic aberration correction, or lens distortion correction on the image data based on the lens driving information. Claim 18 An electronic device according to claim 16, wherein the lens driving information comprises at least one of AF position information for the at least one processor to perform an AF function through the actuator, or OIS position information for the at least one processor to perform an OIS function through the actuator. Claim 19 An electronic device according to claim 16, wherein the lens driving information comprises a first lens driving information obtained by the image sensor from the at least one processor prior to the exposure of the image sensor, and a second lens driving information obtained from the at least one processor prior to the readout of the image data after the exposure of the image sensor, and wherein the lens driving information is included in an embedded header or embedded footer of the image frame. Claim 20 An electronic device according to claim 16, wherein the image frame includes image data corresponding to a plurality of lines, and the lens driving information is included in the packet header or packet footer area of ​​each of the plurality of lines.

Images