Method for providing image, and electronic device supporting same

Abstract

An electronic device (301) according to an embodiment comprises: an inertial sensor (310); a camera (320) including an image sensor; a processor (360); and a controller (330). The controller (330) is configured to, while a camera application is running, acquire the magnitude and direction of movements of the electronic device (301) caused by hand shaking on the basis of sensing data acquired through the inertial sensor (310). The controller (330) is configured to determine, on the basis of the magnitude and direction of the movements of the electronic device (301), a first area for acquiring an image within the entire area of the image sensor. The controller (330) is configured to acquire pixel values of pixels included in the first area. The controller (330) is configured to provide the acquired pixel values to the processor.

Description

Method for providing images and electronic devices supporting the same

[0001] The present disclosure relates to a method for providing an image and an electronic device supporting the same.

[0002] Techniques for image stabilization (also referred to as "camera shake correction") may include optical image stabilization (OIS), digital image stabilization (DIS), and electrical image stabilization (EIS).

[0003] OIS may be a technology for correcting an image by moving a lens and / or an image sensor based on the movement of an electronic device (e.g., a camera) acquired from an inertial sensor (e.g., a gyro sensor, an acceleration sensor). DIS may be a technology for extracting a vector related to the movement of an electronic device based on a difference image between frames of an image, and correcting an image based on the extracted vector. EIS may be a technology for correcting an image based on the movement of an electronic device acquired from an inertial sensor (e.g., a gyro sensor, an acceleration sensor).

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] When using EIS to compensate for hand shake, a camera included in an electronic device may acquire an image (e.g., pixel values ​​of pixels included in an image sensor of the camera) that includes, in addition to a first region having a size corresponding to (e.g., the same as) a size of an image to be output through a display, an additional second region (also referred to as a “margin region”) surrounding the first region, which is used to perform hand shake compensation. For example, the camera may crop a portion including the first region and the second region within an image acquired from all pixels of the image sensor, and transmit the cropped portion to a processor (e.g., an application processor). The processor may determine an area for compensating for hand shake within the cropped portion based on movement of the electronic device (e.g., magnitude and direction of movement of the electronic device) acquired based on sensing data acquired through an inertial sensor (e.g., a gyro sensor or an acceleration sensor). The processor may crop the determined region within the cropped portion and display the cropped region (the determined region) through the display. In this case, since the camera transmits an image having a size larger than the size of the image to be displayed through the display (e.g., the size of the first region) to the processor, the camera must transmit more data to the processor than when transmitting an image having the same size as the size of the image to be displayed through the display to the processor. In addition, since the camera crops a portion including the first region and the second region within an image acquired from all pixels of the image sensor, and then the processor performs a cropping operation to acquire an image with shake correction, the cropping operation is performed redundantly, so more resources may be used and more power may be consumed.

[0006] Various embodiments of the present disclosure relate to a method for providing an image, which determines an area for obtaining an image in which shake is corrected within the entire area of ​​an image sensor included in a camera based on movement of the electronic device, and obtains an image from the determined area (e.g., pixels included in the determined area), and an electronic device supporting the same.

[0007] The technical problems to be solved by the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0008] An electronic device according to one embodiment may include an inertial sensor, a camera including an image sensor, a processor, and a controller. The controller may be configured to acquire a magnitude and direction of movement of the electronic device caused by hand tremor based on sensing data acquired through the inertial sensor while a camera application is running. The controller may be configured to determine a first area for acquiring an image within the entire area of ​​the image sensor based on the magnitude and direction of the movement of the electronic device. The controller may be configured to acquire pixel values ​​of pixels included in the first area. The controller may be configured to provide the acquired pixel values ​​to the processor.

[0009] In one embodiment, a method for providing an image in an electronic device may include an operation of obtaining, by a controller of the electronic device, a magnitude and direction of movement of the electronic device caused by hand tremor based on sensing data obtained through an inertial sensor of the electronic device while a camera application is running. The method may include an operation of determining, by the controller of the electronic device, a first area for obtaining an image within an entire area of ​​an image sensor included in a camera of the electronic device based on the magnitude and direction of the movement of the electronic device. The method may include an operation of obtaining, by the controller of the electronic device, pixel values ​​of pixels included in the first area. The method may include an operation of providing, by the controller of the electronic device, the obtained pixel values ​​to a processor of the electronic device.

[0010] An electronic device according to one embodiment may include an inertial sensor, a camera including an image sensor, a controller, and a processor. The processor may be configured to acquire a magnitude and direction of movement of the electronic device caused by hand tremor based on sensing data acquired through the inertial sensor while a camera application is running. The processor may be configured to determine a first region for acquiring an image within an entire region of the image sensor based on the magnitude and direction of the movement of the electronic device. The processor may provide information about the first region to the controller so that the controller acquires pixel values ​​of pixels included in the first region. The processor may be configured to receive the pixel values ​​acquired by the controller from the controller.

[0011] A computer-readable recording medium according to one embodiment is provided, wherein the computer-readable recording medium may include instructions that, when executed by a controller of an electronic device including an inertial sensor, a camera including an image sensor, a controller, and a processor, cause the electronic device to execute the method.

[0012] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0013] FIG. 2 is a block diagram illustrating a camera module according to one embodiment.

[0014] FIG. 3 is a block diagram of an electronic device according to one embodiment.

[0015] FIG. 4 is a block diagram of an electronic device according to one embodiment.

[0016] FIG. 5 is a flowchart illustrating a method for providing an image according to one embodiment.

[0017] FIG. 6 is a flowchart illustrating a method for determining a first region according to one embodiment.

[0018] FIG. 7 is a diagram illustrating a method for determining a first area according to one embodiment.

[0019] FIG. 8 is a diagram illustrating a method for determining a first area according to one embodiment.

[0020] FIG. 9 is a diagram illustrating a method for performing a binning operation or a remosaic operation according to one embodiment.

[0021] FIG. 10 is a flowchart illustrating a method for providing an image according to one embodiment.

[0022] FIG. 11 is a diagram illustrating a method for providing an image according to one embodiment.

[0023] FIG. 12 is a flowchart illustrating a method for providing an image according to one embodiment.

[0024] FIG. 13 is a block diagram of an electronic device according to one embodiment.

[0025] FIG. 14 is a block diagram of an electronic device according to one embodiment.

[0026] FIG. 15 is a flowchart illustrating a method for providing an image according to one embodiment.

[0027] FIG. 16 is a flowchart illustrating a method for providing an image according to one embodiment.

[0028] FIG. 17 is a diagram illustrating a method for providing an image according to one embodiment.

[0029] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0031] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0032] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0033] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0034] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0035] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0036] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0037] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0038] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0039] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0040] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0041] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0042] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0043] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0044] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0045] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0046] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0047] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0048] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0049] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0050] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0051] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0052] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0053] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0054] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0055] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0056] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0057] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0058] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to one embodiment.

[0059] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0060] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0061] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0062] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be a processor (120). It may be configured as a separate processor that is at least partially composed of, or operates independently of, the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) by the processor (120) as is or after undergoing additional image processing.

[0063] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0064] FIG. 3 is a block diagram of an electronic device (301) according to one embodiment.

[0065] Referring to FIG. 3, in one embodiment, the electronic device (301) may be the electronic device (301) of FIG. 1.

[0066] In one embodiment, the electronic device (301) may include an inertial sensor (310), a camera (320), a controller (330), a display (340), memory (350), and a processor (360).

[0067] In one embodiment, an inertial sensor (310) (referred to as an “inertial measurement unit (IMU) sensor” or a “motion sensor”) may be included in the sensor module (176) of FIG. 1.

[0068] In one embodiment, the inertial sensor (310) may obtain sensing data to obtain information about the movement of the electronic device (301). For example, the inertial sensor (310) may obtain sensing data to obtain the magnitude and / or direction of the movement of the electronic device (301) caused by hand shake (also referred to as “camera shake”). For example, the inertial sensor (310) may include a gyro sensor configured to obtain sensing data related to the angular velocity of the electronic device (301) (e.g., camera (320)) caused by hand shake. However, the present invention is not limited thereto. For example, the inertial sensor (310) may include an acceleration sensor configured to obtain sensing data related to the translational motion of the electronic device (301) caused by hand shake.

[0069] In one embodiment, the camera (320) may be included in the camera module (180) of FIGS. 1 and 2.

[0070] In one embodiment, the camera (320) may include an image sensor comprising a plurality of pixels.

[0071] In one embodiment, the image sensor may include a plurality of pixels in which adjacent pixels are grouped into a single pixel (e.g., adjacent pixels are grouped into a single pixel unit). For example, the image sensor may include a plurality of pixels that enable a binning operation or a remosaic operation to be performed by grouping four adjacent pixels into a single pixel using tetracell technology. Tetracell technology may also be known as Quad Bayer or 4-cell technology. The binning operation and the remosaic operation are described in detail below with reference to FIG. 9.

[0072] In one embodiment, the camera (320) may include multiple cameras. For example, the camera (320) may include a first camera for acquiring an image to be provided through the display (340), and a second camera for acquiring an image to be used for image stabilization so that the first camera (320) acquires the image with the image stabilization compensated. The second camera may acquire an image for digital image stabilization (hereinafter referred to as "DIS") of the image to be acquired through the first camera.

[0073] In one embodiment, the controller (330) may control the overall operation of providing an image. For example, the controller (330) may perform EIS to compensate for the movement of the electronic device (301) caused by hand tremor, based on sensing data acquired through the inertial sensor (310). The operation of compensating for the movement of the electronic device (301) may also be referred to as an operation of compensating for the movement of the electronic device (301).

[0074] In one embodiment, the controller (330) may include a motion processing unit (331), an ROI determination unit (332), and a pixel value acquisition unit (333).

[0075] In one embodiment, the motion processing unit (331) may acquire (e.g., calculate) the magnitude and / or direction of the movement of the electronic device (301) (e.g., the camera (320)) caused by hand tremor based on the sensing data acquired from the inertial sensor (310). For example, the motion processing unit (331) may acquire sensing data related to angular velocity from a gyro sensor. The motion processing unit (331) may integrate the angular velocity based on the acquired sensing data from the time point at which the image sensor included in the camera (320) immediately previously acquired the first image frame (e.g., pixel values) to the time point (e.g., the current time point) at which the second image frame (e.g., the image frame currently to be acquired as the image frame immediately following the first image frame) is acquired, thereby acquiring the magnitude of the movement of the electronic device (301) caused by hand tremor. The motion processing unit (331) can obtain the direction of the movement of the electronic device (301) caused by hand tremor based on the angular velocity (e.g., the direction indicated by the angular velocity). In one embodiment, the size of the time between the time at which the first image frame is obtained and the time at which the second image frame is obtained may be equal to the inverse of the FPS (frame per second) at which the image sensor obtains the image frame (e.g., the time to obtain one image frame, the cycle at which the image frame is obtained). FPS may mean the number of frames per second, i.e., the number of image frames that the camera can obtain per second.

[0076] In one embodiment, the motion processing unit (331) may obtain (e.g., calculate) a motion having the same magnitude as the magnitude of the motion of the electronic device (301) and having a direction opposite to the direction of the motion of the electronic device (301) based on the magnitude and / or direction of the motion of the electronic device (301) caused by hand tremor (hereinafter also referred to as “magnitude and / or direction of the motion of the electronic device (301)”).

[0077] In one embodiment, although FIG. 3 illustrates that the controller (330) (e.g., the motion processing unit (331)) obtains sensing data directly from the inertial sensor (310), this is not limited thereto. For example, the controller (330) (e.g., the motion processing unit (331)) may obtain sensing data directly from the inertial sensor (310), or may receive the sensing data from the processor (360) when the inertial sensor (310) provides the sensing data to the processor (360).

[0078] In one embodiment, the ROI determination unit (332) may determine a region of interest (ROI) (hereinafter also referred to as “ROI” or “first region”) for acquiring an image (e.g., a shake-compensated image) within an area of ​​the image sensor (e.g., the entire area of ​​the image sensor). For example, the ROI determination unit (332) may confirm the size of the ROI determined based on a zoom factor associated with the camera (320). The ROI determination unit (332) may determine (e.g., calculate) coordinates of an ROI having the confirmed size (e.g., the size of the ROI determined based on the zoom factor associated with the camera (320)) within the area of ​​the image sensor based on a movement obtained from the motion processing unit (331) and having the same size as the size of the movement of the electronic device (301) and having a direction opposite to the direction of the movement of the electronic device (301). The coordinates of the above ROI may be coordinates of pixels included in the ROI among all pixels of the image sensor. The coordinates of the above ROI may include the first coordinates of the pixels included in the ROI among all pixels of the image sensor (e.g., the coordinates of the leftmost and topmost pixels among the pixels included in the ROI within the image sensor) and the last coordinates (e.g., the coordinates of the rightmost and bottommost pixels among the pixels included in the ROI within the image sensor) of the pixels included in the ROI.

[0079] In one embodiment, the pixel value acquisition unit (333) can acquire pixel values ​​of pixels included in an ROI within an image sensor.

[0080] In one embodiment, the pixel value acquisition unit (333) may control at least a part of an exposure operation including a photoelectric conversion operation for pixels included in the ROI based on the ROI determined by the ROI determination unit (332). For example, the pixel value acquisition unit (333) may set pixels included in the ROI (or positions of pixels included in the first region) on which an exposure operation including a photoelectric conversion operation (hereinafter, the exposure operation including a photoelectric conversion operation is referred to as an “exposure operation”) is to be performed, such that only pixels included in the ROI among the pixels included in the entire region of the image sensor perform the photoelectric conversion operation that converts received light (also referred to as an “optical signal”) into an electrical signal. In this case, pixels not included in the ROI among all pixels included in the entire region of the image sensor may not perform the photoelectric conversion operation. For example, the pixel value acquisition unit (333) can set the register (e.g., register values) of the camera (320) so that the exposure operation is performed only on the pixels included in the ROI among the pixels included in the entire area of ​​the image sensor. In one embodiment, the pixel value acquisition unit (333) can control so that the read out operation is performed only on the pixels included in the ROI among all the pixels included in the entire area of ​​the image sensor. For example, a photoelectric conversion operation may be performed on the pixels included in the ROI among all the pixels included in the entire area of ​​the image sensor, and a read out operation for acquiring pixel values ​​for the pixels included in the ROI may be performed. The photoelectric conversion operation may not be performed on the pixels not included in the ROI among all the pixels included in the entire area of ​​the image sensor, and a read out operation for acquiring pixel values ​​for the pixels not included in the ROI may not be performed.

[0081] In one embodiment, the pixel value acquisition unit (333) can acquire pixel values ​​of pixels included in the ROI by controlling at least a portion of an exposure operation for pixels included in the ROI. For example, the pixel value acquisition unit (333) can acquire pixel values ​​of pixels included in the ROI through an exposure operation and a read out operation for pixels included in the ROI.

[0082] In one embodiment, the pixel value acquisition unit (333) may provide the pixel values ​​of the pixels included in the acquired ROI to the processor (360).

[0083] In one embodiment, the pixel value acquisition unit (333) can acquire pixel values ​​having a Bayer pattern by performing a binning operation or a remosaic operation on pixel values ​​of pixels included in the ROI. The pixel value acquisition unit (333) can provide pixel values ​​having a Bayer pattern to the processor (360). However, the present invention is not limited thereto. For example, the pixel value acquisition unit (333) can provide pixel values ​​having a non-Bayer pattern, rather than a Bayer pattern, to the processor (360).

[0084] In one embodiment, although FIG. 3 illustrates that the motion processing unit (331), the ROI determination unit (332), and the pixel value acquisition unit (333) included in the controller (330) are independent components, this is not limited thereto. For example, at least some of the motion processing unit (331), the ROI determination unit (332), and the pixel value acquisition unit (333) included in the controller (330) may be implemented independently or integrally.

[0085] In one embodiment, the controller (330) may be implemented independently of the camera (320) or may be included in the camera (320).

[0086] In one embodiment, the processor (360) may be included in the processor (120) of FIG. 1.

[0087] In one embodiment, the processor (360) may include, but is not limited to, an application processor. For example, the processor (360) may include one or more processors (360) (e.g., an application processor and an image signal processor (ISP)).

[0088] In one embodiment, the processor (360) may include an image processing unit (361).

[0089] In one embodiment, the image processing unit (361) may perform an interpolation operation to generate an image (RGB image) to be displayed through the display (340) based on receiving pixel values ​​having a Bayer pattern from the controller (330) (e.g., pixel value acquisition unit). The image processing unit (361) may convert pixel values ​​having a non-Bayer pattern into pixel values ​​having a Bayer pattern based on receiving pixel values ​​having a non-Bayer pattern from the controller (330) (e.g., pixel value acquisition unit). The image processing unit (361) may perform an interpolation operation to generate an RGB image based on the converted pixel values.

[0090] However, the operation performed by the image processing unit (361) is not limited to the interpolation operation. For example, the image processing unit (361) may perform tone mapping, color correction, gamma correction, noise removal operation (e.g., noise removal operation using a spatial filter and / or temporal filter), lens shading correction, white balance adjustment, color space conversion, and / or gamut adjustment on an image (e.g., an RGB image).

[0091] In one embodiment, the display (340) may be included in the display module (160) of FIG. 1.

[0092] In one embodiment, the display (340) can display an image processed by the processor (360). For example, when pixel values ​​of pixels included in the ROI are acquired by the controller (330) on a per-image-frame basis, the display (340) can display a preview image acquired by the processor (360) processing the acquired pixel values.

[0093] In one embodiment, the memory (350) may be included in the memory (130) of FIG. 1.

[0094] In one embodiment, the memory (350) may store information for performing an operation of providing an image.

[0095] In one embodiment, the memory (350) may store instructions that, when individually or collectively executed by at least one controller (e.g., controller (330)) or at least one processor (e.g., processor (360)), cause the electronic device (301) to perform the operations described through FIGS. 3 through 12.

[0096] In one embodiment, although the memory (350) is illustrated as being connected to the processor (360) in FIG. 3, the present invention is not limited thereto. For example, the memory (350) may be connected to the controller (330) and / or the camera (320).

[0097] In one embodiment, although the memory (350) is illustrated in FIG. 3 as being independent of the processor (360), the controller (330), and the camera (320), this is not limiting. For example, the memory (350) may be included in the processor (360), the controller (330), and / or the camera (320).

[0098] In one embodiment, the electronic device (301) is illustrated in FIG. 3 as including, but not limited to, an inertial sensor (310), a camera (320), a controller (330), a display (340), a memory (350), and a processor (360). For example, the electronic device (301) may further include one or more of the components included in the electronic device (101) of FIG. 1. For example, the electronic device (301) may not include some of the components illustrated in FIG. 3 (e.g., the display (340)).

[0099] FIG. 4 is a block diagram of an electronic device (301) according to one embodiment.

[0100] Referring to FIG. 4, in one embodiment, the electronic device (301) may include components for performing OIS in addition to components for performing EIS based on sensing data acquired through the inertial sensor (310). For example, comparing FIGS. 3 and 4, compared to the electronic device (301) of FIG. 3, the electronic device (301) of FIG. 4 may further include an OIS controller (410).

[0101] In one embodiment, the electronic device (301) may include an inertial sensor (310), a camera (320), a controller (330), a display (340), a memory (350), a processor (360), and a controller (410).

[0102] In one embodiment, the inertial sensor (310) is substantially the same as the inertial sensor (310) of FIG. 3, so a redundant description will be omitted.

[0103] In one embodiment, the inertial sensor (310) can provide sensing data acquired by the inertial sensor (310) to the OIS controller (410) as well as the controller (330).

[0104] In one embodiment, the camera (320) is substantially the same as the camera (320) of FIG. 3, so a redundant description will be omitted.

[0105] In one embodiment, the camera (320) may further include, in addition to the image sensor, a Hall sensor (321) and an actuator (322) (also referred to as an “OIS actuator”) to support OIS.

[0106] In one embodiment, the Hall sensor (321) may be a sensor for obtaining the position of a lens (e.g., a lens assembly) included in the camera (320) and / or the position of an image sensor. For example, the Hall sensor (321) may be placed in an actuator (322) in which an OIS coil is placed. The Hall sensor (321) may detect the position of the lens (e.g., a lens assembly) and / or the position of the image sensor (or the position of the actuator (322)) by detecting a magnetic field generated by a magnet placed in an OIS carrier that accommodates the lens and / or the image sensor by the OIS coil. However, the sensor for obtaining the position of the lens (e.g., a lens assembly) included in the camera (320) and / or the position of the image sensor is not limited to the Hall sensor (321).

[0107] In one embodiment, the Hall sensor (321) may acquire sensing data related to the position of the lens and / or the position of the image sensor, and then transmit the acquired sensing data to the OIS controller (410) and the controller (330).

[0108] In one embodiment, the actuator (322) can change the position of the lens and / or the position of the image sensor to perform OIS under the control of the OIS controller (410).

[0109] In one embodiment, the actuator (322) may change the position of the lens and / or the position of the image sensor (e.g., move the lens and / or the image sensor) by using a method using a shape memory alloy (SMA) method, a piezo method, a step motor, or a micro electro mechanical systems (MEMS) method in addition to the method using the above-described OIS coil and magnet (referred to as a “VCM (voice coil motor) method”).

[0110] In one embodiment, a lens and / or image sensor included in the camera (320) may be moved under the control of the OIS controller (410). For example, the OIS controller (410) may control the actuator (322) to shift the lens, shift the image sensor, or tilt the lens and image sensor in a direction opposite to the movement of the electronic device (301) caused by hand shake.

[0111] In one embodiment, the OIS controller (410) can control the OIS. For example, the OIS controller (410) can perform the OIS based on sensing data acquired from the inertial sensor (310) and sensing data acquired from the Hall sensor (321).

[0112] In one embodiment, the OIS controller (410) may include a motion processing unit (411), a movement amount processing unit (412), and a driving unit (413).

[0113] In one embodiment, the motion processing unit (411) may obtain (e.g., calculate) the magnitude and / or direction of movement of the electronic device (301) (e.g., camera (320)) caused by hand tremor based on sensing data obtained from the inertial sensor (310). For example, the motion processing unit (411) may obtain sensing data related to angular velocity from a gyro sensor.

[0114] In one embodiment, the motion processing unit (411) can obtain (e.g., calculate) a motion having the same size as the size of the motion of the electronic device (301) and having a direction opposite to the direction of the motion of the electronic device (301), based on the size and / or direction of the motion of the electronic device (301) caused by hand tremor.

[0115] In one embodiment, the motion processing unit (411) may calculate a target position of a lens and / or an image sensor based on a movement that is opposite to the direction of movement of the electronic device (301). For example, when performing OIS by using a method of shifting a lens, the motion processing unit (411) may calculate a target position (hereinafter referred to as a “target position”) as a final position to which the lens will be moved for OIS when the lens and the image sensor are aligned with respect to the optical axis (e.g., the lens and the image sensor are aligned so that the center of the lens and the center of the image sensor are on the optical axis).

[0116] In one embodiment, the movement amount processing unit (412) can obtain the position of the lens and / or the position of the image sensor based on the sensing data provided from the Hall sensor (321). For example, the movement amount processing unit (412) can obtain the current position of the lens when performing OIS using a method of shifting the lens.

[0117] In one embodiment, the movement amount processing unit (412) may calculate a distance and direction in which the lens and / or the image sensor is to be moved based on the target position calculated by the motion processing unit (411) and the position of the lens and / or the position of the image sensor (e.g., the current position of the lens). For example, the movement amount processing unit (412) may calculate a distance and direction in which to move the lens (e.g., a distance in which to shift the lens) based on the target position of the lens and the current position of the lens.

[0118] In one embodiment, the driving unit (413) can control the actuator (322) based on the distance and direction calculated by the movement amount processing unit (412). For example, the driving unit (413) can control the actuator (322) so that the lens moves by the distance and direction calculated by the movement amount processing unit (412) (e.g., so that the lens moves from the current position by the distance and direction calculated by the movement amount processing unit (412). In one embodiment, the driving unit (413) can be implemented as a driver IC (integrated circuit), but is not limited thereto.

[0119] In one embodiment, in FIG. 4, the OIS controller (410) is illustrated as being a separate component from the controller (330), the camera (320), and the processor (360), but is not limited thereto. For example, the OIS controller (410) may be included in the camera (320) or the processor (360), or may be implemented integrally with the controller (330).

[0120] In one embodiment, the controller (330) may include a motion processing unit (331), a movement amount processing unit (334), an ROI determination unit (332), and a pixel value acquisition unit (333).

[0121] In one embodiment, the motion processing unit (331) has been described through FIG. 3, so a detailed description thereof will be omitted.

[0122] In one embodiment, the movement amount processing unit (334) can obtain the position of the lens and / or the position of the image sensor based on sensing data provided from the Hall sensor (321). For example, the movement amount processing unit (334) can obtain the current position of the lens when performing OIS using a method of shifting the lens.

[0123] In one embodiment, the ROI determination unit (332) performs substantially the same operation as the ROI determination unit (332) of FIG. 3, so a redundant description will be omitted.

[0124] In one embodiment, the ROI determination unit (332) may determine an ROI for obtaining an image in which shake is corrected within the area of ​​the image sensor, based at least in part on information obtained from the motion processing unit (331) and information obtained from the movement amount processing unit (334). For example, the ROI determination unit (332) may obtain, from the motion processing unit (331), a movement that has the same magnitude as the magnitude of the movement of the electronic device (301) caused by the shake and has a direction opposite to the direction of the movement of the electronic device (301). The ROI determination unit (332) may obtain, from the movement amount processing unit (334), a position of the lens and / or a position of the image sensor (e.g., a current position of the lens). The ROI determination unit (332) may calculate coordinates of the ROI based on a movement that has a direction opposite to the direction of the movement of the electronic device (301), taking into account the position of the lens and / or the position of the image sensor.

[0125] In one embodiment, the cycle in which OIS, which compensates for hand shake by moving the position of the lens and / or the position of the image sensor, is performed may be shorter than the cycle in which EIS, which compensates for hand shake based on sensing data acquired through the inertial sensor (310), is performed. The ROI determination unit (332) may determine the ROI so that the movement of the electronic device (301) caused by hand shake (e.g., the movement changed during the time between the time immediately before the time when the image sensor acquires the image frame and the time when the image sensor acquires the image frame) is compensated by considering the position of the lens and / or the position of the image sensor acquired at the time when the ROI is determined (e.g., the current position of the lens moved by performing OIS performed at the time immediately before the time when the ROI is determined).

[0126] In one embodiment, in the examples described above, the ROI determination unit (332) determines the ROI based on a movement having an opposite direction to the direction of movement of the electronic device (301), taking into account the position of the lens and / or the current position of the image sensor, but is not limited thereto. For example, the ROI determination unit (332) may obtain, from the movement amount processing unit (334), an average of the positions of the lens and / or the positions of the image sensor (e.g., an average position of the positions of the lenses calculated during the time from the time immediately before the time the image sensor acquires the image frame to the time the image frame is acquired). The ROI determination unit (332) may determine the ROI based on a movement having an opposite direction to the direction of movement of the electronic device (301), taking into account the average of the positions of the lens and / or the positions of the image sensor acquired.

[0127] In one embodiment, the pixel value acquisition unit (333) performs substantially the same operation as the pixel value acquisition unit (333) of FIG. 3, so a detailed description thereof will be omitted.

[0128] In one embodiment, the display (340), the memory (350), and the processor (360) are substantially the same as the display (340), the memory (350), and the processor (360) of FIG. 3, respectively, and therefore, a detailed description thereof will be omitted.

[0129] In one embodiment, in FIG. 4, it is described that the electronic device (301) performs EIS by considering OIS performed based on sensing data acquired through the inertial sensor (310) and sensing data acquired from the Hall sensor (321), but is not limited thereto. In one embodiment, the electronic device (301) can independently perform OIS and EIS. For example, the OIS controller (410) can perform OIS performed based on sensing data acquired through the inertial sensor (310) and sensing data acquired from the Hall sensor (321). The controller (330) can perform EIS based on sensing data acquired from the inertial sensor (310), regardless of the OIS performed by the OIS controller (410) (e.g., the controller can determine an ROI based on sensing data acquired from the inertial sensor (310)).

[0130] In one embodiment, the electronic device (301) may operate in a first mode for performing EIS as described with reference to FIG. 3, a second mode for performing EIS while considering OIS as described with reference to FIG. 4, or a third mode for performing OIS and EIS independently. For example, the electronic device (301) may determine a region of interest (ROI) and obtain pixel values ​​(e.g., image data with shake correction) of pixels included in the ROI in the first mode, the second mode, or the third mode.

[0131] FIG. 5 is a flowchart (500) for explaining a method of providing an image according to one embodiment.

[0132] In one embodiment, FIG. 5 may be a drawing for explaining operations performed by the electronic device (301) of FIG. 3 to provide an image.

[0133] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0134] Referring to FIG. 5, in operation 501, in one embodiment, the controller (330) may obtain the magnitude and direction of movement of the electronic device (301) caused by hand tremor based on sensing data obtained through the inertial sensor (310) while the camera application is running.

[0135] In one embodiment, the controller (330) may acquire (e.g., calculate) the magnitude and / or direction of movement of the electronic device (301) (e.g., the camera (320)) caused by hand tremor based on sensing data acquired from the inertial sensor (310) based on the execution of the camera application. For example, the controller (330) may acquire sensing data related to angular velocity from a gyro sensor. The controller (330) may integrate the angular velocity from the point in time at which the image sensor included in the camera (320) immediately previously acquired the first image frame (e.g., pixel values) to the point in time (e.g., the current point in time) at which the second image frame (e.g., the image frame immediately following the first image frame) is acquired, thereby acquiring the magnitude of movement of the electronic device (301) caused by hand tremor based on the acquired sensing data. The controller (330) can obtain the direction of movement of the electronic device (301) caused by hand tremor based on the angular velocity. In one embodiment, the time between the time at which the first image frame is obtained and the time at which the second image frame is obtained may be equal to the reciprocal of the FPS (frame per second) at which the image sensor obtains the image frame (e.g., the cycle at which the image frame is obtained).

[0136] In operation 503, in one embodiment, the controller (330) may determine a first region of interest (ROI) for acquiring an image within the area of ​​the image sensor based on the magnitude and direction of the movement of the electronic device (301).

[0137] In one embodiment, the controller (330) may acquire (e.g., calculate) a movement having the same magnitude as the magnitude of the movement of the electronic device (301) and having a direction opposite to the direction of the movement of the electronic device (301) based on the magnitude and / or direction of the movement of the electronic device (301) caused by the hand tremor. The controller (330) may determine a first area based on the movement having the same magnitude as the magnitude of the movement of the electronic device (301) caused by the hand tremor and having a direction opposite to the direction of the movement of the electronic device (301). Hereinafter, an operation of determining the first area will be described in more detail with reference to FIGS. 6 to 8.

[0138] FIG. 6 is a flowchart (600) for explaining a method for determining a first area according to one embodiment.

[0139] FIG. 7 is a diagram illustrating a method for determining a first area according to one embodiment.

[0140] FIG. 8 is a diagram illustrating a method for determining a first area according to one embodiment.

[0141] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0142] Referring to FIGS. 6 through 8, in operation 603, in one embodiment, the controller (330) may obtain (e.g., calculate) a movement displacement based on the magnitude and direction of the movement of the electronic device (301).

[0143] In operation 603, in one embodiment, the controller (330) can determine the size of the first region (ROI) determined based on a zoom factor (hereinafter referred to as “zoom factor”) associated with the camera (320).

[0144] In operation 605, in one embodiment, the controller (330) can determine a first region within the entire area of ​​the image sensor based on the movement displacement obtained in operation 603 and the size of the first region (ROI) identified in operation 603.

[0145] In FIG. 6, operation 601 is illustrated as being performed prior to operation 603, but this is not limited thereto. Operation 603 may be performed prior to operation 601, or operations 601 and 603 may be performed in parallel. Hereinafter, the operation of determining the first region will be described in more detail with reference to FIGS. 7 and 8.

[0146] In one embodiment, FIG. 7 may be a diagram illustrating an operation of determining a first area within the entire area of ​​an image sensor when the zoom magnification associated with the camera (320) is 1.0x.

[0147] In one embodiment, in FIG. 7, arrow (741) represents the magnitude and direction of movement of electronic device (301) caused by hand tremor, and arrow (742) has the same magnitude as the magnitude of movement of electronic device (301) caused by hand tremor and may represent movement in a direction opposite to the direction of movement of electronic device (301). For example, the magnitude (e.g., length) of arrow (742) may be the same as the magnitude (e.g., length) of arrow (741). The direction of arrow (742) may be opposite to the direction of arrow (741).

[0148] In one embodiment, the controller (330) can calculate a movement displacement corresponding to the movement (hereinafter referred to as “movement displacement”), which has the same magnitude as the magnitude of the movement of the electronic device (301) caused by the hand tremor and has a direction opposite to the direction of the movement of the electronic device (301). For example, in FIG. 7, the controller (330) can calculate the movement displacement indicated by the arrow (743).

[0149] In one embodiment, the translation displacement may include a distance and direction for moving an area (720) (hereinafter referred to as a “second area”) having a size determined based on a zoom factor (e.g., having the same center as the center (721) of the entire area (710) of the image sensor) relative to the center (721) of the entire area (710) of the image sensor to compensate (also referred to as “compensation”) for movement of the electronic device (301) caused by hand shake.

[0150] In one embodiment, the direction of the movement displacement may be opposite to the direction of the movement of the electronic device (301) caused by the hand tremor. In one embodiment, the magnitude of the movement displacement may correspond to the magnitude of the movement of the electronic device (301) caused by the hand tremor. As the magnitude of the movement of the electronic device (301) caused by the hand tremor increases, the magnitude of the movement displacement may increase, and as the magnitude of the movement of the electronic device (301) caused by the hand tremor decreases, the magnitude of the movement displacement may decrease.

[0151] In one embodiment, the controller (330) can determine the size of the first area based on the zoom ratio. The size of the first area may be the same as the size of the second area.

[0152] In one embodiment, in FIG. 7, when the zoom magnification is 1x (1.0x), the size of the second area (720) may be determined by considering a margin area (hereinafter referred to as a “margin area”) for compensating for hand shake within the entire area (710) of the image sensor (e.g., an area remaining within the entire area (710) of the image sensor excluding the second area (720)). For example, the size of the second area (720) may be calculated by multiplying the entire area (710) of the image sensor corresponding to a zoom magnification of 1x by a ratio related to the margin area (e.g., a ratio for setting the margin area with respect to the entire area of ​​the image sensor).

[0153] In one embodiment, the operation of calculating the size of the second region (720) (or the operation of calculating the size of the first region (730)) may be performed by the processor (360) (e.g., an application processor) or the controller (330). For example, the processor (360) (e.g., an application processor) may calculate the size of the second region based on a zoom factor set based on a user input (or set as a default). The controller (330) may confirm the size of the second region (and / or the first region) by receiving the size of the second region calculated by the processor (360) from the processor (360). For example, the controller (330) may receive the zoom factor from the processor (360). The controller (330) can determine the size of the second area (and / or the first area) by obtaining (e.g., calculating) the size of the second area (and / or the first area) based on the zoom ratio received from the processor (360).

[0154] In one embodiment, the controller (330) may determine the first region based on the size and displacement of the first region. For example, in FIG. 7, the controller (330) may determine the first region (730) to have a center (731) that is moved from the position of the center (721) of the second region (720) by the displacement indicated by the arrow (743), and to have a size equal to the size of the second region (720) based on the center (731).

[0155] In one embodiment, FIG. 8 may be a diagram illustrating an operation of determining a first area within the entire area of ​​an image sensor when the zoom magnification associated with the camera (320) is 2x (2.0x).

[0156] In one embodiment, in FIG. 8, an arrow (851) indicates a magnitude and direction of a movement of an electronic device (301) caused by hand tremor, and an arrow (852) may have a magnitude equal to the magnitude of the movement of the electronic device (301) caused by hand tremor and may indicate a movement in a direction opposite to the direction of the movement of the electronic device (301). For example, a magnitude (e.g., length) of an arrow (852) may be equal to a magnitude (e.g., length) of an arrow (851). A direction of an arrow (852) may be opposite to a direction of an arrow (851).

[0157] In one embodiment, the controller (330) can calculate a displacement corresponding to the movement, which has the same magnitude as the magnitude of the movement of the electronic device (301) caused by the hand tremor and has a direction opposite to the direction of the movement of the electronic device (301). For example, in FIG. 8, the controller (330) can calculate the displacement indicated by the arrow (853).

[0158] In one embodiment, when the zoom ratio is 2x (2.0x) in FIG. 8, the size of the second region (830) may be determined by considering the margin area with respect to the size of the third region (820) corresponding to the zoom ratio of 2x. For example, when the zoom ratio is 2x (2.0x), the third region (820) may be determined to have a size that is approximately 1 / 4 of the size of the entire region (810) of the image sensor. The size of the second region (830) may be determined by multiplying the third region (820) by a ratio related to the margin area.

[0159] In one embodiment, the operation of calculating the size of the second region (830) (or the operation of calculating the size of the first region (840)) may be performed in the processor (360) (e.g., an application processor) or the controller (330).

[0160] In one embodiment, the controller (330) may determine the first region based on the size and displacement of the first region. For example, in FIG. 8, the controller (330) may determine the first region (840) having a center (841) that is moved from the position of the center (831) of the second region (830) by the displacement indicated by the arrow (853), and having the same size as the second region (830) based on the center (841).

[0161] Referring to FIG. 5, in one embodiment, the operation of determining the first region may include an operation of determining coordinates of the first region within the entire region of the image sensor.

[0162] In one embodiment, the coordinates of the first region may be coordinates of pixels included in the first region among all pixels of the image sensor.

[0163] In one embodiment, the coordinates of the first region may include the first coordinates of pixels included in the first region among all pixels of the image sensor (e.g., the coordinates of the pixel (732) positioned at the leftmost and topmost position among pixels included in the first region (730) in the coordinate system for the entire region (710) of the image sensor in FIG. 7) and the last coordinates (e.g., the coordinates of the pixel (733) positioned at the rightmost and bottommost position among pixels included in the first region (730) in the coordinate system for the entire region (710) of the image sensor in FIG. 7).

[0164] In operation 505, in one embodiment, the controller (330) may obtain pixel values ​​of pixels included in the first region. For example, the controller (330) may obtain only pixel values ​​of pixels included in the first region.

[0165] In one embodiment, the controller (330) may control at least a portion of an exposure operation including a photoelectric conversion operation for pixels included in the first area based on the first area determined through operation 503. For example, the controller (330) may set pixels included in the first area (or positions of pixels included in the first area) on which an exposure operation including a photoelectric conversion operation is to be performed, such that only pixels included in the first area among pixels included in the entire area of ​​the image sensor perform the photoelectric conversion operation of converting received light into an electrical signal. In this case, pixels that are not included in the ROI among all pixels included in the entire area of ​​the image sensor may not perform the photoelectric conversion operation. For example, the controller (330) may set a register (e.g., values ​​of the register) of the camera (320) such that the exposure operation is performed only for pixels included in the first area among pixels included in the entire area of ​​the image sensor. In one embodiment, the controller (330) may control a read out operation to be performed only on pixels included in the ROI among all pixels included in the entire area of ​​the image sensor. For example, a photoelectric conversion operation may be performed on pixels included in the ROI among all pixels included in the entire area of ​​the image sensor, and a read out operation for acquiring pixel values ​​for the pixels included in the ROI may be performed. A photoelectric conversion operation may not be performed on pixels not included in the ROI among all pixels included in the entire area of ​​the image sensor, and a read out operation for acquiring pixel values ​​for pixels not included in the ROI may not be performed.

[0166] In one embodiment, the controller (330) can obtain pixel values ​​of pixels included in the first area by controlling at least a portion of an exposure operation for the pixels included in the first area. For example, the controller (330) can obtain pixel values ​​of pixels included in the first area through an exposure operation and a read out operation for the pixels included in the first area.

[0167] In one embodiment, the controller (330) can obtain pixel values ​​having a Bayer pattern by performing a binning operation or a remosaic operation on pixel values ​​of pixels included in a first region. Hereinafter, a method of performing a binning operation or a remosaic operation on pixel values ​​of pixels included in a first region will be described with reference to FIG. 9.

[0168] FIG. 9 is a diagram illustrating a method for performing a binning operation or a remosaic operation according to one embodiment.

[0169] Referring to FIG. 9, in one embodiment, at reference numeral 901, reference numeral 910 may represent an image sensor including a plurality of pixels, in which four adjacent pixels are grouped into one pixel using tetracell technology. In one embodiment, at reference numeral 910, the image sensor (910) may include pixels such that four pixels correspond to each of the same color filters. For example, as illustrated at reference numeral 910, the image sensor (910) may include pixels such that adjacent pixels (911, 912, 913, 914) correspond to a G (green) color filter and adjacent pixels (915, 916, 917, 918) correspond to a B (blue) color filter.

[0170] Although reference numeral 910 illustrates an image sensor (910) including pixels in which adjacent four pixels are grouped into a single pixel, the image sensor is not limited thereto. For example, the image sensor may include pixels in which a plurality of adjacent pixels are grouped into a single pixel, such as nine adjacent pixels or sixteen adjacent pixels.

[0171] In one embodiment, the controller (330) may perform a binning operation or a remosaic operation on pixel values ​​of pixels included in the first area based on settings set in the electronic device (301) and / or an environment of the electronic device (301).

[0172] In one embodiment, a binning operation may include an operation of converting pixel values ​​of four adjacent pixels within an image sensor into a single pixel value. For example, reference numeral 902 may represent pixel values ​​(920) obtained by a binning operation on an image sensor (910) (e.g., pixel values ​​of pixels included in the image sensor (910). Referring to reference numerals 901 and 902, by summing the pixel values ​​of four adjacent pixels (911, 912, 913, 914) corresponding to a G color filter as part of a binning operation on the image sensor (910), a pixel value (921) corresponding to a G color filter (e.g., corresponding to four pixels (911, 912, 913, 914)) may be obtained. As part of a binning operation for the image sensor (910), a pixel value of one pixel value (922) corresponding to the B color filter can be obtained by summing the pixel values ​​of four adjacent pixels (915, 916, 917, 918) corresponding to the B color filter. However, the present invention is not limited thereto. For example, as part of a binning operation for the image sensor (910), a pixel value of one pixel value (921) corresponding to the G color filter (e.g., corresponding to four pixels (911, 912, 913, 914)) can be obtained by averaging the pixel values ​​of four adjacent pixels (911, 912, 913, 914) corresponding to the G color filter. As part of a binning operation for the image sensor (910), a pixel value of one pixel value (922) corresponding to the B color filter can be obtained by averaging the pixel values ​​of four adjacent pixels (915, 916, 917, 918) corresponding to the B color filter.

[0173] In one embodiment, a binning operation that obtains pixel values ​​corresponding to adjacent pixels by summing the pixel values ​​of adjacent pixels may enable an electronic device (301) in a low-light environment to obtain a brighter image. A binning operation that obtains pixel values ​​corresponding to adjacent pixels by averaging the pixel values ​​of adjacent pixels may enable an electronic device (301) to obtain an image with less noise.

[0174] In one embodiment, reference numeral 903 may represent pixel values ​​(930) obtained by a re-mosaic operation on the image sensor (910) (e.g., pixel values ​​of pixels included in the image sensor (910). For example, by rearranging (also referred to as “re-ordering”) the pixel value of pixel (913) and the pixel value of pixel (915) of the image sensor (910) at reference numeral 901, the pixel value of pixel (933) and the pixel value of pixel (935) at reference numeral 903 may be obtained. By performing a re-mosaic operation that changes the positions of the pixel values ​​of the pixels of the image sensor (910) in this way, the values ​​of pixels (930) having a Bayer pattern may be obtained. For example, as illustrated in reference numeral 903, pixels (931, 932, 933, 934) and pixels (935, 936, 937, 938) may have a Bayer pattern. However, the manner in which the remosaic operation is performed is not limited to the operation of rearranging some of the pixel values ​​of the aforementioned pixels.

[0175] In one embodiment, the re-mosaic operation may enable the electronic device (301) to obtain an image with a higher resolution when the zoom magnification increases. For example, the controller (330) may obtain an image (e.g., image data composed of a number of pixel values ​​corresponding to the resolution A) having a resolution of A (e.g., approximately 12 MP) when the zoom magnification is 1x (1.0x). When the zoom magnification is 2x (2.0x), the controller (330) may perform a re-mosaic operation on pixel values ​​obtained from pixels included in the first area, thereby obtaining an image (e.g., pixel values ​​constituting the image) having a resolution of A (e.g., approximately 12 MP) that is substantially the same as when the zoom magnification is 1x.

[0176] Referring to FIG. 5, in operation 507, in one embodiment, the controller (330) may provide the pixel values ​​obtained through operation 505 to the processor (360).

[0177] In one embodiment, when a binning operation or a re-mosaic operation is performed on pixel values ​​of pixels included in the first region, the controller (330) may provide pixel values ​​obtained by the binning operation or the re-mosaic operation and having a Bayer pattern to a processor (360) (e.g., an application processor, an ISP).

[0178] In one embodiment, the controller (330) may provide pixel values ​​having a non-Bayer pattern, rather than a Bayer pattern, to the processor (360).

[0179] In one embodiment, although not shown in FIG. 5, the processor (360) may perform an interpolation operation to generate an image (RGB image) to be displayed through the display (340) based on receiving pixel values ​​having a Bayer pattern from the controller (330). The processor (360) may convert pixel values ​​having a non-Bayer pattern into pixel values ​​having a Bayer pattern based on receiving pixel values ​​having a non-Bayer pattern from the controller (330). The processor (360) may perform an interpolation operation to generate an RGB image based on the converted pixel values.

[0180] In one embodiment, the processor (360) may perform tone mapping, color correction, gamma correction, noise removal operations (e.g., noise removal operations using spatial filters and / or temporal filters), lens shading correction, white balance adjustment, color space conversion, and / or gamut adjustment on an image (e.g., an RGB image).

[0181] In one embodiment, the processor (360) may display an image (e.g., an RGB image) through the display (340). For example, the processor (360) may receive pixel values ​​from the controller (330) at a cycle corresponding to the FPS set for the camera (320). The processor (360) may obtain an image to be displayed through the display (340) by performing at least some of the operations described above (e.g., an interpolation operation) based on the received pixel values. The processor (360) may display the obtained image as a preview image through the display (340).

[0182] In one embodiment, since the controller (330) transmits the pixel values ​​of the pixels included in the first area that does not include the margin area to the processor (360), pixel values ​​having a smaller amount of data can be transmitted to the processor (360) compared to the comparative example in which the pixel values ​​of the pixels included in the area that includes the margin area are transmitted to the processor (360). In addition, since the controller (330) transmits the pixel values ​​of the pixels included in the first area that does not include the margin area to the processor (360), multiple cropping operations may not be performed. Accordingly, fewer resources may be used, and power consumed by the electronic device (301) may be reduced.

[0183] FIG. 10 is a flowchart (1000) illustrating a method for providing an image according to one embodiment.

[0184] FIG. 11 is a diagram illustrating a method for providing an image according to one embodiment.

[0185] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0186] Referring to FIGS. 10 and 11, in operation 1001, in one embodiment, the controller (330) may obtain the magnitude and direction of movement of the electronic device (301) caused by hand tremor based on sensing data obtained through the inertial sensor (310) while the camera (320) application is running.

[0187] Since operation 1001 is at least partially identical or similar to operation 501 of FIG. 5, a detailed description thereof will be omitted.

[0188] In operation 1003, in one embodiment, the controller (330) can recognize an object corresponding to the subject within an image acquired through another camera.

[0189] In one embodiment, the controller (330) can acquire an image through a camera (hereinafter referred to as a “second camera”) that includes an image sensor from which pixel values ​​of pixels (e.g., pixel values ​​acquired through operation 1009) are acquired (hereinafter referred to as a “first camera”) and another camera (hereinafter referred to as a “second camera”). The controller (330) can detect an object (hereinafter referred to as an “object”) corresponding to a subject in an image acquired through the second camera using a designated algorithm or artificial intelligence model. For example, the second camera can be set such that the FPS of the second camera is greater than the FPS of the first camera (e.g., such that the cycle for acquiring image frames by the first camera is shorter than the cycle for acquiring image frames by the second camera). The controller (330) can identify an image acquired through the second camera at a point in time substantially identical to the point in time when the first camera acquires an image (e.g., pixel values ​​of pixels to be included in a first area within an image sensor) or at the most recent point in time among the points in time when the second camera acquired images (e.g., image frames). The controller (330) can recognize an object within the identified image.

[0190] In operation 1005, in one embodiment, the controller (330) can identify an area (hereinafter also referred to as a “fourth area”) corresponding to an area including an object recognized in an image (e.g., an image acquired through a second camera) (hereinafter referred to as a “first image”) within an area of ​​an image sensor (e.g., an image sensor of a first camera).

[0191] In one embodiment, the controller (330) can identify a fourth region corresponding to an area containing an object (e.g., an object recognized in operation 1003) within the entire area of ​​the image sensor of the first camera, based on a parallax between the first camera and the second camera. For example, the controller (330) can identify a fourth region corresponding to an area containing an object within the first image, based on a difference between a direction in which a center of the field of view of the first camera faces and a direction in which a center of the field of view of the second camera faces, and a difference between a location of the first camera and a location of the second camera within the electronic device (301). For example, the controller (330) can determine a fourth area in which pixel values ​​for a scene including a subject corresponding to an object within the field of view of the first camera are to be acquired, based on a difference between a direction in which the center of the field of view of the first camera faces and a direction in which the center of the field of view of the second camera faces, and a difference between a position of the first camera and a position of the second camera within the electronic device (301).

[0192] In one embodiment, the size of the fourth region may be changeable depending on the size of the object (or the region containing the object) within the first image. For example, if the subject moves forward or backward relative to the direction in which the second camera is facing, the size of an object corresponding to the subject within the image acquired through the second camera may be changed. In this case, the size of the fourth region may be a size that can be changed depending on the size of the object (or the region containing the object) within the first image.

[0193] In one embodiment, the size of the fourth region may be a fixed size. For example, the size of the fourth region may be set to correspond to the zoom factor, and may remain a fixed size when the zoom factor does not change.

[0194] In operation 1007, in one embodiment, the controller (330) may determine a first region of interest (ROI) for acquiring an image within an area of ​​an image sensor (e.g., an image sensor of a first camera (320)) based on the magnitude and direction of the movement of the electronic device (301) acquired through operation 1001 and the fourth region identified through operation 1005.

[0195] In one embodiment, in FIG. 11, an arrow (1141) indicates a magnitude and direction of a movement of an electronic device (301) caused by hand tremor, and an arrow (1142) may have a magnitude equal to the magnitude of the movement of the electronic device (301) caused by hand tremor and may indicate a movement in a direction opposite to the direction of the movement of the electronic device (301). For example, a magnitude (e.g., length) of an arrow (1142) may be equal to a magnitude (e.g., length) of an arrow (1141). A direction of an arrow (1142) may be opposite to a direction of an arrow (1141).

[0196] In one embodiment, in FIG. 11, reference numeral 1120 may indicate a fourth region within the entire region (1110) of the image sensor. The controller (330) may determine the first region (1130) based on the magnitude and direction of the movement of the electronic device (301) and the fourth region (1120). For example, the controller (330) may determine the first region (1130) to have a center (1131) that is moved from the position of the center (1121) of the fourth region (1120) by a displacement indicated by an arrow (1143) (e.g., a displacement corresponding to a movement having the same magnitude as the magnitude of the movement of the electronic device (301) and having a direction opposite to the direction of the movement of the electronic device (301), and having the same size as the size of the fourth region (1120) based on the center (1131).

[0197] In operation 1009, in one embodiment, the controller (330) may obtain pixel values ​​of pixels included in the first region. For example, the controller (330) may obtain only pixel values ​​of pixels included in the first region.

[0198] Since operation 1009 is at least partially identical or similar to operation 505 of FIG. 5, a detailed description thereof will be omitted.

[0199] In operation 1011, in one embodiment, the controller (330) may provide the pixel values ​​obtained through operation 1009 to the processor (360).

[0200] Since operation 1011 is at least partially identical or similar to operation 507 of FIG. 5, a detailed description thereof will be omitted.

[0201] FIG. 12 is a flowchart (1200) illustrating a method of providing an image according to one embodiment.

[0202] In one embodiment, FIG. 12 may be a drawing for explaining operations performed by the electronic device (301) of FIG. 4 to provide an image.

[0203] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0204] Referring to FIG. 12, in operation 1201, in one embodiment, the controller (330) may obtain the position of the lens and / or the position of the image sensor.

[0205] In one embodiment, the OIS controller (410) can perform OIS independently of the operation of the controller (330) performing EIS. The OIS controller (410) can perform OIS based on sensing data acquired from the inertial sensor (310) and sensing data acquired from the Hall sensor.

[0206] In one embodiment, the OIS controller (410) may acquire (e.g., calculate) the magnitude and / or direction of movement of the electronic device (301) (e.g., the camera (320)) caused by hand tremor based on sensing data acquired from the inertial sensor (310). For example, the OIS controller (410) may acquire sensing data related to angular velocity from a gyro sensor.

[0207] In one embodiment, the OIS controller (410) can acquire (e.g., calculate) a motion having the same magnitude as the magnitude of the movement of the electronic device (301) and having a direction opposite to the direction of the movement of the electronic device (301), based on the magnitude and / or direction of the movement of the electronic device (301) caused by hand tremor.

[0208] In one embodiment, the OIS controller (410) may calculate a target position of the lens and / or image sensor based on a movement that is opposite to the direction of movement of the electronic device (301). For example, when performing OIS by shifting the lens, the OIS controller (410) may calculate the target position as the final position to which the lens will be moved for OIS when the lens and the image sensor are aligned with respect to the optical axis (e.g., the lens and the image sensor are aligned such that the center of the lens and the center of the image sensor are on the optical axis).

[0209] In one embodiment, the OIS controller (410) can obtain the position of the lens and / or the position of the image sensor based on sensing data provided from the Hall sensor (321). For example, the OIS controller (410) can obtain the current position of the lens when performing OIS using a method of shifting the lens.

[0210] In one embodiment, the OIS controller (410) may calculate a distance and direction to which the lens and / or the image sensor will move based on the calculated target position and the position of the lens and / or the position of the image sensor (e.g., the current position of the lens). For example, the OIS controller (410) may calculate a distance and direction to move the lens (e.g., a distance to shift the lens) based on the target position of the lens and the current position of the lens.

[0211] In one embodiment, the OIS controller (410) can control the actuator (322) based on the distance and direction calculated by the movement amount processing unit. For example, the OIS controller (410) can control the actuator (322) so that the lens moves by the calculated distance and direction (e.g., so that the lens moves by the calculated distance and direction from the current position).

[0212] In one embodiment, the controller (330) may acquire the position of the lens and / or the position of the image sensor based on sensing data provided from the Hall sensor (321). For example, the controller (330) may acquire the current position of the lens when performing OIS using a method of shifting the lens.

[0213] In operation 1203, in one embodiment, the controller (330) may obtain the magnitude and direction of movement of the electronic device (301).

[0214] Since operation 1203 is at least partially identical or similar to operation 501 of FIG. 5, a detailed description thereof will be omitted.

[0215] In operation 1205, in one embodiment, the controller (330) may determine a first area for acquiring a shake-compensated image within the area of ​​the image sensor based at least in part on the position of the lens and / or the position of the image sensor and the magnitude and direction of movement of the electronic device (301).

[0216] Since operation 1205 is partially identical or similar to operation 503 of FIG. 5, overlapping descriptions will be omitted.

[0217] In one embodiment, the controller (330) can acquire a movement that has the same magnitude as the magnitude of the movement of the electronic device (301) caused by hand tremor and has a direction opposite to the direction of the movement of the electronic device (301). The controller (330) can calculate coordinates of the first area based on the movement that has a direction opposite to the direction of the movement of the electronic device (301), taking into account the position of the lens and / or the position of the image sensor (e.g., the current position of the lens).

[0218] In one embodiment, the cycle in which OIS is performed to compensate for hand shake by moving the position of the lens and / or the position of the image sensor may be shorter than the cycle in which EIS is performed to compensate for hand shake based on sensing data acquired through the inertial sensor (310). The controller (330) may determine the first region so that movement of the electronic device (301) caused by hand shake (e.g., movement changed during the time between the time immediately before the time when the image sensor acquires the image frame and the time when the image sensor acquires the image frame) is compensated by considering the position of the lens and / or the position of the image sensor acquired at the time when the first region is determined (e.g., the current position of the lens moved by performing OIS performed at the time immediately before the time when the ROI is determined).

[0219] In one embodiment, in the examples described above, the controller (330) determines the first area based on a movement that is opposite to the direction of movement of the electronic device (301), taking into account the position of the lens and / or the current position of the image sensor, but is not limited thereto. For example, the controller (330) may obtain an average of the positions of the lens and / or the positions of the image sensor (e.g., an average position of the positions of the lenses calculated during the time from immediately before the time the image sensor acquires the image frame to the time the image frame is acquired). The controller (330) may determine the first area based on a movement that is opposite to the direction of movement of the electronic device (301), taking into account the average of the positions of the lens and / or the positions of the image sensor obtained from the time immediately before the time the image sensor acquires the image frame to the time the image frame is acquired.

[0220] In operation 1207, in one embodiment, the controller (330) may obtain pixel values ​​of pixels included in the first region. The controller (330) may obtain pixel values ​​of pixels included in the first region.

[0221] Since operation 1207 is at least partially identical or similar to operation 505 of FIG. 5, a detailed description thereof will be omitted.

[0222] In operation 1209, in one embodiment, the controller (330) may provide the pixel values ​​obtained through operation 1207 to the processor (360).

[0223] Since operation 1209 is at least partially identical or similar to operation 507 of FIG. 5, a detailed description thereof will be omitted.

[0224] In the examples described above, the operation of providing an image by performing EIS (and OIS) based on sensing data acquired through the inertial sensor (310) has been described, but is not limited thereto. For example, the camera (320) may include a plurality of cameras. The camera (320) may include a first camera for acquiring an image to be provided through the display (340), and a second camera for acquiring an image to be used for image shake correction so that the first camera (320) acquires the image with shake correction. The second camera may acquire images for digital image stabilization (hereinafter referred to as “DIS”) of the image to be acquired through the first camera. The controller (330) can acquire the magnitude and direction of the movement of the electronic device by using images acquired from the second camera (e.g., a difference image between the images) instead of the sensing data acquired through the inertial sensor (310). The controller (330) can acquire an image with shake compensation (e.g., pixel values ​​of pixels included in the first area) by performing the above-described operations based at least in part on the magnitude and direction of the movement of the electronic device.

[0225] FIG. 13 is a block diagram of an electronic device (301) according to one embodiment.

[0226] Referring to FIG. 13, in one embodiment, the electronic device (301) may be the electronic device (101) of FIG. 1.

[0227] In one embodiment, the electronic device (301) may include an inertial sensor (1310), a camera (1320), a controller (1330), a display (1340), memory (1350), and a processor (1360).

[0228] In one embodiment, the inertial sensor (1310), the camera (1320), the display (1340), and the memory (1350) may be substantially identical to the inertial sensor (310), the camera (320), the display (340), and the memory (350) of FIG. 3, respectively. Accordingly, a detailed description of the inertial sensor (1310), the camera (1320), the display (1340), and the memory (1350) will be omitted.

[0229] In one embodiment, comparing FIGS. 3 and 13, some of the functions performed by the controller (330) of FIG. 3 may be performed by the processor (1360). For example, the processor (1360) may further include, in addition to the image processing unit (1363), a motion processing unit (1361) and an ROI determination unit (1362) that perform substantially the same functions as those performed by the motion processing unit (331) and the ROI determination unit (332) included in the controller (330) of FIG. 3.

[0230] In one embodiment, the motion processing unit (1361) may perform substantially the same operation as the motion processing unit (331) included in the controller (330) of FIG. 3. For example, the motion processing unit (1361) may obtain (e.g., calculate) a magnitude and / or direction of a movement of the electronic device (301) (e.g., the camera (1320)) caused by hand tremor, based on sensing data obtained from the inertial sensor (1310). The motion processing unit (1361) may obtain (e.g., calculate) a movement having the same magnitude as the magnitude of the movement of the electronic device (301) and having a direction opposite to the direction of the movement of the electronic device (301), based on the magnitude and / or direction of the movement of the electronic device (301) caused by hand tremor.

[0231] In one embodiment, the ROI determination unit (1362) may perform substantially the same operation as the ROI determination unit (332) included in the controller (330) of FIG. 3. For example, the ROI determination unit (1362) may determine an ROI (first region) for acquiring an image (e.g., a shake-corrected image) within an area of ​​the image sensor (e.g., the entire area of ​​the image sensor). For example, the ROI determination unit (1362) may determine the size of the ROI determined based on a zoom factor associated with the camera (1320). The ROI determination unit (1362) may determine (e.g., calculate) coordinates of an ROI having the determined size (e.g., the size of the ROI determined based on the zoom factor associated with the camera (1320)) within the area of ​​the image sensor based on a movement that is acquired from the motion processing unit (1361) and has an opposite direction to the direction of movement of the electronic device (301).

[0232] In one embodiment, the ROI determination unit (1362) may provide information about the ROI (e.g., coordinates of the ROI) to the pixel value acquisition unit (1331) included in the controller (1330).

[0233] In one embodiment, the pixel value acquisition unit (1331) included in the controller (1330) may perform substantially the same operation as the pixel value acquisition unit (333) included in the controller (330) of FIG. 3. For example, the pixel value acquisition unit (1331) may acquire pixel values ​​of pixels included in an ROI within the image sensor. The pixel value acquisition unit (1331) may control at least a part of an exposure operation for pixels included in the ROI based on the ROI determined by the ROI determination unit. The pixel value acquisition unit (1331) may acquire pixel values ​​of pixels included in the ROI (e.g., only pixel values ​​of pixels included in the ROI) by controlling at least a part of the exposure operation for pixels included in the ROI. The pixel value acquisition unit (1331) may provide the acquired pixel values ​​of pixels included in the ROI to the image processing unit (1363) included in the processor (1360).

[0234] In one embodiment, the pixel value acquisition unit (1331) can acquire pixel values ​​having a Bayer pattern by performing a binning operation or a remosaic operation on pixel values ​​of pixels included in the ROI. The pixel value acquisition unit (1331) can provide pixel values ​​having a Bayer pattern to the processor (1360). However, the present invention is not limited thereto. For example, the pixel value acquisition unit (1331) can provide pixel values ​​having a non-Bayer pattern, rather than a Bayer pattern, to the processor (1360).

[0235] In one embodiment, the image processing unit (1363) may perform an interpolation operation to generate an image (RGB image) to be displayed through the display based on receiving pixel values ​​having a Bayer pattern from the controller (1330) (e.g., pixel value acquisition unit). The image processing unit (1363) may convert pixel values ​​having a non-Bayer pattern into pixel values ​​having a Bayer pattern based on receiving pixel values ​​having a non-Bayer pattern from the controller (1330) (e.g., pixel value acquisition unit (1331)). The image processing unit (1363) may perform an interpolation operation to generate an RGB image based on the converted pixel values. However, the operation performed by the image processing unit (1363) is not limited to the interpolation operation. For example, the image processing unit (1363) may perform tone mapping, color correction, gamma correction, noise removal operation (e.g., noise removal operation using a spatial filter and / or temporal filter), lens shading correction, white balance adjustment, color space conversion, and / or gamut adjustment on an image (e.g., an RGB image).

[0236] FIG. 14 is a block diagram of an electronic device (301) according to one embodiment.

[0237] Referring to FIG. 14, in one embodiment, the electronic device (301) may be the electronic device (101) of FIG. 1.

[0238] In one embodiment, the electronic device (301) may include components for performing OIS in addition to components for performing EIS based on sensing data acquired through the inertial sensor (1310). For example, comparing FIGS. 13 and 14 , compared to the electronic device (301) of FIG. 13 , the electronic device (301) of FIG. 14 may further include an OIS controller (1330).

[0239] In one embodiment, the electronic device (301) may include an inertial sensor (1310), a camera (1320), a controller (1330), a display (1340), a memory (1350), a processor (1360), and an OIS controller (1410).

[0240] In one embodiment, the inertial sensor (1310), the camera (1320), the display (1340), and the memory (1350) may be substantially identical to the inertial sensor (310), the camera (320), the display (340), and the memory (350) of FIG. 4, respectively. Accordingly, a detailed description of the inertial sensor (1310), the camera (1320), the display (1340), and the memory (1350) will be omitted.

[0241] In one embodiment, the memory (1350) may store instructions that, when individually or collectively executed by at least one controller (e.g., controller (1330)) or at least one processor (e.g., processor (1360)), cause the electronic device (301) to perform the operations described through FIGS. 13 through 17.

[0242] In one embodiment, comparing FIGS. 4 and 14, the OIS controller (1410) may perform substantially the same functions as the OIS controller (410) of FIG. 4. For example, the motion processing unit (1411), the movement amount processing unit (1412), and the driving unit (1413) may be substantially the same as the motion processing unit (411), the movement amount processing unit (412), and the driving unit (413) included in the OIS controller (410) of FIG. 4, respectively. Accordingly, a detailed description of the motion processing unit (1411), the movement amount processing unit (1412), and the driving unit (1413) will be omitted.

[0243] In one embodiment, comparing FIGS. 4 and 14, some of the functions performed by the controller (330) of FIG. 4 may be performed by the processor (1360). For example, the processor (1360) may further include, in addition to the image processing unit (1363), a motion processing unit (1361), a motion amount processing unit (1364), and an ROI determination unit (1362), which perform the same functions as the motion processing unit (331), the motion amount processing unit (334), and the ROI determination unit (332) included in the controller (330) of FIG. 4.

[0244] In one embodiment, the motion processing unit (1361) can perform substantially the same operation as the operation performed by the motion processing unit (331) included in the controller (330) of FIG. 4. Accordingly, a detailed description of the motion processing unit (1361) will be omitted.

[0245] In one embodiment, the movement amount processing unit (1364) may perform substantially the same operations as the movement amount processing unit (334) included in the controller (330) of FIG. 4. For example, the movement amount processing unit (1364) may obtain the position of the lens and / or the position of the image sensor based on sensing data provided from a Hall sensor included in the camera (1320).

[0246] In one embodiment, the ROI determination unit (1362) may perform substantially the same operation as the ROI determination unit (332) included in the controller (1330) of FIG. 4. For example, the ROI determination unit (1362) may obtain, from the motion processing unit (1361), a movement having the same magnitude as the magnitude of the movement of the electronic device (301) caused by hand tremor and having a direction opposite to the direction of the movement of the electronic device (301). The ROI determination unit (1362) may obtain, from the movement amount processing unit (1364), the position of the lens and / or the position of the image sensor (e.g., the current position of the lens). The ROI determination unit (1362) may calculate the coordinates of the ROI based on the movement having the direction opposite to the direction of the movement of the electronic device (301), taking into account the position of the lens and / or the position of the image sensor.

[0247] In one embodiment, the ROI determination unit (1362) may provide information about the ROI (e.g., coordinates of the ROI) to the pixel value acquisition unit (1331) included in the controller (1330).

[0248] In one embodiment, the pixel value acquisition unit (1331) included in the controller (1330) may perform substantially the same operation as the pixel value acquisition unit (333) included in the controller (330) of FIG. 4. Accordingly, a detailed description of the pixel value acquisition unit (1331) will be omitted.

[0249] In one embodiment, the image processing unit (1363) included in the processor (1360) may perform operations substantially identical to those performed by the image processing unit (361) included in the processor (360) of FIG. 4. Accordingly, a detailed description of the image processing unit (1331) will be omitted.

[0250] FIG. 15 is a flowchart (1500) illustrating a method of providing an image according to one embodiment.

[0251] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0252] In one embodiment, FIG. 15 may be a drawing for explaining operations performed by the electronic device (301) of FIG. 13 to provide an image.

[0253] Referring to FIG. 15, in operation 1501, in one embodiment, the processor (1360) may obtain the magnitude and direction of movement of the electronic device (301) caused by hand tremor based on sensing data obtained through the inertial sensor (1310) while the camera application is running.

[0254] The operation performed by the processor (1360) in operation 1501 is at least partially the same as or similar to the operation performed by the controller (330) in operation 501 of FIG. 5, so a detailed description thereof will be omitted.

[0255] In operation 1503, in one embodiment, the processor (1360) may determine a first region of interest (ROI) for acquiring an image within the area of ​​the image sensor based on the magnitude and direction of the movement of the electronic device (301).

[0256] The operation performed by the processor (1360) in operation 1503 is at least partially the same as or similar to the operation performed by the controller (330) in operation 503 of FIG. 5, so a detailed description thereof will be omitted.

[0257] In operation 1505, in one embodiment, the processor (1360) may provide information about the first region to the controller (1330). For example, the processor (1360) may provide coordinates of the first region within the entire region of the image sensor to the controller (1330).

[0258] In one embodiment, the controller (1330) may control at least a portion of an exposure operation for pixels included in the first region based on information about the first region received from the processor (1360).

[0259] In one embodiment, the controller (1330) can obtain pixel values ​​of pixels included in the first area by controlling at least a portion of an exposure operation for the pixels included in the first area. For example, the controller (1330) can obtain pixel values ​​of pixels included in the first area through an exposure operation and a read out operation for the pixels included in the first area.

[0260] In operation 1507, in one embodiment, the processor (1360) may receive pixel values ​​from the controller (1330). For example, the processor (1360) may receive pixel values ​​of pixels included in a first region acquired by the controller (1330) from the controller (1330). For example, the processor (1360) may receive pixel values ​​of only pixels included in the first region acquired by the controller (1330) from the controller (1330).

[0261] In one embodiment, the processor (1360) may receive pixel values ​​having a Bayer pattern or a non-Bayer pattern from the controller (1330).

[0262] In one embodiment, the processor (1360) may perform an interpolation operation to generate an image (RGB image) to be displayed through the display based on receiving pixel values ​​having a Bayer pattern from the controller (1330).

[0263] In one embodiment, the processor (1360) may perform tone mapping, color correction, gamma correction, noise removal operations (e.g., noise removal operations using spatial filters and / or temporal filters), lens shading correction, white balance adjustment, color space conversion, and / or gamut adjustment on an image (e.g., an RGB image).

[0264] In one embodiment, the processor (1360) may display an image (e.g., an RGB image) via a display.

[0265] FIG. 16 is a flowchart (1600) illustrating a method of providing an image according to one embodiment.

[0266] In one embodiment, FIG. 16 may be a drawing for explaining operations performed by the electronic device (301) of FIG. 14 to provide an image.

[0267] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0268] Referring to FIG. 16, in operation 1601, in one embodiment, the processor (1360) may obtain the position of the lens and / or the position of the image sensor.

[0269] The operation performed by the processor (1360) in operation 1601 is at least partially the same as or similar to the operation performed by the controller (330) in operation 1201 of FIG. 12, so a detailed description thereof will be omitted.

[0270] In operation 1603, in one embodiment, the processor (1360) may obtain the magnitude and direction of movement of the electronic device (301).

[0271] The operation performed by the processor (1360) in operation 1603 is at least partially the same as or similar to the operation performed by the controller (330) in operation 1203 of FIG. 12, so a detailed description thereof will be omitted.

[0272] In operation 1605, in one embodiment, the processor (1360) may determine a first area for acquiring a shake-compensated image within the area of ​​the image sensor based at least in part on the position of the lens and / or the position of the image sensor and the magnitude and direction of movement of the electronic device (301).

[0273] The operation performed by the processor (1360) in operation 1605 is at least partially identical or similar to the operation performed by the controller (330) in operation 1205 of FIG. 12, so a detailed description thereof will be omitted.

[0274] In operation 1607, in one embodiment, the processor (1360) may provide information about the first region to the controller (1330). For example, the processor (1360) may provide coordinates of the first region within the entire region of the image sensor to the controller (1330).

[0275] Since operation 1605 is at least partially identical or similar to operation 1505 of FIG. 15, a detailed description thereof will be omitted.

[0276] In operation 1609, in one embodiment, the processor (1360) may receive pixel values ​​from the controller (1330). For example, the processor (1360) may receive pixel values ​​of pixels included in a first region acquired by the controller (1330) from the controller (1330). For example, the processor (1360) may receive pixel values ​​of pixels included only in the first region acquired by the controller (1330) from the controller (1330).

[0277] Since operation 1609 is at least partially identical or similar to operation 1507 of FIG. 15, a detailed description thereof will be omitted.

[0278] FIG. 17 is a diagram illustrating a method for providing an image according to one embodiment.

[0279] Referring to FIG. 17, in one embodiment, FIG. 17 may be a diagram illustrating an anyplace zoom function.

[0280] In one embodiment, the anyplace zoom function may be a function that provides an image for an area including a subject (e.g., an object corresponding to a subject to be depicted by a camera). For example, the anyplace zoom function may be a function that can obtain an image representing a scene for a moving subject when the subject moves through an area (hereinafter referred to as a “zoom area” or “region of interest (ROI)”) having a size corresponding to a zoom factor (e.g., 2.0x) within the entire area of ​​the image sensor.

[0281] In one embodiment, when the anyplace zoom function is executed, the resolution of the image acquired through the zoom area may be substantially the same as the resolution of the image acquired through the entire area of ​​the image sensor. For example, when the anyplace zoom function is executed, a re-mosaic operation is performed on the image acquired through the zoom area, so that an image having a resolution substantially the same as the resolution of the image acquired through the entire area of ​​the image sensor and including an object corresponding to the subject may be acquired.

[0282] In one embodiment, reference numeral 1701 of FIG. 17 may represent zoom regions that are sequentially set on a frame-by-frame basis within the entire region (1710) of the image sensor in the camera. For example, ROI 1 (1711), ROI 2 (1712), and ROI 3 (1713) may be sequentially set on a frame-by-frame basis within the entire region (1710) of the image sensor.

[0283] In one embodiment, reference numeral 1702 of FIG. 17 may be a graph representing an operation of acquiring images (e.g., pixel values ​​acquired from each of ROI 1 (1711), ROI 2 (1712), and ROI 3 (1713)) from each of ROI 1 (1711), ROI 2 (1712), and ROI 3 (1713).

[0284] In one embodiment, the X-axis in reference numeral 1702 may represent time (t). The Y-axis in reference numeral 1702 may represent rows of pixels (hereinafter also referred to as “pixel rows”) included in an entire area (1710) of an image sensor included in a camera operating by a rolling shutter method. In reference numeral 1702, a largest value among the Y-axis values ​​may represent a row positioned at the top of the image sensor among the rows of pixels included in the entire area (1710) of the image sensor, and a smallest value among the Y-axis values ​​may represent a row positioned at the bottom of the image sensor among the rows of pixels included in the entire area (1710) of the image sensor.

[0285] In one embodiment, line (1721-1) may represent the time points at which exposure for each of the pixel rows included in ROI 1 (1711) begins, and line (1721-2) may represent the time points at which readout for each of the pixel rows included in ROI 1 (1711) begins. Region (1721-3) may represent the times at which exposure operations are performed for the pixel rows included in ROI 1 (1711). For example, exposure operations for pixel row a may be performed from time (t1) to time (t2).

[0286] In one embodiment, line (1731-1) may represent the time points at which exposure for each of the pixel rows included in ROI 2 (1712) begins, and line (1731-2) may represent the time points at which readout for each of the pixel rows included in ROI 2 (1712) begins. Region (1731-3) may represent the times at which exposure operations are performed for the pixel rows included in ROI 2 (1712).

[0287] In one embodiment, line (1741-1) may represent the time points at which exposure begins for each of the pixel rows included in ROI 3 (1713), and line (1741-2) may represent the time points at which readout begins for each of the pixel rows included in ROI 3 (1713). Region (1741-3) may represent the times at which exposure operations are performed for the pixel rows included in ROI 3 (1713).

[0288] In one embodiment, a portion of the exposure time for ROI 2 (1712) and a portion of the exposure time for ROI 3 (1713) may overlap, such as in the area (1750) indicated by the dotted line in reference numeral 1702. For example, the exposure of the bottommost pixel row of ROI 2 (1712) may end at time (t4), and the exposure of the topmost pixel row of ROI 3 (1713) may begin at time (t3) earlier than time (t4). In this case, a frame drop may occur, in which some of the pixel values ​​acquired from ROI 2 (1712) or at least one of the pixel values ​​acquired from ROI 3 (1713) are not acquired.

[0289] In one embodiment, methods for preventing (or minimizing) frame drops may include reducing the exposure time, reducing the camera's FPS, and reducing the size of the ROI. However, reducing the exposure time may result in images with lower brightness. Reducing the camera's FPS may also present challenges, as it may be difficult to achieve a frame rate below approximately 30 FPS.

[0290] In one embodiment, when acquiring pixel values ​​of pixels included in an area including a margin area within the entire area of ​​the image sensor, the size of the area including the margin area may be larger than the size of a first area (e.g., a first area not including the margin area) determined by considering movement of the electronic device (301) caused by hand shake within the entire area of ​​the image sensor, as described with reference to FIGS. 3 to 16. Accordingly, by determining the first area by considering movement of the electronic device (301) caused by hand shake within the entire area of ​​the image sensor, frame drops can be prevented (or minimized).

[0291] An electronic device (301) according to one embodiment may include an inertial sensor (310), a camera (320) including an image sensor, a processor (360), and a controller (330). The controller (330) may be configured to acquire a magnitude and direction of movement of the electronic device (301) caused by hand tremor based on sensing data acquired through the inertial sensor (310) while a camera application is running. The controller (330) may be configured to determine a first region for acquiring an image within the entire region of the image sensor based on the magnitude and direction of the movement of the electronic device (301). The controller (330) may be configured to acquire pixel values ​​of pixels included in the first region. The controller (330) may be configured to provide the acquired pixel values ​​to the processor (360).

[0292] In one embodiment, the controller (330) may be configured to obtain pixel values ​​of a Bayer pattern by performing a pixel binning operation or a remosaic operation based on the pixel values. The controller (330) may be configured to provide the pixel values ​​of the Bayer pattern to the processor (360).

[0293] In one embodiment, the controller (330) may be configured to obtain the pixel values ​​of the non-Bayer pattern. The controller (330) may be configured to provide the pixel values ​​of the non-Bayer pattern to the processor (360), and the processor (360) may be configured to convert the pixel values ​​of the non-Bayer pattern into pixel values ​​of the Bayer pattern.

[0294] In one embodiment, the controller (330) may be configured to obtain pixel values ​​corresponding to each of the pixels by controlling the camera (320) to perform an exposure operation and a read-out operation for the pixels included in the first area.

[0295] In one embodiment, the controller (330) may be configured to determine a size of the first area, which is determined based on a zoom factor associated with the camera (320). The controller (330) may be configured to determine the first area for acquiring the image in which the shake is corrected within the entire area of ​​the image sensor, based on the determined size of the first area and the magnitude and opposite direction of the movement of the electronic device (301).

[0296] In one embodiment, the electronic device (301) may further include an OIS controller (410) configured to perform OIS (optical image stabilization) of the camera (320) and a sensor (e.g., a Hall sensor (321)) configured to obtain a position of a lens included in the camera (320) and / or a position of the image sensor. The OIS controller (410) may be configured to perform an operation of compensating for a movement of the electronic device (301) caused by the hand shake using the inertial sensor (310) and the sensor. The controller (330) may be configured to determine the first area based on the position of the lens and / or the position of the image sensor obtained through the sensor, and the magnitude and direction of the movement of the electronic device (301).

[0297] In one embodiment, the electronic device (301) may include a camera other than the camera (320). The controller (330) may be configured to recognize an object within a first image acquired through the other camera. The controller (330) may be configured to identify an area including the recognized object within the first image. The controller (330) may be configured to determine the first area based on an area corresponding to the identified area within the entire area of ​​the image sensor and the magnitude and direction of movement of the electronic device (301).

[0298] In one embodiment, the electronic device (301) may include a camera other than the camera (320). The controller (330) may be configured to acquire the magnitude and direction of movement of the electronic device (301) based on second images acquired through the other camera. The controller (330) may be configured to determine the first area based on the magnitude and direction of movement of the electronic device (301) acquired based on the second images.

[0299] In one embodiment, the controller (330) may be included in the camera (320).

[0300] According to one embodiment, a method for providing an image in an electronic device (301) may include an operation of acquiring, by a controller (330) of the electronic device (301), a magnitude and direction of a movement of the electronic device (301) caused by hand tremor based on sensing data acquired through an inertial sensor (310) of the electronic device (301) while a camera application is running. The method may include an operation of determining, by the controller (330) of the electronic device (301), a first area for acquiring an image within an entire area of ​​an image sensor included in a camera (320) of the electronic device (301) based on the magnitude and direction of the movement of the electronic device (301). The method may include an operation of acquiring, by the controller (330) of the electronic device (301), pixel values ​​of pixels included in the first area. The method may include an operation of providing the acquired pixel values ​​to the processor (360) of the electronic device (301) by the controller (330) of the electronic device (301).

[0301] In one embodiment, the operation of obtaining pixel values ​​of pixels included in the first area may include an operation of obtaining pixel values ​​of a Bayer pattern by performing a pixel binning operation or a remosaic operation based on the pixel values. The operation of providing the obtained pixel values ​​to the processor (360) of the electronic device (301) may include an operation of providing pixel values ​​of the Bayer pattern to the processor (360).

[0302] In one embodiment, the operation of obtaining pixel values ​​of pixels included in the first area may include an operation of obtaining pixel values ​​of a non-Bayer pattern. The operation of providing the obtained pixel values ​​to the processor (360) of the electronic device (301) may include an operation of providing the pixel values ​​of the non-Bayer pattern to the processor (360). The method may further include an operation of converting the pixel values ​​of the non-Bayer pattern into pixel values ​​of the Bayer pattern by the processor (360).

[0303] In one embodiment, the operation of obtaining pixel values ​​of pixels included in the first area may include an operation of obtaining pixel values ​​corresponding to each of the pixels by controlling the camera (320) to perform an exposure operation and a read-out operation for the pixels included in the first area.

[0304] In one embodiment, the method may further include an operation of determining a size of the first area, which is determined based on a zoom magnification associated with the camera (320). The operation of determining the first area may include an operation of determining the first area for obtaining the image in which the movement of the electronic device (301) caused by the hand shake is corrected within the entire area of ​​the image sensor, based on the determined size of the first area and the magnitude and opposite direction of the movement of the electronic device (301).

[0305] In one embodiment, the method may further include an operation in which the OIS controller (410) configured to perform OIS of the camera (320) performs an operation of correcting a movement of the electronic device (301) caused by the hand shake by using the inertial sensor (310) and a sensor (e.g., a Hall sensor (321)) configured to obtain a position of a lens included in the camera (320) and / or a position of the image sensor. The operation of determining the first area may include an operation of determining the first area based on the position of the lens and / or the position of the image sensor obtained through the sensor, and the magnitude and direction of the movement of the electronic device (301).

[0306] In one embodiment, the method may further include an operation of recognizing an object within a first image acquired through another camera. The method may further include an operation of identifying an area within the first image that includes the recognized object. The operation of determining the first area may include an operation of determining the first area based on an area corresponding to the identified area within the entire area of ​​the image sensor and a magnitude and direction of movement of the electronic device (301).

[0307] In one embodiment, the method may further include an operation of acquiring a magnitude and direction of movement of the electronic device (301) based on second images acquired through the other camera. The operation of determining the first area may include an operation of determining the first area based on the magnitude and direction of movement of the electronic device (301) acquired based on the second images.

[0308] In one embodiment, the controller (330) may be included in the camera (320).

[0309] An electronic device (301) according to one embodiment may include an inertial sensor (1310), a camera (1320) including an image sensor, a controller (1330), and a processor (1360). The processor (1360) may be configured to acquire a magnitude and direction of movement of the electronic device (301) caused by hand tremor based on sensing data acquired through the inertial sensor (1310) while a camera application is running. The processor (1360) may be configured to determine a first region for acquiring an image within the entire region of the image sensor based on the magnitude and direction of the movement of the electronic device (301). The processor (1360) may provide information about the first region to the controller (1330) so that the controller (1330) may acquire pixel values ​​of pixels included in the first region. The processor (1360) may be configured to receive pixel values ​​acquired by the controller (1330) from the controller (1330).

[0310] In one embodiment, the electronic device (301) may further include an OIS controller (1410) configured to perform OIS of the camera, and a sensor (e.g., a Hall sensor) configured to obtain a position of a lens included in the camera (320) and / or a position of the image sensor. The OIS controller (1410) may be configured to perform an operation of correcting a movement of the electronic device (301) caused by the hand shake through and using the inertial sensor (1310). The processor (1360) may be configured to determine the first area based on the position of the lens and / or the position of the image sensor obtained through the sensor, and the magnitude and direction of the movement of the electronic device (301).

[0311] Additionally, the structure of the data used in the embodiments of the present disclosure described above can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0312] According to one aspect of the present disclosure relating to the electronic device, the electronic device may include an inertial sensor, a camera including an image sensor, a processor, and a controller.

[0313] The controller may be configured to acquire the magnitude and direction of movement of the electronic device caused by hand tremor based on sensing data acquired through the inertial sensor while the camera application is running. The controller may be configured to determine a first area for acquiring an image within the entire area of ​​the image sensor based on the magnitude and direction of the movement of the electronic device. The controller may be configured to acquire pixel values ​​of only pixels included in the first area. The controller may be configured to provide the acquired pixel values ​​to the processor.

[0314] Alternatively or additionally, the processor may be configured to acquire a magnitude and direction of movement of the electronic device caused by hand tremor based on sensing data acquired through the inertial sensor while the camera application is running. The processor may be configured to determine a first region for acquiring an image within the entire region of the image sensor based on the magnitude and direction of the movement of the electronic device. The processor may provide information about the first region to the controller so that the controller acquires pixel values ​​of pixels included in the first region. The processor may be configured to receive the pixel values ​​acquired by the controller from the controller.

[0315] The first region may be smaller than the entire region including all pixels of the image sensor. The first region may be a sub-region of the image sensor. The first region may be a sub-region of the entire region including all pixels of the image sensor.

[0316] According to one aspect of the present disclosure relating to the electronic device, the controller may be configured to obtain pixel values ​​of a Bayer pattern by performing a pixel binning operation or a remosaic operation based on the pixel values, and may be configured to provide the pixel values ​​of the Bayer pattern to the processor.

[0317] According to one aspect of the present disclosure relating to the electronic device, the controller may be configured to obtain the pixel values ​​of the non-Bayer pattern and provide the pixel values ​​of the non-Bayer pattern to the processor, wherein the processor may be configured to convert the pixel values ​​of the non-Bayer pattern into pixel values ​​of the Bayer pattern.

[0318] According to one aspect of the present disclosure relating to the electronic device, the controller may be configured to obtain the pixel values ​​corresponding to each of the pixels by controlling the camera to perform an exposure operation and a read-out operation for the pixels included in the first area.

[0319] According to one aspect of the present disclosure relating to the electronic device, the controller may be configured to determine a size of the first area, which is determined based on a zoom factor associated with the camera. The controller may be configured to determine the first area for acquiring the image in which the shake is corrected within the entire area of ​​the image sensor, based on the determined size of the first area and the magnitude and opposite direction of the movement of the electronic device.

[0320] According to one aspect of the present disclosure relating to the electronic device, the electronic device may further include an OIS controller configured to perform optical image stabilization (OIS) of the camera. The electronic device may further include a sensor configured to acquire a position of a lens included in the camera and / or a position of the image sensor. The OIS controller may be configured to perform an operation of compensating for a movement of the electronic device caused by the hand shake using the inertial sensor and the sensor. The controller may be configured to determine the first area based on the position of the lens and / or the position of the image sensor acquired through the sensor, and the magnitude and direction of the movement of the electronic device.

[0321] According to one aspect of the present disclosure relating to the electronic device, the electronic device may include a camera other than the camera. The controller may be configured to recognize an object in a first image acquired through the other camera, identify an area including the recognized object in the first image, and determine the first area based on an area corresponding to the identified area within the entire area of ​​the image sensor and a magnitude and direction of movement of the electronic device.

[0322] According to one aspect of the present disclosure relating to the electronic device, the electronic device may include a camera other than the camera. The controller may be configured to acquire a magnitude and direction of movement of the electronic device based on second images acquired through the other camera, and determine the first area based on the magnitude and direction of movement of the electronic device acquired based on the second images.

[0323] According to one aspect of the present disclosure relating to the electronic device, the controller may be included in the camera.

[0324] According to another aspect of the present disclosure relating to a method for providing an image by an electronic device, the method may include, for example, an operation of acquiring, by a controller of the electronic device, a magnitude and direction of a movement of the electronic device caused by hand tremor, based on sensing data acquired, for example, through an inertial sensor of the electronic device, while a camera application is running. The method may include, for example, an operation of determining, by the controller of the electronic device, a first region for acquiring an image within an entire region of an image sensor included in a camera of the electronic device, based on the magnitude and direction of the movement of the electronic device. The method may include, for example, an operation of acquiring, by the controller of the electronic device, pixel values ​​of pixels included in the first region. The method may include, for example, an operation of providing, by the controller of the electronic device, the acquired pixel values ​​to, for example, a processor of the electronic device. The first region may be smaller than an entire region including all pixels of the image sensor. The first region may be a sub-region of the image sensor. The first region may be a sub-region of the entire region including all pixels of the image sensor.

[0325] According to another aspect of the present disclosure related to the method, the operation of obtaining pixel values ​​of pixels included in the first area may include an operation of obtaining pixel values ​​of a Bayer pattern by performing a pixel binning operation or a remosaic operation based on the pixel values. The operation of providing the obtained pixel values ​​to a processor of the electronic device may include an operation of providing pixel values ​​of the Bayer pattern to the processor.

[0326] According to another aspect of the present disclosure related to the method, the operation of obtaining pixel values ​​of pixels included in the first area may include the operation of obtaining the pixel values ​​of a non-Bayer pattern. The operation of providing the obtained pixel values ​​to a processor of the electronic device may include the operation of providing the pixel values ​​of the non-Bayer pattern to the processor. The processor may convert the pixel values ​​of the non-Bayer pattern into pixel values ​​of a Bayer pattern.

[0327] According to another aspect of the present disclosure related to the method, the operation of obtaining pixel values ​​of pixels included in the first area may include an operation of obtaining the pixel values ​​corresponding to each of the pixels by controlling the camera to perform an exposure operation and a read-out operation for the pixels included in the first area.

[0328] According to another aspect of the present disclosure related to the method, the method may further include an operation of determining a size of the first area, which is determined based on a zoom factor associated with the camera. The operation of determining the first area may include an operation of determining the first area for obtaining the image in which the movement of the electronic device caused by the hand shake is corrected within the entire area of ​​the image sensor, based on the determined size of the first area and the magnitude and opposite direction of the movement of the electronic device.

[0329] According to another aspect of the present disclosure related to the method, the method may further include an operation in which an OIS controller configured to perform OIS of the camera performs an operation of compensating for a movement of the electronic device caused by the hand shake using the inertial sensor and a sensor configured to acquire a position of a lens included in the camera and / or a position of the image sensor. The operation of determining the first area may include an operation of determining the first area based on the position of the lens and / or the position of the image sensor acquired through the sensor and the magnitude and direction of the movement of the electronic device.

[0330] According to another aspect of the present disclosure related to the method, the method may further include an operation of recognizing an object within a first image acquired through another camera. The method may further include an operation of identifying an area including the recognized object within the first image. The operation of determining the first area may include an operation of determining the first area based on an area corresponding to the identified area within the entire area of ​​the image sensor and a magnitude and direction of movement of the electronic device.

[0331] According to another aspect of the present disclosure related to the method, the method may further include an operation of acquiring a magnitude and direction of movement of the electronic device based on second images acquired through the other camera. The operation of determining the first area may include an operation of determining the first area based on the magnitude and direction of movement of the electronic device acquired based on the second images.

[0332] In one embodiment, the controller may be included in the camera.

[0333] According to another aspect of the present disclosure related to the electronic device, the electronic device may include an inertial sensor, a camera including an image sensor, a controller, and a processor. The processor may be configured to acquire a magnitude and direction of a movement of the electronic device caused by hand tremor based on sensing data acquired through the inertial sensor while a camera application is running. The processor may be configured to determine a first region for acquiring an image within the entire region of the image sensor based on the magnitude and direction of the movement of the electronic device. The processor may provide information about the first region to the controller so that the controller acquires pixel values ​​of pixels included in the first region. The processor may be configured to receive the pixel values ​​acquired by the controller from the controller.

[0334] According to one aspect of the present disclosure relating to the electronic device, in particular, according to another aspect of the present disclosure, the electronic device may include an OIS controller configured to perform OIS of the camera. The electronic device may include a sensor configured to acquire a position of a lens included in the camera and / or a position of the image sensor. The OIS controller may be configured to perform an operation of compensating for a movement of the electronic device caused by the hand shake through and using the inertial sensor. The processor may be configured to determine the first area based on the position of the lens and / or the position of the image sensor acquired through the sensor, and the magnitude and direction of the movement of the electronic device.

[0335] The first region may be smaller than the entire region including all pixels of the image sensor. The first region may be a sub-region of the image sensor. The first region may be a sub-region of the entire region including all pixels of the image sensor.

[0336] Remosaicing, also known as demosaicing or color reconstruction, can be a digital image processing algorithm used to reconstruct a full color image from incomplete color samples output from an image sensor overlaid on a color filter array (CFA), such as a Bayer filter. This may also be known as CFA interpolation or debayering. Tetracell technology may also be known as Quad Bayer or 4-cell technology.

Claims

1. In the electronic device (301), Inertial sensor (310); A camera (320) including an image sensor; Processor (360); and A controller (330) is included, wherein the controller (330) comprises: While the camera application is running, based on the sensing data acquired through the inertial sensor (310), the size and direction of the movement of the electronic device (301) caused by hand tremor are acquired, Based on the magnitude and direction of the movement of the electronic device (301), a first area for acquiring an image within the entire area of ​​the image sensor is determined, Obtaining pixel values ​​of pixels included in the first region, and An electronic device (301) configured to provide the acquired pixel values ​​to the processor (360).

2. In paragraph 1, The above controller (330): By performing a pixel binning operation or a remosaic operation based on the above pixel values, pixel values ​​of a Bayer pattern are obtained, and An electronic device (301) configured to provide pixel values ​​of the above Bayer pattern to the processor (360).

3. In paragraph 1 or 2, The above controller (330): Obtain the above pixel values ​​of the non-Bayer pattern, and configured to provide the pixel values ​​of the non-Bayer pattern to the processor (360), The processor (360) is an electronic device (301) configured to convert the pixel values ​​of the non-Bayer pattern into pixel values ​​of the Bayer pattern.

4. In any one of paragraphs 1 to 3, The above controller (330): An electronic device (301) configured to obtain pixel values ​​corresponding to each of the pixels by controlling the camera (320) to perform an exposure operation and a read out operation for the pixels included in the first area.

5. In any one of paragraphs 1 to 4, The above controller (330) Check the size of the first area, which is determined based on the zoom ratio associated with the camera (320), An electronic device (301) configured to determine the first area for obtaining the image in which the movement of the electronic device (301) caused by the hand shake within the area of ​​the image sensor is corrected based on the size of the first area confirmed above, the size of the movement of the electronic device (301) and the opposite direction of the direction.

6. In any one of paragraphs 1 to 5, An OIS controller (410) configured to perform OIS (optical image stabilization) of the above camera (320); and Further comprising a sensor configured to obtain the position of the lens included in the camera (320) and / or the position of the image sensor, The OIS controller (410) is configured to perform an operation of compensating for movement of the electronic device (301) caused by hand tremor using the inertial sensor (310) and the sensor. The above controller (330): An electronic device (301) configured to determine the first area based on the position of the lens and / or the position of the image sensor obtained through the sensor, and the magnitude and direction of movement of the electronic device (301).

7. In any one of paragraphs 1 to 6, Including a camera other than the above camera (320), The above controller (330): Recognize an object within a first image acquired through the other camera, Identifying an area containing the recognized object within the first image, and An electronic device (301) configured to determine the first area based on the area corresponding to the identified area within the entire area of ​​the image sensor and the magnitude and direction of movement of the electronic device (301).

8. In any one of paragraphs 1 to 7, Including a camera other than the above camera (320), The above controller (330): Obtaining the size and direction of movement of the electronic device (301) based on the second images acquired through the other camera, and An electronic device (301) configured to determine the first area based on the magnitude and direction of movement of the electronic device (301) obtained based on the second images.

9. In any one of paragraphs 1 to 8, The above controller (330) is an electronic device (301) included in the above camera (320).

10. In a method for providing an image from an electronic device (301), An operation of acquiring the size and direction of movement of the electronic device (301) caused by hand tremor based on sensing data acquired through the inertial sensor (310) of the electronic device (301) while the camera application is running by the controller (330) of the electronic device (301); An operation of determining a first area for acquiring an image within the entire area of ​​an image sensor included in a camera (320) of the electronic device (301) based on the size and direction of movement of the electronic device (301) by the controller (330) of the electronic device (301); An operation of obtaining pixel values ​​of pixels included in the first area by the controller (330) of the electronic device (301); and A method including an operation of providing the acquired pixel values ​​to a processor (360) of the electronic device (301) by the controller (330) of the electronic device (301).

11. In paragraph 10, The operation of obtaining pixel values ​​of pixels included in the first region includes an operation of obtaining pixel values ​​of a Bayer pattern by performing a pixel binning operation or a remosaic operation based on the pixel values. A method in which the operation of providing the acquired pixel values ​​to the processor (360) of the electronic device (301) includes the operation of providing the pixel values ​​of the Bayer pattern to the processor (360).

12. In paragraph 10 or 11, The operation of obtaining pixel values ​​of pixels included in the first region includes an operation of obtaining pixel values ​​of a non-Bayer pattern, and The operation of providing the acquired pixel values ​​to the processor (360) of the electronic device (301) includes the operation of providing the pixel values ​​of the non-Bayer pattern to the processor (360), A method further comprising an operation of converting the pixel values ​​of the non-Bayer pattern into pixel values ​​of the Bayer pattern by the processor (360).

13. In any one of paragraphs 10 to 12, A method in which the operation of obtaining pixel values ​​of pixels included in the first area includes an operation of obtaining pixel values ​​corresponding to each of the pixels by controlling the camera (320) so that the camera (320) performs an exposure operation and a read-out operation for the pixels included in the first area.

14. In any one of paragraphs 10 to 13, Further comprising an operation of checking the size of the first area, which is determined based on a zoom ratio associated with the camera (320); A method in which the operation of determining the first area includes an operation of determining the first area for obtaining the image in which the movement of the electronic device (301) caused by the hand shake within the area of ​​the image sensor is corrected based on the size of the confirmed first area and the magnitude and opposite direction of the movement of the electronic device (301).

15. A computer-readable recording medium comprising instructions that, when executed by a controller (330) of an electronic device (301) including an inertial sensor (310), a camera (320) including an image sensor, a controller (330), and a processor (360), cause the electronic device (301) to execute the method of any one of claims 10 to 14.

Images