Electronic device including camera module and operating method thereof

Abstract

An electronic device according to various embodiments includes: a motion sensor for outputting a signal for acquiring motion data corresponding to a motion of the electronic device; a camera module including an image sensor and an OIS module for performing optical image stabilization (OIS) based on a motion of the electronic device; and at least one processor operably connected to the motion sensor and the camera module, where the at least one processor determines motion state information based on a size of the motion data if an exposure time of the image sensor is less than or equal to a predetermined exposure time while the image frames are continuously acquired by the camera module, a suppression ratio related to a driving range of the OIS module is determined based on the motion state information, and an operation of the OIS module is controlled based on the determined suppression ratio.

Description

Technical Field

[0001] The present disclosure relates to an electronic device including a camera module and a control method thereof. Background Art

[0002] As photographing functions of mobile devices have been recently enhanced, technology for correcting photographs or videos taken by using the mobile devices is being developed.

[0003] For example, mobile devices are provided with optical image stabilization (OIS) and video digital image stabilization (VDIS) functions to enable enhanced photos or videos to be obtained.

[0004] Here, OIS is a method for reducing shaking by shifting a lens assembly or an image sensor included in a camera module, and VDIS is a method for reducing shaking through digital processing in a mobile device. Summary of the invention

[0005] Technical issues

[0006] Electronic devices can solve defects caused by vibration during video recording through the OIS function. However, the movement range of the lens assembly or image sensor for the OIS function is limited. For example, the maximum value of the physical correction angle for performing the OIS function is 1 degree. In this case, the electronic device may not be able to cover large movements of the electronic device due to the limitation on the correction angle.

[0007] To address the above-described limitations, the electronic device may improve jitter by adaptively adjusting a rejection ratio according to a relative position of a lens assembly or an image sensor.

[0008] Electronic devices can solve the problem of jitter to a certain extent by adaptively adjusting the suppression ratio. However, it may be difficult for electronic devices to quickly apply the suppression ratio every moment, which leads to the problem of micro-shaking.

[0009] In this case, the electronic device may enlarge the maximum value of a physical correction angle for performing the OIS function, but may have problems caused by the enlargement of the correction angle.

[0010] Solution to the problem

[0011] According to an embodiment, an electronic device may include: a motion sensor configured to output a signal for acquiring motion data corresponding to the motion of the electronic device; a camera module including an image sensor and an optical image stabilization (OIS) module, wherein the OIS module is configured to perform OIS based on the motion of the electronic device; and at least one processor operably connected to the motion sensor and the camera module, and the at least one processor may determine motion state information based on the size of the motion data when an exposure time of the image sensor is less than or equal to a predefined exposure time while image frames are continuously acquired through the camera module, may determine a suppression ratio related to an actuation range of the OIS module based on the motion state information, and may control the operation of the OIS module based on the determined suppression ratio.

[0012] According to an embodiment, the electronic device may further include a memory, the motion state information may include the size of the motion data based on the motion level in which it is divided stepwise, the memory may store a table in which the suppression ratio is assigned by motion level, and at least one processor may determine the suppression ratio on the OIS module based on the stored table.

[0013] According to an embodiment, when the exposure time is longer than a predefined exposure time, the at least one processor may adaptively adjust the suppression ratio based on the correction angle position of the OIS module.

[0014] According to an embodiment, when the exposure time is less than or equal to a predefined exposure time and the motion level is greater than or equal to a predefined value, the at least one processor may adaptively adjust the suppression ratio based on the corrected angular position of the OIS module.

[0015] According to an embodiment, when the exposure time is less than or equal to a predefined exposure time and the motion level is greater than or equal to a predefined value, the at least one processor may adaptively adjust the suppression ratio based on the correction angle amount required by the OIS module.

[0016] According to an embodiment, the camera module may expand the actuation range of the OIS module from a first range to a second range.

[0017] According to an embodiment, the at least one processor may acquire the motion level by processing the motion data based on the actuation range of the OIS module expanded from the first range to the second range.

[0018] According to an embodiment, at least one processor may control the OIS module based on a suppression ratio that increases in stages according to a motion level.

[0019] According to an embodiment, the motion sensor may include at least one of an acceleration sensor that outputs a signal for obtaining acceleration data or a gyroscope sensor that outputs a signal for obtaining angular velocity data, and at least one processor may determine a motion level corresponding to the motion of the electronic device based on at least one of the acceleration data or the angular velocity data.

[0020] According to an embodiment, the electronic device may further include an illuminance sensor configured to output a signal for measuring the ambient illuminance of the electronic device; and at least one processor may determine an exposure time based on the measured illuminance, and may obtain a motion level by processing motion data when the exposure time is less than or equal to a predefined exposure time.

[0021] According to an embodiment, an operating method of an electronic device may include: continuously acquiring image frames through a camera module; acquiring motion data corresponding to the motion of the electronic device; when an exposure time of an image sensor is less than or equal to a predefined exposure time while the image frames are continuously acquired, determining motion state information based on the size of the motion data; determining a suppression ratio related to an actuation range of an OIS module based on the motion state information; and controlling the operation of the OIS module based on the determined suppression ratio.

[0022] According to an embodiment, determining the motion state information based on the size of the motion data may include: determining the suppression ratio on the OIS module based on a table in which the suppression ratio is allocated by motion level, and the motion state information may include the size of the motion data based on the motion level divided into stages.

[0023] According to an embodiment, the operating method may further include adaptively adjusting the suppression ratio based on the correction angle position of the OIS module when the exposure time is longer than a predefined exposure time.

[0024] According to an embodiment, the operating method may further include adaptively adjusting the suppression ratio based on the corrected angle position of the OIS module when the exposure time is less than or equal to a predefined exposure time and the motion level is greater than or equal to a predefined value.

[0025] According to an embodiment, the operating method may further include adaptively adjusting the suppression ratio based on an amount of correction angle required on the OIS module when the exposure time is less than or equal to a predefined exposure time and the motion level is greater than or equal to a predefined value.

[0026] According to an embodiment, acquiring the motion level by processing the motion data may include acquiring the motion level by processing the motion data when the actuation range of the OIS module is expanded from the first range to the second range.

[0027] According to an embodiment, an electronic device may include: a motion sensor configured to output a signal for acquiring motion data corresponding to a motion of the electronic device; a camera module including a lens module and configured to perform an OIS function; an OIS module configured to shift the lens module within an actuation range of the OIS function; and at least one processor electrically connected to the motion sensor, the camera module and the OIS module, and the at least one processor may control the OIS module to shift the lens module when motion of the electronic device occurs while image frames are continuously acquired through the camera module, may acquire a correction angle of the OIS function at a position to which the lens module is shifted, may determine a transformation angle for perspective transform of the image frame based on the correction angle, and may perform video digital image stabilization (VDIS) based on the transformation angle.

[0028] According to an embodiment, the OIS module may shift the lens module according to a second range of an actuation range of the OIS function, wherein the second range is greater than the first range.

[0029] According to an embodiment, at least one processor may acquire correction angles at regular intervals while acquiring image frames, and the correction angles may include a first correction angle and a second correction angle acquired after the first correction angle, and at least one processor may determine a transformation angle based on an average value between the first correction angle and the second correction angle.

[0030] According to an embodiment, at least one processor may perform lens distortion correction (LDC) before transforming the image frame based on the transformation angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment;

[0032] Figure 2 is a block diagram showing a camera module according to an embodiment;

[0033] Figure 3 is a control block diagram of an electronic device including a camera module according to an embodiment;

[0034] Figure 4 is a view showing an OIS actuation range in which a camera module performs OIS according to an embodiment;

[0035] Figure 5 is a view showing an example of determining a suppression rate by a camera module according to an embodiment;

[0036] Figure 6is a view showing an example of determining a suppression ratio by a camera module when a required OIS correction amount exceeds an OID actuation range according to an embodiment;

[0037] Figure 7 is a view showing an example of determining a suppression ratio when an OIS maximum amplitude adjustment range of a camera module is expanded according to an embodiment;

[0038] Figure 8 is a flowchart of a method for operating an electronic device according to an embodiment;

[0039] Fig. 9 is a diagram showing the priority between jitter and micro vibration at exposure time;

[0040] Fig.10 is a diagram showing an example of determining a suppression ratio when the exposure time is relatively long;

[0041] Fig.11 is a diagram showing an example of determining a suppression ratio when the exposure time is relatively short;

[0042] Fig.12 is a view showing an OIS operation when there is no movement of the electronic device according to an embodiment;

[0043] Fig.13 is a view showing a state in which an OIS function is turned off when there is a motion of an electronic device according to an embodiment;

[0044] Fig.14 is a view showing a state in which an OIS function is turned on when there is a motion of an electronic device according to an embodiment;

[0045] Fig.15 is a flowchart of a method for operating an electronic device according to an embodiment;

[0046] Fig.16 are views showing examples of perspective transformation; and

[0047] Fig.17 It is shown that the export is used Fig.16 FIG. 1 is a view of an example of the process of perspective transformation of the OIS angle. DETAILED DESCRIPTION

[0048] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0049] refer to Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one component (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added in the electronic device 101. In some embodiments, some components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (eg, display module 160).

[0050] The processor 120 may execute, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of the electronic device 101 coupled to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that may operate independently of the main processor 121 or in conjunction with the main processor 121. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or be specific to a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0051] The auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display module 160, sensor module 176, or communication module 190) among the components of the electronic device 101 instead of the main processor 121 when the main processor 121 is in an inactive (e.g., sleep) state, or control at least some of the functions or states related to at least one component (e.g., display module 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121 when the main processor 121 is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as a part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be generated by machine learning. Such learning may be performed, for example, by the electronic device 101 in which artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. The artificial intelligence model may additionally or alternatively include a software structure in addition to a hardware structure.

[0052] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0053] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0054] The input module 150 may receive a command or data to be used by another component (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0055] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing recordings. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a part of the speaker or as a part of the speaker.

[0056] The display module 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0057] The audio module 170 can convert sound into an electrical signal, and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly coupled to the electronic device 101.

[0058] The sensor module 176 can detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0059] The interface 177 may support one or more designated protocols for the electronic device 101 to be directly (e.g., wired) or wirelessly coupled to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0060] The connection terminal 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0061] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or movement) or electric stimulation that can be recognized by the user via his tactile sense or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0062] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0063] The power management module 188 may manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0064] The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0065] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors that can operate independently of the processor 120 (e.g., an application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth™, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can use the subscriber information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 196 to identify and authenticate the electronic device 101 in a communication network such as the first network 198 or the second network 199.

[0066] The wireless communication module 192 can support 5G networks and next-generation communication technologies after the 4G network, for example, new radio (NR) access technology. NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC) or ultra-reliable and low-latency communication (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., mmWave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple input and multiple output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., an electronic device 104) or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or more), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0067] The antenna module 197 may transmit a signal or power to or receive a signal or power from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna, wherein the antenna includes a radiation element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network such as the first network 198 or the second network 199 may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). Then, a signal or power may be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component other than the radiation element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197.

[0068] According to various embodiments, the antenna module 197 may form a millimeter wave (mmWave) antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., millimeter wave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving signals of the specified high frequency band.

[0069] At least some of the components described above may be coupled to each other and transmit signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0070] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via a server 108 coupled to the second network 199. Each of the electronic devices 102 or 104 may be a device of the same type or a different type as the electronic device 101. According to an embodiment, all or some of the operations to be performed at the electronic device 101 may be performed at one or more of the external electronic devices 102, 104, or 108. For example, if the electronic device 101 should automatically perform a function or service, or in response to a request from a user or another device, instead of performing the function or service, or in addition to performing the function or service, the electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a part of the reply to the request with or without further processing of the result. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or health care) based on 5G communication technology or IoT-related technologies.

[0071] Figure 2is a block diagram 200 illustrating a camera module 180 according to various embodiments.

[0072] refer to Figure 2 , the camera module 180 may include a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., a buffer memory), or an image signal processor 260. The lens assembly 210 may collect light emitted or reflected from an object whose image is to be captured. The lens assembly 210 may include one or more lenses. According to an embodiment, the camera module 180 may include a plurality of lens assemblies 210. In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (e.g., viewing angle, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from another lens assembly. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

[0073] The flashlight 220 may emit light for enhancing light reflected from an object. According to an embodiment, the flashlight 220 may include one or more light emitting diodes (LEDs) (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared (IR) LEDs, or ultraviolet (UV) LEDs) or xenon lamps. The image sensor 230 may obtain an image corresponding to an object by converting light emitted or reflected from an object and transmitted via the lens assembly 210 into an electrical signal. According to an embodiment, the image sensor 230 may include one selected from image sensors having different properties (such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0074] The image stabilizer 240 may move the image sensor 230 or at least one lens included in the lens assembly 210 in a particular direction, or control the operating properties of the image sensor 230 (e.g., adjust the readout timing) in response to the movement of the camera module 180 or the electronic device 101 including the camera module 180. This allows compensation for at least a portion of the negative effects (e.g., image blur) caused by the movement on the image being captured. According to an embodiment, the image stabilizer 240 may sense such movement by the camera module 180 or the electronic device 101 using a gyroscope sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180. According to an embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer. The memory 250 may at least temporarily store at least a portion of the image obtained via the image sensor 230 for subsequent image processing tasks. For example, if image capture is delayed due to shutter lag or multiple images are captured quickly, the obtained original image (e.g., Bayer patterned image, high-resolution image) may be stored in the memory 250, and its corresponding copy image (e.g., low-resolution image) may be previewed via the display module 160. Thereafter, if a specified condition is satisfied (e.g., by a user's input or a system command), at least a portion of the original image stored in the memory 250 may be obtained and processed, for example, by the image signal processor 260. According to an embodiment, the memory 250 may be configured as at least a portion of the memory 130, or as a separate memory that operates independently of the memory 130.

[0075] The image signal processor 260 may perform one or more image processing on an image obtained via the image sensor 230 or an image stored in the memory 250. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 260 may perform control (e.g., exposure time control or readout timing control) on at least one of the components included in the camera module 180 (e.g., the image sensor 230). The image processed by the image signal processor 260 may be stored back in the memory 250 for further processing, or may be provided to an external component (e.g., the memory 130, the display module 160, the electronic device 102, the electronic device 104, or the server 108) outside the camera module 180. According to an embodiment, the image signal processor 260 may be configured as at least a part of the processor 120, or may be configured as a separate processor that operates independently of the processor 120. If the image signal processor 260 is configured as a separate processor from the processor 120 , at least one image processed by the image signal processor 260 may be displayed by the processor 120 via the display module 160 as is or after being further processed.

[0076] According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 having different properties or functions. In this case, at least one of the plurality of camera modules 180 may form, for example, a wide-angle camera, and at least another of the plurality of camera modules 180 may form a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may form, for example, a front camera, and at least another of the plurality of camera modules 180 may form a rear camera.

[0077] Figure 3 is a control block diagram of an electronic device including a camera module according to an embodiment.

[0078] According to an embodiment, Figure 3 1 shows schematic components included in the electronic device 101, and according to Figure 3 The electronic device 101 may include Figure 1 The components of the electronic device 101 shown in FIG. 1 are the same as or similar to those of the electronic device 101 shown in FIG. 1 .

[0079] refer to Figure 3 , the electronic device 101 may include a motion sensor 211 , an illumination sensor 212 , a processor 270 , and a camera module 180 .

[0080] According to an embodiment, the electronic device 101 may include a motion sensor 211. The processor 270 may detect the motion of the electronic device 101 via the motion sensor 211. The motion sensor 211 may provide the processor 270 with motion data corresponding to the motion of the electronic device 101.

[0081] According to an embodiment, the motion sensor 211 may provide motion data to the processor 270 so that the processor 270 processes the motion data and acquires a motion level. The motion level may be a standard for the motion of the electronic device 101 and may be used to determine a suppression ratio in an OIS function, which will be described later. The motion data may be data related to the physical momentum of the electronic device 101 and may be acquired via a signal output from the motion sensor 211. Specifically, the motion data may include at least one piece of data generated by processing a signal generated from an acceleration sensor, a gyroscope sensor (gyroscope), a magnetic sensor, or a Hall sensor.

[0082] In an embodiment, the motion sensor 211 may include at least one of an acceleration sensor, a gyroscope sensor (gyroscope), a magnetic sensor, or a Hall sensor. For example, the acceleration sensor may measure the acceleration acting on three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device 101. In another example, the gyroscope sensor may measure the rotation angle or slope on the three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device 101. However, the sensors described above are exemplary, and the motion sensor 211 may also include at least one other type of sensor.

[0083] In an embodiment, the electronic device 101 may include an illumination sensor 212. The processor 270 may measure the ambient illumination (or brightness) of the electronic device 101 via the illumination sensor 212. In an embodiment, the processor 270 may determine whether the measured illumination is less than a specified value corresponding to a low illumination environment. In an embodiment, the processor 270 may determine whether the measured illumination is greater than or equal to a specified value corresponding to a high illumination environment.

[0084] According to an embodiment, it can be understood that the processor 270 includes at least one processor. In an embodiment, the processor 270 may include an OIS control circuit 271 and may exchange various data with an application processor (AP) 290. In an embodiment, the AP 290 may provide an exposure time to the processor 270, wherein the exposure time is information set for the camera module 180. As a similar concept, the AP 290 in the embodiment may provide a shutter speed to the processor 270, wherein the shutter speed is information set for the camera module 180. In an embodiment, the processor 270 may also include at least one of an image signal processor 260 and a communication processor (CP).

[0085] According to an embodiment, the OIS control circuit 271 may control the camera module 180 to perform an OIS operation. For example, the OIS control circuit 271 may control the camera module 180 to perform an OIS function based on motion data acquired from the motion sensor 211. In an embodiment, the OIS control circuit 271 may be disposed adjacent to the camera module 180. For example, the OIS control circuit 271 may be disposed on a certain surface inside the camera module 180.

[0086] According to an embodiment, the OIS control circuit 271 may obtain acceleration data from the acceleration sensor. The OIS control circuit 271 may obtain information about the speed (acceleration) of the electronic device vibration by performing an integration operation on the acceleration data. The OIS control circuit 271 may control the OIS function of the camera module 180 based on the information about the acceleration.

[0087] According to an embodiment, the OIS control circuit 271 may acquire angular velocity data from the gyro sensor. The OIS control circuit 271 may acquire information about the angle at which the electronic device 101 is shaken by performing an integration operation on the angular velocity data. The OIS control circuit 271 may control the OIS function of the camera module 180 based on the information about the angle.

[0088] According to an embodiment, the image signal processor 260 (see Figure 2 ) may continuously acquire image frames while the camera module 180 is performing the OIS function. The image signal processor 260 may acquire image frames for which the OIS function is performed from the camera module 180 and may provide the image frames to the AP 290.

[0089] According to an embodiment, the AP 290 may acquire an image frame for which the OIS function is performed. The AP 290 may perform video digital image stabilization (VDIS) on the image frame for which the OIS function is performed. In another embodiment, the image signal processor 260 may provide the AP 290 with an image frame for which VDIS is also performed.

[0090] According to an embodiment, the camera module 180 may acquire image frames under the control of the processor 270. In an embodiment, the camera module 180 may acquire image frames while the OIS function is performed by the OIS control circuit 271. In an embodiment, the camera module 180 may provide the image frames acquired while performing the OIS function to the processor 270.

[0091] According to an embodiment, the camera module 180 may use a lens shift method to shift the lens assembly 210 through the OIS function (see Figure 2). For example, the processor 270 may shift the lens assembly 210 through the actuator 182. The processor 270 may detect the position to which the lens assembly 210 is shifted via a Hall sensor (not shown). According to an embodiment, the camera module 180 may shift the image sensor 181 through the OIS function using a sensor shift method. For example, the processor 270 may shift the image sensor 181 through the actuator 182. The processor 270 may detect the position to which the image sensor 181 is shifted via a Hall sensor.

[0092] According to an embodiment, the camera module 180 may perform the OIS function within a first range. The first range may be an actuation range of the OIS function and may be a maximum actuation range that allows the camera module 180 to shift the lens assembly 210 to compensate for the movement of the electronic device 101.

[0093] According to an embodiment, the camera module 180 may perform the OIS function within a second range greater than the first range. The second range may be an expansion of the maximum actuation range greater than the first range. The second range may be provided by expanding the displacement space of the lens assembly 210 in the production step of the camera module 180. For example, if the maximum actuation range of the lens assembly 210 of the prior art is 1 degree, the maximum actuation range in the present disclosure may be set to 3 degrees. However, the numerical values ​​described above are merely examples. The maximum actuation range of the OIS function may be a fixed value when the camera module 180 is manufactured, and may be expanded from the first range to the second range according to a command of the AP 200, or may be reduced from the second range to the first range.

[0094] According to an embodiment, the memory 250 (see Figure 2 ) Various programming languages ​​or instructions may be stored by the processor 270. For example, the processor 270 may execute an application by executing a code written in a programming language stored in the memory 250, and may control various hardware. In addition, the processor 270 may set and support an appropriate shooting mode so that the camera module 180 performs the operation desired by the user. In an embodiment, the processor 270 may store an image frame for which VDIS is performed in the memory 250. In another embodiment, the processor 270 may store an image frame for which an OIS function is performed in the memory 250.

[0095] Figure 4 An OIS actuation range in which a camera module performs OIS according to an embodiment is illustrated.

[0096] According to an embodiment, the camera module 180 (see Figure 2 ) can perform the OIS function within the first range. In an embodiment, the processor 270 (see Figure 3) may shift the lens assembly 210 (or the image sensor 230) included in the camera module 180 within a first range. For example, the OIS control circuit 271 may rotate the lens assembly 210 within the first range via the actuator 182. The OIS control circuit 271 may detect the position to which the lens assembly 210 is shifted via a Hall sensor (not shown).

[0097] According to an embodiment, the camera module 180 may perform an OIS function within a second range greater than the first range. As the OIS maximum motion range is enlarged, the camera module 180 may effectively improve jitter even when a large motion occurs in the electronic device 101. Jitter refers to a phenomenon in which a specific object in a video flickers due to different degrees of blur in each image frame.

[0098] refer to Figure 4 , the OIS maximum actuation range is an example of a maximum actuation range in which the camera module 180 performs the OIS function. Figure 4 The first range and the second range are shown for comparison. The OIS actuation range is an example of an actuation range in which the camera module 180 performs the OIS function. The OIS actuation range may be less than or equal to the OIS maximum actuation range. If the OIS maximum actuation range is the second range, the OIS actuation range may be less than or equal to the second range.

[0099] In the present disclosure, it is understood that the OIS maximum actuation range is the second range. In an embodiment, the processor 270 may perform the OIS function for an OIS actuation range included in the OIS maximum actuation range due to physical limitations of the camera module 180 .

[0100] Figure 5 An example of determining a suppression ratio by a camera module according to an embodiment is shown.

[0101] refer to Figure 5 When the movement of the electronic device 101 is relatively small and the required OIS correction amount is less than or equal to 1 degree (region 510), the processor 270 can shift the lens assembly 210 within a range of 1 degree (see Figure 2 ) to improve the jitter. On the other hand, if the movement of the electronic device 101 is relatively large and the required OIS correction amount exceeds 1 degree, the processor 27 may not perform the OIS function. That is, if a large movement occurs in the electronic device 101 during video recording, the OIS function may be interrupted and the jitter phenomenon may not be improved.

[0102] In order to solve the above-described problem, in an embodiment, if the movement of the electronic device 101 is relatively large, the jitter phenomenon can be improved by adjusting the suppression ratio. Figure 6 This is described in detail.

[0103] Figure 6 An example of determining a suppression ratio by the camera module when a required OIS correction amount exceeds an OIS maximum adjustable range according to an embodiment is shown.

[0104] refer to Figure 6 , if the movement of the electronic device 101 is relatively large and the required OIS correction amount exceeds the OIS maximum movement range, then Figure 5 Here, the required OIS correction amount may be determined based on the motion data acquired from the motion sensor 211. If the required OIS correction amount is less than the OIS maximum motion range (area 610), the processor 270 may adjust the suppression ratio in accordance with the above. Figure 5 The OIS function is performed in the same way as in FIG.

[0105] When the movement of the electronic device 101 is relatively large (when the required OIS correction amount is greater than or equal to 1 degree), the required correction amount and suppression ratio may be determined based on the following equations:

[0106] OIS maximum adjustable range = Required OIS correction amount x suppression ratio… Equation 1

[0107] According to an embodiment, when the required OIS correction amount is 2 degrees (620), the processor 270 may determine the suppression ratio to be 0.5. In an embodiment, the processor 270 may control the camera module 180 to perform the OIS function within a range of 1 degree based on the suppression ratio of 0.5. For example, when the electronic device 101 is vibrating 2 degrees, whenever the electronic device 101 moves to require a correction amount corresponding to 1 degree, the processor 270 may shift the lens assembly 210 (or the image sensor 230) to perform an OIS correction corresponding to 0.5 degrees.

[0108] According to an embodiment, when the required OIS correction amount is 5 degrees, the processor 270 may determine the suppression ratio to be 0.2 based on Equation 1. In an embodiment, the processor 270 may control the camera module 180 to perform the OIS function within a range of 1 degree based on the suppression ratio of 0.2. For example, when the electronic device 101 is vibrating 5 degrees, the processor 270 may control the camera module 180 to shift the lens assembly 210 (or the image sensor 230) by 0.2 degrees whenever the electronic device moves 1 degree.

[0109] According to an embodiment, when the required correction amount is greater than 5 degrees, the processor 270 may not perform the OIS operation or may limit the OIS operation based on a suppression ratio of 0.2.

[0110] That is, the processor 270 may resolve the jitter that occurs when the motion of the electronic device 100 is large through an adaptive suppression ratio (ASR).

[0111] In ASR, since the position of the lens assembly 210 (or the image sensor 230) is constantly changed every short time, the suppression ratio should be changed every time. If the processor 270 does not provide an appropriate suppression ratio, micro vibration may occur in the image frame to be provided to the VDIS. In addition, if the OIS maximum actuation range is small, since the suppression ratio is only 0.2 and the lens assembly 210 is moved at a small suppression ratio, when a large motion occurs in the electronic device 101, the electronic device 101 may have a problem of being susceptible to jitter.

[0112] Therefore, by expanding the OIS maximum actuation range to increase the minimum suppression ratio, the present disclosure can improve jitter even in response to large motions of the electronic device 101. Figure 7 This is described in detail.

[0113] Figure 7 An example of determining a suppression ratio when an OIS maximum amplitude range of a camera module is expanded according to an embodiment is shown.

[0114] like Figure 7 As shown, when the OIS maximum amplitude range is 3 degrees, the camera module 180 according to the embodiment can set the suppression ratio to be reduced to 0.5 in the section where the required OIS correction amount is 3 degrees to 6 degrees. When the movement of the electronic device 101 is relatively large and the required OIS correction amount exceeds 6 degrees, the camera module 180 may not perform the OIS operation, or may limit the OIS operation based on the minimum suppression ratio of 0.5. For the relationship between the required OIS correction amount and the suppression ratio, refer to Equation 1. For the area 720 where the required OIS correction amount is higher than the OIS maximum amplitude range, refer to Figure 6 An embodiment of region 620 .

[0115] In the present disclosure, the OIS maximum amplitude range of the camera module 180 is enlarged so that blurring is reduced and shaking is improved, but another problem of micro vibration may occur in the area 710 where the movement of the electronic device 101 is relatively small. This is because when the OIS maximum amplitude range is enlarged, the displacement amount of the lens assembly 210 becomes relatively large.

[0116] When the motion of the electronic device 101 is relatively small, micro-vibration may occur, and when the motion is relatively large, jitter may take precedence over micro-vibration. The motion of the electronic device 101 is important for the OIS function, but since the degree of blur varies with the exposure time, the exposure time should be considered in addition to the motion of the electronic device 101 to perform effective OIS.

[0117] In order to solve the above-described problem, in addition to adjusting the suppression ratio based on the position of the OIS, the present disclosure determines the suppression ratio by using the exposure time and motion information of the electronic device.

[0118] Figure 8 is a flowchart 800 illustrating an operating method of an electronic device according to an embodiment. Fig. 9 shows the priority between jitter and micro vibration at exposure time, Fig.10 An example of determining the suppression ratio when the exposure time is relatively long is shown, and Fig.11 An example of determining the suppression ratio when the exposure time is relatively short is shown. Figure 8 For examples, refer to Figures 9 to 11 .

[0119] The processor 270 may continuously acquire image frames through the camera module 180 ( 801 ).

[0120] According to an embodiment, the processor 270 may acquire motion data from the motion sensor 211 while continuously acquiring image frames through the camera module 180 .

[0121] In addition, according to an embodiment, the processor 270 may acquire illumination data from the illumination sensor 212 while continuously acquiring image frames through the camera module 180. The processor 270 may process the illumination data to determine whether the electronic device 101 is in a low illumination environment or a high illumination environment.

[0122] According to an embodiment, the processor 270 may actuate the camera module 180 and may continuously acquire image frames through the image sensor 181. For example, the processor 270 may shoot a video through the camera module 180 and may acquire continuous image frames included in the video data.

[0123] The processor 270 may determine whether to perform OIS by prioritizing shaking or micro-vibration according to the exposure time of the camera module 180. Fig. 9 , when the exposure time is relatively long relative to the predefined exposure time T_set (region A), shake priority OIS can be performed, and when the exposure time is relatively short (region B), micro-vibration priority OIS can be performed. The exposure time is a value determined according to the shutter speed, and a fast shutter speed means a short exposure time. Therefore, based on the exposure time is the same as based on the shutter speed, and in the present disclosure, the OIS process will be described with reference to the exposure time.

[0124] According to an embodiment, when the exposure time is less than or equal to the predefined exposure time T_set (802), the processor 270 may obtain the motion level by processing the motion data (803). When the exposure time is relatively short, it may be predicted that micro vibration is more prioritized than jitter. Therefore, the processor 270 may determine the degree of motion of the electronic device 101 through the motion level, and may perform the OIS function by suppressing the ratio in stages according to the motion level.

[0125] The processor 270 according to an embodiment may determine a motion level corresponding to the motion of the electronic device by processing at least one of acceleration data or angular velocity data.The processor 270 according to an embodiment may determine a suppression ratio based on a motion level received from the AP 290 without directly calculating the motion level.

[0126] The processor 270 according to an embodiment may determine the exposure time according to a control command of the AP 290. For example, the exposure time may be determined according to a user input set through an application.

[0127] In addition, the processor 270 according to the embodiment may determine the exposure time based on the illuminance measured via the illuminance sensor 212, and when the exposure time is less than or equal to the predefined exposure time, the processor 270 may obtain the motion level by processing the motion data. The processor 270 may set the exposure time to be long in a low illumination environment, and may set the exposure time to be short in a high illumination environment.

[0128] The motion level may be a standard for motion of an electronic device acquired by processing motion data, and may indicate a result value obtained by processing a signal generated in an acceleration sensor, a gyro sensor (gyro), a magnetic sensor, or a Hall sensor.

[0129] The motion level may indicate the amount of change in pixels constituting an image when the size of a signal obtained by the motion sensor 211, which is proportional to the degree of motion of the electronic device, is reflected on the image. Therefore, the motion level may be weighted according to the performance information of the camera module 180. The processor 270 according to an embodiment may apply weighting to the motion level based on information about the field of view (FOV) of the camera module 180 or information about the resolution of the display module 160.

[0130] In an embodiment, when the FOV of the camera module 180 is smaller than a predefined FOV range, the processor 270 may apply a first weighting to the motion level.

[0131] In an embodiment, when the FOV of the camera module 180 is greater than a predefined FOV range, the processor 270 may apply a second weight greater than the first weight to the motion level.

[0132] In an embodiment, the processor 270 may apply different weights to the motion level based on the resolution of the display module 180. For example, when the resolution is high, the processor 270 may apply a relatively small weight to the motion level, and when the resolution is low, the processor 270 may apply a relatively large weight to the motion level.

[0133] On the other hand, when the exposure time is relatively long ( Fig. 9 and Fig.10 In the area A), since the movement of the electronic device is relatively unimportant, the processor 270 according to the embodiment can omit the process of processing the motion data. When the exposure time is longer than the predefined exposure time T_set, the processor 270 according to the embodiment can perform the OIS function (805) through an adaptive suppression ratio (ASR). In this case, in order to prioritize jitter rather than micro-vibration, the processor 270 can adaptively adjust the suppression ratio based on the position of the correction angle of the OIS function.

[0134] Meanwhile, when the motion level is lower than the predefined motion level L_set (804), the processor 270 according to an embodiment may determine a suppression ratio based on the motion level (806). Specifically, when the motion level is lower than the predefined motion level, the processor 270 may perform the OIS function based on a suppression ratio that increases in stages according to the motion level.

[0135] For example, refer to Fig.11 , Region B is where the exposure time is relatively short and micro vibration takes precedence over jitter (see Fig. 9 ), and the processor 270 can adjust the suppression ratio in stages in the segment where the motion level is less than or equal to 2. The electronic device 101 can solve the problem of micro vibration by adjusting the suppression ratio in stages. However, when the movement of the electronic device is so large that the motion level is greater than or equal to 3, the predicted jitter is prioritized again, and the processor 270 according to the embodiment can perform the OIS function through ASR. However, Fig.11 The suppression ratio at each motion level proposed in is merely an example for convenience of explanation and may be set differently according to data of the camera module 180 .

[0136] According to an embodiment, the processor 270 may acquire an image frame for which an OIS function is performed ( 807 ).

[0137] According to an embodiment, when the exposure time of the image sensor 181 obtained based on the shutter speed while the image frame is being obtained by the camera module 180 is less than or equal to the predefined exposure time, the processor 270 may determine the suppression ratio based on the motion level, and may acquire the image frame for which the OIS function is performed according to the determined suppression ratio. In this case, the processor 270 may determine the suppression ratio of the OIS function based on the table stored in the memory 250. The table according to the embodiment may be data indicating the suppression ratio assigned to the motion level.

[0138] When the exposure time is less than or equal to the predefined exposure time and the motion level is less than or equal to the predefined motion level, the processor 270 according to an embodiment may perform the OIS function by suppressing the ratio in stages. The processor 270 may perform the OIS function based on the suppression ratio increasing in stages according to the motion level.

[0139] When the exposure time is longer than the predefined exposure time, the processor 270 according to the embodiment may adaptively adjust the suppression ratio based on the correction angle position of the OIS function. That is, the processor 270 may continuously change the suppression ratio according to a predetermined period (e.g., readout timing) based on at least one of the position of the OIS correction angle or the required OIS correction amount.

[0140] When the exposure time is less than or equal to the predefined exposure time and the motion level is greater than or equal to the predefined value, the processor 270 according to the embodiment can adaptively adjust the suppression ratio based on the position of the OIS correction angle. In the corresponding section, micro vibration is prioritized, but it is effective when the jitter is prioritized due to the violent movement of the electronic device. Therefore, the processor 270 according to the embodiment can continuously change the suppression ratio according to a predetermined period (e.g., readout timing) based on at least one of the position of the OIS correction angle or the required OIS correction amount.

[0141] According to an embodiment, the actuation range of the OIS function of the camera module 180 can be expanded from the first range to the second range. For example, the OIS maximum actuation range of the camera module 180 to which the embodiment of the present disclosure is applied may be 3 degrees instead of 1 degree. The OIS maximum actuation range may be a piece of data fixed during the manufacturing process of the camera module 180, but the OIS maximum actuation range may be changed later by setting. For example, the OIS maximum actuation range may be set to the second range by default, but may be reduced from the second range to the first range according to the setting. When the OIS maximum actuation range of the camera module 180 is the second range, the OIS maximum actuation range may be changed from the first range to the second range or from the second range to the first range according to the control command of the AP 290.

[0142] When the actuation range of the OIS function is expanded from the first range to the second range, the processor 270 according to an embodiment may acquire a motion level by processing motion data.

[0143] In the above, a motion correction method for solving jitter or micro vibration in the OIS process as the maximum vibration range of OIS is expanded has been described. In addition to the OIS function for correcting vibration, the electronic device 101 can also correct vibration of a predetermined level or higher through VDIS.

[0144] At the same time, the enlargement of the maximum actuation range of OIS may additionally lead to problems regarding perspective distortion during VDIS. Figure 12 to Figure 14 Describe this.

[0145] Fig.12 illustrates an OIS operation when there is no movement of the electronic device according to an embodiment, Fig.13 shows a state in which the OIS function is turned off when there is motion of the electronic device, Fig.14 A state in which the OIS function is turned on when there is a motion of the electronic device according to an embodiment is shown.

[0146] refer to Fig.12 , when there is no movement of the electronic device, light reflected from the object S may pass through the lens assembly 210 and may be collected at the center of the image sensor 230 which is a right position.

[0147] On the other hand, reference Fig.13 , when there is movement of the electronic device in a state in which the OIS function is turned off, since the surface incident angle on the lens assembly 210 is changed, light reflected from the object S may be collected outside the center of the image sensor 230.

[0148] refer to Fig.14 , in a state in which the OIS function is turned on, the lens assembly 210 is shifted relative to the image sensor 230, thereby allowing light reflected from the object S to be collected at the center of the image sensor 230. As described above, the OIS maximum adjustment range may be expanded from the first range 210-1 to the second range 210-2. As the OIS maximum adjustment range is expanded and the movement distance of the lens assembly 210 increases, more severe perspective distortion may occur in the image frame. Perspective distortion is a phenomenon in which an object or the surroundings of an object are distorted or deformed within an image frame, and may occur as the focal length changes according to the angle of the incident light. Perspective distortion may occur due to OIS wobbling.

[0149] Fig.15 is a flowchart 1500 of a method for operating an electronic device according to an embodiment. Fig.16 shows an example of perspective transformation, and Fig.17 Shows the export for Fig.16 An example of the process of perspective transformation of OIS angle in . Fig.16 and Fig.17 To describe Fig.15 .

[0150] The processor 270 according to the embodiment may acquire an image frame to which the OIS function is applied (1501). The camera module 180 may acquire an image frame under the control of the processor 270. In an embodiment, the camera module 180 may acquire an image frame while performing the OIS function through the OIS control circuit 271. In an embodiment, the camera module 180 may provide the image frame obtained while performing the OIS function to the processor 270. The processor 270 may control the OIS module (actuator 182) to shift the lens module (lens assembly 210) while performing the OIS function. Here, the OIS maximum actuation range of the OIS actuation module may be an expansion from the first range to the second range. The processor 270 may acquire the OIS correction angle of the position to which the lens module is shifted.

[0151] The processor 270 according to an embodiment may perform lens distortion correction (LDC) (1502), and then may perform perspective transformation (1503). That is, the processor 270 may first perform LDC before perspective transformation.

[0152] The processor 270 according to an embodiment may acquire an OIS correction angle of a position to which the lens module is shifted to perform perspective transformation. The processor 270 may calculate a transformation angle (θ) for perspective transformation based on the acquired OIS correction angle (see Fig.16 ). The relationship between the OIS correction angle and the transformation angle can be defined based on various mathematical techniques, and the transformation angle can be derived from a neural network model by using various algorithms of machine learning, wherein the neural network model has the OIS correction angle as input data and has the transformation angle as output data.

[0153] The processor 270 can acquire a plurality of OIS correction angles with different values ​​according to the readout timing while generating one image frame. Fig.17, the processor 270 may acquire a plurality of OIS correction angles (θ(i) ​​to θ(n+k), θ(n+k+1) to θ(f)) each time data is read out. According to an embodiment, the processor 270 may acquire the OIS correction angle at intervals of 1 KHz. In this case, while the OIS function is being executed, the OIS correction angle sent within one image frame may have different values. The processor 270 according to an embodiment may acquire the OIS correction angle at regular intervals while acquiring the image frame. The processor 270 may determine the transformation angle based on the average value between the first correction angle closest to the center within the image frame and the second correction angle obtained after the second correction angle. That is, interpolation may be used to estimate the OIS correction angle at the center of the image frame.

[0154] The processor 270 according to an embodiment may acquire an image frame for which VDIS is performed (1504). The processor 270 according to an embodiment may perform VDIS on an image frame in which perspective distortion occurs based on a transformation angle.

[0155] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0156] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and include various changes, equivalents or replacements of the corresponding embodiments. With regard to the description of the accompanying drawings, similar reference numerals may be used to indicate similar or related elements. It should be understood that, unless otherwise clearly stated in the relevant context, the singular form of the noun corresponding to the project may include one or more things. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in a corresponding phrase. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled to another element (e.g., a second element)”, “coupled to another element (e.g., the second element)”, “connected to another element (e.g., the second element)”, or “connected to another element (e.g., the second element)”, with or without the term “operably” or “communicatively”, it means that the element can be coupled to another element directly (e.g., by wire), wirelessly, or via a third element.

[0157] As used in conjunction with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integral component or its smallest unit or portion suitable for performing one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0158] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute it with or without one or more other components under the control of the processor. This allows the machine to be operated to perform at least one function according to at least one instruction called. One or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between a location where data is semi-permanently stored in the storage medium and a location where data is temporarily stored in the storage medium.

[0159] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online (e.g., downloaded or uploaded) via an application store (e.g., PlayStoreTM), or distributed directly between two user devices (e.g., smart phones) (e.g., downloaded or uploaded). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0160] According to various embodiments, each component (e.g., module or program) in the components described above may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the components described above may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as they were performed by a corresponding one of the multiple components before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed or omitted in a different order, or one or more other operations may be added.

Claims

1. An electronic device, comprising: a motion sensor configured to output a signal for acquiring motion data corresponding to the motion of the electronic device; a camera module comprising an image sensor and an optical image stabilization (OIS) module, wherein the OIS module is configured to perform OIS based on a motion of the electronic device; and at least one processor operably connected to the motion sensor and the camera module, Wherein, the at least one processor is configured to: determining motion state information based on the size of the motion data when the exposure time of the image sensor is less than or equal to a predefined exposure time while image frames are continuously acquired by the camera module; determining a suppression ratio associated with an actuation range of the OIS module based on the motion state information; and The operation of the OIS module is controlled based on the determined suppression ratio.

2. The electronic device according to claim 1, further comprising a memory, in, The motion state information includes the size of the motion data based on the motion level divided into stages, wherein the memory is configured to store a table in which suppression ratios are assigned by motion level, The at least one processor is configured to determine a suppression ratio on the OIS module based on a stored table.

3. The electronic device according to claim 1, wherein: The at least one processor is configured to adaptively adjust the suppression ratio based on a corrected angular position of the OIS module when the exposure time is longer than the predefined exposure time.

4. The electronic device according to claim 1, wherein: The at least one processor is configured to adaptively adjust the suppression ratio based on a corrected angular position of the OIS module when the exposure time is less than or equal to the predefined exposure time and a motion level is greater than or equal to a predefined value.

5. The electronic device according to claim 1, wherein: The at least one processor is configured to adaptively adjust the suppression ratio based on an amount of correction angle required by the OIS module when the exposure time is less than or equal to the predefined exposure time and the motion level is greater than or equal to a predefined value.

6. The electronic device according to claim 1, wherein: The camera module has an actuation range of the OIS module extending from a first range to a second range.

7. The electronic device according to claim 6, wherein: The at least one processor is configured to obtain a motion level by processing the motion data based on an actuation range of the OIS module extending from the first range to the second range.

8. The electronic device according to claim 2, wherein: The at least one processor is configured to control the OIS module based on the suppression ratio increasing in stages according to the motion level.

9. The electronic device according to claim 1, wherein: The motion sensor includes at least one of an acceleration sensor that outputs a signal for acquiring acceleration data or a gyro sensor that outputs a signal for acquiring angular velocity data, and The at least one processor is configured to determine a motion level corresponding to the motion of the electronic device based on at least one of the acceleration data or the angular velocity data.

10. The electronic device according to claim 1, further comprising: an illumination sensor configured to output a signal for measuring an ambient illumination of the electronic device, The at least one processor is configured to determine the exposure time based on the measured illumination, and to obtain the motion level by processing the motion data when the exposure time is less than or equal to the predefined exposure time.

11. An operating method of an electronic device, the operating method comprising: Continuously acquire image frames through the camera module; Acquiring motion data corresponding to the motion of the electronic device; When the exposure time of the image sensor is less than or equal to the predefined exposure time while the image frames are continuously acquired, determining the motion state information based on the size of the motion data; determining a suppression ratio associated with an actuation range of the OIS module based on the motion state information; and The operation of the OIS module is controlled based on the determined suppression ratio.

12. The operating method according to claim 11, wherein: Determining the motion state information based on the size of the motion data includes determining a suppression ratio on the OIS module based on a table in which suppression ratios are allocated by motion level, and the motion state information includes a motion level based on which the size of the motion data is divided by stages.

13. The operating method according to claim 11, further comprising: When the exposure time is longer than the predefined exposure time, the suppression ratio is adaptively adjusted based on a corrected angular position of the OIS module.

14. The operating method according to claim 11, further comprising: When the exposure time is less than or equal to the predefined exposure time and the motion level is greater than or equal to a predefined value, the suppression ratio is adaptively adjusted based on the corrected angular position of the OIS module.

15. An electronic device, comprising: a motion sensor configured to output a signal for acquiring motion data corresponding to the motion of the electronic device; A camera module including a lens module and configured to perform an OIS function; An OIS actuation module configured to displace the lens module within an actuation range of the OIS function; and at least one processor electrically connected to the motion sensor, the camera module and the OIS actuation module, Wherein, the at least one processor is configured to: When movement of the electronic device occurs while image frames are continuously acquired by the camera module, controlling the OIS actuation module to shift the lens module; Acquiring a correction angle of the OIS function at the position where the lens module is shifted; determining a transformation angle for perspective transformation of the image frame based on the correction angle; and Video digital image stabilization (VDIS) is performed based on the transformed angle.

Images