Electronic device for automatically switching camera and operating method therefor

Abstract

An electronic device according to one embodiment may comprise: a first camera supporting a first field of view (FOV); a second camera supporting a second field of view that is an angle wider than the first field of view; a display; and at least one processor including processing circuitry. The electronic device according to one embodiment may: display a preview image on the display or store an image in a memory on the basis of an image acquired through the first camera while the second camera is deactivated; acquire first depth information on the basis of phase difference information acquired using the first camera; activate the second camera on the basis of the value of the first depth information being less than a first threshold value; and, after the second camera is activated, determine third depth information on the basis of at least one of the first depth information or second depth information based on phase difference information acquired using the second camera.

Description

Electronic device for performing automatic camera switching and method of operation thereof

[0001] The present disclosure relates to an electronic device including a camera and a method of operating the same.

[0002] The electronic device is generally equipped with multiple cameras of different focal lengths and can switch the main camera manually via user input or automatically according to certain conditions through internal functions. To switch cameras, the electronic device requires distance information between the device and the subject.

[0003] The absolute distance between an electronic device and an object can be measured using distance sensors (ToF: Time-of-Flight, LiDAR: Laser-induced Detection and Ranging). However, using them in conjunction with a camera causes operational issues such as signal synchronization and changes in corresponding points on the image depending on the distance. Additionally, distance sensors require separate mounting space and may result in problems such as additional power consumption and increased costs.

[0004] If camera focus information is used, a separate distance sensor is not required; however, the accuracy and stability of the switching operation may decrease due to errors caused by changes in the characteristics of the optical system.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] Various aspects of the present disclosure are intended to address at least the problems and / or disadvantages mentioned above and to provide at least the advantages described below. Accordingly, one aspect of the present disclosure is to provide an electronic device comprising a camera and a method of operating the same.

[0007] Additional aspects are mentioned in part of the description below, some of which may become apparent from the description or be acquired through the execution of the presented embodiments.

[0008] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) may include a first camera supporting a first field of view (FoV), a second camera supporting a second field of view which is a wider angle than the first field of view, at least one processor including a display and processing circuitry, and a memory for storing instructions. The instructions may be executed individually or collectively by at least one processor so that the electronic device: displays a preview image on a display or stores an image in memory based on an image acquired through the first camera while the second camera is disabled. Instructions according to one embodiment may acquire first depth information based on phase difference information acquired using the first camera. Instructions according to one embodiment may enable the second camera based on the value of the first depth information being less than a first threshold. The above instructions may determine third depth information based on at least one of first depth information or second depth information based on phase difference information obtained using the second camera after the second camera is activated. Instructions according to one embodiment may switch the camera used to display a preview image on a display or store an image in memory to the second camera based on the activation of the second camera. Instructions according to one embodiment may determine whether to switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the third depth information.

[0009] A method of operation of an electronic device according to one embodiment may include an operation of displaying a preview image on a display or storing an image in memory based on an image acquired through a first camera while the second camera is in a deactivated state. A method of operation of an electronic device according to one embodiment may include an operation of acquiring first depth information based on phase difference information acquired using the first camera, and an operation of activating the second camera based on the fact that the value of the first depth information is less than a first threshold. A method of operation of an electronic device according to one embodiment may include an operation of determining third depth information based on at least one of the first depth information or second depth information based on phase difference information acquired using the second camera after the second camera is activated. A method of operation of an electronic device according to one embodiment may include an operation of switching the camera used to display a preview image on a display or store an image in memory to the second camera based on the fact that the second camera is activated. A method of operation of an electronic device according to one embodiment may include an operation of determining whether to switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the third depth information.

[0010] A recording medium according to one embodiment may be one or more non-transitory computer-readable recording media storing one or more computer programs that include computer execution instructions that cause the electronic device to perform an operation when executed by one or more processors of an electronic device including a first camera and a second camera. The operation may include an operation of displaying a preview image on a display or storing an image in memory based on an image acquired through the first camera while the second camera is deactivated. The operation may include an operation of acquiring first depth information based on phase difference information acquired using the first camera and an operation of activating the second camera based on the value of the first depth information being less than a first threshold. The operation may include, after the second camera is activated: an operation of determining third depth information based on at least one of the first depth information or second depth information based on phase difference information acquired using the second camera. The operation may include an operation of switching the camera used to display a preview image on a display or store an image in memory to the second camera based on the second camera being activated. The above operation may include an operation to determine whether to switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the third depth information.

[0011] Other aspects, advantages, and key features of the present disclosure will become apparent to those skilled in the art from the following detailed description provided together with the accompanying drawings.

[0012] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description, which is referenced together with the accompanying drawings.

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0014] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0015] FIG. 3 is a control block diagram of an electronic device according to one embodiment.

[0016] FIG. 4 is a flowchart of an operation method of an electronic device according to one embodiment.

[0017] FIG. 5 is a flowchart of a camera switching operation method of an electronic device according to one embodiment.

[0018] FIG. 6 is a signal flow diagram of depth information of an electronic device according to one embodiment.

[0019] FIGS. 7a and 7b illustrate various error types and error types that may occur depending on the camera switching direction in an electronic device according to one embodiment.

[0020] FIG. 8 illustrates a first error type that occurs when the camera switching direction of an electronic device according to one embodiment is a first direction.

[0021] FIG. 9 illustrates a second-1 error type that occurs when the camera switching direction of an electronic device according to one embodiment is the first direction.

[0022] FIG. 10 illustrates a second-2 error type that occurs when the camera switching direction of an electronic device according to one embodiment is the first direction.

[0023] FIG. 11 illustrates a third error type that occurs when the camera switching direction of an electronic device according to one embodiment is a second direction.

[0024] FIG. 12 illustrates a 4-1 error type that occurs when the camera switching direction of an electronic device according to one embodiment is the second direction.

[0025] FIG. 13 illustrates a 4-2 error type that occurs when the camera switching direction of an electronic device according to one embodiment is the second direction.

[0026] FIG. 14 is a flowchart of an operation in which an electronic device according to one embodiment corrects depth information based on lens position information.

[0027] FIG. 15 is a control block diagram of a depth information acquisition unit according to one embodiment.

[0028] FIG. 16 is a flowchart for dynamic correction according to one embodiment.

[0029] It should be noted that the same reference numbers are used throughout the drawings to denote identical or similar elements, features, and structures.

[0030] The following description, with reference to the attached drawings, is provided to aid in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid understanding, but are merely illustrative. Accordingly, those skilled in the art will recognize that various modifications and changes to the various embodiments described herein may be made without departing from the scope and spirit of the present disclosure. Additionally, descriptions of well-known functions and structures may be omitted for the sake of clarity and brevity.

[0031] Preferred embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, this document is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present disclosure is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0032] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0033] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0034] It should be understood that the blocks and flowchart combinations of each flowchart may be executed by one or more computer programs containing computer execution instructions. The entirety of one or more computer programs may be stored in a single memory device, or different parts may be distributed and stored in multiple memory devices.

[0035] Any function or operation described herein may be processed by a single processor or a combination of processors. The single processor or combination of processors is a circuit that performs processing and includes an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural network processing unit (NPU, e.g., an artificial intelligence (AI) chip), a wireless LAN (Wi-Fi) chip, a Bluetooth™ chip, a satellite navigation system (GPS) chip, a near-field communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio CODEC chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system-on-chip (SoC), an IC, or a similar circuit.

[0036] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0037] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0038] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0039] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0043] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0044] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0045] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0049] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0051] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0054] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0055] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0056] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0057] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, wide lenses or tele lenses.

[0059] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0060] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0061] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). One or more of the above image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) is at least part of the processor (120). It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after additional image processing by the processor (120).

[0062] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0063] FIG. 3 is a control block diagram of an electronic device according to one embodiment.

[0064] The overall operation of the electronic device (101) according to one embodiment may be controlled by a processor (320). The processor (320) may control the operation of the components provided in the electronic device (101) for capturing images. Specific control functions and operations of such a processor (320) will be described later.

[0065] The electronic device (101) may include a first camera (181) and a second camera (182) for capturing images. The first camera (181) and the second camera (182) may support different fields of view (FoV). According to one embodiment, the first camera (181) may support a first field of view, and the second camera (182) may support a second field of view that is wider than the first field of view. For example, the first camera (181) may be a wide camera, and the second camera (182) may be an ultra-wide camera. Additionally, the first camera (181) may be a telephoto camera, and the second camera (182) may be a wide camera. Although FIG. 3 is illustrated and described based on two cameras (180) for convenience of explanation, the electronic device (101) according to one embodiment may include three cameras, and each camera may have a different field of view.

[0066] According to one embodiment, the first camera (181) can acquire first phase difference information so that the processor (320) acquires first depth information, which is the distance to the object. Additionally, according to one embodiment, the second camera (182) can acquire second phase difference information so that the processor (320) acquires second depth information, which is the distance to the object. Here, even though the actual distance between the electronic device (101) and the object is the same, different phase difference information is acquired between the first camera (181) and the second camera (182), so the first depth information based on the first camera (181) and the second depth information based on the second camera may be different.

[0067] A memory (330) according to one embodiment can store first depth information and second depth information (including first phase difference information and second phase difference information) obtained through a camera (180). Additionally, a memory (330) according to one embodiment can store the first depth information and second depth information together with an image obtained through a camera (180).

[0068] A display (360) according to one embodiment can output an image (including a preview image) obtained through a camera (180).

[0069] The image sensor of the camera (180) (e.g., the image sensor (230) of FIG. 2) can acquire phase difference. The image sensor (230) can acquire phase difference information through focus detection. For example, the image sensor (230) may be provided in a matrix form. The arrangement of pixels in the image sensor (230) is sufficient if it can acquire data for focus detection, and it is obvious that it can be varied by a person with ordinary knowledge in the technical field of this document.

[0070] The image sensor (230) can provide phase difference data. In particular, the phase difference data is information used for focus detection by a phase difference focus detection method and can be used as focus information. For example, the focus information may include phase difference information provided by the image sensor (230). The phase difference data can be provided to the processor (320) through A / D (analog / digital) conversion. And, the operation of the image sensor (230) can be controlled by the processor (320) that processes focus detection.

[0071] The processor (320) can control the operation for capturing video (or images). The processor (320) can initiate the operation when the execution of a camera application is requested. The processor (320) can perform an autofocus (hereinafter referred to as 'AF') autofocus detection operation. Accordingly, the processor (320) can control the operation of an image sensor (230, FIG. 2) and an image stabilizer (240, FIG. 2) to detect autofocus of input image data. For example, the processor (320) can collect image data for focus detection by adjusting the operation of the image sensor (230). To do this, the processor (320) can adjust the exposure time, sensitivity (ISO), frame rate, etc. of the image sensor (230), and may issue a command to acquire high-resolution data for focus detection in a specific area. Additionally, the processor (320) can operate the image stabilizer (240) to correct camera shake and support more accurate focus detection. For example, a hand shake correction function may be activated to minimize image shaking during the autofocus process. Additionally, the operation of the stabilizer (240) may be temporarily paused or adjusted to analyze a specific focus area more accurately. For example, the processor (320) may check multiple focus information (e.g., phase difference information) provided by the image sensor (230) based on the lens position (hereinafter referred to as the 'initial lens position') during the initial process in which the image shooting operation of the electronic device (101) is initiated. Then, the processor (320) may estimate a lens position (hereinafter referred to as the 'first lens position') corresponding to a designated area (e.g., image center area or main subject area) (or an area designated by user input) based on the multiple focus information (e.g., phase difference value).The processor (320) can determine the relationship between the initial lens position and the estimated first lens position, and based on this, determine information necessary to move the lens from the initial lens position to the estimated first lens position (hereinafter referred to as 'lens movement information'). For example, the lens movement information may include at least one of the lens movement direction, lens movement distance, and lens movement speed. The processor (320) can use this lens movement information to move the lens to the estimated first lens position.

[0072] An image signal processor (260, FIG. 2) can provide focus information or lens movement information (e.g., lens movement direction, lens movement distance, lens movement speed, etc.) obtained from an operation to set autofocus to a processor (320). Then, when the AF function is performed, the processor (320) can use the focus information or lens movement information (e.g., lens movement direction, lens movement distance, lens movement speed, etc.) to create a focus movement indicator user interface (UI) that displays the movement of the lens and provide it to a display (360).

[0073] An image sensor (230) included in a camera module (180) according to one embodiment may obtain phase difference information for an external object using image data generated from a plurality of photodiodes, such as a processor (320), which has a plurality of light receiving units (not shown).

[0074] An image sensor (230) according to one embodiment can determine phase difference information regarding an external object by using a plurality of light receiving units that constitute a single unit pixel. For example, the image sensor (230) can determine phase difference information regarding an external object by using two photodiodes (not shown) that constitute a single unit pixel.

[0075] A processor (320) according to one embodiment can determine depth information, which is the distance to an external object, based on an external object identified using phase difference information identified through an image sensor (230) and a plurality of light receiving units composed of unit pixels.

[0076] A processor (320) according to one embodiment can generate one image data by combining two photodiodes when an input for image capture is detected. A unit pixel of the image sensor (230) can be composed of two photodiodes or four photodiodes.

[0077] According to one embodiment, the camera module (180) can acquire a light signal corresponding to an external object recognized through the image sensor (230). For example, the camera module (180) can identify phase difference information regarding the external object by using a plurality of light receiving units, such as at least two photodiodes, included in each pixel constituting the pixel array of the image sensor (230). For example, two or more PDs may be placed below the microlens. For example, in a 2PD method, a first PD and a second PD may be placed, and in a 4PD method, a first PD, a second PD, a third PD, and a fourth PD may be placed. A plurality of images can be acquired using the information received from each photodiode (PD). For example, in a 2PD method, a first image acquired from the first PD and a second image acquired from the second PD can be generated, and in a 4PD method, each image can be generated by utilizing combinations of the first PD + third PD and the second PD + fourth PD. The focus direction and focus position can be calculated by analyzing the phase difference between multiple acquired images. The phase difference is determined by comparing the temporal or spatial difference between signals acquired from each PD, and based on this, optimal focus adjustment can be performed. The image sensor (230) controls a lens driving device (e.g., an actuator) using the detected phase difference information, and thereby performs automatic focusing to achieve optimal focus.

[0078] The processor (320) can obtain depth information between the electronic device (101, or lens) and an external object based on the confirmed phase difference information. The processor (320) can store the depth information in memory (330).

[0079] FIG. 4 is a flowchart (400) of an operation method of an electronic device according to one embodiment.

[0080] FIG. 4 illustrates the operation when the first camera (181) and the second camera (182) are simultaneously activated. Simultaneous activation of the first camera (181) and the second camera (182) may occur temporarily during the camera switching process, and distance may be measured with only one camera activated based on third depth information.

[0081] An electronic device (101) according to one embodiment obtains phase difference information (first phase difference information) from a first camera (181) (410), and can obtain first depth information based on the phase difference information (420).

[0082] An electronic device (101) according to one embodiment can obtain phase difference information (second phase difference information) from a second camera (182) (430) and obtain second depth information based on the phase difference information (440).

[0083] An electronic device (101) according to one embodiment can simultaneously activate a first camera (181) and a second camera (182) to simultaneously acquire first depth information (and first phase difference information) and second depth information (and second phase difference information). Additionally, an electronic device (101) according to one embodiment may acquire only one depth information by activating only one of the first camera (181) and the second camera (182) to distinguish whether the distance to an external object increases or decreases. For example, an electronic device (101) according to one embodiment may acquire second depth information by activating only the second camera (182) while the first camera (181) is deactivated, or acquire first depth information by activating only the first camera (181) while the second camera (182) is deactivated.

[0084] An electronic device (101) according to one embodiment may determine a third depth information based on a first depth information or a second depth information (450). The third depth information is information that serves as a reference for camera switching, and either the first depth information or the second depth information may be considered as the third depth information, thereby allowing the selection of either the first camera (181) or the second camera (182) based on the third depth information.

[0085] Meanwhile, the third depth information may be the first or second depth information, but a correction value may be reflected in the first or second depth information. Therefore, the third depth information may have a value greater than or smaller than the first or second depth information. The correction value is a value determined by the camera's structure or external factors, and specific examples will be described later.

[0086] An electronic device (101) according to one embodiment can determine whether to switch cameras based on third depth information (460).

[0087] For example, a processor (320) according to one embodiment may disable the first camera (181) and enable the second camera (182) in accordance with a change in third depth information (e.g., a decrease in the distance between an external object and an electronic device) while outputting an image (preview image) to a display (360) through the first camera (181), thereby outputting an image obtained from the second camera (182) to the display (360). Additionally, a processor (320) according to one embodiment may disable the second camera (182) and enable the first camera (181) in accordance with a change in third depth information (e.g., an increase in the distance between an external object and an electronic device) while outputting an image (preview image) to a display (360) through the second camera (182), thereby outputting an image obtained from the first camera (181) to the display (360).

[0088] In another example, a processor (320) according to one embodiment may maintain the activation of the first camera (181) in accordance with a change in third depth information (e.g., an increase in the distance between an external object and an electronic device) while outputting an image (preview image) to a display (360) through the first camera (181). Additionally, a processor (320) according to one embodiment may maintain the activation of the second camera (182) in accordance with a change in third depth information (e.g., a decrease in the distance between an external object and an electronic device) while outputting an image (preview image) to a display (360) through the second camera (182).

[0089] According to one embodiment, the electronic device (101) may determine a third depth information based on at least one of whether the first camera (181) is active, whether the second camera (182) is active, or the result of comparing the first depth information and the second depth information.

[0090] FIG. 5 is a flowchart of a method for switching cameras in an electronic device according to one embodiment (500).

[0091] The embodiment of FIG. 5 illustrates an embodiment of depth information determination and camera switching.

[0092] In the embodiment of FIG. 5, the camera on which information is displayed on the display is defined as the main camera, and the camera on which information is not displayed is defined as the auxiliary camera, and the first camera (181) and the second camera (182) can operate as the main camera and the auxiliary camera according to the switching operation. Here, when the first camera (181) is ultra-wide, the second camera (182) can become wide, or when the first camera (181) is tele, the second camera (182) can become wide, or when the first camera (181) is tele, the second camera (182) can become ultra-wide.

[0093] The automatic switching function can repeatedly perform the operation of automatically switching to a second camera with a relatively smaller minimum focus distance to prevent image blurring when shooting close subjects due to the limitation of the first camera's minimum focus distance when the user initially sets the first camera as the default camera, and returning to the first camera when the distance increases.

[0094] An electronic device (101) according to one embodiment can execute a main camera-based function when an auxiliary camera is deactivated (510). For example, when a camera application is executed, the electronic device (101) according to one embodiment can prioritize activating the first camera (181) and output an image (preview image) obtained through the first camera (181) to a display (360).

[0095] An electronic device (101) according to one embodiment can acquire depth information (main depth information) through a main camera (520). The depth information is acquired through phase difference information, as described above. In the following, the main depth information is depth information acquired from the main camera, and the auxiliary depth information may be depth information acquired from the auxiliary camera.

[0096] An electronic device (101) according to one embodiment can check whether the final depth information exceeds a threshold (540). Here, the final depth information stores the depth information (main depth information) and the final depth calculated during a switching operation where both cameras are simultaneously activated, and uses the existing final depth when only the main camera is activated (530), and ignores it when there is no existing switching operation. Here, the threshold is a value determined based on the first camera (181), which has a relatively long minimum focal distance, and if the distance to the subject is closer than the threshold, the first camera cannot focus, so image blur may occur.

[0097] The above minimum focus distance refers to the distance at which the lens of the first camera (181) or the lens of the second camera (182) can get closest to an external object, and refers to the minimum distance between the lens and the subject at which no image blur occurs in the first camera (181) and the second camera (182), respectively, and for example, the minimum focus distance of the tele camera may be longer than the minimum focus distance of the wide camera.

[0098] According to one embodiment, the electronic device (101) can prepare for camera switching by activating an auxiliary camera when the final depth information exceeds the threshold of the main camera (540) (550).

[0099] According to one embodiment, the electronic device (101) can acquire depth information based on the main camera without activating the auxiliary camera when the final depth information does not exceed the threshold of the main camera (No of 540).

[0100] According to one embodiment, the electronic device (101) can acquire two depth information (main depth information and auxiliary depth information) simultaneously with the main camera and the auxiliary camera during a certain period of time when the auxiliary camera is activated (550) and stabilized during the switching process (560).

[0101] For example, the electronic device (101) can determine the final depth information based on at least one of two depth information (main depth information and auxiliary depth information) simultaneously acquired during a certain period of time during a switching operation (570).

[0102] According to one embodiment, the electronic device (101) switches the image (preview image) displayed on the display to the auxiliary camera when the distance of the subject exceeds the threshold of the main camera and the auxiliary camera is activated and reaches a stabilized state after a certain period. At this time, the existing main camera is deactivated and only the auxiliary camera is switched to the main camera to maintain an activated state (590).

[0103] According to one embodiment, the electronic device (101) can maintain a state where the main camera and the auxiliary camera are simultaneously activated when the auxiliary camera is not yet stabilized (No of 580), and can continuously acquire main depth information and auxiliary depth information.

[0104] FIG. 6 is a signal flow diagram (600) of depth information of an electronic device according to one embodiment.

[0105] An electronic device (101, FIG. 1) according to one embodiment may include a first camera (181) and a second camera (182) for capturing images. The first camera (181) and the second camera (182) may support different fields of view (FoV). According to one embodiment, the first camera (181) may support a first field of view, and the second camera (182) may support a second field of view that is wider than the first field of view. Meanwhile, FIG. 6 is illustrated and described based on two cameras for convenience of explanation, but the electronic device (101) according to one embodiment may include a third camera (not shown), and each camera may have a different field of view.

[0106] An electronic device (101) according to one embodiment may select only one of the first camera (181) or the second camera (182) as the main camera and acquire (or store) image and depth information through data acquired based on the main camera. An electronic device (101) according to one embodiment may acquire third depth information based on an image acquired from the first camera (181), an image acquired from the second camera (182), phase difference information, lens position information, correction value, and / or temperature information for automatic switching of cameras.

[0107] Meanwhile, an electronic device (101) according to one embodiment may include a first depth information acquisition unit (610) and a second depth information acquisition unit (620) to individually process phase difference information acquired from each of the first camera (181) and the second camera (182) to determine third depth information. The first depth information acquisition unit (610) may acquire first depth information based on phase difference information acquired through an image sensor (not shown) included in the first camera (181). The second depth information acquisition unit (620) may acquire second depth information based on phase difference information acquired through an image sensor (not shown) included in the second camera (182).

[0108] According to one embodiment, the first depth information acquisition unit (610) or the second depth information acquisition unit (620) can output each depth information and reliability based on data input from the first camera (or second camera) with relatively high distance accuracy and the second camera (or first camera) with relatively low distance accuracy, the static / dynamic calibration coefficient of each camera, and additional information such as sensing data and temperature data acquired from an inertia measurement unit (IMU).

[0109] The dynamic correction unit (630) calculates the distance error using a correction coefficient for the distance and reliability of each region of interest (ROI) calculated for each camera, calculates the value of the correction coefficient, and applies it to the first depth information or the second depth information to obtain more accurate third depth information.

[0110] The depth determination unit (640) may determine either the first depth information or the second depth information as the third depth information. The criteria for determining the third depth information are based on a set threshold and are as described above in FIG. 5. The depth determination unit (640) may output the third depth information, which is the final depth, based on the respective depth information (distance between the electronic device and the external object) calculated separately by the first depth information acquisition unit (610) and the second depth information acquisition unit (620). Meanwhile, although FIG. 6 describes determining the final depth information (third depth information) based on depth information, the final distance information may be determined based on distance information, and camera switching may be performed based on the final distance information. The depth information corresponds to the relative degree of defocus of a region of interest or pixel within the image, and the distance information may correspond to the distance between the camera and the external object.

[0111] During the camera switching process, a plurality of blocks (610 to 640) are operated in the section where the first camera (181) and the second camera (182) are turned on simultaneously. After the switching is completed, if only one of the first camera (181) or the second camera (182) is operated after a certain amount of time has elapsed, the depth information acquisition unit (610 or 620), dynamic correction unit (630), and / or depth determination unit (640) associated with the camera without input may be turned off.

[0112] FIGS. 7A and 7B illustrate various error types and flowcharts regarding error types that may occur depending on the camera switching direction in an electronic device according to one embodiment. Cases that may occur depending on the depth information of the first camera (Wide) and the second camera (Ultra Wide) during camera switching are illustrated. In the initial state where the first camera, the Wide camera, is activated by user settings, when the distance to the subject becomes close to or below the first threshold (F2N Threshold), the second camera, the Ultra Wide camera, is activated. At this time, the main camera displayed on the display remains as the Wide camera for a certain period, and during this period, the Wide and Ultra Wide cameras are simultaneously activated (720 in FIG. 7B). In the first direction of switching from the Wide camera to the Ultra Wide camera, there are three cases depending on the depth values ​​of the two cameras (730, 740, and 750 in FIG. 7B). When the depth values ​​of the two cameras are the same (730 in FIG. 7B), no error occurs as this is an ideal operation. On the other hand, if the wide depth value is large and the ultra-wide depth value is small (740 in FIG. 7b), if the ultra-wide value is set to the third depth, an error occurs during the second direction change where the switch occurs at a distance greater than the second threshold (N2F Threshold) (770 in FIG. 7b). If the ultra-wide depth value is larger than the wide depth value (750 in FIG. 7b), toggling occurs because the third depth value, which is ultra-wide, is greater than the second threshold. Even in the case of the second direction change from ultra-wide to wide (730, 760, 770 in FIG. 7b), ideal operation is achieved when the two values ​​are the same (730 in FIG. 7b).When the ultra-wide value is greater than the wide value (760 in FIG. 7b), switching occurs at a distance smaller than the second threshold (N2F Threshold). If the wide depth value is set to the third depth, a toggling error occurs again when it is smaller than the first threshold (F2N Threshold) (760 in FIG. 7b). Conversely, an error occurs when switching occurs at a distance greater than the second threshold (N2F Threshold) (770 in FIG. 7b).

[0113] FIGS. 8 to 13 illustrate a third depth (po) detection method that prevents toggling or switching distance errors by referencing at least one piece of information in each case where the first camera of FIG. 7 is wide and the second camera is ultra-wide. pw, pu, and po in FIGS. 8 to 13 represent the first depth information, second depth information, and third depth information of FIG. 6. During the first direction switching process, an inaccurate depth information section appears due to an unstable transient section at the beginning when the ultra-wide camera is activated (E0).

[0114] FIG. 8 illustrates a first error type (Case 1) occurring when the camera switching direction of an electronic device according to one embodiment is a first direction, FIG. 9 illustrates a second-1 error type (Case 2-1) occurring when the camera switching direction of an electronic device according to one embodiment is a first direction, FIG. 10 illustrates a second-2 error type (Case 2-2) occurring when the camera switching direction of an electronic device according to one embodiment is a first direction, FIG. 11 illustrates a third error type (Case 3) occurring when the camera switching direction of an electronic device according to one embodiment is a second direction, FIG. 12 illustrates a fourth-1 error type (Case 4-1) occurring when the camera switching direction of an electronic device according to one embodiment is a second direction, and FIG. 13 illustrates a fourth-2 error type (Case 4-2) occurring when the camera switching direction of an electronic device according to one embodiment is a second direction. Various error types of FIG. 7b will be explained with reference to FIG. 8 to FIG. 13.

[0115] According to one embodiment, the electronic device (101) can detect the occurrence of a camera switching event (710 in FIG. 7a). The camera switching event may occur to determine the camera for outputting an image (preview image) or to determine depth information used in the image when the distance between the electronic device (101) and an external object increases or decreases.

[0116] A camera switching event may occur when the third depth information decreases below the first threshold (F2N Threshold) during the execution of the first camera-based function, or when the third depth information exceeds the second threshold (N2F Threshold) during the execution of the second camera-based function. The first threshold (F2N Threshold) is set to a value smaller than the second threshold (N2F Threshold) by a certain margin to prevent toggling caused by noise as a hysteresis threshold.

[0117] According to one embodiment, the electronic device (101) can determine the camera switching direction (720 in FIG. 7a). Here, the first direction is when the distance between the electronic device (101) and an external object becomes close (F2N, Far to Near), meaning the case where the camera is switched from the first camera (181) to the second camera (182), and the second direction is when the distance between the electronic device (101) and an external object becomes far (N2F, Near to Far), meaning the case where the camera is switched from the second camera (182) to the first camera (181).

[0118] When the camera's switching direction is the first direction, the electronic device (101) according to one embodiment can activate the second camera (182) (731 in FIG. 7a). The second camera (182) has a smaller minimum focal length than the first camera (181), so that image blurring that may occur when shooting with the first camera (181) can be prevented when the distance between the electronic device (101) and an external object is relatively close.

[0119] In one embodiment, the electronic device (101) can check second depth information obtained through the second camera (182) when the first camera (181) is not activated (733 in FIG. 7a or the second camera activation section in FIG. 7b) (735 in FIG. 7a or 760, 770 in FIG. 7b). According to one embodiment, the electronic device (101) can determine a first reference value as third depth information instead of the second depth information obtained through the second camera (182) (737 in FIG. 7a or 740 in FIG. 7b).

[0120] In this regard, referring to FIG. 9, in section E2 of the transition from the first camera (Wide) to the second camera (Ultra Wide), the first reference value (p2w0), which is the depth information last acquired from the first camera, can be determined as the third depth information, which is the final depth information. In section E2, the second camera is actually activated, but since there is a gap between the depth information acquired from the first camera and the depth information acquired from the second camera, the depth information last acquired from the first camera (the first reference value), which has relatively high distance accuracy, is maintained as the third depth information. Then, at the point where the second depth becomes the same value as the third depth information, the third depth information follows the second depth value, thereby preventing unnecessary camera re-transition.

[0121] In one embodiment, the electronic device (101) can check the second depth information obtained through the second camera (182) when the first camera (181) is not activated (733 in FIG. 7a) (735 in FIG. 7a). According to one embodiment, the electronic device (101) can determine the second depth information obtained through the second camera (182) as the third depth information (739 in FIG. 7a). In this regard, referring to FIG. 9, in section E2 of the section switched from the first camera (Wide) to the second camera (Ultra Wide), a first reference value (p2 w0 After applying depth information for a certain period of time based on ), the reliability of the depth information based on the second camera can be high. Therefore, in the section after E2-, the second depth information obtained from the second camera can be determined as the third depth information.

[0122] According to one embodiment, the electronic device (101) may determine the first depth information as the third depth information when the first camera (181) is activated after the second camera (182) is activated in a state where the first direction of the camera's switching direction (when the distance between the electronic device and the subject decreases). The electronic device (101) according to one embodiment may deactivate the first camera (181) when the first depth information reaches a first reference value (p2w0) (the section after E2 in FIG. 9 or the section between E3-1 and E3-2 in FIG. 10). Then, after the first camera (181) is deactivated, the first reference value (p2w0) may be determined as the third depth information based on the fact that the second depth information is greater than the first reference value (p2w0) (the section after E2 in FIG. 9). According to one embodiment, the electronic device (101) may determine the second depth information as the third depth information based on whether the second depth information is less than or equal to a first reference value (p2w0) or whether the difference between the second depth information and the minimum value of the second depth information is greater than a second threshold (N2F Threshold) (section E3-2 of FIG. 10).

[0123] Meanwhile, according to one embodiment, FIG. 10 illustrates a case where the second depth is greater than the first depth when the first direction is changed, and the camera's direction of movement from the subject changes while the first camera is deactivated after the E3-1 section. In Case 2-1 of FIG. 9, the camera's direction of movement is the same, so the difference between the second depth and the third depth decreases over time, but in Case 2-2 of FIG. 10, the camera's direction of movement changes, so the difference between the second depth and the third depth value increases. In this case as well, to prevent re-switching, the third depth is made to follow the second depth when it moves further away than a certain threshold (TH) relative to the minimum value (p2min).

[0124] The electronic device (101) has the first depth information as a first reference value (p2) when the first camera (181) is activated (example of FIG. 7a, 733) (first camera activation section of FIG. 7b). w0 It can be checked whether it is smaller than (753 in FIG. 7a). In this embodiment, the second camera (182) is in an activated state, and the first camera (181) is in an activated state along with the activation of the second camera (182), or the first depth information is a first reference value (p2 w0 It may be in a reactivated state after being deactivated (751 in FIG. 7a) based on being smaller than (example of 753 in FIG. 7a). In one embodiment, the first depth information is a first reference value (p2 w0 If it is not smaller than ), the first depth information can be determined as the third depth information (755 in FIG. 7a). That is, the first depth information is the first reference value (p2 w0 If the depth information based on the second camera is larger than ) and there is a significant gap, the first depth information can be determined as the final depth information to prevent future re-conversion. Related cases may include section E1 in FIG. 8, the section prior to E2 in FIG. 9, and section E3-1 in FIG. 10.

[0125] FIG. 8 illustrates the process of switching from a main camera (Wide) to an auxiliary camera (Ultra Wide). Initially, the first depth information (Pw) of the main camera continuously decreases, and eventually, the first depth information falls below the first threshold (F2N Threshold). When the first depth information falls below the first threshold, the electronic device (101) can acquire the second depth information (Pu) of the auxiliary camera. Immediately after switching, there may be a temporary difference in distance between the main camera and the auxiliary camera, but since the electronic device (101) performs camera switching based on the third depth information (Po, see FIG. 6), stable camera switching can be performed without toggling.

[0126] When the camera's switching direction is the second direction, the electronic device (101) according to one embodiment can activate the first camera (181) (741). The first camera (181) has a relatively smaller angle of field of view than the second camera (182), and the operation is to return to the first camera initially set by the user when the distance between the electronic device (101) and an external object is relatively far.

[0127] In one embodiment, the electronic device (101) can obtain first depth information from the first camera (181) (743).

[0128] In one embodiment, the electronic device (101) may determine the switching depth value (p2u0 in FIG. 12) as third depth information (745 in FIG. 7a). Referring to FIG. 12 in relation thereto, after switching from the second camera (182) to the first camera (181), if the first depth information of the first camera (181) is smaller than the first threshold (F2N Threshold) (section E5-1), toggling may occur if the third depth information follows the first depth. Accordingly, the electronic device (101) may determine the switching depth value, which is the last depth information when the camera is switched, as third depth information.

[0129] In one embodiment, the electronic device (101) may activate the first camera based on the fact that the second depth information is greater than the second threshold (N2F Threshold) while the first camera is deactivated and the second camera is activated (wide section of FIG. 12 and FIG. 13). Based on the fact that the first camera is activated, the electronic device (101) may switch the camera used to display a display preview image or store an image in memory to the first camera while the third depth information is the switching depth value. After the first camera is activated, the electronic device (101) may determine the switching depth value as the third depth information based on the fact that the first depth information is smaller than the switching depth value (E5-2 of FIG. 12). Additionally, the electronic device (101) can prevent toggling that may occur when the camera's direction of movement changes by determining the first depth information as the third depth information based on whether the first depth information is greater than or equal to the switching depth value, or whether the difference between the maximum value of the first depth information and the first depth information is greater than or equal to the fourth threshold (TH in FIG. 10 or TH in FIG. 13).

[0130] Meanwhile, FIG. 13 illustrates a case where there is a risk of malfunction because the depth information of the auxiliary camera is acquired unstably after switching from the main camera (Ultra Wide) to the auxiliary camera (Wide). The electronic device (101) can perform a switch from the main camera to the auxiliary camera when the depth information of the main camera increases and exceeds a second threshold (N2F Threshold). Immediately after the switch, the initial depth information (Pw) acquired from the auxiliary camera has a significant difference from the depth information acquired from the main camera. In this case, the camera switch can be performed based on the depth information of the main camera until the change in depth information in the reversed direction exceeds a certain value (TH).

[0131] According to one embodiment, the electronic device (101) can determine whether to switch the camera used to display a preview image on a display or store an image in memory to a second camera based on third depth information.

[0132] Meanwhile, embodiments have been described above for determining at least one of the first depth information and the second depth information as the third depth information when a camera switching event occurs. Before determining the third depth information, the electronic device (101) can derive more accurate third depth information by applying a correction value to the first depth information or the second depth information. Below, a method is described for minimizing errors and preventing malfunctions in various situations by detecting errors that may occur during the camera operation process in real time and changing the correction coefficient.

[0133] FIG. 14 is a flowchart (1400) of an operation in which an electronic device according to one embodiment corrects depth information based on lens position information.

[0134] For example, the camera lens moves according to the focus by means of an actuator, and the moved lens position can be detected by a Hall sensor included in the camera module. Meanwhile, if a positional difference occurs due to minute gaps between the mechanical structures for lens fixation, the lens may shift its position due to gravity. If the amount of lens movement changed by gravity is not accurately detected through the Hall sensor, errors may occur in the first depth information and / or the second depth information. Therefore, correction considering the positional difference of the lens is required.

[0135] According to one embodiment, the electronic device (101) can acquire lens position information (1410). The lens position information can be acquired through a signal from a Hall sensor provided in a camera module.

[0136] According to one embodiment, the electronic device (101) can determine a lens stroke value from lens position information by performing at least one compensation operation (1420). The lens position information may correspond to the absolute position of the lens determined by an Optical Image Stabilization (OIS) or Auto Focus (AF) function. That is, the lens position information represents the coordinates or physical position where the lens is currently located within the optical system, and this can be controlled by a lens driving device (actuator, etc.). For example, an OIS system may move the lens to a specific position to compensate for external vibrations or movements, and an AF system may move the lens forward or backward to focus on a subject. The lens position information measured during this process can be stored and utilized as the absolute spatial position value of the lens. Meanwhile, the lens stroke value (relative to the AF axis) may correspond to a relative value of the actual movement of the lens. That is, the lens stroke value may be defined as a value representing the distance or displacement amount of the lens moved relative to a specific reference position (e.g., initial lens position). This is a value for quantitatively measuring changes in lens position and can reflect relative position changes that occur according to specific lens drive commands (e.g., AF drive, OIS adjustment). For example, if the lens moves forward by 0.5mm from its initial position, the stroke value of the lens can be recorded as +0.5mm, and if it moves backward, it can be represented as a negative value.

[0137] At least one compensation operation may include: processing a bilateral filter to reduce noise by averaging highly correlated regions of interest (ROIs) when there are multiple ROIs in an image; synchronization compensation to remove spike noise values ​​that occur when acquiring depth information for each image frame and obtaining an average value; defocus compensation to acquire depth information by restricting the ROI to a certain area; distortion compensation to acquire depth information by considering that the amount of defocus varies according to the image height, which is the distance from the center of the image, taking into account lens characteristics; hysteresis compensation to consider displacement errors that occur according to the direction of movement due to the mechanical characteristics of the lens actuator; lens position compensation to consider the difference in lens position; and temperature compensation to consider the change in refractive index according to the lens temperature. Here, since it is difficult to directly measure the temperature of the lens itself, temperature compensation can be indirectly estimated through the temperature of the image sensor.

[0138] According to one embodiment, the electronic device (101) can determine depth information based on a lens stroke value and / or focal length (1430). Here, the lens stroke value may be a value that reflects at least one compensation operation described above. Accordingly, the depth information determined in operation 1430 corresponds to a value that reflects a correction value in the first depth information and / or second depth information. The focal position may correspond to the position of at least one lens related to the focal point of a lens unit composed of multiple lenses. That is, the position where the focal point is formed may be determined according to the arrangement and movement of individual lenses within the lens unit. Additionally, the position of the corresponding lens may be adjusted through the physical movement of the lens, and the arrangement and movement of the lens may be controlled using control values ​​(e.g., lens position, lens movement amount, stroke, etc.) to control this.

[0139] According to one embodiment, an electronic device (101) can acquire lens position information from a first camera (or a second camera) and synchronize the lens position information with phase difference information acquired from the first camera (or the second camera). Here, the lens position information may be acquired through a Hall sensor or the like included in the camera. The electronic device (101) can acquire first depth information (or second depth information) based on the lens position information synchronized with the phase difference information.

[0140] According to one embodiment, the electronic device (101) may acquire lens position information from a first camera (or a second camera) and determine a correction value for compensating for hysteresis of the first camera (or a second camera) based on the lens position information. The electronic device (101) may acquire first depth information (or second depth information) based on the determined correction value.

[0141] According to one embodiment, the electronic device (101) may acquire movement information regarding the movement of the electronic device and determine a correction value based on the movement information. The electronic device (101) may acquire first depth information (or second depth information) based on the determined correction value.

[0142] FIG. 15 is a control block diagram of a depth information acquisition unit (1500) according to one embodiment.

[0143] In FIG. 15, the depth information acquisition unit (1500) may include the first depth information acquisition unit (610) or the second depth information acquisition unit (620) of FIG. 6.

[0144] First, the depth information acquisition unit (1500) receives image data from a camera (first camera or second camera) and can change the image size in pixels through image resizing (1501) and transmit it to a spatial filter (1503, Bilateral filter). The spatial filter (1503) can reduce noise in the image signal by calculating the correlation coefficient between multiple regions of interest (ROI) in the image and averaging at least one ROI with a correlation.

[0145] Distortion compensation (1505) is a correction value (Δz) to compensate for defocus according to the image height, which is the distance from the center of the image due to the characteristics of the lens. d0 ) can be provided to the spatial filter (1503).

[0146] Defocus compensation (1507) is a correction value (k) for compensating for depth information by restricting the region of interest to a certain area. md ) can be provided to the spatial filter (1503).

[0147] Statistics (1509) are based on the sum of absolute differences (SAD) with noise reduced by the spatial filter (1503), and the disparity (Dp) and confidence (R bo You can obtain ).

[0148] Correction value (Δz) to compensate for defocus d0 ) and the correction value (k), which is a coefficient for converting the defocus amount in disparity md Since ) has different values ​​for each ROI, spatial filter (1503) processing is performed, such as SAD (Sum of Absolute Difference). SAD, synchronization compensation (1511), and delay block (1513) can suppress spike noise caused by asynchronous operation.

[0149] Meanwhile, the lens actuator (not shown) has a hysteresis characteristic in which its position changes depending on the direction. The disparity measured when moving from near to far and from far to near relative to an external object may have different values. Hysteresis compensation (1515) is a third offset value (Δz) for compensating for the hysteresis characteristic. hd Based on ), the hysteresis compensation value (Δz q ) can be provided. In addition, the orientation difference compensation (1517) can provide a lens position compensation value (Δz) to compensate for the orientation difference of the lens. p It can provide ).

[0150] Temperature compensation (1519) can provide a second offset value (Δq0) to compensate for changes in refractive index according to the temperature of the lens. If the lens is plastic, the error caused by changes in refractive index according to temperature can be estimated by modeling the temperature (Ts) of the image sensor with an IIR (Infinite Impulse Response) filter and converting it into the lens temperature (Tl).

[0151] As described above, a depth information acquisition unit (1500) according to one embodiment can perform at least one compensation operation (at least one of 1503 to 1519). The depth information acquisition unit (1500) can acquire a lens stroke value (q) through the compensation operation. And, depth information (p) can be acquired by a lens model (1521).

[0152] The focal length (f) can remove spike noise using a median value filter (1523), and after determining the distance (Main Depth) of an external object (1525), additional noise can be removed using a temporal filter (Temporal Filter, 1527).

[0153] Overall, the depth information acquisition unit (1500) can acquire depth information (p) and the distance (q) between the image and the lens based on the following mathematical formulas 1 and 2.

[0154] [Mathematical Formula 1]

[0155]

[0156] (f: Focal length [mm]

[0157] p: Depth from object to lens [mm]

[0158] q: image to lens distance [mm]

[0159] Δp (1st offset value): Depth error [mm])

[0160] [Mathematical Formula 2]

[0161]

[0162] q: image to lens distance [mm]

[0163] z q0 : Current lens position [code]

[0164] Δz q: Hysteresis compensation value[code]

[0165] Δz D : Defocus compensation value [code]

[0166] Δz d : Distortion compensation value [code]

[0167] Δz p : Lens position compensation value [code]

[0168] k q : Lens position (z q ) to effective lens stroke (q) coefficient [mm / code]

[0169] q0: Lens position (z q ) to Lens stroke (q) offset at 0 o C [mm]

[0170] Δq T : Lens stroke difference by temperature [mm]

[0171] α d : k md compensation factors

[0172] k md1,0 : Disparity (D p ) to Lens position difference (Δz D ) coefficient by module calibration

[0173] D p : Disparity [pixel]

[0174] Δz d0 : Difference of onfocus position from center ROI

[0175] az : Normalized z directional acceleration

[0176] Δz pz : z axis directional lens movement by acceleration

[0177] q 0i : Lens position (z q ) to Lens stroke (q) offset at T0[mm]

[0178] ΔT s :: Module calibration correction temperature [ o C]

[0179] k T : Temperature compensation coefficient [mm / o C]

[0180] Δt: Elapse time from sensor turn on [sec]

[0181] T i (Δt): Temperature of lens [ o C]

[0182] )

[0183] Meanwhile, in addition to the depth information acquisition unit (1500) performing at least one compensation operation, the present disclosure implements a dynamic calibration function to detect the distance error between cameras occurring during the actual operation process in real time and change the calibration coefficient, thereby resolving the problems that occur in static calibration. Below, dynamic calibration will be described with reference to FIG. 16.

[0184] FIG. 16 is a flowchart (1600) for dynamic correction according to one embodiment.

[0185] According to one embodiment, the electronic device (101) can obtain first depth information (or second depth information) (1610).

[0186] According to one embodiment, the electronic device (101) can convert a region of interest for a first image (or a second image) into coordinates within the second image (or the first image) (1620). Specifically, the electronic device (101) can map to the image coordinates of the second camera (or the first camera) using the distance of an object per region of interest of the first camera (or the second camera) and stereo camera parameters.

[0187] According to one embodiment, the electronic device (101) can determine the position between coordinates (1630). Specifically, the center of the region of interest of the first image (or second image) can be reflected in the coordinates for the second image (or first image) by the mapping described above. Unlike the fixed center of the region of interest of the second image, the region of interest of the first image can be mapped unevenly due to distortion and coordinate differences based on distance. Here, the coordinate difference refers to the difference in the position of an object on the image (due to parallax) caused by the difference between the viewpoint of the first camera and the viewpoint of the second camera. The distance error between the first image and the second image can be calculated for each region of interest corresponding to each object within the field of view. To this end, the operation of calculating the region of interest of the first image (or second image) corresponding to the region of interest of the second image (or first image) is 1630. A nearest neighbor can be used to select the region of interest in the first image (or second image) closest to the region of interest in the second image (or first image) using interpolation.

[0188] According to one embodiment, the electronic device (101) can determine the distance error between cameras (1640). Specifically, to obtain the distance error between the first camera and the second camera, the distance of each camera can be averaged with different weights depending on the location, reliability, and distance from the object of each region of interest.

[0189] According to one embodiment, the electronic device (101) determines at least one parameter (1650) based on a determined error. The at least one parameter is a first offset value (Δp), a second offset value (Δq0), or a third offset value (ΔZ hd It may include at least one of ).

[0190] According to one embodiment, the electronic device (101) can obtain second depth information (or first depth information) based on at least one parameter (1660).

[0191] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs.

[0192] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1) may include a first camera (e.g., the first camera (181) of FIG. 3) that supports a first field of view (FoV), a second camera (e.g., the second camera (182) of FIG. 3) that supports a second field of view that is wider than the first field of view, a display (e.g., the display module (160) of FIG. 1), at least one processor (e.g., the processor (120) of FIG. 1) that includes processing circuitry, and a memory (e.g., the memory (130) of FIG. 1) that stores instructions. The above instructions may be executed individually or collectively by at least one processor to enable an electronic device to: display a preview image on a display or store an image in memory based on an image acquired through a first camera while the second camera is disabled; acquire first depth information based on phase difference information acquired using the first camera; and enable the second camera based on the value of the first depth information being less than a first threshold. The above instructions may enable, after the second camera is enabled: to determine a third depth information based on at least one of the first depth information or the second depth information based on phase difference information acquired using the second camera; and to determine whether to switch the camera used to display a preview image on a display or store an image in memory to the second camera based on the fact that the second camera is enabled, and to switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the third depth information.

[0193] Instructions according to one embodiment may enable an electronic device to: determine at least one parameter based on first depth information, and obtain second depth information from phase difference information obtained using a second camera based on the determined at least one parameter.

[0194] According to one embodiment, at least one parameter may include at least one of a first offset value for a determined depth, a second offset value for lens stroke information, or a third offset value associated with a compensation action for hysteresis.

[0195] Instructions according to one embodiment may cause an electronic device to determine a third depth information based on at least one of whether the first camera is activated, whether the second camera is activated, or the result of comparing the first depth information and the second depth information.

[0196] Instructions according to one embodiment may cause an electronic device to: determine the first depth information as the third depth information when the first camera is activated after the second camera is activated, and to deactivate the first camera when the first depth information corresponds to a first reference value. The instructions may cause the first reference value to be determined as the third depth information based on the fact that the second depth information is greater than the first reference value after the first camera is deactivated, and to determine the second depth information as the third depth information based on the fact that the second depth information is less than or equal to the first reference value or that the difference between the second depth information and the minimum value of the second depth information is greater than a second threshold.

[0197] Instructions according to one embodiment may cause an electronic device to: enable the first camera based on the fact that the third depth information is greater than the second threshold when the first camera is disabled and the second camera is enabled, and switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the fact that the first camera is enabled and the third depth information is a transition depth value. The instructions may, after the first camera is enabled: determine the transition depth value as the third depth information based on the fact that the first depth information is less than the transition depth value, determine the first depth information as the third depth information based on the fact that the first depth information is greater than or equal to the transition depth value or the difference between the maximum value of the first depth information and the first depth information is greater than or equal to the fourth threshold, and determine whether to switch the camera used to display a preview image on a display or store an image in memory to the second camera based on the third depth information.

[0198] Instructions according to one embodiment may enable an electronic device to: acquire temperature information from an image sensor of a first camera, determine a correction value based on the temperature information, and acquire first depth information based on the determined correction value.

[0199] Instructions according to one embodiment may enable an electronic device to: acquire lens position information from a first camera, synchronize the lens position information with phase difference information acquired from the first camera, and acquire first depth information based on the lens position information synchronized with the phase difference information.

[0200] Instructions according to one embodiment may obtain lens position information from a first camera, determine a correction value for compensating for hysteresis phenomena of the first camera based on the lens position information, and obtain first depth information based on the determined correction value.

[0201] Instructions according to one embodiment may enable an electronic device to: acquire movement information regarding the movement of the electronic device, determine a correction value based on the movement information, and acquire first depth information based on the determined correction value.

[0202] A method of operation of an electronic device according to one embodiment may include, when the second camera is deactivated, an operation of displaying a preview image on a display or storing an image in memory based on an image acquired through a first camera, an operation of acquiring first depth information based on phase difference information acquired using the first camera, and an operation of activating the second camera based on the fact that the value of the first depth information is less than a first threshold. The method of operation of the electronic device may include, after the second camera is activated: an operation of determining third depth information based on at least one of the first depth information or second depth information based on phase difference information acquired using the second camera, an operation of switching the camera used to display a preview image on a display or store an image in memory to the second camera based on the fact that the second camera is activated, and an operation of determining whether to switch the camera used to display a preview image on a display or store an image in memory to the first camera based on the third depth information.

[0203] A method of operation of an electronic device according to one embodiment may further include an operation of determining at least one parameter based on first depth information and an operation of obtaining second depth information from phase difference information obtained using a second camera based on the determined at least parameter.

[0204] According to one embodiment, at least one parameter may include at least one of a first offset value for a determined depth, a second offset value for lens stroke information, or a third offset value associated with a compensation action for hysteresis.

[0205] A method of operation of an electronic device according to one embodiment may further include an operation of determining a third depth information based on at least one of whether the first camera is in an activated state, whether the second camera is in an activated state, or the result of comparing the first depth information and the second depth information.

[0206] A method of operation of an electronic device according to one embodiment may include: after the second camera is activated: when the first camera is activated, determining the first depth information as the third depth information; and deactivating the first camera when the first depth information corresponds to a first reference value. The method of operation of the electronic device may further include: after the first camera is deactivated: determining the first reference value as the third depth information based on the fact that the second depth information is greater than the first reference value; and determining the second depth information as the third depth information based on the fact that the second depth information is less than or equal to the first reference value, or that the difference between the second depth information and the minimum value of the second depth information is greater than a second threshold.

[0207] A method of operation of an electronic device according to one embodiment may include: an operation of activating the first camera based on the fact that the third depth information is greater than a second threshold when the first camera is deactivated and the second camera is activated; and an operation of switching the camera used to display a preview image on a display or store an image in memory to the first camera based on the fact that the first camera is activated and the third depth information is a switching depth value. The method of operation of the electronic device may further include: after the first camera is activated: an operation of determining the switching depth value as the third depth information based on the fact that the first depth information is less than the switching depth value; an operation of determining the first depth information as the third depth information based on the fact that the first depth information is greater than or equal to the switching depth value, or that the difference between the maximum value of the first depth information and the first depth information is greater than or equal to a fourth threshold; and an operation of determining whether to switch the camera used to display a preview image on a display or store an image in memory to the second camera based on the third depth information.

[0208] A method of operation of an electronic device according to one embodiment may further include an operation of acquiring temperature information from an image sensor of a first camera, an operation of determining a correction value based on the temperature information, and an operation of acquiring first depth information based on the determined correction value.

[0209] A method of operation of an electronic device according to one embodiment may further include an operation of acquiring lens position information from a first camera, an operation of synchronizing the lens position information with phase difference information acquired from the first camera, and an operation of acquiring first depth information based on the lens position information synchronized with the phase difference information.

[0210] A method of operation of an electronic device according to one embodiment may further include the operation of acquiring lens position information from a first camera, the operation of determining a correction value for compensating for hysteresis phenomena of the first camera based on the lens position information, and the operation of acquiring first depth information based on the determined correction value.

[0211] A method of operating an electronic device according to one embodiment may further include an operation of acquiring movement information regarding the movement of the electronic device, an operation of determining a correction value based on the movement information, and an operation of acquiring first depth information based on the determined correction value.

[0212] According to the disclosed embodiment, an automatic switching function between multiple cameras can be implemented using the subject distance calculated solely from camera information without using a separate distance sensor (e.g., ToF, LiDAR). Since a separate distance sensor is not required, the effects of reduced cost, reduced mounting space, and reduced power consumption can be achieved.

[0213] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0214] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0215] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0216] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0217] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0218] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0219] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0220] It will be understood that various embodiments of the present disclosure according to the claims and the description of the specification may be implemented in the form of hardware, software, or a combination of hardware and software.

[0221] Such software may be stored on a non-transitory computer-readable storage medium (recording medium). The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs include computer execution instructions that cause the electronic device to perform the method of the present disclosure when executed by one or more processors of the electronic device.

[0222] Such software may be stored in a volatile or non-volatile storage form, for example, in a storage device such as read-only memory (ROM), regardless of whether it is erasable or rewritable, or in a memory such as random access memory (RAM), memory chips, devices, or integrated circuits (IC), or in an optical or magnetic reading medium such as a compact disc (CD), digital video disc (DVD), magnetic disc, magnetic tape, etc. It will be understood that the storage device and storage medium are various embodiments of a non-transient machine-readable storage medium suitable for storing computer programs or computer programs that include instructions for implementing various embodiments of the present disclosure at execution. Accordingly, various embodiments provide a program including code for implementing an apparatus or method described in any claim of this specification, and a non-transient machine-readable storage medium for storing such a program.

[0223] Although the present disclosure has been described and illustrated with reference to various embodiments, those skilled in the art will understand that various changes in form and detail are possible without departing from the spirit and scope of the present disclosure as defined by the appended claims and equivalents.

Claims

1. In an electronic device, A first camera supporting a first field of view (FoV); A second camera that supports a second field of view, which is an angle wider than the first field of view; display; At least one processor including processing circuitry; and It includes memory for storing instructions, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: When the second camera is deactivated, a preview image is displayed on the display or an image is stored in the memory based on an image obtained through the first camera. First depth information is obtained based on phase difference information obtained using the first camera, and Activating the second camera based on the fact that the value of the first depth information is less than the first threshold, and After the above second camera is activated: A third depth information is determined based on at least one of the first depth information or the second depth information based on phase difference information obtained using the second camera, and Based on the activation of the second camera, the camera used to display a preview image on the display or store an image in the memory is switched to the second camera, and An electronic device that determines whether to switch the camera used to display a preview image on the display or store an image in the memory to the first camera based on the third depth information.

2. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Determine at least one parameter based on the above first depth information, and An electronic device that obtains the second depth information from the phase difference information obtained using the second camera based on at least the parameters determined above.

3. In Paragraph 2, An electronic device wherein the above-mentioned at least one parameter comprises at least one of a first offset value for a determined depth, a second offset value for lens stroke information, or a third offset value associated with a compensation operation for hysteresis.

4. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: An electronic device that determines the third depth information based on at least one of whether the first camera is activated, whether the second camera is activated, or the result of comparing the first depth information and the second depth information.

5. In Paragraph 1 or Paragraph 4, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: After the above second camera is activated: When the first camera is activated, the first depth information is determined as the third depth information, and The first camera is disabled while the first depth information corresponds to a first reference value, and After the above-mentioned first camera is deactivated: Based on the fact that the second depth information is greater than the first reference value, the first reference value is determined as the third depth information, and An electronic device that determines the second depth information as the third depth information based on whether the second depth information is less than or equal to the first reference value, or whether the difference between the second depth information and the minimum value of the second depth information is greater than the second threshold.

6. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: With the first camera deactivated and the second camera activated, the first camera is activated based on the fact that the third depth information is greater than the second threshold, and Based on the activation of the first camera, the camera used to display a preview image on the display or store an image in the memory is switched to the first camera while the third depth information is a switched depth value, and After the first camera mentioned above is activated: Based on the fact that the first depth information is smaller than the transition depth value, the transition depth value is determined as the third depth information, and Based on the fact that the first depth information is greater than or equal to the conversion depth value, or that the difference between the maximum value of the first depth information and the first depth information is greater than or equal to the fourth threshold, the first depth information is determined as the third depth information. An electronic device that determines whether to switch the camera used to display the display preview image or store the image in the memory to the second camera based on the third depth information.

7. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Temperature information is obtained from the image sensor of the first camera, and A correction value is determined based on the above temperature information, and An electronic device that obtains the first depth information based on the above-determined correction value.

8. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Lens position information is obtained from the first camera, and The above lens position information is synchronized with the phase difference information obtained from the first camera, and An electronic device that obtains the first depth information based on the lens position information synchronized with the phase difference information.

9. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Lens position information is obtained from the first camera, and Based on the above lens position information, a correction value for compensating for hysteresis of the first camera is determined, and An electronic device that obtains the first depth information based on the above-determined correction value.

10. In Paragraph 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Acquiring movement information regarding the movement of the above electronic device, and A correction value is determined based on the above movement information, and An electronic device that obtains the first depth information based on the above-determined correction value.

11. A method of operating an electronic device including a first camera and a second camera, An operation of displaying a preview image on a display or storing an image in memory based on an image acquired through the first camera while the second camera is deactivated; An operation of acquiring first depth information based on phase difference information acquired using the first camera; An operation to activate the second camera based on the fact that the value of the first depth information is less than a first threshold; After the above second camera is activated: An operation of determining third depth information based on at least one of the first depth information or second depth information based on phase difference information obtained using the second camera; An operation of switching the camera used to display a preview image on the display or store an image in the memory to the second camera based on the activation of the second camera; and A method of operation of an electronic device comprising: determining whether to switch the camera used to display a preview image on the display or store an image in the memory to the first camera based on the third depth information.

12. In Paragraph 11, An operation of determining at least one parameter based on the above-mentioned first depth information; and A method of operation of an electronic device further comprising: an operation of obtaining the second depth information from the phase difference information obtained using the second camera based on at least the parameters determined above.

13. In Paragraph 12, A method of operating an electronic device, wherein the above-mentioned at least one parameter comprises at least one of a first offset value for a determined depth, a second offset value for lens stroke information, or a third offset value associated with a compensation operation for hysteresis.

14. In Paragraph 11, A method of operation of an electronic device further comprising: an operation of determining the third depth information based on at least one of whether the first camera is in an activated state, whether the second camera is in an activated state, or a result of comparing the first depth information and the second depth information.

15. In Paragraph 11 or Paragraph 14, After the above second camera is activated: When the first camera is activated, the operation of determining the first depth information as the third depth information; An operation to disable the first camera when the first depth information corresponds to a first reference value; After the above-mentioned first camera is deactivated: An operation of determining the first reference value as the third depth information based on the fact that the second depth information is greater than the first reference value; and A method of operation of an electronic device further comprising: an operation of determining the second depth information as the third depth information based on whether the second depth information is less than or equal to the first reference value, or whether the difference between the second depth information and the minimum value of the second depth information is greater than the second threshold.

Images