Electronic device comprising plurality of cameras, and control method therefor

Abstract

An electronic device is disclosed. The electronic device of the present disclosure comprises: a first camera; a second camera which is disposed to be adjacent to the first camera and which has an angle of view that is different from the angle of view of the first camera; a display; at least one processor including a processing circuit; and a memory for storing instructions, wherein, when executed individually or collectively by the at least one processor, the instructions can cause the electronic device to: acquire a first image of a first zoom magnification through the first camera; display at least a portion of the first image on the display; identify a target zoom magnification on the basis of a user input for changing zoom magnification; on the basis that the target zoom magnification deviates from a set range, generate an interpolation frame on the basis of the first image and at least a portion of a second image of a second zoom magnification acquired through the second camera; and display the interpolation frame on the display by replacing a region of the first image corresponding to the zoom magnification for some of the time for which the zoom magnification changes.

Description

Electronic device including multiple cameras and method for controlling the same

[0001] The present disclosure relates to an electronic device including a plurality of cameras and a method for controlling the same.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] An electronic device (e.g., a smartphone) may include multiple cameras. For example, the electronic device may include an ultra-wide camera, a wide camera, and a telephoto camera, each with different primary zoom ratios.

[0004] The above multiple cameras may have different focal lengths. The electronic device can capture images of a wider scene by using a camera with a shorter focal length (e.g., an ultra-wide-angle camera). Conversely, the electronic device can capture higher-quality images of subjects located further away from the electronic device by using a telephoto camera, compared to using an ultra-wide-angle camera or a wide-angle camera.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] According to one embodiment, an electronic device may include a first camera, a second camera disposed around the first camera and having a different angle of view from that of the first camera, a display, at least one processor including a processing circuit, and a memory storing instructions.

[0007] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire a first image at a first zoom magnification through the first camera.

[0008] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display at least a portion of the first image on the display.

[0009] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a target zoom factor based on a user input that changes the zoom factor.

[0010] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate an interpolated frame based on at least a portion of a second image of a second zoom ratio acquired through the first image and the second camera, based on the target zoom ratio being outside a set range.

[0011] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0012] According to one embodiment, a method of controlling an electronic device may include an operation of acquiring a first image at a first zoom ratio through a first camera of the electronic device.

[0013] According to one embodiment, a method of controlling an electronic device may include an action of displaying at least a portion of the first image on a display of the electronic device.

[0014] In one embodiment, a method of controlling an electronic device may include an action of determining a target zoom factor based on a user input that changes the zoom factor.

[0015] In one embodiment, a method of controlling an electronic device may include generating an interpolated frame based on at least a portion of a second image of a second zoom ratio obtained through a second camera positioned around the first camera and having a different angle of view from that of the first camera, based on the target zoom ratio being outside a set range.

[0016] According to one embodiment, a method of controlling an electronic device may include displaying the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0017] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include instructions for causing an electronic device to acquire a first image at a first zoom ratio through a first camera of the electronic device.

[0018] In one embodiment, the one or more programs may include instructions that cause the electronic device to display at least a portion of the first image on a display of the electronic device.

[0019] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a target zoom factor based on a user input that changes the zoom factor.

[0020] In one embodiment, the one or more programs may include instructions that cause the electronic device to generate an interpolated frame based on at least a portion of a second image of a second zoom ratio obtained through a second camera positioned around the first camera and having a different angle of view from that of the first camera, based on the target zoom ratio being outside a set range.

[0021] In one embodiment, the one or more programs may include instructions that cause the electronic device to display the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0022] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0023] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0024] FIG. 3 is a flowchart illustrating an operation of displaying an interpolated frame when a camera that acquires an image displayed on a display changes depending on a target zoom ratio in an electronic device according to one embodiment.

[0025] FIG. 4A is a diagram illustrating an operation of displaying an interpolated frame when a camera that acquires an image displayed on a display changes depending on a target zoom ratio in an electronic device according to one embodiment.

[0026] FIG. 4b is a diagram for explaining an operation of generating an interpolation frame in an electronic device according to one embodiment.

[0027] FIG. 5A is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0028] FIG. 5b is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0029] FIG. 5c is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0030] FIG. 6 is a diagram for explaining an operation of cropping images acquired from multiple cameras before generating an interpolation frame in an electronic device according to one embodiment.

[0031] FIG. 7 is a diagram illustrating an operation of scaling the cropped images of FIG. 6 to the same size before generating an interpolation frame in an electronic device according to one embodiment.

[0032] FIG. 8 is a diagram for explaining an operation of obtaining parallax data and ratio data based on the scaled images of FIG. 7 using an artificial intelligence model of an electronic device according to one embodiment.

[0033] FIG. 9 is a diagram for explaining an operation of obtaining time difference data and ratio data using a plurality of artificial intelligence models configured in a joint form in an electronic device according to one embodiment.

[0034] FIG. 10 is a diagram for explaining an operation of generating an interpolation frame using time difference data of an electronic device according to one embodiment.

[0035] FIG. 11 is a drawing for explaining images displayed on a display of an electronic device according to one embodiment.

[0036] FIG. 12 is a diagram for explaining an operation of generating an interpolation frame using parallax data and ratio data of an electronic device according to an embodiment of the present invention.

[0037] FIG. 13 is a flowchart illustrating an operation of displaying an interpolated frame according to a target zoom ratio of an electronic device according to one embodiment.

[0038] FIG. 14 is a flowchart illustrating an operation of determining a zoom curve according to a target zoom ratio of an electronic device according to one embodiment.

[0039] FIG. 15 is a diagram for explaining an operation of displaying parallax-compensated interpolated frames of an electronic device according to one embodiment.

[0040] FIG. 16 is a drawing for explaining an operation of displaying an exposure-compensated interpolated frame of an electronic device according to one embodiment.

[0041] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0042] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0043] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0044] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0045] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0046] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0047] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0048] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0049] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0050] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0051] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0052] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0053] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0054] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0055] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0056] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0057] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0058] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0059] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0060] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0061] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0062] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0063] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments.

[0064] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0065] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0066] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display device display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0067] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or may be provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) It may be configured as at least a part of the processor (120), or may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display device display module (160) by the processor (120) either as is or after undergoing additional image processing.

[0068] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0069] FIG. 3 is a flowchart illustrating an operation of displaying an interpolated frame when a camera that acquires an image displayed on a display changes depending on a target zoom ratio in an electronic device according to one embodiment.

[0070] Referring to FIG. 3, in operation 310, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) can obtain a first image of a first zoom ratio through a first camera (e.g., a camera module (180) of FIG. 1 or a camera module (180) of FIG. 2).

[0071] According to one embodiment, the electronic device can obtain a second image at a second zoom magnification through a second camera (e.g., the camera module (180) of FIG. 1 or the camera module (180) of FIG. 2). According to one embodiment, the first camera and the second camera can have different angles of view and different optical zoom magnifications. For example, the first camera can have an optical zoom magnification of a first value (e.g., 1x), and the second camera can have an optical zoom magnification of a second value (e.g., 3x). According to one embodiment, the first camera can obtain an image corresponding to a zoom magnification of a first range (e.g., 1x or more and less than 3x) through digital zoom, and the second camera can obtain an image corresponding to a zoom magnification of a second range (e.g., 3x or more and less than 10x) through digital zoom.

[0072] In one embodiment, the first camera and the second camera may be positioned on one side (e.g., the front or back) of the electronic device. For example, the second camera may be positioned around the first camera.

[0073] In one embodiment, when a user input for executing a camera application is received, the electronic device may activate two cameras from among a plurality of cameras. For example, when a user input for executing a camera application is received, the electronic device may activate a first camera from among a plurality of cameras included in the electronic device, the first camera having an optical zoom ratio corresponding to a current zoom ratio (e.g., a last used zoom ratio) or a default zoom ratio, and a second camera having an optical zoom ratio that has a smallest difference from the optical zoom ratio of the first camera.

[0074] According to one embodiment, when a user input for executing a camera application is received, the electronic device may activate a first camera among a plurality of cameras included in the electronic device that has an optical zoom ratio corresponding to a current zoom ratio (e.g., a last used zoom ratio) or a default zoom ratio. According to one embodiment, when a user input for changing a zoom ratio is received (e.g., changing the zoom ratio through physical button operation, changing the zoom ratio through a plurality of buttons or a slide bar superimposed on a preview screen, changing the zoom ratio through a touch input on the display (e.g., pinch input), changing the zoom ratio through an air gesture, changing the zoom ratio through a button of an external electronic device such as an electronic pen, or voice input for changing the zoom ratio), the electronic device may activate a second camera. For example, when a touch for changing the zoom ratio is received through the execution screen of the camera application or a multi-touch for pinch input is input, the electronic device may activate the second camera. In one embodiment, the second camera may be a camera for acquiring an image of an identified target zoom factor based on a user input for changing the zoom factor.

[0075] In one embodiment, based on a user input for changing a zoom factor, if the changed zoom factor falls within a set range, the electronic device may activate the second camera. In one embodiment, the set range of zoom factors for activating the second camera is a value between the current zoom factor and the digital zoom factor of the first camera (e.g., 3x) (e.g., 2.5x or more and less than 3x), and an image may be acquired through the first camera.

[0076] In one embodiment, the electronic device may activate at least two of the plurality of cameras independently of the execution of the camera application.

[0077] In one embodiment, the electronic device may acquire a first image at a first zoom factor (e.g., 1x) through the first camera and a second image at a second zoom factor (e.g., 3x) through the second camera, while both the first camera and the second camera are activated.

[0078] In one embodiment, in operation 320, the electronic device may display at least a portion of the first image. In one embodiment, the electronic device may display a portion of the first image acquired through the first camera if the current zoom ratio is within a first range (e.g., 1x or more and less than 3x). For example, the electronic device may crop a portion of the first image to be displayed on a display (e.g., display (160) of FIG. 1) based on the current zoom ratio, and display the cropped portion on the display by scaling it to fit the size of the display.

[0079] According to one embodiment, a second image acquired through a second camera together with the acquisition of a first image through a first camera may be stored in a memory (e.g., memory (130) of FIG. 1).

[0080] In one embodiment, in operation 330, the electronic device can determine a target zoom factor based on a user input that changes the zoom factor.

[0081] According to one embodiment, the electronic device may display a plurality of icons (or buttons) on the display, each corresponding to a plurality of zoom ratios (e.g., 0.5x, 1x, 2x, 3x, 5x, and 10x). According to one embodiment, the electronic device may determine the zoom ratio corresponding to the selected icon as the target zoom ratio based on receiving a user input for selecting one of the plurality of icons. An embodiment in which a plurality of icons are displayed according to one embodiment will be described below with reference to FIG. 5A.

[0082] According to one embodiment, the electronic device may display a slide bar on the display that allows for changing the zoom ratio. According to one embodiment, the electronic device may determine the target zoom ratio based on a user input that selects or drags a specific location on the slide bar. An embodiment in which the slide bar is displayed will be described below with reference to FIG. 5B.

[0083] According to one embodiment, the electronic device may determine a zoom factor corresponding to the distance between the start and end points of the pinch input as a target zoom factor based on the user input of the pinch input received. For example, when a pinch input for changing the zoom factor is received through a preview screen in the execution screen of a camera application, the electronic device may determine the target zoom factor based on the change in the distance between the two fingers when the pinch input starts and the distance between the two fingers when the pinch input ends.

[0084] According to one embodiment, the electronic device may determine a zoom ratio corresponding to the speed of the pinch input as a target zoom ratio based on the reception of a user input, which is a pinch input. For example, when a pinch input for changing a zoom ratio is received through a preview screen in the execution screen of a camera application, the electronic device may determine the speed of the pinch input based on the distance between two fingers when the pinch input starts, the amount of change in the distance between two fingers when the pinch input ends, and the time taken for the pinch input, and may determine the target zoom ratio based on the speed of the pinch input. According to one embodiment, an embodiment of determining a target zoom ratio through a pinch input will be described below with reference to FIG. 5C.

[0085] In one embodiment, the pinch input may be terminated when the two-finger touch is released, or when the two touch locations are maintained for a set period of time even if the touch is not released. In one embodiment, if the two-finger touch locations change after the pinch input is terminated while the two-finger touch is not released, the electronic device may determine that a new pinch input has been received. By receiving multiple pinch inputs while maintaining the touch in this manner, the electronic device may provide an experience similar to continuous pinch input.

[0086] In one embodiment, a change in the zoom factor may be a change in magnification (e.g., zooming in) or a change in reduction (e.g., zooming out). For example, a change in magnification may be selecting an icon corresponding to a zoom factor greater than the current zoom factor, or a change in the zoom factor corresponding to a pinch input that moves the two fingers apart. In one embodiment, a change in zoom factor may be selecting an icon corresponding to a zoom factor smaller than the current zoom factor, or a change in the zoom factor corresponding to a pinch input that moves the two fingers apart.

[0087] In one embodiment, in operation 340, the electronic device may generate an interpolated frame based on at least a portion of a second image of a second zoom ratio acquired through the first image and the second camera, based on a target zoom ratio being outside a set range (e.g., a first range of zoom ratios obtainable by the first camera (e.g., 1x or more and less than 3x)).

[0088] According to one embodiment, the electronic device may input a first input image obtained based on a first image and a second input image obtained based on a second image as input data to an artificial intelligence model based on a target zoom ratio being out of a set range, and may obtain first disparity data per pixel of the first input image for the second input image, second disparity data per pixel of the second input image for the first input image, and ratio data as output data of the artificial intelligence model.

[0089] In one embodiment, the electronic device may determine that when the target zoom factor is outside a first range that can be acquired by the first camera, the camera that acquires the image to be displayed on the display when the target zoom factor is applied is changed. For example, when the target zoom factor is outside a set range, the electronic device may determine that at least a portion of a first image currently acquired by the first camera, which is within the set range of the zoom factor, is displayed, and at least a portion of a second image acquired by the second camera, which is at the target zoom factor, is displayed.

[0090] In one embodiment, when it is determined that the camera that acquires the image to be displayed on the display has changed when applying the target zoom factor, the electronic device may display interpolated frames in the operation of changing the zoom factor to the target zoom factor. In one embodiment, the interpolated frames may correspond to intermediate zoom factors, respectively, in the operation of changing the zoom factor.

[0091] According to one embodiment, the electronic device can obtain parallax data based on a first image and a second image to generate an interpolated frame, and ratio data to which the first image and the second image are to be applied to generate the interpolated frame.

[0092] According to one embodiment, the electronic device can obtain a first input image and a second input image by preprocessing the first image and the second image to obtain disparity data and ratio data.

[0093] According to one embodiment, the first input image and the second input image may be obtained by cropping a first region of the first image that matches the second image and a second region of the second image that matches the first image, and resizing the first region and the second region so that the sizes of the first region and the second region correspond. According to one embodiment, the operation of obtaining the first input image and the second input image will be described in more detail with reference to FIGS. 6 and 7 below.

[0094] According to one embodiment, an artificial intelligence model (e.g., the artificial intelligence model (820) of FIG. 8 or the second artificial intelligence model (921) of FIG. 9) may be trained to use multiple input images as input data to obtain, as output data, multiple disparity data for each of the multiple input images and data related to the ratio at which each pixel of the multiple input images is to be applied.

[0095] According to one embodiment, parallax data (e.g., optical flow) may be data related to a difference in the position (e.g., pixel coordinates) of the same object included in the first input image and the second input image, which is generated due to different arrangement positions of two cameras. According to one embodiment, parallax data may include data related to pixel-by-pixel movement information between the input images (e.g., the first input image and the second input image). For example, the electronic device may include, through an artificial intelligence model, first parallax data including pixel-by-pixel movement information when the position of an object in the first input image moves to the position of an object in the second input image, and second parallax data including pixel-by-pixel movement information when the position of an object in the second input image moves to the position of the object in the first input image.

[0096] According to one embodiment, the ratio data (e.g., occlusion mask) may relate to a ratio at which pixel values ​​of the first input image are applied for each pixel and a ratio at which pixel values ​​of the second input image are applied. For example, if the ratio data of a specific pixel (x, y) is 0, only the pixel value of (x, y) of the first input image will be applied when generating an interpolation frame, either to a portion included in the first input image or to a portion not included in the second input image. According to one embodiment, if the ratio data of a specific pixel (x, y) is 0.5, an average value of the pixel value of (x, y) of the first input image and the pixel value of (x, y) of the second input image may be applied when generating an interpolation frame.

[0097] According to one embodiment, the electronic device may acquire the first input image and the second input image through an artificial intelligence model stored in a memory (e.g., the first artificial intelligence model (920) of FIG. 9). According to one embodiment, the artificial intelligence model for acquiring the input images may be separate from the artificial intelligence model for acquiring the plurality of disparity data and the data related to the ratio at which pixels of each of the plurality of input images are to be applied. According to one embodiment, the artificial intelligence model for acquiring the input images may be trained to acquire the plurality of images as output data by resizing each matching area between the plurality of images to the same size using the plurality of images as input data.

[0098] According to one embodiment, the electronic device may use the first image and the second image as input data of the first artificial intelligence model, and obtain the first input image and the second input image as output data. In this way, when obtaining the first input image and the second input image through the artificial intelligence model, the electronic device may obtain disparity data and ratio data by configuring the two artificial intelligence models in a joint form. According to one embodiment, the operation of obtaining disparity data and ratio data through the two artificial intelligence models configured in a joint form will be described in more detail below with reference to FIG. 9.

[0099] According to one embodiment, the electronic device can generate an interpolated frame based on a first input image, first disparity data, a second input image, second disparity data, and ratio data.

[0100] In one embodiment, the electronic device can determine a set number of intermediate zoom ratios between the first zoom ratio and the second zoom ratio. In one embodiment, a difference between adjacent intermediate zoom ratios can be constant. For example, when the zoom ratio changes from 1x to 3x, the intermediate zoom ratios can be 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75. In one embodiment, when the zoom ratio changes from 3x to 1x, the intermediate zoom ratios can be 2.75, 2.5, 2.25, 2, 1.75, 1.5, 1.25.

[0101] According to one embodiment, the electronic device may generate a set number of first input images and a set number of second input images. According to one embodiment, the first input images may correspond to a set number of intermediate zoom ratios, respectively, and the second input images may correspond to a set number of intermediate zoom ratios, respectively.

[0102] According to one embodiment, the electronic device may generate a set number of interpolated frames corresponding to each of a set number of intermediate zoom ratios. For example, the electronic device may generate a set number of interpolated frames corresponding to each of a set number of intermediate zoom ratios using a set number of first input images and a set number of second input images, respectively. According to one embodiment, the number of interpolated frames may be greater than or equal to a set minimum number (e.g., 3).

[0103] In one embodiment, the electronic device may determine the set number based on the time required to change from the first magnification to the target zoom factor. For example, if the time required to change from the first magnification to the target zoom factor is long, the number of intermediate zoom factors may be increased to naturally display the screen being enlarged or reduced by the zoom change.

[0104] In one embodiment, the electronic device may determine the number of intermediate zoom ratios based on the difference between the first magnification and the target zoom ratio. For example, if the difference between the first magnification and the target zoom ratio is large (e.g., the difference in magnification is 3 or more), the number of intermediate zoom ratios may increase to naturally display the screen being enlarged or reduced by the zoom change.

[0105] In one embodiment, the number of intermediate zoom ratios may be determined based on the type of camera activated based on the target zoom ratio. For example, if a camera with a 1x magnification is activated based on the first magnification of 1x, and the camera activated based on the target zoom ratio is a 10x camera, the number of intermediate zoom ratios may be greater than the number of intermediate zoom ratios if the camera activated is a 3x camera.

[0106] In one embodiment, if the electronic device is a visual see-through (VST), virtual reality (VR), or augmented reality (AR) device, such as a head-mounted display (HMD), the electronic device may be equipped with more cameras than a smartphone, and the cameras may be spaced farther apart. The farther the cameras are spaced apart, the greater the chance of camera transitions being interrupted due to parallax. Therefore, the number of intermediate zoom ratios may increase to achieve a smooth transition.

[0107] In one embodiment, the electronic device may generate an interpolated frame based on a plurality of zoom curves stored in a memory. The zoom curve may be related to the timing at which an intermediate zoom factor is applied (e.g., a parallax estimation point or a synthesis timing). For example, if the zoom curve is linear, a screen with an intermediate zoom factor applied may be displayed at regular intervals, making it appear that the zoom change is performed at a constant rate. In one embodiment, if the zoom curve is non-linear, a section with a large slope of the zoom curve may appear to perform a zoom change at a fast rate, and a section with a small slope of the zoom curve may appear to perform a zoom change at a slow rate.

[0108] According to one embodiment, the electronic device may identify one of a plurality of zoom curves based on a difference between a first zoom factor and a target zoom factor. For example, if the difference between the first zoom factor and the target zoom factor is large, a non-linear zoom curve including a section with a large slope may be selected.

[0109] In one embodiment, the electronic device may store a zoom curve mapped to the difference between the first zoom factor and the target zoom factor. In one embodiment, the electronic device may select a zoom curve corresponding to the difference between the first zoom factor and the target zoom factor.

[0110] According to one embodiment, the electronic device may determine the parallaxes to which a set number of intermediate zoom factors are applied, respectively, based on the identified zoom curve. For example, if the zoom curve is linear, the difference between the parallaxes to which the intermediate zoom factors are applied may be a constant value. According to one embodiment, if the zoom curve is non-linear, the difference between the parallaxes to which the intermediate zoom factors are applied may be large in a section where the zoom curve has a large slope, and the difference between the parallaxes to which the intermediate zoom factors are applied may be small in a section where the zoom curve has a small slope.

[0111] In one embodiment, when a high refresh rate (or frame rate (FPS; frames per second)) is required, such as in VST, VR, or AR devices, the electronic device may determine the number and / or timing of intermediate zoom factors based on the refresh rate (or frame rate).

[0112] In one embodiment, in operation 350, the electronic device may display an interpolated frame by replacing a portion of the first image corresponding to the zoom factor, at least during a portion of the zoom factor being changed.

[0113] According to one embodiment, the electronic device may display a portion of the first image corresponding to the current zoom factor while changing from a first zoom factor to a set zoom factor between the first and second zoom factors. According to one embodiment, the electronic device may sequentially display a set number of interpolated frames to replace a portion of the first image while changing from the set zoom factor to the second zoom factor. For example, after the zoom factor is changed, the electronic device may display a portion of the first image acquired by the first camera by digitally zooming and scaling it until a certain zoom factor, and from the certain zoom factor onward until the second zoom factor, the electronic device may display the interpolated frames instead of the first image. According to one embodiment, from the second zoom factor onward, at least a portion of the second image acquired by the second camera may be displayed.

[0114] In one embodiment, the electronic device may sequentially display a set number of interpolated frames in place of at least a portion of the first image while changing from a first zoom ratio to a second zoom ratio. For example, the electronic device may display interpolated frames in place of the first image acquired by the first camera from the time the zoom ratio starts changing until the second zoom ratio is reached. In one embodiment, from the second zoom ratio onward, the electronic device may display at least a portion of the second image acquired by the second camera.

[0115] In this way, even if the camera that acquires the image displayed on the display changes according to the change in zoom ratio, the camera change can be performed naturally without any sense of incongruity by displaying an interpolated frame in the zoom ratio change operation.

[0116] In one embodiment, in a flex zoom function that automatically recognizes people, backgrounds, and objects in a preview during video capture and automatically determines a zoom ratio, the electronic device may check the automatically determined zoom ratio as a target zoom ratio, and if camera switching is required based on the current zoom ratio and the target zoom ratio, the electronic device may generate and display interpolated frames based on images acquired by the two cameras.

[0117] FIG. 4A is a diagram illustrating an operation of displaying an interpolated frame when a camera that acquires an image displayed on a display changes depending on a target zoom ratio in an electronic device according to one embodiment.

[0118] Referring to FIG. 4A, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) may include a preview system (400) that displays an image acquired from a camera (180-1, 180-2) (e.g., a camera module (180) of FIG. 1 or a camera module (180) of FIG. 2) on a display (160) (e.g., a display module (160) of FIG. 1).

[0119] According to one embodiment, the preview system (400) may include an image preprocessing operation (410), a spatial alignment transformation operation (420), an image parallax compensation operation (430), and an image postprocessing operation (440). According to one embodiment, each operation may be performed by a single chip via software, or at least one of each operation may be performed by a separate chip.

[0120] According to one embodiment, the image preprocessing operation (410) may include an operation for calculating a valid area in each camera (180-1, 180-2) and acquiring an image of the area based on camera parameters (411) and other information. According to one embodiment, the camera parameters (411) may include at least one of angle of view information, resolution information, or aspect ratio information of the cameras (180-1, 180-2). According to one embodiment, the other information may include display (e.g., display module (160) of FIG. 1) information and / or currently requested zoom magnification information.

[0121] For example, the electronic device may acquire a first image of the valid area to be displayed on the display from among the images acquired from the first camera (180-1) and acquire a second image of the valid area to be displayed on the display from among the images acquired from the second camera (180-2) based on the camera parameters (411) of each camera (180-1, 180-2) and the currently requested zoom magnification information through the image preprocessing operation (410). For example, if the first camera (180-1) (or the second camera (180-2)) is a 12MP (mega pixel) camera of 4000x3000 and a 1.1x zoom magnification is required, the electronic device may calculate a 3636x2727 part in the center of the image acquired by the first camera (180-1) as the valid area and crop the area.

[0122] According to one embodiment, the spatial alignment transformation operation (420) may calculate an area where the angles of view of the images match based on the images acquired through the image preprocessing operation (410) and metadata (e.g., app request, camera mode, sensor mode). According to one embodiment, the electronic device may calculate an area where the angles of view of the images match through an image matching algorithm.

[0123] For example, the electronic device can calculate coordinates of corresponding areas (e.g., areas containing the same object) of the first image and the second image cropped in the image preprocessing operation (410) through the spatial alignment transformation operation (420).

[0124] According to one embodiment, the electronic device may select a reference image from among the first and second images cropped in the image preprocessing operation (410) through a spatial alignment transformation operation (420), and may calculate coordinates of an area of ​​the remaining images in which the angles of view of the reference image and the remaining images match as much as possible. According to one embodiment, the reference image may be an image whose change in image is small even when the current zoom ratio is changed. For example, when an image having a zoom ratio of 1x or more and less than 3x is acquired by the first camera (180-1), and an image having a zoom ratio of 3x or more and less than 10x is acquired by the second camera (180-2), and the change in the zoom ratio is 1x or more and less than 3x, the electronic device may select the image acquired by the second camera (180-2) as the reference image.

[0125] According to one embodiment, the electronic device may determine a representative image to be displayed on the display after calculating an area where the angles of view of the first image and the second image match through a spatial alignment transformation operation (420). According to one embodiment, the electronic device may determine the representative image among the first image and the second image based on a currently required zoom ratio. For example, if the currently required zoom ratio is greater than or equal to 1x and less than 3x, the electronic device may determine the first image as the representative image and determine to display an area among the first image where the angles of view of the second image match on the display (160).

[0126] According to one embodiment, the image parallax compensation operation (430) may include an operation of obtaining images of an area calculated so that the angles of view of the first image and the second image match as much as possible in the spatial alignment transformation operation (420), and compensating for the remaining angle difference and parallax between the images.

[0127] According to one embodiment, the electronic device may align the angles of view of two images by cropping a calculated area of ​​a first image and a calculated area of ​​a second image through a parallax compensation operation (430), scale the cropped images to align their sizes, estimate the parallax between the scaled images, and generate an interpolation frame to compensate for the estimated parallax. According to one embodiment, the parallax compensation operation (430) will be described in more detail below with reference to FIG. 4B.

[0128] According to one embodiment, the image post-processing operation (440) may include an operation of resizing interpolated images obtained through the image parallax compensation operation (430) or a calculated area within a representative image obtained through the spatial alignment transformation operation (420) based on the resolution of the display (160).

[0129] According to one embodiment, the electronic device may, through an image post-processing operation (440), crop the images by applying the transformation information obtained in the spatial alignment transformation operation (420) to each of the images obtained in the image pre-processing operation (410) when the zoom ratio does not change, and may adjust the size of a representative image among the cropped images to match the resolution of the display (160) and transmit the same to the display (160). According to one embodiment, when the zoom ratio changes, interpolated images obtained through the image parallax compensation operation (430) may be adjusted in size to match the resolution of the display (160) and transmitted to the display (160).

[0130] According to one embodiment, the display (160) can display images received from the preview system (400) as a preview screen.

[0131] FIG. 4b is a diagram for explaining an operation of generating an interpolation frame in an electronic device according to one embodiment.

[0132] Referring to FIG. 4b, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) can perform an image parallax compensation operation (430) to obtain an interpolated image as an output image (438) using input images (431).

[0133] According to one embodiment, the image parallax compensation operation (430) may include a crop operation (432), a scale operation (433), a parallax estimation operation (434), and a parallax compensation operation (435).

[0134] According to one embodiment, the cropping operation (432) may crop each of the input images (431) (e.g., the first image and the second image) based on image alignment information (436) of the input images (431) obtained through a spatial alignment transformation operation (e.g., the spatial alignment transformation operation (420) of FIG. 4A). For example, the electronic device may include, through the cropping operation (432), an operation of cropping corresponding regions of the input images (431) calculated through the spatial alignment transformation operation. According to one embodiment, the electronic device may align the angles of view of the two images by cropping corresponding regions (e.g., regions containing the same object) of the cropped first image and the second image through an image preprocessing operation (e.g., the image preprocessing operation (410) of FIG. 4A). According to one embodiment, the operation of cropping corresponding regions will be described in more detail below with reference to FIG. 6.

[0135] According to one embodiment, the scaling operation (433) may include an operation of resizing cropped images of different sizes to the same size. According to one embodiment, the electronic device may align the sizes by resizing the corresponding areas between the input images (431) so that the sizes of the cropped images are the same. According to one embodiment, the operation of scaling the corresponding areas of the cropped images so that the sizes of the cropped images are the same will be described in more detail below with reference to FIG. 7.

[0136] According to one embodiment, the parallax estimation operation (434) may include an operation of estimating parallax between images whose angles and sizes are aligned through a crop operation (432) and a scale operation (433). According to one embodiment, the electronic device may use an artificial intelligence model (or an artificial neural network) utilized for frame-rate conversion (FRC) for parallax estimation. According to one embodiment, the parallax information acquired through the artificial intelligence model may include an optical flow and an occlusion mask. According to one embodiment, the optical flow may include parallax data related to a difference in position within the image for the same object included in each of the two images. According to one embodiment, the occlusion mask may be related to a ratio at which the two images are applied to each pixel of the interpolation frame.

[0137] In one embodiment, optical flow and occlusion masks may be generated for a set number of intermediate zoom factors between a current zoom factor and a target zoom factor. In one embodiment, the number of intermediate zoom factors may be determined based on the time required to change from the current zoom factor to the target zoom factor, and the plurality of intermediate zoom factors may correspond to each of a plurality of interpolated frames.

[0138] In one embodiment, the electronic device may estimate a parallax between two images at an intermediate zoom factor based on time information (time T) (437) associated with each intermediate zoom factor. In one embodiment, the time information (time T) (437) associated with each intermediate zoom factor may be determined based on a zoom curve determined based on a difference between a current zoom factor and a target zoom factor.

[0139] According to one embodiment, the parallax compensation operation (435) may include an operation of generating interpolation frames to compensate for parallax using the estimated parallax information. According to one embodiment, the electronic device may generate a set number of compensation frames, each corresponding to a set number of intermediate zoom ratios in the parallax estimation operation (434). According to one embodiment, the operation of generating a set number of compensation frames will be described in more detail with reference to FIGS. 10, 11, and 12 below.

[0140] In one embodiment, the electronic device may omit the parallax estimation operation (434) and generate an interpolated frame on which parallax compensation is performed, depending on the type of artificial intelligence model.

[0141] FIG. 5A is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0142] Referring to FIG. 5A, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may display a preview screen through the execution screen of a camera application. According to one embodiment, the preview screen may be at least a portion of a first image of a first zoom ratio acquired through a first camera (e.g., the camera module (180) of FIG. 1 or the camera module (180) of FIG. 2).

[0143] According to one embodiment, the electronic device may display a plurality of icons (or buttons) (510) on a display (e.g., the display module (160) of FIG. 1) each corresponding to a plurality of zoom ratios (e.g., 0.5x, 1x, 2x, 3x, 5x, 10x) for changing the zoom ratio by overlapping at least a portion of the preview screen.

[0144] According to one embodiment, the electronic device can identify a zoom ratio corresponding to a selected icon among a plurality of icons (510) as a target zoom ratio.

[0145] FIG. 5b is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0146] Referring to FIG. 5B, the electronic device may display a slide bar (520) for changing the zoom ratio, overlapping at least a portion of the preview screen. According to one embodiment, the electronic device may determine the zoom ratio corresponding to the selected position as the target zoom ratio based on a user input (521) selecting a specific position of the slide bar.

[0147] In one embodiment, the electronic device may change the zoom ratio based on a user input received by selecting a specific location on the slide bar (520) and then dragging it left or right, and determine the zoom ratio when the touch is released after the drag as the target zoom ratio.

[0148] FIG. 5c is a diagram illustrating a user input for changing a zoom ratio of an electronic device according to one embodiment.

[0149] Referring to FIG. 5c, the electronic device may display a slide bar (520) for changing the zoom ratio, overlapping at least a portion of the preview screen. In one embodiment, the electronic device may change the zoom ratio selected in the slide bar (520) based on the distance between the start and end points of the pinch input (522).

[0150] In one embodiment, the electronic device can change the zoom factor selected from the slide bar (520) based on the speed of the pinch input.

[0151] In one embodiment, the electronic device can determine the zoom factor when the pinch input is released as the target zoom factor.

[0152] FIG. 6 is a diagram for explaining an operation of cropping images acquired from multiple cameras before generating an interpolation frame in an electronic device according to one embodiment.

[0153] Referring to FIG. 6, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may acquire a first image (610) from a first camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the first camera (180-1) of FIG. 4A). According to an embodiment, the electronic device may acquire a second image (620) from a second camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the second camera (180-2) of FIG. 4A). According to an embodiment, the first image (610) may have a first angle of view obtainable through the first camera, and the second image (620) may have a second angle of view obtainable through the second camera. According to an embodiment, the second angle of view may be different from the first angle of view.

[0154] According to one embodiment, the electronic device may identify and / or crop a first valid area (611), which is an area displayed on a display (e.g., a display module (160) of FIG. 1), among a first image (610), through an image preprocessing operation (e.g., an image preprocessing operation (410) of FIG. 4A). According to one embodiment, the electronic device may identify and / or crop a second valid area (621), which is an area displayed on a display (e.g., a display module (160) of FIG. 1), among a second image (620), through an image preprocessing operation.

[0155] According to one embodiment, the electronic device can identify an area (612, 622) where the mutual field of view of the first valid area (611) and the second valid area (621) match through a spatial alignment transformation operation (e.g., the spatial alignment transformation operation (420) of FIG. 4A). According to one embodiment, the electronic device can calculate an area (612, 622) where the field of view of the first valid area (611) and the second valid area (621) match through an image matching algorithm. According to one embodiment, the area (612, 622) where the field of view matches may be an area containing the same object within the first valid area (611) and the second valid area (621).

[0156] According to one embodiment, the electronic device may crop areas (612, 622) having matching field of view in the first valid area (611) and the second valid area (621), respectively. According to one embodiment, the sizes of the cropped area (612) in the first valid area (611) and the cropped area (622) in the second valid area (621) may be different. According to one embodiment, the electronic device may scale the sizes of the cropped areas (612, 622) having different sizes, as illustrated in FIG. 7 below.

[0157] FIG. 7 is a diagram illustrating an operation of scaling the cropped images of FIG. 6 to the same size before generating an interpolation frame in an electronic device according to one embodiment.

[0158] Referring to FIG. 7, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may scale (or resize) the different sizes of cropped areas (612, 622) illustrated in FIG. 6 so that they have the same size. According to an embodiment, the electronic device may resize the size of the cropped area (612) in the first effective area based on the size of the cropped area (622) in the second effective area, which has a smaller size change while changing from the current zoom ratio to the target zoom ratio among the plurality of cropped areas (612, 622). According to an embodiment, the electronic device may obtain the first input image (710) and the second input image (720) having the same size through the scaling operation.

[0159] FIG. 8 is a diagram for explaining an operation of obtaining parallax data and ratio data based on the scaled images of FIG. 7 using an artificial intelligence model of an electronic device according to one embodiment.

[0160] Referring to FIG. 8, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) can obtain parallax data (830, 831) and ratio data (832) by inputting the scaled input images (810, 811) of FIG. 7 into an artificial intelligence model (820). According to one embodiment, the artificial intelligence model (820) may include a CNN model or a vision transformer (VIT).

[0161] According to one embodiment, the first disparity data (830) may be pixel-by-pixel data for the second input image (811) of the first input image (810), and the second disparity data (831) may be pixel-by-pixel data for the first input image (810) of the second input image (811). According to one embodiment, the ratio data (832) may be related to a ratio at which pixel values ​​of the first input image (810) are applied for each pixel when generating an interpolation frame and a ratio at which pixel values ​​of the second input image (811) are applied for each pixel.

[0162] In this way, by estimating the disparity between two input images (810, 811) through an artificial intelligence model, it is possible to additionally perform field of view alignment that is insufficient even when performing a spatial alignment transformation operation (e.g., spatial alignment transformation (420) of FIG. 4a).

[0163] According to an embodiment, the two input images (810, 811) of FIG. 8 may be aligned in angle of view and size through an algorithm. According to an embodiment, the electronic device may additionally use an artificial intelligence model to obtain parallax data and ratio data using as input data the valid area of ​​the first image acquired from the first camera before the angle of view and size alignment (e.g., the first valid area (611) of FIG. 6) and the valid area of ​​the second image acquired from the second camera (e.g., the second valid area (621) of FIG. 6). According to an embodiment, an embodiment in which the artificial intelligence model for aligning the angle of view and size and the artificial intelligence model for obtaining the parallax data and the ratio data are configured in a joint form will be described in more detail below with reference to FIG. 9.

[0164] FIG. 9 is a diagram for explaining an operation of obtaining time difference data and ratio data using a plurality of artificial intelligence models configured in a joint form in an electronic device according to one embodiment.

[0165] Referring to FIG. 9, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) can obtain parallax data (930, 931) and ratio data (932) by inputting an image (910) of a valid area of ​​a first image obtained from a first camera (e.g., the first valid area (611) of FIG. 6) and an image (911) of a valid area of ​​a second image obtained from a second camera (e.g., the second valid area (621) of FIG. 6) into a joint-type artificial intelligence model (920, 921). The joint-type artificial intelligence model (920, 921) may include a first artificial intelligence model (920) for aligning the angle of view and parallax between images using an image (910) of a valid area of ​​a first image acquired from a first camera and an image (911) of a valid area of ​​a second image acquired from a second camera as input data, and a second artificial intelligence model (921) for estimating the parallax of images whose angles of view and parallax are aligned to obtain parallax data (930, 931) and ratio data (932) as output data. According to an embodiment, the first artificial intelligence model (920) and / or the second artificial intelligence model (921) may include a CNN model or a vision transformer (VIT).

[0166] According to one embodiment, although the artificial intelligence models (920, 921) are illustrated as being in a joint form in FIG. 9, according to one embodiment, the electronic device may obtain output data of images in which the angle of view and parallax are aligned by using the image (910) of the valid area of ​​the first image acquired from the first camera and the image (911) of the valid area of ​​the second image acquired from the second camera as input data of the first artificial intelligence model (920), and may obtain parallax data (930, 931) and ratio data (932) as output data by using the images in which the angle of view and parallax are aligned acquired as output data of the first artificial intelligence model (920) as input data of the second artificial intelligence model (921).

[0167] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may execute a neural network model as an artificial intelligence model (920, 921). The neural network model may include a deep learning model that performs a specific purpose operation based on a result of learning learning data. For example, the neural network model may include at least one of various types of neural network models, such as a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted boltzmann machine (RBM), a fully convolutional network, a long short-term memory (LSTM) network, or a classification network.

[0168] FIG. 10 is a diagram for explaining an operation of generating an interpolation frame using time difference data of an electronic device according to one embodiment.

[0169] Referring to FIG. 10, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may obtain a first input image (1010) and a second input image (1020). According to an embodiment, the first input image (1010) may be obtained from a first camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the first camera (180-1) of FIG. 4A), and the second input image (1020) may be obtained from a second camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the second camera (180-2) of FIG. 4A).

[0170] In one embodiment, depending on the different placement positions of the first camera and the second camera, the position of the object (1011) included in the first input image (1010) and the position of the object (1021) included in the second input image (1020) may be different.

[0171] According to one embodiment, the electronic device may identify the first input image (1010) as the nth frame to be displayed on the display, and identify the second input image (1020) as the n+1th frame to be displayed on the display. According to one embodiment, the electronic device may obtain parallax information (1030) (or motion information) based on the difference between the position of the object (1011) included in the first input image (1010) and the position of the object (1021) included in the second input image (1020).

[0172] According to one embodiment, the electronic device can compensate for parallax by generating interpolated frames between the first input image (1010) and the second input image (1020). For example, the electronic device can generate a continuous image by generating interpolated frames (1040, 1050, 1060) corresponding to times n+0.25, n+0.50, n+0.75 between the n-th frame of the first input image (1010) and the n+1-th frame of the second input image (1020).

[0173] According to one embodiment, the interpolated frame (1040) corresponding to the n+0.25th time may include an object at a position that has moved 0.25 times the distance from the location of the object (1011) included in the nth frame to the location of the object (1021) included in the n+1th frame. According to one embodiment, the interpolated frame (1050) corresponding to the n+0.5th time may include an object at a position that has moved 0.5 times the distance from the location of the object (1011) included in the nth frame to the location of the object (1021) included in the n+1th frame. According to one embodiment, the interpolated frame (1060) corresponding to the n+0.75th time may include an object at a position that has moved 0.75 times the distance from the location of the object (1011) included in the nth frame to the location of the object (1021) included in the n+1th frame.

[0174] In Fig. 10, the number of interpolation frames is 3, and the difference in the time at which the interpolation frames are generated is constant (e.g., a difference of 0.25), but this is not limited to the above, and the number of interpolation frames may be 4 or more, and the difference in the time at which the interpolation frames are generated may not be constant. For example, the interpolation frame number and timing determination operation as illustrated in Fig. 14 below may be applied.

[0175] FIG. 11 is a drawing for explaining images displayed on a display of an electronic device according to one embodiment.

[0176] Referring to FIG. 11, an electronic device (e.g., an electronic device (101) of FIG. 1 or a processor (120) of FIG. 1) can acquire first images (1110) through a first camera (e.g., a camera module (180) of FIG. 1, a camera module (180) of FIG. 2, or a first camera (180-1) of FIG. 4A) at the same time, and can acquire second images (1120) through a second camera (e.g., a camera module (180) of FIG. 1, a camera module (180) of FIG. 2, or a second camera (180-2) of FIG. 4A).

[0177] According to one embodiment, the electronic device may display some (1111) of the first images (1110) on a display (e.g., the display module (160) of FIG. 1) at a current zoom factor (e.g., 1x).

[0178] According to one embodiment, the electronic device may display interpolated frames (1130) generated based on the first images (1110) and the second images (1120) on the display in a section where the zoom ratio changes (e.g., from 1x to 3x). According to one embodiment, the electronic device may apply different parallax estimation points of the interpolated frames (1130) in proportion to the changed zoom ratio.

[0179] For example, when displaying an interpolated frame corresponding to a 1.25x zoom, the electronic device may estimate parallax information corresponding to a 0.125 point in time so that the 6th image among the first images (1110) and the 6th image among the second images (1120) are more similar to the 6th image among the first images (1110), and may generate and display the 6th image as one of the interpolated frames (1130). According to an embodiment, when displaying an interpolated frame corresponding to a 1.5x zoom, the electronic device may estimate parallax information corresponding to a 0.25 point in time so that the 7th image among the 7th image among the first images (1110) and the 7th image among the second images (1120) are more similar to the 7th image among the first images (1110), and may generate and display the 7th image as one of the interpolated frames (1130). According to one embodiment, when displaying an interpolated frame corresponding to 2x zoom, the electronic device may estimate parallax information corresponding to a point of time of 0.5 among the 9th image among the first images (1110) and the 9th image among the second images (1120) so that the 9th image among the first images (1110) is more similar to the 9th image among the first images (1110), and display the 9th image as one of the interpolated frames (1130). In FIG. 11, the number of interpolated frames (1130) is illustrated as 7, but is not limited thereto, and may be 6 or less or 8 or more.

[0180] In one embodiment, when the current zoom ratio is greater than or equal to the zoom ratio of the second camera (e.g., 3x or more), the electronic device may display some (1121) of the second images (1120) on the display.

[0181] In this way, when the zoom ratio is changed, a natural and smooth camera transition effect can be provided by displaying interpolated frames (1130) instead of displaying an image corresponding to the zoom ratio among the first images (1110).

[0182] FIG. 12 is a diagram for explaining an operation of generating an interpolation frame using parallax data and ratio data of an electronic device according to an embodiment of the present invention.

[0183] Referring to FIG. 12, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may obtain an interpolated image (1250) by using input images (1210, 1220) whose angles and sizes are aligned (e.g., the first input image (810) of FIG. 8 and the second input image (811) of FIG. 8), parallax data (1211, 1221) obtained through an artificial intelligence model (e.g., the first parallax data (820) of FIG. 8 and the second parallax data (821) of FIG. 8), and ratio data (1230). According to one embodiment, the parallax data (1211, 1221) and ratio data (1230) may be parallax information corresponding to an intermediate zoom ratio at which an interpolated frame is generated.

[0184] According to one embodiment, the electronic device may obtain a first warped image (1212) by warping a first input image (1210) using first parallax data (1211), and may obtain a second warped image (1222) by warping a second input image (1220) using second parallax data (1221).

[0185] According to one embodiment, the electronic device may generate an interpolated image (1250) by synthesizing (1240) a first warping image (1212), a second warping image (1222), and ratio data (1230). According to one embodiment, the ratio data (1230) may include data related to a ratio at which pixel values ​​of the first warping image (1212) are to be applied and a ratio at which pixel values ​​of the second warping image (1222) are to be applied, for each pixel, when generating the interpolated image (1250).

[0186] FIG. 13 is a flowchart illustrating an operation of displaying an interpolated frame according to a target zoom ratio of an electronic device according to one embodiment.

[0187] Referring to FIG. 13, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may, in operation 1301, acquire images of a valid area from a camera. According to an embodiment, the operation of acquiring images of a valid area is the same as the image preprocessing operation (410) of FIG. 4A, and thus, a duplicate description is omitted. According to an embodiment, only one camera may be activated, and the electronic device may acquire an image of a valid area that can be displayed on a display among the images acquired from the activated camera.

[0188] In one embodiment, in operation 1302, the electronic device may calculate an appropriate field of view for the acquired images. In one embodiment, the electronic device may calculate an angle of view to be displayed among the acquired images based on a required zoom ratio. For example, the electronic device may calculate an angle of view to be displayed among the images acquired through digital zoom based on a required zoom ratio.

[0189] In one embodiment, in operation 1303, the electronic device may determine whether the zoom factor has changed. In one embodiment, the electronic device may determine that the zoom factor has changed when a user input selecting one of a plurality of buttons each associated with a plurality of zoom factors, a user input selecting or dragging a specific area of ​​a slide bar for changing the zoom factor, or a pinch input (e.g., pinch-in input, pinch-out input) for changing the zoom factor is received.

[0190] In one embodiment, if the zoom factor does not change (operation 1303 - No), in operation 1304, the electronic device may crop the image based on the calculated field of view. In one embodiment, if no user input for changing the zoom factor (e.g., selecting a button, selecting or dragging a specific area of ​​a slide bar, or pinch input) is received, the electronic device may determine that the zoom factor does not change. In one embodiment, the electronic device may crop the image based on the calculated field of view based on the current zoom factor.

[0191] In one embodiment, in operation 1305, the electronic device may display an image scaled to fit the display. In one embodiment, the electronic device may display a cropped image on the display by resizing (e.g., scaling) the image to fit the display.

[0192] In one embodiment, when the zoom factor is changed (operation 1303 - Yes), in operation 1306, the electronic device may determine whether a camera change is required. In one embodiment, when a user input for changing the zoom factor is received (e.g., selecting a button, selecting or dragging a specific area of ​​a slide bar, or pinch input), the electronic device may determine whether a camera change is required based on the target zoom factor. For example, when a first camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the first camera (180-1) of FIG. 4A) acquires an image having a zoom magnification of 1x or more and less than 3x, and a second camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the second camera (180-2) of FIG. 4A) acquires an image having a zoom magnification of 3x or more and less than 10x, if the target zoom magnification is 3x or more while a preview image having a current zoom magnification of 1x is displayed, the electronic device can determine that a change of the camera is required.

[0193] In one embodiment, when only one camera is activated and the zoom factor is changed or a camera change is determined to be necessary, the electronic device may additionally activate a camera adjacent to the activated camera (or a camera capable of acquiring an image at the target zoom factor).

[0194] In one embodiment, if it is determined that a camera change is unnecessary (operation 1306 - No), in operation 1307, the electronic device may recalculate the image field of view based on the changed zoom ratio. For example, if the current zoom ratio is 1x and the target zoom ratio is less than 3x based on a user input for changing the zoom ratio, the electronic device may determine that a camera change is unnecessary and recalculate the field of view corresponding to the target zoom ratio among the images acquired by the first camera.

[0195] According to one embodiment, if it is determined that a camera change is required (operation 1306 - Yes), in operation 1308, the electronic device may align the field of view between the images. According to one embodiment, the operation of aligning the field of view between the images is identical to the spatial alignment transformation operation (420) of FIG. 4A, and thus, a duplicate description is omitted.

[0196] According to one embodiment, in operation 1309, the electronic device may crop images based on the aligned field of view. According to one embodiment, the operation of cropping images based on the aligned field of view is identical to the crop operation (432) of FIG. 4B, and thus, a duplicate description is omitted.

[0197] According to one embodiment, in operation 1310, the electronic device can scale the sizes of cropped images to be the same. According to one embodiment, the operation of scaling the cropped images is identical to the scaling operation (433) of FIG. 4B, and thus, a duplicate description is omitted.

[0198] According to one embodiment, in operation 1311, the electronic device may estimate the parallax of images. According to one embodiment, the operation of estimating the parallax is identical to the parallax estimation operation (434) of FIG. 4B, and thus, a duplicate description is omitted.

[0199] According to one embodiment, in operation 1312, the electronic device may acquire a parallax-compensated image. According to one embodiment, the operation of acquiring the parallax-compensated image is identical to the parallax compensation operation (435) of FIG. 4B, and thus, a duplicate description is omitted.

[0200] In one embodiment, the electronic device may, after acquiring an interpolated frame, which is a parallax-compensated image, proceed to operation 1305 to display the interpolated frame by scaling it to fit the display. For example, at least a portion of a zoom ratio change from a current zoom ratio to a target zoom ratio may be displayed as interpolated frames instead of digitally zooming an image acquired from an activated camera.

[0201] FIG. 14 is a flowchart illustrating an operation of determining a zoom curve according to a target zoom ratio of an electronic device according to one embodiment.

[0202] Referring to FIG. 14, an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) may, in operation 1401, determine whether a camera change is necessary when a zoom ratio is changed. According to one embodiment, determining whether a camera change is necessary is the same as operation 1306 of FIG. 13, and thus, a duplicate description will be omitted.

[0203] In one embodiment, if a camera change is not required when the zoom factor is changed (operation 1401 - No), the electronic device may proceed to operation 1307 of FIG. 13 to recalculate the field of view of the image based on the changed zoom factor.

[0204] In one embodiment, if a camera change is required when a zoom factor is changed (operation 1401 - Yes), the electronic device may determine a zoom curve in operation 1402. In one embodiment, the zoom curve may be related to a timing (e.g., a parallax estimation timing or a synthesis timing) at which an intermediate zoom factor is applied between a current zoom factor and a target zoom factor. For example, if the zoom curve is linear, a screen with an intermediate zoom factor applied may be displayed at regular intervals so that the zoom change may appear to be performed at a constant speed. In one embodiment, if the zoom curve is non-linear, a section with a large slope of the zoom curve may appear to perform a zoom change at a fast speed, and a section with a small slope of the zoom curve may appear to perform a zoom change at a slow speed.

[0205] For example, when changing from a 1x zoom state to a 3x zoom state, the electronic device may have 7 intermediate zoom levels between 1x and 3x. In one embodiment, the number of interpolation frames may be 7, each corresponding to an intermediate zoom level.

[0206] According to one embodiment, the parallax estimation time for each step may be performed linearly or nonlinearly as shown in [Table 1] below. [Table 1] may be an example in which the number of interpolation frames between a 1x camera and a 3x camera is set to 7, and the parallax estimation time (synthesis timing) between input images is set linearly and nonlinearly for each intermediate zoom ratio.

[0207] Request zoom factor 1.0001.2501.5001.7502.0002.2502.5002.7503.000 Leading parallax estimation point -0.1250.2500.3750.5000.6250.7500.875- Non-leading parallax estimation point -0.3130.5280.6750.7770.8470.8950.928-

[0208] According to one embodiment, the electronic device may identify one of a plurality of zoom curves based on the difference between the current zoom factor and the target zoom factor. For example, if the difference between the current zoom factor and the target zoom factor is large, a non-linear zoom curve including a section with a large slope may be selected. According to one embodiment, the electronic device may store a zoom curve mapped to the difference between the current zoom factor and the target zoom factor. According to one embodiment, the electronic device may select a zoom curve corresponding to the difference between the current zoom factor and the target zoom factor.

[0209] According to one embodiment, the electronic device may determine parallax estimation points at which a set number of intermediate zoom factors are applied, respectively, based on the identified zoom curve. For example, if the zoom curve is linear, the difference between the parallaxes to which the intermediate zoom factors are applied may be a constant value. According to one embodiment, if the zoom curve is non-linear, the difference between the parallaxes to which the intermediate zoom factors are applied may be large in a section where the slope of the zoom curve is large, and the difference between the parallaxes to which the intermediate zoom factors are applied may be small in a section where the slope of the zoom curve is small.

[0210] In one embodiment, the electronic device may determine a zoom factor corresponding to a distance between a start point and an end point of a pinch zoom input or a speed of the pinch zoom input as a target zoom factor.

[0211] In one embodiment, when changing the zoom ratio through a pinch input, when a pinch input is received, two cameras close to the current zoom ratio are selected, and the optical zoom ratios of the two cameras and the requested zoom ratio are linearly calculated as in [Mathematical Formula 1] to select an estimated time point.

[0212]

[0213] (Here, Optical Zoom1 < Request Zoom < Optical Zoom 2) )

[0214] For example, if 1x zoom, 3x zoom, and 10x zoom cameras are arranged in an electronic device, and 1.25x is requested by pinch zoom input from the current 1x zoom state, the electronic device can select the 1x zoom camera and the 3x zoom camera, and calculate the parallax estimation point of the images acquired by the 1x zoom camera and the 3x zoom camera as 0.125.

[0215] In one embodiment, if the current zoom state is 3x and 5.1x is requested by pinch zoom input, the electronic device may select a 3x zoom camera and a 10x zoom camera, and calculate the parallax estimation point of the images acquired by the 3x zoom camera and the 10x zoom camera as 0.3.

[0216] In one embodiment, when determining the target zoom factor based on the speed of the pinch zoom input, the electronic device may skip the 3x zoom and determine the target zoom factor as 7.3x zoom, which is close to 10x zoom, based on the fast speed of the pinch zoom input in the current 1x zoom state. In one embodiment, the electronic device may select the current 1x zoom camera and the 10x zoom camera, which are close to the target zoom factor, instead of selecting the two cameras, the 3x zoom camera and the 10x zoom camera, which are close to the target zoom factor. In one embodiment, the electronic device may calculate the parallax estimation point of time of the images acquired by the 1x zoom camera and the 10x zoom camera as 0.7.

[0217] In one embodiment, for zoom by pinch input, the electronic device may determine the number of interpolated frames to be one for each request received while the zoom is changed.

[0218] In one embodiment, at operation 1403, the electronic device may determine the number and timing of FRC frames (or interpolated frames).

[0219] According to one embodiment, the electronic device may proceed to operation 1308 of FIG. 13 after determining the number and timing of interpolation frames to align the angles of view between images acquired by the plurality of cameras.

[0220] FIG. 15 is a diagram for explaining an operation of displaying parallax-compensated interpolated frames of an electronic device according to one embodiment.

[0221] Referring to FIG. 15, when an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) switches from a first image (1501) acquired by a first camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the first camera (180-1) of FIG. 4A) to a second image (1502) acquired by a second camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the second camera (180-2) of FIG. 4A), the electronic device may display interpolated frames (1510) between the first image (1501) and the second image (1502).

[0222] According to one embodiment, the corresponding positions (1512) included in each of the interpolated frames (1510) gradually move away from the parallel reference line (1511) as they get closer to the second image (1502) from the first image (1501), thereby compensating for the parallax between the first image (1501) and the second image (1502).

[0223] In this way, by compensating for parallax when switching cameras, you can create smooth camera switching and zoom effects by correcting for the video jerkiness or shaking that may occur due to the switching parallax that occurs when switching cameras.

[0224] FIG. 16 is a drawing for explaining an operation of displaying an exposure-compensated interpolated frame of an electronic device according to one embodiment.

[0225] Referring to FIG. 16, when an electronic device (e.g., the electronic device (101) of FIG. 1 or the processor (120) of FIG. 1) switches from a first image (1601) acquired by a first camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the first camera (180-1) of FIG. 4A) to a second image (1602) acquired by a second camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, or the second camera (180-2) of FIG. 4A), the electronic device may display interpolated frames (1610) between the first image (1601) and the second image (1602).

[0226] In one embodiment, when images are acquired with the exposure value (EV) of the first camera set to -1 and the exposure value of the second camera set to +1, the matching performance between the two images may deteriorate. If the image matching performance deteriorates, information acquired in a spatial alignment transformation operation (e.g., the spatial alignment transformation operation (420) of FIG. 4A) may be inaccurate, which may cause the subject to pop out or shake when the camera is switched due to a zoom change. Even when the alignment is successful, there may be a difference in brightness between the images acquired by the first camera and the images acquired by the second camera, and a rapid change in brightness may occur when the camera is switched.

[0227] According to one embodiment, by generating interpolated frames (1610) while adjusting the application ratios of the first image (1601) and the second image (1602), the exposure difference between the first image (1601) and the second image (1602) can be compensated for as the application ratio of the first image (1601) decreases and the application ratio of the second image (1602) increases as the first image (1601) gets closer to the second image (1602).

[0228] For example, when performing a zoom change from 0.6x to 1.0x using a first camera that is a 0.6x zoom camera and a second camera that is a 1x zoom camera, the electronic device can display images acquired by the first camera from the first image (1601) to the fifth image, and display images acquired by the second camera for the sixth image.

[0229] In one embodiment, when the first camera has an exposure value set to -1 and the second camera has an exposure value set to +1, the electronic device determines a parallax estimation point in proportion to the zoom ratio, and generates interpolation frames (1610) according to a synthesis ratio of the two images based on the parallax estimation point, thereby allowing gradual brightness changes and compensating for abrupt exposure changes.

[0230] According to one embodiment, when a camera change is required due to a change in zoom ratio while the focal points of the first camera and the second camera are different from each other, the electronic device determines a parallax estimation point in proportion to the zoom ratio, and generates interpolated frames according to a synthesis ratio of the two images based on the parallax estimation point, so that a gradual focus change can be made and rapid subject shaking can be corrected even when the focus changes due to the camera change.

[0231] In one embodiment, a third party (3 rd Even when a filter function (e.g., face masking or background effect) is applied through the camera application, the filter function can be applied even if the camera changes according to the change in zoom ratio by generating an interpolated frame.

[0232] Electronic devices can capture images with a wide range of zoom ratios using multiple cameras. Depending on the user's desired zoom ratio, the camera used to capture the image to be displayed can be changed.

[0233] Even when multiple cameras simultaneously capture the same subject, the physical characteristics and placement of the cameras differ. Therefore, the images captured by each camera have different angles of view and parallax, and a discontinuity in the preview screen may be perceived when switching cameras. Therefore, technology may be needed to compensate for the abrupt changes when switching cameras.

[0234] The technical problems to be solved by the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the description below.

[0235] According to one embodiment, an electronic device may include a first camera, a second camera disposed around the first camera and having a different angle of view from that of the first camera, a display, at least one processor including a processing circuit, and a memory storing instructions.

[0236] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire a first image at a first zoom magnification through the first camera.

[0237] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display at least a portion of the first image on the display.

[0238] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a target zoom factor based on a user input that changes the zoom factor.

[0239] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate an interpolated frame based on at least a portion of a second image of a second zoom ratio acquired through the first image and the second camera, based on the target zoom ratio being outside a set range.

[0240] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0241] In one embodiment, the memory may further store an artificial intelligence model.

[0242] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to input a first input image obtained based on the first image and a second input image obtained based on the second image as input data to the artificial intelligence model based on the target zoom factor being outside the set range, and to obtain, as output data of the artificial intelligence model, first disparity data per pixel of the first input image with respect to the second input image, second disparity data per pixel of the second input image with respect to the first input image, and ratio data.

[0243] In one embodiment, the ratio data may relate to a ratio at which pixel values ​​of the first input image are to be applied pixel by pixel and a ratio at which pixel values ​​of the second input image are to be applied pixel by pixel.

[0244] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the interpolated frame based on the first input image, the first disparity data, the second input image, the second disparity data, and the ratio data.

[0245] According to one embodiment, the artificial intelligence model may be trained to use a plurality of input images as input data and obtain, as output data, a plurality of disparity data with respect to each other of the plurality of input images and data related to a ratio to which pixels of each of the plurality of input images are to be applied.

[0246] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a set number of intermediate zoom factors between the first zoom factor and the second zoom factor.

[0247] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the set number of first input images and the set number of second input images.

[0248] According to one embodiment, the first input images may correspond to the set number of intermediate zoom ratios, respectively, and the second input images may correspond to the set number of intermediate zoom ratios, respectively.

[0249] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the set number of interpolated frames corresponding to each of the set number of intermediate zoom factors.

[0250] In one embodiment, the memory may store a plurality of zoom curves.

[0251] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify one of the plurality of zoom curves based on a difference between the first zoom factor and the target zoom factor.

[0252] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine, based on the identified zoom curve, respective parallaxes to which the set number of intermediate zoom factors are applied.

[0253] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a portion of the first image corresponding to a current zoom factor while changing from the first zoom factor to a set zoom factor between the first zoom factor and the second zoom factor.

[0254] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to sequentially display the set number of interpolated frames by replacing a portion of the first image while changing from the set zoom factor to the second zoom factor.

[0255] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to sequentially display the set number of interpolated frames in place of at least a portion of the first image while changing from the first zoom factor to the second zoom factor.

[0256] According to one embodiment, the first input image and the second input image may be obtained by cropping a first region of the first image that matches the second image and a second region of the second image that matches the first image, and resizing the first region and the second region so that the sizes of the first region and the second region correspond.

[0257] According to one embodiment, the memory may further store a second artificial intelligence model trained to acquire output data by resizing each matching area between the plurality of images to the same size using a plurality of images as input data.

[0258] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain the first input image and the second input image as output data using the second artificial intelligence model, using the first image and the second image as input data.

[0259] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a plurality of icons, each corresponding to a plurality of zoom factors, overlapping at least a portion of the first image displayed on the display.

[0260] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, based on receiving the user input selecting one of the plurality of icons, identify a zoom factor corresponding to the selected icon as the target zoom factor.

[0261] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, based on the user input being received as a pinch input, determine a zoom factor corresponding to a distance between a start point and an end point of the pinch input as the target zoom factor.

[0262] According to one embodiment, a method of controlling an electronic device may include an operation of acquiring a first image at a first zoom ratio through a first camera of the electronic device.

[0263] According to one embodiment, a method of controlling an electronic device may include an action of displaying at least a portion of the first image on a display of the electronic device.

[0264] In one embodiment, a method of controlling an electronic device may include an action of determining a target zoom factor based on a user input that changes the zoom factor.

[0265] In one embodiment, a method of controlling an electronic device may include generating an interpolated frame based on at least a portion of a second image of a second zoom ratio obtained through a second camera positioned around the first camera and having a different angle of view from that of the first camera, based on the target zoom ratio being outside a set range.

[0266] According to one embodiment, a method of controlling an electronic device may include displaying the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0267] According to one embodiment, a control method of an electronic device may further include an operation of inputting a first input image obtained based on the first image and a second input image obtained based on the second image as input data to an artificial intelligence model stored in a memory of the electronic device based on the target zoom ratio being out of the set range, and obtaining first disparity data per pixel of the first input image with respect to the second input image, second disparity data per pixel of the second input image with respect to the first input image, and ratio data as output data of the artificial intelligence model.

[0268] In one embodiment, the ratio data may relate to a ratio at which pixel values ​​of the first input image are to be applied pixel by pixel and a ratio at which pixel values ​​of the second input image are to be applied pixel by pixel.

[0269] According to one embodiment, the artificial intelligence model may be trained to use a plurality of input images as input data and obtain, as output data, a plurality of disparity data with respect to each other of the plurality of input images and data related to a ratio to which pixels of each of the plurality of input images are to be applied.

[0270] According to one embodiment, the operation of generating the interpolated frame may include an operation of generating the interpolated frame based on the first input image, the first disparity data, the second input image, the second disparity data, and the ratio data.

[0271] In one embodiment, the act of generating the interpolated frame may include the act of determining a set number of intermediate zoom factors between the first zoom factor and the second zoom factor.

[0272] According to one embodiment, the operation of generating the interpolated frame may include the operation of generating the set number of first input images and the set number of second input images.

[0273] According to one embodiment, the first input images may correspond to the set number of intermediate zoom ratios, respectively, and the second input images may correspond to the set number of intermediate zoom ratios, respectively.

[0274] According to one embodiment, the operation of generating the interpolated frame may include an operation of generating the set number of interpolated frames corresponding to each of the set number of intermediate zoom ratios.

[0275] According to one embodiment, the memory of the electronic device can store a plurality of zoom curves.

[0276] In one embodiment, the act of generating the interpolated frame may include the act of identifying one of the plurality of zoom curves based on a difference between the first zoom factor and the target zoom factor.

[0277] In one embodiment, the operation of generating the interpolated frame may include an operation of determining, based on the identified zoom curve, respective parallaxes to which the set number of intermediate zoom ratios are applied.

[0278] In one embodiment, the act of displaying the interpolated frame on the display may include an act of displaying a portion of the first image corresponding to a current zoom factor while changing from the first zoom factor to a set zoom factor between the first zoom factor and the second zoom factor.

[0279] In one embodiment, the operation of displaying the interpolated frames on the display may include an operation of sequentially displaying the set number of interpolated frames by replacing a portion of the first image while changing from the set zoom ratio to the second zoom ratio.

[0280] In one embodiment, the act of displaying the interpolated frames on the display may include an act of sequentially displaying the set number of interpolated frames in place of at least a portion of the first image while changing from the first zoom factor to the second zoom factor.

[0281] According to one embodiment, the first input image and the second input image may be obtained by cropping a first region of the first image that matches the second image and a second region of the second image that matches the first image, and resizing the first region and the second region so that the sizes of the first region and the second region correspond.

[0282] According to one embodiment, the control method of the electronic device may further include an operation of displaying a plurality of icons, each corresponding to a plurality of zoom ratios, overlapping at least a portion of the first image displayed on the display.

[0283] According to one embodiment, the operation of checking the target zoom ratio may include an operation of checking a zoom ratio corresponding to the selected icon as the target zoom ratio based on receiving the user input of selecting one icon from the plurality of icons.

[0284] According to one embodiment, the operation of checking the target zoom ratio may include an operation of checking a zoom ratio corresponding to a distance between a start point and an end point of the pinch input as the target zoom ratio, based on the user input being received as a pinch input.

[0285] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may include instructions for causing an electronic device to acquire a first image at a first zoom ratio through a first camera of the electronic device.

[0286] In one embodiment, the one or more programs may include instructions that cause the electronic device to display at least a portion of the first image on a display of the electronic device.

[0287] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a target zoom factor based on a user input that changes the zoom factor.

[0288] In one embodiment, the one or more programs may include instructions that cause the electronic device to generate an interpolated frame based on at least a portion of a second image of a second zoom ratio obtained through a second camera positioned around the first camera and having a different angle of view from that of the first camera, based on the target zoom ratio being outside a set range.

[0289] In one embodiment, the one or more programs may include instructions that cause the electronic device to display the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

[0290] In one embodiment, the memory may further store an artificial intelligence model.

[0291] According to one embodiment, the one or more programs may include instructions that cause the electronic device to input a first input image obtained based on the first image and a second input image obtained based on the second image as input data to the artificial intelligence model based on the target zoom ratio being out of the set range, and to obtain, as output data of the artificial intelligence model, first disparity data per pixel of the first input image with respect to the second input image, second disparity data per pixel of the second input image with respect to the first input image, and ratio data.

[0292] In one embodiment, the ratio data may relate to a ratio at which pixel values ​​of the first input image are to be applied pixel by pixel and a ratio at which pixel values ​​of the second input image are to be applied pixel by pixel.

[0293] According to one embodiment, the one or more programs may include instructions that cause the electronic device to generate the interpolated frame based on the first input image, the first disparity data, the second input image, the second disparity data, and the ratio data.

[0294] According to one embodiment, the artificial intelligence model may be trained to use a plurality of input images as input data and obtain, as output data, a plurality of disparity data with respect to each other of the plurality of input images and data related to a ratio to which pixels of each of the plurality of input images are to be applied.

[0295] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine a set number of intermediate zoom factors between the first zoom factor and the second zoom factor.

[0296] According to one embodiment, the one or more programs may include instructions that cause the electronic device to generate the set number of first input images and the set number of second input images.

[0297] According to one embodiment, the first input images may correspond to the set number of intermediate zoom ratios, respectively, and the second input images may correspond to the set number of intermediate zoom ratios, respectively.

[0298] In one embodiment, the one or more programs may include instructions that cause the electronic device to generate the set number of interpolated frames corresponding to each of the set number of intermediate zoom factors.

[0299] In one embodiment, the memory may store a plurality of zoom curves.

[0300] In one embodiment, the one or more programs may include instructions that cause the electronic device to identify one of the plurality of zoom curves based on a difference between the first zoom factor and the target zoom factor.

[0301] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine, based on the identified zoom curve, respective parallaxes to which the set number of intermediate zoom factors are applied.

[0302] In one embodiment, the one or more programs may include instructions that cause the electronic device to display a portion of the first image corresponding to a current zoom factor while changing from the first zoom factor to a set zoom factor between the first zoom factor and the second zoom factor.

[0303] In one embodiment, the one or more programs may include instructions that cause the electronic device to sequentially display the set number of interpolated frames by replacing a portion of the first image while changing from the set zoom factor to the second zoom factor.

[0304] In one embodiment, the one or more programs may include instructions that cause the electronic device to sequentially display the set number of interpolated frames in place of at least a portion of the first image while changing from the first zoom factor to the second zoom factor.

[0305] According to one embodiment, the first input image and the second input image may be obtained by cropping a first region of the first image that matches the second image and a second region of the second image that matches the first image, and resizing the first region and the second region so that the sizes of the first region and the second region correspond.

[0306] According to one embodiment, the memory may further include instructions for storing a second artificial intelligence model trained to acquire output data by resizing each matching region between the plurality of images to the same size using a plurality of images as input data.

[0307] According to one embodiment, the one or more programs may include instructions that cause the electronic device to obtain the first input image and the second input image as output data, using the second artificial intelligence model, using the first image and the second image as input data.

[0308] According to one embodiment, the one or more programs may include instructions that cause the electronic device to display a plurality of icons, each corresponding to a plurality of zoom magnifications, overlapping at least a portion of the first image displayed on the display.

[0309] In one embodiment, the one or more programs may include instructions that cause the electronic device to, based on receiving a user input selecting one of the plurality of icons, determine a zoom factor corresponding to the selected icon as the target zoom factor.

[0310] In one embodiment, the one or more programs may include instructions that cause the electronic device to determine, based on the user input being a pinch input, a zoom factor corresponding to a distance between a start point and an end point of the pinch input as the target zoom factor.

[0311] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0312] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0313] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0314] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0315] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0316] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0317] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, Camera 1; A second camera arranged around the first camera and having a different angle of view from that of the first camera; display; At least one processor comprising a processing circuit; and memory for storing instructions; The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining a first image with a first zoom ratio through the first camera, displaying at least a portion of the first image on the display; Determine the target zoom factor based on user input that changes the zoom factor, Based on the above target zoom ratio being out of the set range, generating an interpolated frame based on at least a part of the second image of the second zoom ratio acquired through the first image and the second camera, and An electronic device that causes the interpolated frame to be displayed on the display by replacing a portion of the first image corresponding to the zoom ratio during at least a portion of the time during which the zoom ratio is changed.

2. In paragraph 1, The above memory further stores artificial intelligence models, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the above target zoom ratio being out of the set range, a first input image obtained based on the first image and a second input image obtained based on the second image are input as input data to the artificial intelligence model, and first disparity data per pixel of the first input image with respect to the second input image, second disparity data per pixel of the second input image with respect to the first input image, and ratio data are obtained as output data of the artificial intelligence model, wherein the ratio data is related to a ratio to which pixel values ​​of the first input image are applied and a ratio to which pixel values ​​of the second input image are applied for each pixel, and Generate the interpolated frame based on the first input image, the first disparity data, the second input image, the second disparity data, and the ratio data, The above artificial intelligence model is, An electronic device that is trained to use multiple input images as input data to obtain multiple parallax data with respect to each other of the multiple input images and data related to a ratio to which each pixel of the multiple input images is to be applied as output data.

3. In paragraph 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Determine a set number of intermediate zoom ratios between the first zoom ratio and the second zoom ratio, Generating the set number of first input images and the set number of second input images, wherein the first input images correspond to the set number of intermediate zoom ratios, and the second input images correspond to the set number of intermediate zoom ratios, and An electronic device that generates a set number of interpolation frames corresponding to each of the set number of intermediate zoom ratios.

4. In paragraph 3, The above memory stores multiple zoom curves, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Identifying one of the plurality of zoom curves based on the difference between the first zoom ratio and the target zoom ratio, and An electronic device that determines the parallaxes to which the set number of intermediate zoom ratios are applied, based on the identified zoom curve.

5. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: While changing from the first zoom ratio to a set zoom ratio between the first zoom ratio and the second zoom ratio, displaying a portion of the first image corresponding to the current zoom ratio, and An electronic device that sequentially displays the set number of interpolated frames by replacing a portion of the first image while changing from the set zoom ratio to the second zoom ratio.

6. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that sequentially displays the set number of interpolated frames by replacing at least a portion of the first image while changing from the first zoom ratio to the second zoom ratio.

7. In paragraph 2, The first input image and the second input image are, An electronic device obtained by cropping a first region of the first image that matches the second image and a second region of the second image that matches the first image, and resizing the first region and the second region so that the sizes of the first region and the second region correspond.

8. In paragraph 2, The above memory further stores a second artificial intelligence model trained to acquire output data by resizing each matching area between the plurality of images to the same size using a plurality of images as input data; The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that uses the first image and the second image as input data, and obtains the first input image and the second input image as output data using the second artificial intelligence model.

9. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: A plurality of icons corresponding to a plurality of zoom ratios are displayed overlapping at least a portion of the first image displayed on the display, and An electronic device that, based on receiving a user input for selecting one of the plurality of icons, confirms a zoom ratio corresponding to the selected icon as the target zoom ratio.

10. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that, based on the user input being received as a pinch input, determines a zoom ratio corresponding to the distance between the start point and the end point of the pinch input as the target zoom ratio.

11. In a method for controlling an electronic device, An operation of acquiring a first image of a first zoom ratio through a first camera of the electronic device; An action of displaying at least a portion of the first image on a display of the electronic device; An action to determine a target zoom factor based on user input that changes the zoom factor; An operation of generating an interpolated frame based on at least a portion of a second image of a second zoom ratio obtained through a second camera positioned around the first camera and having a different angle of view from that of the first camera, based on the target zoom ratio being out of a set range; and A control method of an electronic device, comprising: an operation of displaying the interpolated frame on the display by replacing a portion of the first image corresponding to the zoom ratio during at least a portion of the time during which the zoom ratio is changed.

12. In paragraph 11, An operation of inputting a first input image obtained based on the first image and a second input image obtained based on the second image as input data to an artificial intelligence model stored in a memory of the electronic device based on the fact that the target zoom ratio is out of the set range, and obtaining first disparity data per pixel of the first input image for the second input image, second disparity data per pixel of the second input image for the first input image, and ratio data as output data of the artificial intelligence model; further comprising; The above ratio data is related to the ratio at which the pixel values ​​of the first input image are applied for each pixel and the ratio at which the pixel values ​​of the second input image are applied, The above artificial intelligence model is, It is learned to use multiple input images as input data to obtain multiple parallax data for each of the multiple input images and data related to the ratio to which each pixel of the multiple input images is applied as output data, The operation of generating the above interpolated frame is: A control method of an electronic device, comprising an operation of generating the interpolated frame based on the first input image, the first disparity data, the second input image, the second disparity data, and the ratio data.

13. In paragraph 12, The operation of generating the above interpolated frame is: An operation of determining a set number of intermediate zoom ratios between the first zoom ratio and the second zoom ratio; An operation of generating the set number of first input images and the set number of second input images, wherein the first input images correspond to the set number of intermediate zoom ratios, respectively, and the second input images correspond to the set number of intermediate zoom ratios, respectively; and A control method of an electronic device, comprising an operation of generating a set number of interpolation frames corresponding to each of the set number of intermediate zoom ratios.

14. In paragraph 13, The memory of the above electronic device stores a plurality of zoom curves, The operation of generating the above interpolated frame is: An operation of identifying one of the plurality of zoom curves based on a difference between the first zoom ratio and the target zoom ratio; and A control method of an electronic device, comprising an operation of determining parallaxes to which the set number of intermediate zoom ratios are applied, based on the identified zoom curve.

15. A non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs are: Obtaining a first image with a first zoom magnification through a first camera of the electronic device, displaying at least a portion of the first image on a display of the electronic device; Determine the target zoom factor based on user input that changes the zoom factor, Based on the above target zoom ratio being out of the set range, generating an interpolated frame based on at least a part of a second image of a second zoom ratio acquired through a second camera positioned around the first image and the first camera and having a different angle of view from that of the first camera, and A recording medium comprising instructions for causing the interpolated frame to be displayed on the display by replacing a portion of the first image corresponding to the zoom factor during at least a portion of the time during which the zoom factor is changed.

Images