Electronic device and method for outputting image data of camera

Abstract

Provided are an electronic device and a method for outputting image data of a camera. The electronic device comprises a display, a camera, a memory, and at least one processor. Instructions, when individually or collectively executed by the at least one processor, cause the electronic device to generate the original image data including pixel data of a plurality of pixels acquired through an image sensor of the camera. The instructions cause the electronic device to identify a currently set first zoom magnification. The instructions cause the electronic device to determine a first area to be cropped from the original image data according to the first zoom magnification . The instructions cause the electronic device to generate, from the original image data, first image data in which pixel data of pixels of a second area, which is the remaining area excluding the first area, is set as padding data. The instructions cause the electronic device to transmit the first image data from the camera to the processor through an interface. The instructions cause the electronic device to receive the first image data from the processor. The instructions cause the electronic device to display a preview image on the display on the basis of pixel data of pixels of the first area of the first image data.

Description

Method for outputting image data of electronic devices and cameras

[0001] This document relates to an electronic device, and, for example, to a method for outputting image data acquired from a camera of an electronic device.

[0002] Portable electronic devices, such as smartphones and tablet PCs (hereinafter referred to as "electronic devices"), may include a camera for capturing images of the surrounding environment to provide various user experiences. For example, the electronic device may include at least one camera positioned on the front and / or rear.

[0003] Since portability is critical for electronic devices, the size of the camera is inevitably limited; consequently, it may be difficult to incorporate high-magnification optical zoom lenses with a wide variable focal length range, as is the case with general digital cameras (e.g., DSLR (digital single-lens reflex) cameras). Accordingly, electronic devices may provide a digital crop zoom function that crops and enlarges the original image acquired from the camera according to a zoom ratio set based on user input.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] When an electronic device provides a digital crop zoom function, the camera provides the original image acquired from the image sensor to the processor through an interface, and the processor can process the zoom by cropping an area corresponding to the zoom magnification ratio from the original image. In other words, the camera can output data of an area larger than the area actually needed in a zoom scenario to the processor. In this case, current consumption may occur because the analog operation of the image sensor's pixels and circuit parts, as well as the sensor read-out operation, must continue even for difference lines that are not actually used.

[0006] An electronic device according to the present disclosure (or specification, invention) may include a display, a camera including an image sensor, a memory, and at least one processor operatively connected to the display, the camera, and the memory.

[0007] According to one embodiment, the camera is configured to generate original image data including pixel data of a plurality of pixels acquired through the image sensor, check a currently set first zoom magnification, determine a first area to be cropped in the original image data corresponding to the first zoom magnification, and generate first image data in which pixel data of pixels of a second area, which is the remaining area excluding the first area in the original image data, is set as padding data, and can be configured to transmit the first image data to the processor through an interface.

[0008] According to one embodiment, the processor may be configured to receive the first image data and, based on the pixel data of the pixels of the first region of the first image data, display a preview image on the display.

[0009] A method performed by an electronic device according to various embodiments of the present document may include: generating original image data including pixel data of a plurality of pixels acquired through an image sensor by a camera of the electronic device; checking a first zoom magnification currently set by the camera; determining a first area to be cropped in the original image data corresponding to the first zoom magnification by the camera; generating first image data by the camera in which pixel data of pixels of a second area, which is the remaining area excluding the first area, is set as padding data by the camera; transmitting the first image data to a processor through an interface by the camera; receiving the first image data by the processor; and displaying a preview image on the display by the processor based on pixel data of pixels of the first area of ​​the first image data.

[0010] According to various embodiments of the present document, by inserting padding data into the remaining area outside the actual field of view area and transmitting it to a processor, the electronic device and a method for outputting image data acquired from the camera of the electronic device can be provided, which can reduce current consumption while satisfying the requirements of seamless mode switching for the transmission of image data of the same size.

[0011] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0012] FIG. 2 is a block diagram of a camera module according to various embodiments.

[0013] FIG. 3 illustrates cameras positioned on the rear of an electronic device according to one embodiment.

[0014] FIG. 4 is a block diagram of an electronic device according to various embodiments.

[0015] FIG. 5 illustrates a method for switching seamless modes of a camera according to one embodiment.

[0016] FIG. 6 illustrates image data acquired, transmitted, or processed by a camera, interface, and processor according to a zoom magnification according to one embodiment.

[0017] FIG. 7 illustrates the format of image data according to one embodiment.

[0018] FIG. 8 illustrates image data acquired, transmitted, or processed by a camera, interface, and processor according to a zoom magnification according to one embodiment.

[0019] FIG. 9 illustrates the format of image data according to one embodiment.

[0020] FIG. 10 illustrates a method in which a camera generates partial padding image data according to each frame in a zoom-in scenario, according to one embodiment.

[0021] FIG. 11 illustrates a method in which a camera generates partial padding image data according to each frame in a zoom-out scenario, according to one embodiment.

[0022] FIG. 12 illustrates an image provided according to the subject tracking zoom function of an electronic device according to one embodiment.

[0023] FIG. 13 illustrates a method for generating partial padding image data when an electronic device according to one embodiment provides a subject tracking zoom function.

[0024] FIG. 14 is a flowchart of a method for outputting image data of a camera of an electronic device according to one embodiment.

[0025] FIG. 15 is a flowchart of a method for a camera to output image data when changing the zoom magnification according to one embodiment.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0028] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0029] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0030] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0031] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0035] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0036] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0037] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0041] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0043] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0046] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0047] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0048] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0049] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050] FIG. 2 is a block diagram (200) illustrating a camera module according to various embodiments.

[0051] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a camera memory (250) (e.g., buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the subject of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0052] According to one embodiment, the flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different properties such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0053] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction in response to the movement of the camera module (180) or the electronic device (101) containing it, or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing). This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented, for example, as an optical image stabilizer. The camera memory (250) may temporarily store at least a portion of an image acquired through the image sensor (230) for subsequent image processing operations. For example, when image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the camera memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display device (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the camera memory (250) may be acquired and processed, for example, by an image signal processor (260). According to one embodiment, the camera memory (250) may be configured as at least a portion of the memory (130) of FIG. 1, or as a separate memory that operates independently thereof.

[0054] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the camera memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the camera memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display device (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be composed of at least a part of the processor (120), or It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display device (160) as is or after additional image processing by the processor (120).

[0055] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0056] FIG. 3 illustrates cameras positioned on the rear of an electronic device according to one embodiment.

[0057] According to one embodiment, the electronic device (300) may include at least one front camera disposed on the front of a housing where a display (e.g., the display module (160) of FIG. 1) is disposed, and at least one rear camera disposed on the rear of a housing where a rear cover is disposed.

[0058] Referring to FIG. 3, the electronic device (300) may include three cameras (e.g., a first camera (312), a second camera (314), and a third camera (316)) positioned adjacent to each other on the rear of the housing (e.g., the upper left rear), but the number and / or position of the cameras are not limited thereto. The first camera (312), the second camera (314), and the third camera (316) may include at least some of the configuration and / or functions of the camera module (180) of FIG. 1 and / or the camera module (180) of FIG. 2. According to one embodiment, the first camera (312), the second camera (314), and the third camera (316) are positioned adjacent to each other so as to be able to photograph in substantially the same direction.

[0059] According to one embodiment, the first camera (312), the second camera (314), and the third camera (316) may have different optical characteristics. For example, the first camera (312), the second camera (314), and the third camera (316) may differ in at least some of the lens optical characteristics, such as the angle of view, focal length, aperture, lens material, structure, refractive index, or refraction / diffraction characteristics, and / or at least some of the sensor characteristics, such as sensor pitch and number of pixels.

[0060] According to one embodiment, the first camera (312), the second camera (314), and the third camera (316) can capture images with different fields of view. Here, the field of view may refer to the range of a scene that the camera can capture at once. For example, the first camera (312) may be an ultra-wide camera (or UW (ultra-wide) camera) that captures a very wide field of view (e.g., about 120 degrees), the second camera (314) may be a wide camera (or W (wide) camera) that captures a wide field of view (e.g., about 84 degrees), and the third camera (316) may be a telephoto camera (or T (telescope) camera) that captures a narrow field of view (e.g., about 20 degrees). Accordingly, when the electronic device captures the same shooting scene with the first camera (312) and the second camera (314), when shooting with the first camera (312), the environment with a relatively wide field of view is captured, so the size of a specific subject may be small, and when shooting with the second camera (314), the environment with a relatively narrow field of view is captured, so the size of a specific subject may be large.

[0061] According to one embodiment, when a camera application (or an application using camera resources) is executed, the electronic device (300) activates one of the first camera (312), the second camera (314), or the third camera (316), and the electronic device (300) can display an image obtained from the camera in real time on a display as a preview (or viewfinder).

[0062] According to one embodiment, the electronic device (300) may provide a user interface (UI) that can set the zoom magnification when a camera application is executed. For example, the UI may be provided as selectable items corresponding to various zoom magnifications (e.g., x0.6, x1.0, x3.0), provided in the form of a scrollable bar where a specific zoom magnification can be selected, and / or provided so that the zoom magnification can be adjusted according to the user's multi-touch interaction (e.g., pinch to zoom). According to one embodiment, the electronic device (300) may acquire an image with a default zoom magnification value (e.g., x1.0) when a camera application is executed, and change the zoom magnification based on user input through the UI.

[0063] According to one embodiment, the electronic device (300) may provide a hybrid zoom function. Here, the hybrid zoom function may include analog optical zoom and digital zoom (or digital crop zoom), and may be a method in which analog optical zoom is used in some zoom magnification ranges and digital zoom is used in other zoom magnification ranges. Analog optical zoom may be a method of magnifying or reducing the field of view using a camera lens and optical elements. Digital zoom may be a method of magnifying or reducing the field of view of an image to be displayed or stored by cropping a part of the original image through digital processing of the original image acquired from an image sensor without changing the optical elements. For example, the first camera (312), the second camera (314), and the third camera (316) each acquire original images with analog optical zoom magnifications of x0.6, x1.0, and x3.0, respectively, and the electronic device (300) can acquire an image using the first camera (312) in the range where the current zoom magnification is less than the optical zoom magnification of the second camera (314) (e.g., x1.0) (e.g., x0.6 to x1.0), acquire an image using the second camera (314) in the range where the current zoom magnification is greater than or equal to the optical zoom magnification of the second camera (314) and less than the optical zoom magnification of the third camera (316) (e.g., x1.0 to x3.0), and acquire an image using the third camera (316) in the range where the current zoom magnification is greater than or equal to the optical zoom magnification of the third camera (316) (e.g., x3.0 to x30.0). The electronic device (300) crops a portion of the acquired image and displays it as a preview image in correspondence with the currently set zoom ratio, and can change the size of the area to be cropped from the image when the zoom ratio changes.

[0064] FIG. 4 is a block diagram of an electronic device according to various embodiments.

[0065] Referring to FIG. 4, an electronic device (400) according to various embodiments may include a camera (410), a display (450), a processor (490), and a memory (480). Various embodiments of this document may be implemented even if at least some of the illustrated configurations are omitted or replaced with other configurations. In addition to the illustrated configurations, the electronic device (400) may further include at least some of the configurations and / or functions of the electronic device (101) of FIG. 1. At least some of each of the illustrated (or unillustrated) components of the electronic device (400) (e.g., processor (490), memory (480)) may be placed within the housing of the electronic device (400), and at least some of the other components (e.g., display (450), camera (410)) may be visually exposed to the outside of the housing. At least some of each of the components of the electronic device (400) may be operatively, functionally, and / or electrically connected to one another.

[0066] According to one embodiment, the display (450) can display image information provided by the processor (490). The display (450) may be implemented as any one of a liquid crystal display (LCD), a light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display, but is not limited thereto. The display (450) may be configured as a touch screen that detects touch and / or proximity touch (or hovering) input using a part of the user's body (e.g., finger) or an input device (e.g., stylus pen). The display (450) may include at least some of the configuration and / or functions of the display module (160) of FIG. 1.

[0067] According to one embodiment, the display (450) may be a flexible display in which at least a portion is flexible. The electronic device (400) may be implemented in various form factors, such as a foldable device, a multi-foldable device, or a rollable device, in which the size of the display area can be changed by utilizing the characteristics of the flexible display.

[0068] According to one embodiment, the electronic device (400) may include at least one camera (410) on the front and / or rear respectively. As described through FIG. 3, the electronic device (400) may include three cameras (e.g., a first camera (312), a second camera (314), and a third camera (316)) on the rear of the housing where the rear cover is placed, but the number of cameras included in the electronic device (400) is not limited thereto. When the electronic device (400) includes a plurality of cameras (312, 314, 316) as in FIG. 3, each camera may capture images with different angles of view. For example, the first camera may be an ultra-wide camera (or UW (ultra-wide) camera) that captures a very wide angle of view (e.g., about 120 degrees), the second camera may be a wide camera (or W (wide) camera) that captures a wide angle of view (e.g., about 84 degrees), and the third camera may be a telephoto camera (or T (telescope) camera) that captures a narrow angle of view (e.g., about 20 degrees).

[0069] According to one embodiment, the camera (410) may include at least some of the configurations and / or functions of the camera module (180) of FIG. 2, such as a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a camera memory (250), and an image signal processor (260).

[0070] According to one embodiment, the image sensor (420) (e.g., the image sensor (230) of FIG. 2) may be implemented as a sensor such as a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The image sensor (420) includes a plurality of pixels in a matrix form and can generate image data by converting light detected at each pixel into an electrical signal.

[0071] According to one embodiment, the camera (410) may temporarily store at least a portion of the image data acquired through the image sensor (420) in the camera memory (430) (e.g., the camera memory (250) of FIG. 2). For example, the image acquired from the image sensor (420) and stored in the camera memory (430) includes pixel data for each pixel, and the camera (410) may generate original image data by performing processing such as binning and remosaic on the image stored in the camera memory (430). In this document, original image data may refer to image frames that have undergone image processing according to each shooting mode, such as DCG (dual conversion gain) mode, remosaic output mode, and LN (low noise). Original image data may also be referred to as full field of view image data or original full field of view image raw data.

[0072] According to one embodiment, the camera memory (430) can store various instructions that can be executed in the processor (490). The operations of the camera (410) described below can be performed by loading instructions stored in the camera memory (430) or memory (480).

[0073] According to one embodiment, the memory (480) may include volatile memory and non-volatile memory, and may store various data temporarily or permanently. The memory (480) may include at least some of the configuration and / or functions of the memory (130) of FIG. 1 and may store the program (140) of FIG. 1. The memory (480) may store various instructions that can be executed by the processor (490). Such instructions may include control commands such as arithmetic and logical operations, data movement, and input / output that can be recognized by the processor (490).

[0074] According to one embodiment, the processor (490) may be configured to perform operations or data processing regarding the control and / or communication of each component of the electronic device (400), and may be composed of one or more processors. The processor (490) may include at least some of the configuration and / or functions of the processor (120) of FIG. 1. There is no limit to the operations and data processing functions that the processor (490) can implement on the electronic device (400), but in this document, various embodiments for reducing power consumption occurring during the image data transmission process between the camera (410) and the processor (490) using partial padding image data will be described in detail. The operations of the processor (490) described below may be performed by loading instructions stored in the memory (480).

[0075] In this document, the description that a processor (490) can perform a certain operation (or function, task, or operation) may be interpreted substantially as meaning that an instruction (or command, computer program) causing the electronic device (400) (or processor (490)) to perform said operation is stored in memory (480) (e.g., non-volatile memory, storage). Additionally, the description that a processor (490) can perform a certain operation may be interpreted substantially as meaning that at least one processor, without a fixed number, can perform said operation.

[0076] Additionally, the description that the camera (410) can perform any operation (or function, operation, task) may be interpreted substantially as having the same meaning as that an instruction (or command, computer program) causing the electronic device (400) (or camera (410)) to perform the said operation is stored in the camera memory (430) and / or memory (480).

[0077] According to one embodiment, the camera (410) can transmit image data to the processor (490) through the interface (440). For example, the interface (440) may include, but is not limited to, a mobile industry processor interface (MIPI). Additionally, data other than image data, such as control commands between the camera (410) and the processor (490) and camera (410) setting values, may be transmitted and received via I2C (inter-integrated circuit) or I3C (improved inter-integrated circuit) communication.

[0078] According to one embodiment, the camera (410) may support a seamless change mode. The seamless change mode may also be referred to as FCM (fast change mode). The seamless change mode may be a technology that enables a smooth transition without delay in image output when switching the shooting mode of the camera (410). In contrast to the seamless change mode, the normal mode involves a stream off method in which, when switching modes, the image frame of the previous mode is finished and the set file of the changed mode is loaded in a SW standby state, thereby causing a delay during mode switching. In contrast, when operating in the seamless change mode, the camera (410) does not stop transmitting image frames to the processor (490) and can perform the seamless setting required for mode switching while transmitting image frames of the previous mode. The difference between the seamless change mode and the normal mode will be explained in more detail through FIG. 5.

[0079] According to one embodiment, for the electronic device (400) to operate in a seamless switching mode, it may be required that the size, number of pixels, number of bits, size of PDAF (phase detection auto focus) data, and / or MIPI (mobile industry processor interface) rate of each image frame transmitted from the camera (410) to the processor (490) be the same. In a situation where the device operates in a seamless switching mode, if the output size of the image data differs before and after the switching of the shooting mode, for example, if at least some of the image frame size, number of pixels, number of bits, size of PDAF data, or MIPI differs, the device cannot operate in a seamless switching mode, and only a mode switching accompanied by a stream-off in the normal mode may be possible. Even in the process of cropping only the actual field of view area corresponding to the zoom magnification and transmitting it to the processor (490), the camera (410) may transmit the remaining area filled with padding data, and accordingly, image data of the same size may be transmitted to the processor (490) before and after the change in the zoom magnification.

[0080] According to one embodiment, the camera (410) can generate original image data including pixel data of a plurality of pixels acquired through an image sensor (420). Here, the original image data may be image data that has undergone processing such as binning or remosaic depending on the shooting mode.

[0081] According to one embodiment, the camera (410) can determine an actual field of view area (or first area) to be cropped in correspondence with a first zoom magnification currently set in the original image data.

[0082] According to one embodiment, the electronic device (400) may provide a digital crop zoom function. For example, the electronic device (400) may acquire original image data using one of a plurality of cameras (e.g., the first camera (312), the second camera (314), and the third camera (316) of FIG. 3) that corresponds to a set zoom ratio, and the processor (490) may crop and enlarge an area corresponding to the zoom ratio from the original image data transmitted from the camera (410) and display it through a display (450).

[0083] According to one embodiment, the size of the actual field of view area in the original image data may correspond to the currently set zoom magnification. For example, when the image sensor (420) acquires an image corresponding to an optical zoom magnification (or field of view (FOV)) of x1.0, the actual field of view area corresponding to a zoom magnification of x1.5 may include an area corresponding to 1 / 1.5 based on the diagonal of the image among the center areas of the image acquired by the image sensor (420).

[0084] According to one embodiment, the camera (410) may generate partial padding image data (or first image data) in which pixel data of pixels in the remaining area (or second area), excluding the actual field of view area, is set as padding data. In this document, partial padding image data may refer to image data in which some of the pixels among the total pixels of the image data are filled with padding data. For example, when the zoom magnification is set to x1.5, pixels in the area outside the actual field of view area corresponding to 1 / 1.5 of the diagonal of the image among the center area of ​​the image acquired by the image sensor (420) may be designated as the remaining area. Padding data may also be referred to by terms such as dummy data, filler data, or dummy extension data.

[0085] According to one embodiment, padding data is data used to fit an image frame to a predetermined size and can be distinguished from the pixel data of the actual image. According to one embodiment, padding data may have a range of values ​​different from the range of values ​​of the pixel data of the original image data. For example, the camera (410) may add a pedestal value (or offset value) to the pixel data of each pixel to express the dark level of the pixel data obtained from each pixel of the image sensor (420) in detail. The camera (410) may set the padding data to be applied to the pixels of the remaining area to be less than or equal to the pedestal value so as to be distinguished from the pixel data of the pixels of the actual field of view area. Accordingly, the pixel data of the pixels of the actual field of view area and the padding data of the pixels of the remaining area may have different ranges of data.

[0086] According to one embodiment, the camera (410) can transmit the generated partial padding image data to the processor (490) through the interface (440). In the partial padding image data, the pixels in the actual field of view area may have pixel data of the pixels of the original image data, and the pixels in the remaining area may have padding data. The method and format for generating / transmitting the partial padding image data will be described in more detail through FIGS. 8 and 9.

[0087] According to one embodiment, the processor (490) can receive partial padding image data transmitted from the camera (410) through the interface (440). According to one embodiment, the processor (490) can display a preview image on the display (450) based on the pixel data of the actual field of view area of ​​the received partial padding image data, and can capture and save an image based on a user's shooting command.

[0088] According to one embodiment, the processor (490) may read pixels in the actual viewing angle area from the partial padding image data and not read padding data set in the pixels of the remaining area.

[0089] According to one embodiment, the processor (490) can identify the pixels in the remaining area set as padding data in the current frame based on the header information of the previous image frame. For example, the camera (410) can transmit information to the processor (490) indicating the area of ​​pixels into which padding data is inserted in the image frame to be transmitted next (or two image frames later) after the image frame, through a next frame information (NFI) function using metadata of a specific image frame (e.g., embedded data lines (EMB)). Alternatively, the camera (410) can transmit to the processor (490) information indicating the area of ​​pixels into which padding data is inserted by inserting it into the metadata of the current image frame, and the processor (490) can identify the remaining area into which padding data is inserted for each frame and not read the data in that area.

[0090] According to one embodiment, the processor (490) can identify the pixels in the remaining area set as padding data in the partial padding image data based on the currently set zoom magnification. For example, in a situation where a specific zoom magnification is maintained, the camera (410) is configured to insert padding data into the remaining area other than the actual field of view area and transmit it to the processor (490), so the processor (490) can recognize the remaining area corresponding to the zoom magnification and recognize the pixels with inserted padding data without separate readout.

[0091] According to one embodiment, the camera (410) may generate partial padded image data with padding data inserted and transmit it to the processor (490) in order to continuously transmit image data of the same size (or number of pixels) to the processor (490) when operating in a seamless switching mode. When the camera (410) operates in a normal mode that combines stream off rather than a seamless switching mode, it may transmit only pixel data of the actual field of view area, transmit original image data, or transmit partial padded image data as in a seamless switching mode.

[0092] According to one embodiment, as the camera (410) transmits partial padded image data with padding data inserted to the processor (490) through the interface (440), the current consumed for transmitting the image data may be reduced. Since the camera (410) inserts padding data outside the actual field of view area, the extra line with padding data inserted may not be processed, thereby reducing the current consumed by the power supply of that line. For example, for the line set with padding data, the analog voltage (e.g., AVDD) supplied to the analog pixel, and the RST (reset transistor), TG (transfer gate), and CDS (correlated double sampling) circuits may be reduced, and / or the digital voltage (e.g., DVDD) supplied to the digital counter and logic, and the BPC (bad pixel correction), HW remosaic, or artifact tune may be reduced. Accordingly, during the process of changing the shooting mode, image data of the same size is transmitted from the camera (410) to the processor (490), enabling seamless mode switching, while reducing read-out and processing operations for unnecessary sensor areas, thereby improving current consumption.

[0093] According to one embodiment, when the camera (410) transmits partial padding image data to the processor (490), the current consumption can be improved by using only about 0.5 to 0.6 times the current consumption compared to when the original image data is transmitted. For example, when the camera (410) transmits original image data to the processor (490) and when it transmits partial padding image data, the analog voltage and power can be measured as shown in Table 1.

[0094] Mode (AVDD 2.2V 기준) AVDD Current (mA) AVDD Power (mW) Original Image Data 119.6263.12 Partial Padding Image Data (Zoom magnification x2.0) 75.65166.42

[0095] According to one embodiment, when the zoom ratio changes from a first zoom ratio to another zoom ratio, the camera (410) can change the remaining area to be set as padding data corresponding to the changed zoom ratio starting from an image frame after a predetermined number of frames. For example, when the camera (410) receives information indicating the changed zoom ratio from the processor (490) via I2C communication, it can maintain the remaining area to be inserted as before without changing it for M frames from the frame at that point in time, and change the remaining area to be inserted as padding data corresponding to the changed zoom ratio starting from the image frame after M frames.

[0096] According to one embodiment, in the case of a zoom-in scenario in which the zoom magnification increases, considering zoom continuity and operational stability, the camera (410) maintains the existing output for a predetermined number of frames (e.g., M) after the zoom change trigger, and after frames corresponding to the zoom magnification before the change continue, it can transmit image data corresponding to the changed zoom magnification to the processor (490).

[0097] According to one embodiment, the camera (410) may transmit first partial padding image data corresponding to the first zoom magnification to the processor (490) for at least one image frame of a predetermined number in response to a change in the zoom magnification from a first zoom magnification to a second zoom magnification higher than the first zoom magnification. For image frames after the predetermined number, the camera (410) may determine an actual field of view area (or third area) to be cropped corresponding to the second zoom magnification changed from the original image data, generate second partial padding image data (or second image data) in which pixel data of the remaining area (or fourth area) excluding the actual field of view area is set as padding data, and transmit it to the processor (490).

[0098] The image data transmitted from the camera (410) to the processor (490) for each image frame in the zoom-in scenario will be explained in more detail through FIG. 10.

[0099] According to one embodiment, in a zoom-out scenario where the zoom magnification decreases, considering zoom continuity and operational stability, the camera (410) can perform a seamless mode transition by outputting original image data corresponding to the optical zoom magnification of the camera (410) for several frames, since the camera (410) does not know how far the zoom-out will proceed. After a specific zoom magnification has continued for M frames, the camera (410) can output partial padded image data again, in which the area excluding the actual required angle of view of the image corresponding to the changed zoom magnification is filled with padding values.

[0100] According to one embodiment, the camera (410) can transmit original image data to the processor (490) for at least one image frame of a predetermined number in response to a change in zoom magnification from a first zoom magnification to a third zoom magnification lower than the first zoom magnification. For image frames after the predetermined number, the camera (410) can determine an actual field of view area (or a fifth area) to be cropped in the original image data corresponding to the third zoom magnification, and generate third partial padded image data (or third image data) by setting the pixel data of the remaining area (or sixth area) excluding the actual field of view area as padding data, and transmit it to the processor (490).

[0101] The image data transmitted from the camera (410) to the processor (490) for each image frame in the zoom-out scenario will be explained in more detail through FIG. 11.

[0102] According to one embodiment, the electronic device (400) may provide a subject tracking zoom function. The subject tracking zoom function may be referred to as anyplace zoom or de-centered remosaic crop zoom. In this document, the subject tracking zoom function may be a technology that magnifies and outputs a desired area of ​​interest within an image through the dual output of a multi-camera (410) or a single camera (410). When the electronic device (400) performs the subject tracking zoom operation, the position and / or size of the subject within the image may change according to the movement of the subject, and the position and / or size of the cropped area displayed according to zoom-in / zoom-out may vary depending on the subject. The subject tracking zoom function will be explained in more detail through FIG. 12.

[0103] According to one embodiment, during a subject tracking zoom operation, the camera (410) identifies a region of interest containing the subject in the original image data and sets pixel data of a non-interest region, which is the remaining region excluding the region of interest, as padding data, and transmits partial padding image data (or fourth image data) to the processor (490). The image data transmitted from the camera (410) to the processor (490) during the subject tracking zoom operation will be explained in more detail through FIG. 13.

[0104] Instructions for performing the operation of the electronic device (400) (or processor (490)) described above may be stored in a computer-readable recording medium. The recording medium may be tangible and non-transitory. The recording medium may store one or more computer programs containing the instructions.

[0105] FIG. 5 illustrates a method for switching seamless modes of a camera according to one embodiment.

[0106] According to one embodiment, a camera (e.g., camera (410) of FIG. 4) of an electronic device (e.g., electronic device (400) of FIG. 4) may support a seamless change mode (or fast change mode (FCM)). A seamless change mode may be a technology that enables a smooth transition without delay in image output when switching the camera's shooting mode. For example, the camera may provide various shooting modes such as a dual conversion gain (DCG) mode, a remosaic output mode, and a low noise (LN) mode, and image data may be acquired by selecting an appropriate shooting mode according to ambient light or zoom scenarios. Examples of switching the camera's shooting mode are not limited to the examples described above, and various parameters applied to the camera may be changed continuously or discontinuously for each frame.

[0107] Figure 5(a) illustrates a mode switching method in normal mode.

[0108] According to one embodiment, the general mode is contrasted with the seamless switching mode and may be a method of performing stream off in parallel when changing the shooting mode. Here, stream off may be a method of temporarily suspending the transmission of image data to the processor when switching modes, and then retransmitting image data according to the changed mode.

[0109] Referring to FIG. 5(a), the camera can sequentially transmit image frames acquired through the image sensor to the processor using camera parameters in mode A (510). When the shooting mode is switched to mode B, the camera can stop transmitting image frames (or stream off) and check whether the acquisition of image frames in mode A has been successfully completed (515). For example, after the image frames in mode A are finished, the camera can check whether the frame counter has reached a set address (e.g., 0xFF).

[0110] According to one embodiment, when the normal termination of Mode A is confirmed, the camera can set the shooting parameters to Mode B and start transmitting (or stream on) image frames again. (520) For example, after the frame counter reaches a set address, the camera can load a set file corresponding to Mode B while in SW standby mode. The camera can apply the set file and sequentially transmit image frames captured in Mode B to the processor. (525)

[0111] In a normal mode that performs stream off in parallel as in Fig. 5(a), compared to the seamless switching mode in Fig. 5(b), the switching speed is slow, so delay may occur from a UX perspective. For example, when switching from DCG preview to full remosaic capture mode, a relatively high delay time (e.g., about 60-70ms) may be required.

[0112] Figure 5(b) illustrates a mode switching method in seamless switching mode.

[0113] Referring to FIG. 5(b), the camera can sequentially transmit image frames acquired through the image sensor to the processor using camera parameters in mode A. (550)

[0114] When the camera switches the shooting mode to mode B, it can perform the seamless setting required for switching to mode B without stopping the transmission of image frames. (555) For example, while the camera is acquiring and transmitting image frames in mode A, it can pre-load setting values ​​using internal memory (e.g., SRAM). In this case, a certain number of frames (e.g., 1 or 2 frames) may be required to switch to mode B. That is, the camera can transmit image frames in mode A for a certain number of frames after switching to mode B and perform the operation to switch to mode B at least partially simultaneously.

[0115] According to one embodiment, once the mode switching setting is complete, the camera can sequentially transmit image frames captured in mode B to the processor starting from a predetermined number of frames (e.g., N+1 frame or N+2 frame). (560)

[0116] In this way, when switching modes seamlessly through a seamless switching mode, image frames captured in the switched mode can be transmitted to the processor without interruption without stream-off when switching modes according to AE (auto exposure), zoom, or camera usage scenarios, thereby providing the user with a camera experience with minimized latency.

[0117] According to one embodiment, in order to operate in a seamless switching mode, the camera may transmit image data of the same output size in the mode before switching and the mode after switching to the processor. If the output size of the image data differs before and after the mode switching, for example, if the number of pixels in the image frames differs, a delay may occur during the image processing and output process of the processor, or some frames may be dropped.

[0118] According to one embodiment, when the zoom magnification is changed, the camera can transmit image data of the same output size to the processor before and after the change in zoom magnification. For example, in a remosaic crop zoom scenario, when the zoom magnification is changed from x1.9x to x2.0x, the output size (or number of pixels) can be maintained by transmitting image data processed by 12M remosaic crop from image data processed by 12M binning.

[0119] According to one embodiment, in order to minimize current consumption while maintaining the same output size of image data, the camera may insert padding data into the pixel data of the pixels in the area to be removed by cropping at the corresponding zoom magnification and transmit it to the processor. For example, the camera may determine the actual field of view area (or first area) to be cropped corresponding to the first zoom magnification in the original image data, generate partial padding image data (or first image data) in which the pixel data of the remaining area (or second area) excluding the actual field of view area is set as padding data, and transmit the partial padding image data to the processor through an interface. Since the output size of the partial padding image data is the same as that of the original image data, the electronic device can perform a shooting mode switching according to a seamless switching mode even when changing from the original image data to the partial padding image data.

[0120] FIGS. 6 and 7 illustrate an embodiment in which a camera transmits original image data to a processor.

[0121] FIG. 6 illustrates image data acquired, transmitted, or processed by a camera, interface, and processor according to a zoom magnification according to one embodiment.

[0122] According to one embodiment, an electronic device (e.g., the electronic device (400) of FIG. 4) can display and capture an image corresponding to a set zoom ratio by using a digital crop zoom function. According to one embodiment, the electronic device (400) includes a plurality of cameras having different viewing angles (e.g., the first camera (312), the second camera (314), and the third camera (316) of FIG. 3), and can acquire original image data using one of the cameras corresponding to the set zoom ratio, and can crop and enlarge an area corresponding to the zoom ratio in the original image data and display it through a display. For example, when the zoom ratio is set to x1.5, the electronic device (400) can take an image corresponding to a zoom ratio x1.0 using one of the cameras (e.g., the second camera (314) or the W camera of FIG. 3), and the processor (490) can crop and enlarge an actual viewing angle area corresponding to the zoom ratio x1.5 in the original image data and display it through a display.

[0123] According to one embodiment, the camera (410) can acquire original image data including image information of the entire field of view of the optical zoom magnification of the camera (410) using an image sensor (610). The original image data may include pixel data corresponding to each of the pixels included in the image sensor. For example, the original image data may include pixel data corresponding to each of the N * M pixels of the image sensor, which consists of N rows and M columns. The original image data may be image data generated by undergoing processing such as binning and remosaic from an image acquired through the image sensor.

[0124] According to one embodiment, the original image data may further include phase detection autofocus (PDAF) data obtained from at least some pixels.

[0125] According to one embodiment, the camera (410) can transmit original image data to the processor (490) through the interface (440). (620) The camera (410) can transmit original image data acquired from the image sensor of the camera (410) to the processor (490) even when the zoom magnification is reduced within the digital zoom range (e.g., x1.0 or more and less than x3.0 in the case of a W camera).

[0126] According to one embodiment, the processor (490) can crop the actual field of view area corresponding to the zoom magnification in the original image data (630). For example, the original image data may be of a size corresponding to the optical zoom magnification (e.g., x1.0) of the camera (410), and the processor (490) may recognize the pixels in the area corresponding to the zoom magnification set according to user input in the original image data as the actual field of view area.

[0127] According to one embodiment, the processor (490) can enlarge the actual field of view area and display it on the display (640). For example, the processor (490) can enlarge the actual field of view area to correspond to the size of the preview area displayed on the display.

[0128] FIG. 7 illustrates the format of image data according to one embodiment.

[0129] According to one embodiment, a camera (e.g., camera (410) of FIG. 4) can transmit image data acquired using an image sensor (e.g., image sensor (420) of FIG. 4) to a processor through an interface (e.g., interface (440) of FIG. 4). The interface between the camera and the processor may be a mobile industry processor interface (MIPI), but is not limited thereto.

[0130] According to one embodiment, the camera may configure one image frame constituting image data (700) into at least one packet and transmit it to the processor. FIG. 7 illustrates the structure of one image frame or packet transmitted from the camera to the processor in the embodiment of FIG. 6.

[0131] According to one embodiment, the packet header (710) may include data related to the packet, such as a packet number, frame number, data length, and / or synchronization information. The packet footer (715) may include data such as an end of packet (EOP) and an error detection code.

[0132] According to one embodiment, FS (720) (frame start) indicates the start of a frame and may be included in the first packet of the frame. FS (720) may include metadata such as a frame number and a timestamp. FE (725) (frame end) indicates the end of a frame and may be included in the last packet of the frame. The processor can recognize the start and end points of the frame through FS (720) and FE (725).

[0133] According to one embodiment, the embedded data lines (730) (EMB) may include data related to the settings of the image sensor. For example, image sensor setting information such as the field of view, exposure time, gain, and / or white balance may be included in the embedded data lines (730). The embedded data lines (730) may also include information about the current frame and / or information about other frames received thereafter.

[0134] According to one embodiment, the active data (740, 742, 744) may include actual image data obtained through the pixels of an image sensor. For example, the camera may process pixel data obtained from the pixels of the image sensor according to a shooting mode (e.g., DCG, RCP, LN) and include it in the active data (740, 742, 744).

[0135] According to one embodiment, PDAF data (750, 752, 754) may include phase detection autofocus (PDAF) data obtained using some pixels of an image sensor. PDAF data (750, 752, 754) includes phase difference information collected by specific pixels, and a processor can set the focus of the camera based on the PDAF data (750, 752, 754).

[0136] According to one embodiment, at least one line of image data may be allocated for recording PDAF data (750, 752, 754), and the remaining area after all PDAF data is recorded in the allocated line may be filled with PDAF dummy (760, 762, 764). The PDAF dummy (760, 762, 764) may be recorded as a data pattern that can be distinguished from the PDAF data (750, 752, 754).

[0137] According to one embodiment, the active data (740, 742, 744) of the image data (700) may be divided into a plurality of parts, each containing a plurality of lines, and PDAF data (750, 752, 754) may be included in the next line of each part. Referring to FIG. 7, a first part (740) of the active data may be included in a predetermined number of lines following the embedded data lines (730), and a first part (750) of the PDAF data may be included in the next line of the first part (740) of the active data. Additionally, a second part (742) of the active data, a second part (752) of the PDAF data, a third part (744) of the active data, and a third part (754) of the PDAF data may be recorded sequentially. The number of parts of the active data is not limited thereto.

[0138] FIGS. 8 and 9 illustrate an embodiment in which a camera transmits partial padding image data, in which padding data is inserted into the remaining area according to the zoom magnification, to a processor.

[0139] FIG. 8 illustrates image data acquired, transmitted, or processed by a camera, interface, and processor according to a zoom magnification according to one embodiment.

[0140] According to one embodiment, a camera (410) (e.g., the camera (410) of FIG. 4) can determine an actual field of view area (or first area) to be cropped in the original image data corresponding to the currently set zoom magnification, generate partial padding image data in which pixel data of the remaining area (or second area) excluding the actual field of view area is set as padding data, and transmit it to a processor (490) (e.g., the processor (490) of FIG. 4). For example, the camera (410) can output a camera (410) scenario in which padding data is seamlessly inserted into the remaining area excluding the actual field of view area corresponding to the actual field of view of the image required for each image frame.

[0141] According to one embodiment, the camera (410) can obtain data of the actual field of view area from the original image data from the image sensor. (810) The original image data may include pixel data corresponding to each of the pixels included in the image sensor. The original image data may correspond to the optical zoom magnification (e.g., x1.0) of the camera (410). The original image data may include pixel data corresponding to each of the N * M pixels of the image sensor, which consists of N rows and M columns.

[0142] According to one embodiment, the camera (410) can identify the actual field of view area corresponding to the current zoom magnification in the original image data. The camera (410) can receive data of the currently set zoom magnification in real time from the processor (490) via I2C (inter-integrated circuit) communication.

[0143] According to one embodiment, when a zoom magnification is set according to user input on a display, the electronic device may provide a digital crop zoom function that crops a portion of the center area corresponding to the zoom magnification in the original image data and displays it enlarged. For example, the original image data obtained through the camera (410) may correspond to x1.0, which is the optical zoom magnification of the camera (410), and when the zoom magnification set by the user is x1.5, an area corresponding to 1 / 1.5 of the diagonal of the image within the center area of ​​the original image data may be designated as the actual field of view area.

[0144] According to one embodiment, the camera (410) can identify the remaining area excluding the actual field of view area corresponding to the zoom magnification in the original image data. For example, when the zoom magnification is x1.5, pixels in the area outside the area corresponding to 1 / 1.5 of the diagonal of the image in the center area of ​​the original image data, which is the actual field of view area, may be designated as the remaining area.

[0145] According to one embodiment, the camera (410) can generate partial padding image data by inserting pixel data of pixels in the remaining area as padding data (820). Here, padding data is data used to fit an image frame to a predetermined size and can be distinguished from the pixel data of the actual image.

[0146] According to one embodiment, padding data may have a value different from the pixel data of the pixels of the original image data. For example, the camera (410) may add a pedestal value (or offset value) to the pixel data of each pixel to express the dark level of the pixel data obtained from each pixel of the image sensor in detail. The camera (410) may set the padding data to be applied to the pixels of the remaining area to be less than or equal to the pedestal value so as to be distinguished from the pixel data of the pixels of the actual field of view area. Accordingly, the pixel data of the pixels of the actual field of view area and the padding data of the pixels of the remaining area may have different ranges of values.

[0147] According to one embodiment, in partial padding image data, the pixels in the actual field of view area have pixel data of the pixels of the original image data, and the pixels in the remaining area may have padding data.

[0148] According to one embodiment, the camera (410) can transmit partial padding image data to the processor (490) via an interface (440) (e.g., MIPI). (830)

[0149] According to one embodiment, the processor (490) can read pixel data of pixels in the actual field of view area corresponding to the zoom magnification in the received partial padding image data. (840)

[0150] According to one embodiment, the camera (410) can transmit data to the processor (490) indicating the area of ​​pixels into which padding data is inserted. For example, the camera (410) can transmit information to the processor (490) indicating the area of ​​pixels into which padding data is inserted in an image frame to be transmitted next (or two image frames later) through a next frame information (NFI) function using metadata (e.g., embedded data lines (EMB)) of a specific image frame.

[0151] According to another embodiment, the processor (490) may read only the area corresponding to the zoom scale in the received partial padding image data and not read the pixel data of the remaining area.

[0152] According to one embodiment, the processor (490) can enlarge the actual field of view area and display it on the display (850). For example, the processor (490) can enlarge the actual field of view area to correspond to the size of the preview area displayed on the display.

[0153] FIG. 9 illustrates the format of image data according to one embodiment.

[0154] FIG. 9 illustrates the format of partial padding image data (900), wherein pixels in the actual field of view area corresponding to the zoom magnification include pixel data of the original image data, and pixels in the remaining area excluding the actual field of view include padding data. In FIG. 9, the description of parts identical to the format of the image data (700) of FIG. 7 may be omitted.

[0155] According to one embodiment, the packet header (910), packet footer (915), FS (920) (frame start), and FE (925) (frame end) may each be identical to the packet header (710), packet footer (715), FS (720) (frame start), and FE (725) (frame end) of FIG. 7.

[0156] According to one embodiment, in partial padding image data (900), the pixels of the actual field of view area may have the pixels of the original image data, and the pixels of the remaining area may have padding data. For example, the original image data obtained through a camera may correspond to the optical zoom magnification of the camera, x1.0, and when the zoom magnification set by the user is x1.5, the area corresponding to 1 / 1.5 of the diagonal of the image within the center area of ​​the original image data may be designated as the actual field of view area, and the area outside the actual field of view area may be designated as the remaining area.

[0157] Referring to FIG. 9, among the three parts constituting the active data in the original image data, the first part (970) and the third part (976) may be included in the remaining area that does not belong to the actual field of view area according to the currently set zoom magnification. Accordingly, the first part (970) and the third part (976) of the active data may be filled with active dummy data (or padding data). For example, the camera may set the dummy data of the pixels in the remaining area to a value less than or equal to the pedestal value applied to the pixel data. Additionally, PDAF data lines (960, 962) corresponding to the first part (970) and the third part (976) of the active data may also be filled with PDAF dummy data.

[0158] According to one embodiment, the camera may record pixels corresponding to the actual field of view area in the second part (972, 940, 974) of the active data as active data (940) based on the pixel data of the original image data, and pixels corresponding to the remaining area as active dummy (972, 974). The camera may record PDAF data (950) measured in the actual field of view area in the next at least one line of the second part (972, 940, 974) of the active data, and fill the remaining part with PDAF dummy (964).

[0159] According to one embodiment, the embedded data line (930) of the partial padding image data (900) may include information related to the area (or address) (e.g., 940) where active data is recorded in the corresponding frame (or the next frame). The output position of the embedded data line (930) may be located at the beginning of the active data, and the output status may be determined by turning the value of a specific address (e.g., 0x0118) on / off.

[0160] A processor (e.g., the processor (490) of FIG. 4) that receives partial padding image data (900) can identify the areas (970, 972, 974, 976) where padding data is recorded in the corresponding frame in the embedded data line (930), and as a result of the identification, the data may not be processed for image display. Alternatively, the camera can transmit information related to the areas (970, 972, 974, 976) of pixels where padding data is inserted in the image frame to be transmitted next (or after 2 image frames) through the NFI (next frame information) function using the embedded data line (930) to the processor.

[0161] FIG. 10 illustrates a method in which a camera generates partial padding image data according to each frame in a zoom-in scenario, according to one embodiment.

[0162] According to one embodiment, when the current zoom magnification is a first zoom magnification, the camera (e.g., camera (410) of FIG. 4) determines an actual field of view area (or first area) to be cropped corresponding to the first zoom magnification from the original image data obtained through the image sensor (e.g., image sensor (420) of FIG. 4), and generates a first partial padded image data (or first image data) in which pixel data of the remaining area (or second area) excluding the actual field of view area is set as padding data, and transmits it to a processor (e.g., processor (490) of FIG. 4).

[0163] According to one embodiment, when the zoom magnification is changed, the camera can change the area to be set as padding data corresponding to the changed zoom magnification starting from a predetermined number of image frames.

[0164] According to one embodiment, in response to a change in zoom magnification from a first zoom magnification to a second zoom magnification higher than the first zoom magnification, the camera may transmit first partial padding image data corresponding to the first zoom magnification to a processor for at least one image frame of a predetermined number. For image frames after the predetermined number, the camera may determine an actual field of view area (or third area) to be cropped corresponding to the second zoom magnification changed from the original image data, and generate and transmit second partial padding image data (or second image data) in which pixel data of the remaining area (or fourth area) excluding the actual field of view area is set as padding data.

[0165] Figure 10 illustrates image data transmitted from the camera to the processor during the process of zooming in from x1.5 to x1.8.

[0166] According to one embodiment, when the camera is set to a zoom magnification of x1.5, it can identify an actual field of view area corresponding to the field of view of the zoom magnification of x1.5 in the original image data. For example, the original image data corresponds to a zoom magnification of x1.0, and the actual field of view area of ​​the zoom magnification of x1.5 may be an area corresponding to 1 / 1.5 of the diagonal of the image within the center area of ​​the original image data.

[0167] According to one embodiment, the camera can identify a remaining area, which is the area excluding the actual field of view area. For example, at a zoom ratio of x1.5, pixels in the area outside the area corresponding to 1 / 1.5 of the image, which is the actual field of view area, may be included in the remaining area.

[0168] According to one embodiment, the camera can generate first partial padding image data (1010) in which the pixel data of the pixels in the remaining area is set as padding data. For example, the camera can generate first partial padding image data (1010) in which the pixels in the actual field of view area corresponding to the field of view of the first zoom magnification have pixel data of the pixels of the original image data, and the pixels in the remaining area have padding data. When the zoom magnification is maintained at the first zoom magnification, the camera can generate the first partial padding image data (1010) from sequentially acquired image frames and transmit them to a processor via an interface (e.g., MIPI).

[0169] According to one embodiment, the processor can confirm that the zoom magnification is zoomed in from x1.5 to x1.8 based on user input. The processor can transmit information about the changed zoom magnification to the camera via I2C communication.

[0170] According to one embodiment, the camera may maintain the remaining area corresponding to the zoom magnification prior to the change without changing the remaining area for inserting padding data up to a predetermined number of at least one image frame from the image frame at the time when information of the changed zoom magnification is received. For example, when it is confirmed that the zoom magnification changes from x1.5 to x1.8, the camera may generate and transmit to a processor a third part padding image data (1020), in which M image frames from that time include pixel data of the original image data in the actual field of view area corresponding to the zoom magnification x1.5 and padding data in the remaining area. The active area containing image information in the third part padding image data (1020) may be the same as the first part padding image data (1010).

[0171] According to one embodiment, when a third-part padding image data (1020) is received from a camera, the processor may crop and enlarge an area corresponding to a currently set zoom ratio of x1.8 and display it. For example, after the zoom ratio is changed from x1.5 to x1.8, the third-part padding image data (1020) corresponding to x1.5 may be received for M image frames, and the processor may crop and enlarge only the area corresponding to x1.8 from this and display it through a display.

[0172] According to one embodiment, for image frames after a predetermined number from the image frame at the time when information on the changed zoom magnification is received, the camera may specify an actual field of view area and a remaining area based on the changed zoom magnification, and insert padding data into the remaining area to generate second partial padding image data (1030). For example, the camera may generate second partial padding image data (1030) by including pixel data of the original image data in the actual field of view area corresponding to the zoom magnification x1.5 and including padding data in the remaining area, starting from image frames after M from the time when it is confirmed that the zoom magnification has changed to x1.8, and transmit it to a processor.

[0173] In a zoom-in scenario such as Fig. 10, considering zoom continuity and operational stability, the camera may maintain the existing output (e.g., third-part padded image data (1020)) for a predetermined number of frames (e.g., M) after the zoom change trigger, and then transmit image data corresponding to the changed zoom ratio (e.g., second-part padded image data (1030)) to the processor after frames corresponding to the zoom ratio before the change have continued.

[0174] FIG. 11 illustrates a method in which a camera generates partial padding image data according to each frame in a zoom-out scenario, according to one embodiment.

[0175] According to one embodiment, a camera (e.g., camera (410) of FIG. 4) may transmit original image data to a processor (e.g., processor (490) of FIG. 4) for at least one image frame of a predetermined number in response to a change in zoom magnification from a first zoom magnification to a third zoom magnification lower than the first zoom magnification. For image frames after the predetermined number, the camera may determine an actual field of view area (or a fifth area) to be cropped in the original image data corresponding to the third zoom magnification, and generate and transmit to the processor a third partial padded image data (or third image data) in which pixel data of the remaining area (or a sixth area) excluding the actual field of view area is set as padding data.

[0176] Figure 11 illustrates image data transmitted from the camera to the processor during the process of zooming out from x1.5 to x1.2.

[0177] According to one embodiment, when the zoom magnification is set to x1.5, a first partial padding image data (1110) is generated in which pixels in the actual viewing angle area corresponding to x1.5 contain original image data and pixels in the remaining area contain padding data, and can be transmitted to a processor through an interface.

[0178] According to one embodiment, the processor can confirm that the zoom magnification is zoomed out from x1.5 to x1.2 based on user input. The processor can transmit information about the changed zoom magnification to the camera via I2C communication.

[0179] According to one embodiment, the camera may transmit original image data (1120) without padding data inserted to a processor from the image frame at the time when information on the changed zoom magnification is received up to a predetermined number of at least one image frame. The original image data (1120) may be raw data containing an image of the entire field of view of the camera's optical zoom magnification. For example, if the camera detects that the zoom magnification is zoomed out to a lower zoom magnification than x1.5, it may transmit original image data (1120) corresponding to x1.0, which is the camera's optical zoom magnification, to the processor, since it is not possible to know in advance how far the zoom-out will proceed from that point onward. The camera may transmit original image data to the processor until the point when the zoom magnification is fixed at a specific zoom magnification.

[0180] According to one embodiment, the camera can confirm that the zoom magnification is fixed at x1.2 for M image frames after the change in zoom magnification. The camera can continuously transmit original image data (1120) to the processor for M image frames.

[0181] According to one embodiment, the camera can generate and transmit to the processor third-part padding image data (1130) corresponding to the zoom magnification x1.2 for M subsequent image frames from the point in time when the zoom magnification is fixed at x1.2. For example, the camera can generate and transmit to the processor third-part padding image data (1130) by including pixel data of the original image data in the actual field of view area corresponding to the zoom magnification x1.2 and padding data in the remaining area, starting from M subsequent image frames from the point in time when it is confirmed that the zoom magnification is fixed at x1.2.

[0182] In a zoom-out scenario such as Fig. 11, considering zoom continuity and operational stability, the camera may transition to a seamless mode by outputting original image data of x1.0 for several frames, as it does not know how far the zoom-out will proceed. After a specific zoom magnification has persisted for M frames, the camera may output partial padding image data (e.g., third partial padding image data (1130)) in which the area excluding the actual required image angle of view corresponding to the changed zoom magnification is filled with padding values.

[0183] FIG. 12 illustrates an image provided according to the subject tracking zoom function of an electronic device according to one embodiment.

[0184] According to one embodiment, an electronic device (e.g., the electronic device (400) of FIG. 4) may provide a subject tracking zoom function. In this document, the subject tracking zoom function may be a technology that magnifies and outputs a desired area of ​​interest within an image through the dual output of a multi-camera or single camera. The subject tracking zoom function may be referred to as anyplace zoom or de-centered remosaic crop zoom. For example, a camera can utilize remosaic crop zoom to acquire the movement of a subject in real-time as a high-resolution image without changing the shooting angle of view. In the case of a general digital zoom function, it may be configured to magnify only the center part of the original image, and if a user wishes to magnify an area of ​​a specific subject, they may need to change the entire angle of view to that area of ​​the subject and then adjust the zoom magnification. In contrast, the subject tracking zoom function can display and output an image magnified from a part of the entire preview area as if it were a full-angle preview image, and can output a high-resolution image when a part of the area is magnified.

[0185] Referring to FIG. 12 (a), the electronic device can track the position of a subject by acquiring a wide-angle image (1210). For example, the electronic device can track in real time a region of interest (1220) where a specific subject is located in an image (1210) acquired from a first camera having a wide angle (e.g., the first camera (312) of FIG. 3 or a UW camera).

[0186] Referring to FIG. 12(b), the electronic device can acquire an image (1230) containing a subject using a second camera having a narrower field of view than the first camera (e.g., the second camera (314) of FIG. 3 or a W camera). For example, the camera can enlarge the screen while maintaining detail without degradation of image quality by zooming the area of ​​interest containing the subject using remosiac.

[0187] According to one embodiment, the electronic device may use a single camera to provide the image (1210) of FIG. 12 (a) and the image (1230) of FIG. 12 (b).

[0188] According to one embodiment, when the electronic device performs a subject tracking zoom operation, the position and / or size of the region of interest (1220) containing the subject may change according to the movement of the subject.

[0189] FIG. 13 illustrates a method for generating partial padding image data when an electronic device according to one embodiment provides a subject tracking zoom function.

[0190] As shown in FIG. 12, when an electronic device performs a subject tracking zoom operation, the position and / or size of the subject within the image may change according to the movement of the subject. According to one embodiment, there may be a constraint that the processor must receive fixed-size image data from the camera in order to operate in a seamless change mode. Accordingly, considering the transition delay time such as streaming on / off, only an area with a fixed zoom magnification can be output during the subject tracking zoom operation, and additional zoom-in / zoom-out on the subject tracking zoom screen may not be provided to the user.

[0191] FIG. 13(a) illustrates an embodiment in which a camera transmits an image of a fixed angle of view to a processor when the region of interest changes.

[0192] According to one embodiment, the camera may determine the region of interest as the tracking result of the subject as ROI 1 (1315). In this case, the camera may crop the region corresponding to ROI 1 (1315) excluding the remaining region (1317) from the first image data (1310) obtained through the image sensor and transmit it to the processor.

[0193] According to one embodiment, the camera can detect that the region of interest has changed from ROI 1 (1315) to ROI 2 (1325) as the subject moves, such as in the second image data (1320). In this case, to enable seamless mode switching, the camera can crop the image data of ROI 2 (1325), which is the same size as ROI 1 (1315) excluding the remaining area (1327) from the second image data (1320), and transmit it to the processor.

[0194] Unlike in FIG. 13 (a), if a mode switch accompanied by steam on / off is performed without a seamless mode switch, there may be a problem with a large parallax (e.g., about 30ms) during the mode switch. Additionally, as in FIG. 13 (a), if the camera transmits image data of regions of interest (1315, 1325) of the same size to the processor and then the processor upscales or downscales the image of ROI 2 for use, image quality degradation occurs, so the effect of providing high-quality images to the processor through remosaic processing by the camera may be reduced. As in FIG. 13 (b), the camera can transmit image data of the same size to the processor even when the region of interest changes by inserting padding data into non-regions of interest (1357, 1367).

[0195] FIG. 13(b) illustrates an embodiment in which the camera transmits partial padding image data to the processor when the region of interest changes.

[0196] According to one embodiment, the camera can identify a region of interest containing a subject in the original image data and transmit partial padding image data (or fourth image data) to the processor, in which pixel data of a non-interest region, which is the remaining region excluding the region of interest, is set as padding data.

[0197] Referring to FIG. 13(b), the camera can determine the region of interest as ROI 1 (1355) based on the tracking results of the subject. In this case, the camera can take pixels of the region corresponding to ROI 1 (1355) from the first image data (1350) obtained through the image sensor, and fill the pixels of the non-interest region (1357), excluding ROI 1 (1355), with padding data. For example, the camera can insert padding data of less than or equal to the pedestal value for the pixels of the non-interest region (1357) so as to distinguish them from the pixel data of the pixels of the region of interest ROI 1 (1355).

[0198] According to one embodiment, the camera can detect that the region of interest has changed from ROI 1 (1355) to ROI 2 (1365) as the subject moves, such as in the second image data (1360). In this case, the camera can take pixel data of the pixels in the region corresponding to ROI 2 (1365) from the second image data (1360) and fill the pixels of the non-interest region (1367), excluding ROI 2 (1365), with padding data. In this way, when padding data is filled into the non-interest region (1357, 1367), image data of the same size can be transmitted to the processor even if the sizes of ROI 1 (1355) and ROI 2 (1365) are different. The processor can read the image of the region of interest (1355, 1365) from the received image data (1350, 1360), and since image data of the same size filled with padding data is input continuously, seamless mode switching may be possible.

[0199] By operating as shown in Fig. 13(b), the electronic device overcomes the limitation that a fixed zoom magnification must be used due to processor constraints even in shooting modes where the area of ​​interest changes, such as subject tracking zoom, thereby providing the user with the ability to additionally zoom in / out on the area of ​​interest. Additionally, the processor can reduce read-out and processing operations for unnecessary pixel lines where padding data is inserted, thereby reducing current consumption.

[0200] FIG. 14 is a flowchart of a method for outputting image data of a camera of an electronic device according to one embodiment.

[0201] According to one embodiment, the illustrated method may be performed by an electronic device (e.g., the electronic device (400) of FIG. 4), and the technical features described above may be omitted from the description below.

[0202] According to one embodiment, in operation 1410, the camera of the electronic device may generate original image data. The original image data may include image information of the entire field of view of the optical zoom magnification of the camera. For example, the camera may generate original image data by performing processing such as binning and remosaic on an image acquired through an image sensor. The format of the original image data has been described through FIG. 7.

[0203] According to one embodiment, in operation 1420, the camera can check the currently set first zoom magnification. For example, when a camera application is executed, the electronic device can change the zoom magnification based on user input, such as pinch and zoom actions, or touch and drag actions on a slider. The camera can receive information indicating the zoom magnification from the processor in real time via I2C communication.

[0204] According to one embodiment, in operation 1430, the camera can determine a first area (or actual field of view area) to be cropped in the original image data corresponding to a first zoom magnification. The size of the first area may correspond to the currently set zoom magnification. When the zoom magnification increases with zoom-in, the actual field of view area in the original image data may become narrower, and when the zoom magnification decreases with zoom-out, the actual field of view area may become wider.

[0205] According to one embodiment, in operation 1440, the camera may generate first image data (or partially padded image data) in which the pixel data of the pixels of a second region (or remaining region) is set as padding data. Here, the second region may include the remaining region excluding the first region from the original image data. The first image data (or partially padded image data) may have the same size (or number of pixels) as the original image data. In the first image data (or partially padded image data), the pixels of the first region (or actual field of view region) may have the pixel data of the pixels of the original image data, and the pixels of the second region (or remaining region) may have padding data. According to one embodiment, the camera may set the padding data to a value different from the pixel data of the pixels of the original image data.

[0206] According to one embodiment, in operation 1450, the camera may transmit first image data (or partially padded image data) to the processor. The camera may transmit the first image data to the processor through an interface (e.g., MIPI).

[0207] According to one embodiment, a processor can read pixels of a first region in first image data (or partial padding image data) and display an image based on pixel data of the pixels of the first region. The processor may not read padding data set on pixels of the remaining region (e.g., a second region).

[0208] According to one embodiment, instructions for performing each operation constituting the method may be stored on a tangible and non-transitory computer-readable recording medium.

[0209] FIG. 15 is a flowchart of a method for a camera to output image data when changing the zoom magnification according to one embodiment.

[0210] According to one embodiment, the illustrated method may be performed by an electronic device (e.g., the electronic device (400) of FIG. 4), and the technical features described above may be omitted from the description below.

[0211] According to one embodiment, in operation 1510, a camera of an electronic device (e.g., the camera of FIG. 4) may generate first partial padding image data corresponding to a first zoom magnification and transmit it to a processor. For example, in the first partial padding image data, the pixels of the actual field of view area corresponding to the first zoom magnification may have pixel data of the pixels of the original image data, and the pixels of the remaining area may have padding data.

[0212] According to one embodiment, in operation 1520, the camera can check whether the zoom magnification is changed. For example, the electronic device can change the zoom magnification based on user input, such as pinch and zoom actions, or touch and drag actions on a slider. The camera can receive information indicating the zoom magnification in real time from the processor via I2C communication. In a state where the zoom magnification is maintained (operation 1520 - No), the camera can transmit first partial padding image data corresponding to the first zoom magnification to the processor.

[0213] According to one embodiment, when the zoom magnification is changed (operation 1520 - yes), as in operation 1530, when zoom-in (operation 1530 - yes), operations 1540 and 1550 are performed, and when zoom-out is performed (operation 1530 - no), operations 1560, 1570, and 1580 may be performed.

[0214] According to one embodiment, when zoom-in is confirmed in operation 1530 (operation 1530 - yes), in operation 1540, the camera may generate and transmit first partial padding image data corresponding to the first zoom magnification for a predetermined number (e.g., M frames) of frames from a trigger for zoom-in to a second zoom magnification higher than the first zoom magnification. For example, when the zoom magnification is zoomed in from the first zoom magnification to the second zoom magnification, the camera may generate and transmit to the processor first partial padding image data consisting of an actual field of view area corresponding to the first zoom magnification and a remaining area for M frames from the time the zoom-in trigger is confirmed.

[0215] According to one embodiment, in operation 1550, the camera may generate and transmit to the processor second partial padding image data corresponding to the second zoom magnification ratio starting from a predetermined number (e.g., M frames). For example, starting from frames after M frames, the camera may specify an actual field of view area and a remaining area based on the changed second zoom magnification ratio, insert padding data into the remaining area, and generate and transmit second partial padding image data to the processor.

[0216] The image data transmitted from the camera to the processor for each image frame in the zoom-in scenario has been explained through Fig. 10.

[0217] According to one embodiment, if it is confirmed in operation 1530 that it is zoom-out rather than zoom-in (operation 1530 - No), in operation 1560, the camera may transmit original image data to the processor until the zoom magnification is fixed during zoom-out. For example, if it is confirmed that the camera is zooming out to a lower zoom magnification than the first zoom magnification, since it is not possible to know in advance how far the zoom-out will proceed from that point onward, the camera may transmit original image data corresponding to the optical zoom magnification of the camera to the processor. The camera may transmit original image data to the processor until the zoom magnification is fixed at a specific zoom magnification.

[0218] According to one embodiment, in operation 1570, the camera can transmit original image data for a predetermined number (e.g., M) of frames from the point where it is fixed at a third zoom magnification.

[0219] According to one embodiment, in operation 1580, the camera may generate and transmit to the processor third partial padding image data corresponding to the third zoom magnification from frames after a predetermined number (e.g., M frames). The camera may generate and transmit to the processor third partial padding image data by including pixel data of the original image data in the actual field of view area corresponding to the third zoom magnification and padding data in the remaining area from image frames after M frames from the point in time when it is confirmed that the zoom magnification is fixed at the third zoom magnification.

[0220] The image data transmitted from the camera to the processor for each image frame in a zoom-out scenario has been explained through Fig. 11.

[0221] According to one embodiment, instructions for performing each operation constituting the method may be stored on a tangible and non-transitory computer-readable recording medium.

[0222] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs.

[0223] An electronic device (400) according to various embodiments of the present document may include a display (450), a camera (410) including an image sensor (420), a memory (480), and at least one processor (490) operatively connected to the display (450), the camera (410), and the memory (480).

[0224] According to one embodiment, the camera (410) is configured to generate original image data including pixel data of a plurality of pixels obtained through the image sensor (420), check a currently set first zoom magnification, determine a first area to be cropped in the original image data corresponding to the first zoom magnification, and generate first image data in which pixel data of a second area, which is the remaining area excluding the first area in the original image data, is set as padding data, and is configured to transmit the first image data to the processor (490) through the interface (440).

[0225] According to one embodiment, the processor (490) may be configured to receive the first image data and, based on the pixel data of the pixels of the first region of the first image data, display a preview image on the display.

[0226] According to one embodiment, the camera (410) may be configured to change the area to be set as padding data corresponding to the changed zoom ratio starting from a predetermined number of image frames when the zoom ratio changes from the first zoom ratio to another zoom ratio.

[0227] According to one embodiment, the camera (410) may be configured to transmit the first image data corresponding to the first zoom ratio to the processor (490) for at least one image frame of a predetermined number in response to a change in the zoom ratio from the first zoom ratio to a second zoom ratio higher than the first zoom ratio, and for image frames after the predetermined number, to determine a third area to be cropped in the original image data corresponding to the second zoom ratio, and to generate and transmit to the processor (490) second image data in which pixel data of the pixels of a fourth area, which is the remaining area excluding the third area, is set as padding data.

[0228] According to one embodiment, the camera (410) may be configured to transmit the original image data to the processor (490) for at least one image frame of a predetermined number in response to a change in the zoom ratio from the first zoom ratio to a third zoom ratio lower than the first zoom ratio, and for image frames after the predetermined number, to determine a fifth area to be cropped in the original image data corresponding to the third zoom ratio, and to generate and transmit to the processor (490) third image data in which pixel data of the pixels of a sixth area, which is the remaining area excluding the fifth area, is set as padding data.

[0229] According to one embodiment, the padding data may have a value different from the pixel data of the pixels of the original image data.

[0230] According to one embodiment, the original image data and the first image data may include data corresponding to substantially the same number of pixels.

[0231] According to one embodiment, the processor (490) may be configured not to read padding data set in the pixels of the second region.

[0232] According to one embodiment, the processor (490) may be configured to check the pixels of the second region set as padding data based on the header information of the previous image frame.

[0233] According to one embodiment, the processor (490) may be configured to check the pixels of the second region set as padding data in the first image data based on the first zoom magnification.

[0234] According to one embodiment, the camera (410) may be configured to identify a region of interest containing a subject in the original image data, generate a fourth image data in which pixel data of a non-interest region, which is the remaining region excluding the region of interest, is set as padding data, and transmit it to the processor (490).

[0235] According to one embodiment, the camera (410) may be configured to generate the first image data and transmit it to the processor (490) when operating in a seamless change mode.

[0236] A method performed by an electronic device according to various embodiments of the present document may include: generating original image data including pixel data of a plurality of pixels acquired through an image sensor by a camera of the electronic device; checking a first zoom magnification currently set by the camera; determining a first area to be cropped in the original image data corresponding to the first zoom magnification by the camera; generating first image data by the camera in which pixel data of pixels of a second area, which is the remaining area excluding the first area, is set as padding data by the camera; transmitting the first image data to a processor through an interface by the camera; receiving the first image data by the processor; and displaying a preview image on the display by the processor based on pixel data of pixels of the first area of ​​the first image data.

[0237] According to one embodiment, the method may further include an operation of changing the area to be set as padding data corresponding to the changed zoom ratio starting from a predetermined number of image frames when the zoom ratio is changed from the first zoom ratio to another zoom ratio.

[0238] According to one embodiment, the method may further include the operation of transmitting the first image data corresponding to the first zoom ratio to the processor for at least one image frame of a predetermined number in response to a change in the zoom ratio from the first zoom ratio to a second zoom ratio higher than the first zoom ratio, and for image frames after the predetermined number, determining a third area to be cropped in the original image data corresponding to the second zoom ratio, and generating and transmitting the second image data to the processor in which pixel data of a fourth area, which is the remaining area excluding the third area, is set as padding data.

[0239] According to one embodiment, the method may further include the operation of transmitting the original image data to the processor for at least one image frame of a predetermined number in response to a change in the zoom ratio from the first zoom ratio to a third zoom ratio lower than the first zoom ratio, and the operation of generating and transmitting to the processor, for image frames after the predetermined number, a third image data in which a fifth region to be cropped in the original image data corresponding to the third zoom ratio is determined, and pixel data of pixels of a sixth region, which is the remaining region excluding the fifth region, is set as padding data.

[0240] According to one embodiment, the padding data may have a value different from the pixel data of the pixels of the original image data.

[0241] According to one embodiment, the original image data and the first image data may include data corresponding to substantially the same number of pixels.

[0242] According to one embodiment, the operation of displaying the preview image may include the operation of displaying the preview image based on the pixel data of the pixels of the first area without reading the padding data set in the pixels of the second area set as the padding data.

[0243] According to one embodiment, the method may further include the operation of identifying a region of interest containing a subject in the original image data, generating a fourth image data in which pixel data of a non-interest region, which is the remaining region excluding the region of interest, is set as padding data, and transmitting it to the processor.

[0244] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0245] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0246] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0247] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0248] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0249] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0250] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; A camera including an image sensor; Memory comprising one or more storage media and storing instructions; The device includes at least one processor that is operatively connected to the display, the camera, and the memory, and includes a processing circuit. When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Original image data including pixel data of a plurality of pixels acquired through the image sensor of the camera is generated, and Check the currently set first zoom magnification, and A first area to be cropped in the above original image data corresponding to the first zoom ratio is determined, and In the above original image data, it is configured to generate a first image data in which the pixel data of the pixels of the second region, which is the remaining region excluding the first region, is set as padding data, and The first image data is transmitted from the camera to the processor through an interface, and The processor receives the first image data, and An electronic device that displays a preview image on the display based on pixel data of the pixels of the first region of the first image data.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that changes the area to be set as padding data corresponding to the changed zoom ratio starting from a predetermined number of image frames when the zoom ratio changes from the first zoom ratio to another zoom ratio.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: In response to a change in the zoom magnification from the first zoom magnification to a second zoom magnification higher than the first zoom magnification, for a predetermined number of at least one image frames, the first image data corresponding to the first zoom magnification is transmitted from the camera to the processor, and An electronic device configured to determine a third area to be cropped corresponding to the second zoom ratio in the original image data for image frames after the above-determined number of frames, generate second image data in which pixel data of the remaining area excluding the third area, which is a fourth area, is set as padding data, and transmit it from the camera to the processor.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: In response to the zoom magnification changing from the first zoom magnification to a third zoom magnification lower than the first zoom magnification, for a predetermined number of at least one image frame, the original image data is transmitted from the camera to the processor, and An electronic device that, for image frames after the above-determined number, determines a fifth region to be cropped in the original image data corresponding to the third zoom ratio, generates third image data in which pixel data of the remaining region excluding the fifth region, which is a sixth region, is set as padding data, and transmits it from the camera to the processor.

5. In Paragraph 1, The above padding data is an electronic device having a value different from the pixel data of the pixels of the above original image data.

6. In Paragraph 1, An electronic device comprising data corresponding to substantially the same number of pixels, wherein the original image data and the first image data.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that prevents padding data set in the pixels of the second region from being read.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that identifies pixels of the second region set as padding data based on header information of a previous image frame.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that identifies pixels of the second region set as padding data in the first image data based on the first zoom magnification.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that identifies a region of interest containing a subject in the original image data, generates a fourth image data in which pixel data of a non-interest region (the remaining area excluding the region of interest) is set as padding data, and transmits it from the camera to the processor.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device configured to generate the first image data and transmit it from the camera to the processor when operating in a seamless change mode.

12. In a method performed by an electronic device, The operation of generating original image data including pixel data of a plurality of pixels acquired through the image sensor by the camera of the electronic device; An operation to check the currently set first zoom magnification by the above camera; An operation to determine a first area to be cropped in the original image data corresponding to the first zoom magnification ratio by the camera above; An operation to generate first image data by the above camera, wherein pixel data of pixels in a second region, which is the remaining region excluding the first region from the original image data, is set as padding data; The operation of transmitting the first image data to a processor via an interface by the camera above; The operation of receiving the first image data by the above processor; and A method comprising the operation of displaying a preview image on the display based on pixel data of pixels of the first region of the first image data by the above processor.

13. In Paragraph 12, A method further comprising, when the zoom ratio changes from the first zoom ratio to another zoom ratio, changing the area to be set as padding data corresponding to the changed zoom ratio starting from a predetermined number of image frames.

14. In Paragraph 12, An operation of transmitting the first image data corresponding to the first zoom ratio to the processor for a predetermined number of at least one image frame in response to the change of the zoom ratio from the first zoom ratio to a second zoom ratio higher than the first zoom ratio; and A method further comprising, for image frames after the above-determined number, determining a third area to be cropped in the original image data corresponding to the second zoom ratio, generating second image data in which pixel data of the remaining area excluding the third area, which is a fourth area, is set as padding data, and transmitting to the processor.

15. In a computer-readable non-transient recording medium, The operation of generating original image data including pixel data of a plurality of pixels acquired through the image sensor by the camera of the electronic device; An operation to check the currently set first zoom magnification by the above camera; An operation to determine a first area to be cropped in the original image data corresponding to the first zoom magnification ratio by the camera above; An operation to generate first image data by the above camera, wherein pixel data of pixels in a second region, which is the remaining region excluding the first region from the original image data, is set as padding data; The operation of transmitting the first image data to a processor via an interface by the camera above; The operation of receiving the first image data by the above processor; and A recording medium storing instructions for performing an operation to display a preview image on the display based on pixel data of pixels in the first region of the first image data by the above processor.

Images