Electronic device including camera, operating method thereof, and recording medium

Abstract

An electronic device is provided. The electronic device may comprise a first camera module including a first image sensor. The electronic device may comprise at least one processor including a processing circuit. The electronic device may comprise a memory for storing instructions. The electronic device may identify first coordinates of a region of interest on the first image sensor. The electronic device may acquire a first control signal for acquiring image data including the region of interest. The electronic device may acquire first line data from a first pixel line including at least a portion of the region of interest, on the basis of the first coordinates among pixel lines of the first image sensor, in response to the first control signal. The electronic device may perform first image processing using the first line data. The electronic device may acquire second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor. The electronic device may acquire a first frame by performing second image processing using the first line data and the second line data.

Description

Electronic device including a camera, method of operation thereof and recording medium

[0001] The present disclosure relates to an electronic device including a camera, an operating method thereof, and a recording medium.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. Electronic devices can provide various functions. For example, electronic devices can run at least one application in the foreground and / or background to provide at least one function.

[0003] Electronic devices run on a specified operating system (e.g., Android operating system). TM An electronic device can provide various functions using a camera. For example, an electronic device can support multiple functions provided using a camera.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0005] Aspects of the present disclosure are intended to address at least the problems and / or disadvantages mentioned above and provide at least the advantages described below. Accordingly, one aspect of the present disclosure provides an electronic device including a camera, a method of operating the same, and a recording medium.

[0006] Additional aspects will be set forth in part in the description below, and in part will be apparent from the description or may be learned by practicing the embodiments provided.

[0007] According to one aspect of the present disclosure, an electronic device is provided. The electronic device may include a first camera module including a first image sensor. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory storing instructions. The electronic device may identify first coordinates of a region of interest on the first image sensor. The electronic device may obtain a first control signal for obtaining image data including the region of interest. In response to the first control signal, the electronic device may obtain first line data from a first pixel line including at least a portion of the region of interest based on the first coordinates among pixel lines of the first image sensor. The electronic device may perform first image processing using the first line data. The electronic device may obtain second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor. The electronic device may obtain a first frame by performing second image processing using the first line data and the second line data.

[0008] According to one aspect of the present disclosure, a method performed by an electronic device is provided. The method may include an operation of identifying a first coordinate of a region of interest on a first image sensor of a first camera module. The method may include an operation of obtaining a first control signal for obtaining image data including the region of interest. The method may include an operation of obtaining, in response to the first control signal, first line data from a first pixel line including at least a portion of the region of interest based on the first coordinate among pixel lines of the first image sensor. The method may include an operation of performing first image processing using the first line data. The method may include an operation of obtaining second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor. The method may include an operation of performing second image processing using the first line data and the second line data to obtain a first frame.

[0009] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium having recorded thereon instructions for controlling an electronic device is provided. The recording medium may include instructions for identifying first coordinates of a region of interest on a first image sensor of a first camera module. The recording medium may include instructions for obtaining a first control signal for obtaining image data including the region of interest. The recording medium may include instructions for obtaining, in response to the first control signal, first line data from a first pixel line including at least a portion of the region of interest based on the first coordinates among pixel lines of the first image sensor. The recording medium may include instructions for performing first image processing using the first line data. The recording medium may include instructions for obtaining second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor. The recording medium may include instructions for performing second image processing using the first line data and the second line data to obtain a first frame.

[0010] Other aspects, advantages and important features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which discloses various embodiments of the present disclosure together with the accompanying drawings.

[0011] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0012] FIG. 1 is a block diagram illustrating an electronic device within a network environment according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram illustrating a camera module according to one embodiment of the present disclosure.

[0014] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment of the present disclosure.

[0016] FIG. 5 is a drawing for explaining the configuration of an image sensor that acquires an image using a rolling shutter method according to one embodiment of the present disclosure.

[0017] FIG. 6 is a drawing for explaining an operation of an image sensor that acquires an image using a rolling shutter method according to one embodiment of the present disclosure.

[0018] FIG. 7 is a diagram for explaining an operation of an electronic device controlling a camera module using an image acquired in a rolling shutter manner according to one embodiment of the present disclosure.

[0019] FIG. 8 is a diagram for explaining the configuration of an image sensor that acquires an image using a global shutter method according to one embodiment of the present disclosure.

[0020] FIG. 9 is a diagram illustrating sensor pixels of an image sensor that acquires images using a global shutter method according to one embodiment of the present disclosure.

[0021] FIG. 10 is a diagram for explaining sensor pixels of an image sensor that acquires images using a global shutter method according to one embodiment of the present disclosure.

[0022] FIG. 11 is a diagram for explaining an operation of an image sensor that acquires an image using a global shutter method according to one embodiment of the present disclosure.

[0023] FIG. 12 is a diagram for explaining an operation of an electronic device controlling a camera module using an image acquired using a global shutter method, according to one embodiment of the present disclosure.

[0024] FIG. 13 is a diagram for explaining data output from an image sensor that acquires an image using a global shutter method according to one embodiment of the present disclosure.

[0025] FIG. 14 is a diagram for explaining an artificial intelligence model used by an electronic device according to one embodiment of the present disclosure.

[0026] FIG. 15 is a diagram illustrating an operation of an electronic device using an artificial intelligence model to identify an object in a region of interest according to an embodiment of the present disclosure.

[0027] FIG. 16 is a flowchart of an operation method for an electronic device to acquire an image according to one embodiment of the present disclosure.

[0028] FIG. 17 is a flowchart of an operation method for an electronic device to acquire images using multiple camera modules according to one embodiment of the present disclosure.

[0029] FIG. 18 is a drawing for explaining an operation of an electronic device acquiring an image using a plurality of camera modules according to one embodiment of the present disclosure.

[0030] Similar reference numerals may be used throughout the drawings to describe identical or similar components, features and structures.

[0031] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While this disclosure includes numerous specific details to facilitate this understanding, these should be considered merely illustrative. Accordingly, those skilled in the art will recognize that various modifications and variations can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0032] The terms and words used in the following description and claims are not intended to be limited by their bibliographic meanings, but are merely used by the inventors to facilitate a clear and consistent understanding of the present invention. Accordingly, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure, which is defined by the appended claims and their equivalents.

[0033] The singular forms "a," "an," and "the" should be understood to include the plural unless the context clearly indicates otherwise. Thus, for example, a reference to a "component surface" includes a reference to one or more such surfaces.

[0034] Electronic devices can capture two-dimensional or three-dimensional images of objects through a camera module. The image sensor of the camera module can create an image of the object using a photoelectric conversion element that responds to the intensity of light reflected from the object.

[0035] An electronic device can set a region of interest on an image acquired through a camera module. The electronic device can control the AF operation of the camera module or perform image processing based on the region of interest.

[0036] An image sensor may include a plurality of pixel lines. A pixel line may include a line of image sensors in which sensor pixels are arranged. A sensor pixel may include a component of an image sensor corresponding to a light-receiving element.

[0037] Multiple pixel lines can be sequentially exposed so that pixel values ​​can be sequentially read out. This method can be referred to as a rolling shutter method. An image sensor that generates an image using a rolling shutter method sequentially exposes pixel lines arranged from a first position (e.g., the top or bottom of the image sensor) to a second position (e.g., the bottom or top of the image sensor), and pixel values ​​can be read out.

[0038] Image sensors that generate images using a rolling shutter method can sequentially read pixel values ​​from pixel lines, regardless of the location of the region of interest. Pixel values ​​in the region of interest may be acquired earlier or later, depending on the region's location. Depending on the region of interest's location, the camera module's AF operation control or image processing may be delayed.

[0039] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains from the description of the present disclosure.

[0040] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0041] Each block of the flowchart and the combination of flowcharts can be performed by one or more computer programs containing instructions. One or more computer programs may be stored entirely in a single memory device, or one or more computer programs may be divided into multiple parts and stored in multiple memory devices.

[0042] All functions or operations described in this disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and may be an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), or a wireless fidelity (Wi-Fi) chip. TM ) chip, Bluetooth TM Circuits such as chips, global positioning system (GPS) chips, near field communication (NFC) chips, connection chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio codec chips, universal serial bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), systems on chips (SoCs), ICs, etc.

[0043] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment of the present disclosure. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0044] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0045] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0046] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0047] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0048] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0049] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0050] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0051] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0052] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0053] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0054] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0055] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0056] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0057] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0058] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0059] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0060] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0061] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0062] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0063] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0064] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or server (108)). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0065] FIG. 2 is a block diagram of an electronic device (200) including a camera module (180) according to one embodiment of the present disclosure. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260).

[0066] The lens assembly (210) can collect light emitted from a subject that is the target of image capture. The lens assembly (210) can include one or more lenses.

[0067] In one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In such a case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0068] The flash (220) can emit light that is used to enhance light emitted or reflected from a subject. In one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or a xenon lamp.

[0069] The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled dEvice) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0070] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image.

[0071] According to one embodiment, the image stabilizer (240) can detect movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) placed inside or outside the camera module (180). For example, the image stabilizer (240) can be implemented as an optical image stabilizer.

[0072] The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130), or as a separate memory that operates independently therefrom.

[0073] According to one embodiment, the memory (250) can at least temporarily store images acquired and output as preview images. The preview images can include images provided by the electronic device (200) that enable the user to confirm the location, lighting, or composition of the subject to be photographed so that the user can acquire the desired image.

[0074] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). In one embodiment, the image signal processor (260) may be at least a part of the processor (120). It may be configured as a separate processor that is configured or operates independently from the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0075] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. For example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0076] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0077] Referring to FIG. 3, an electronic device (101) according to one embodiment may include a camera module (380), a memory (330), and a processor (320). The electronic device (101) according to one embodiment may further include a display (360). The display (360) may be replaced with an external display connected to the electronic device (101). The electronic device (101), the processor (320), the memory (330), the camera module (380), and the display (360) may each correspond to the electronic device (101), the processor (120), the memory (130), the camera module (180), and the display module (160) described above with reference to FIGS. 1 and 2 . However, the components of the electronic device (101) illustrated in FIG. 3 are for describing one embodiment, and the electronic device (101) may include more components than the components illustrated in FIG. 3 or may include other components that may replace at least some of the components. For example, the memory (330) is not limited to a storage medium included in the electronic device (101), but may include a cloud storage external to the electronic device (101).

[0078] According to one embodiment, the camera module (380) may include a lens unit (381) including at least one lens for focusing light (e.g., lens assembly (210) of FIG. 2) and an image sensor (383) (e.g., image sensor (230) of FIG. 2) that converts an optical signal passing through the lens unit (381) into a digital signal.

[0079] The image sensor (383) may include a color filter array including a plurality of light-receiving elements, a plurality of micro-lenses, and a plurality of color channels. The plurality of light-receiving elements may include photodiodes arranged to correspond to one micro-lens in an array having M rows and N columns. Here, M and N may each be a natural number greater than or equal to 1.

[0080] The color filter array included in the image sensor (383) may be configured with a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexa-deca pattern). The colors of the element groups of the color filter array configured with a non-Bayer pattern may be matched to correspond to the Bayer pattern.

[0081] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a non-Bayer pattern. For example, the image sensor (383) including a color filter array composed of a non-Bayer pattern can output an image signal composed of a non-Bayer pattern by determining a pixel value using an output value of a light receiving element corresponding to an element. The image signal output from the image sensor (383) can be data in which the color pattern is maintained by not changing the color order of the color pattern of the image sensor (383).

[0082] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) configured in a Bayer pattern. For example, an image sensor (383) including a color filter array configured in a non-Bayer pattern can output an image signal configured in a Bayer pattern by binning output values ​​of light-receiving elements corresponding to groups of elements constituting the color filter array.

[0083] According to one embodiment, the image sensor (383) may operate in a high-resolution mode or a crop mode. The high-resolution mode may include a mode in which each of the output values ​​of the light-receiving elements included in the image sensor (383) is used as the pixel value of the pixel corresponding to each of the light-receiving elements. In the present disclosure, 'pixel' may mean the smallest unit constituting a digital image. The resolution of an image may be expressed by the number of pixels included in the image. For example, if an image is composed of axb pixels arranged in a rows and b columns, the resolution of the image may be indicated as axb.

[0084] For example, an electronic device (101) can obtain high-resolution image data by using an image sensor (383) composed of 50 Mp photodetectors and using each of the output values ​​of the 50 Mp photodetectors as the pixel value of the pixel corresponding to the photodetectors. The high-resolution mode can be understood as a full pixel mode.

[0085] The crop mode may include a mode in which the output values ​​of a predetermined number of light-receiving elements (e.g., a predetermined number of light-receiving elements located at the center of the image sensor) among the light-receiving elements are used as pixel values. The electronic device (101) can obtain an image with a narrowed field of view (FOV) through the crop mode, and can provide the user with an experience similar to that of a zoom-in function. For example, when the electronic device (101) operates in a crop mode in which the output values ​​of 12.5 Mp light-receiving elements located at the center of the image sensor (383) composed of 50 Mp light-receiving elements are used as pixel values, the electronic device (101) can provide the user with an experience similar to that of a 2x zoom-in function.

[0086] According to one embodiment, the image sensor (383) can operate in a low-light mode or a multi-frame synthesis mode. The low-light mode may include a mode in which the pixel values ​​of the first pixels corresponding to the first light-receiving elements are used based on the values ​​output from the first light-receiving elements corresponding to the elements of the color filter array matched with the same color of the light-receiving elements included in the image sensor (383). The multi-frame synthesis mode may include a mode in which the image sensor (383) acquires a plurality of frames in which there is a difference in exposure value.

[0087] According to one embodiment, the memory (330) can store instructions that can be executed by the processor (320). The processor (320) can perform operations or control components of the electronic device (101) by executing the instructions stored in the memory (330).

[0088] In the present disclosure, the operation of the electronic device (101) may be understood as being performed by at least one processor (320) executing instructions. According to one embodiment, the processor (320) may include at least one of an application processor (AP), a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, the at least one processor (320) may include an application processor. The at least one processor (320) may acquire image data based on an image frame including information read out from an image sensor (383).

[0089] According to one embodiment, at least one processor (320) may control the camera module (380) by executing a camera application. For example, the processor (320) may initiate the operation of the camera module (380) through the camera application. For example, the processor (320) may provide the camera module (380) with a request for acquisition of at least one frame (e.g., a frame for preview, a frame for capture) through the camera application. For example, the processor (320) may acquire one or more frames through the camera application. For example, the processor (320) may identify a user input received through the camera application (e.g., a user input for acquisition of a capture image). For example, the processor (320) may acquire image data (e.g., preview image data, draft image data, capture image data, video data) through the camera application. For example, the processor (320) may display the acquired image data using the display (360).

[0090] According to one embodiment, at least one processor (320) can transmit a control signal to the image sensor (383) to perform a read operation. For example, at least one processor (320) can transmit the control signal based on a designated communication method (e.g., I2C (inter-integrated circuit), I3C (improved inter-integrated circuit)). At least one processor (320) can obtain image data output from the image sensor (383). The image data can include an image frame. For example, at least one processor (320) can receive the image data through an interface (e.g., MIPI (mobile industry processor interface)) connected to the image sensor (383).

[0091] According to one embodiment, at least one processor (320) may control the image sensor (383) to acquire an image corresponding to a user input. The processor (320) may control the image sensor (383) to operate based on a selected shooting mode. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of the light-receiving elements constituting the image sensor (383) based on a first shooting mode (e.g., high-resolution shooting mode) so that the output corresponds to each pixel corresponding to each of the light-receiving elements. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of a predetermined number of first light-receiving elements among a plurality of light-receiving elements so that the output corresponds to each pixel corresponding to each of the first light-receiving elements based on a second shooting mode (e.g., crop shooting mode). For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out the output of a first group including a plurality of light-receiving elements so that it corresponds to a first pixel corresponding to the first group based on a third shooting mode (e.g., low-light shooting mode). For example, the image sensor (383) can output a short-exposure image signal and a long-exposure image signal by controlling the exposure time of the image sensor (383) so that a preset difference in exposure exists based on a fourth shooting mode (e.g., HDR shooting mode).

[0092] According to one embodiment, at least one processor (320) can obtain image data by performing an operation on an image signal (e.g., raw image data) output from an image sensor (383). For example, the processor (320) can perform at least one operation (e.g., demosaicing (or Bayer interpolation), an operation for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement) on the image signal.

[0093] For example, the processor (320) may generate a second image signal (e.g., raw image data composed of a non-Bayer pattern) by performing remosaicking on a first image signal (e.g., raw image data composed of a non-Bayer pattern) output from the image sensor (383). For example, the processor (320) may input an image signal output from the image sensor (383) to an image signal processor (e.g., the image signal processor (260) of FIG. 2) to perform at least one operation (e.g., demosaicing, an operation for adjusting white balance, contrast, saturation value, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement). The processor (320) may store the generated image data in the memory (330).

[0094] For example, the processor (320) can identify an object included in an image using an image signal. For example, the processor (320) can identify an object included in an image by performing an object recognition operation on the image signal. For example, the processor (320) can identify a main object and a sub-object among the objects included in the image. For example, the processor (320) can identify a face included in an image by performing face recognition.

[0095] According to one embodiment, at least one processor (320) can initiate operation of the camera module (380). For example, the processor (320) can initiate operation of the camera module (380) by providing a control signal to a hardware abstraction layer (HAL) that includes an ID (identification) of the camera and a command to initiate operation of the camera.

[0096] According to one embodiment, at least one processor (320) can acquire one or more frames. For example, the processor (320) can acquire one or more frames by controlling the camera module (380) through a request to acquire a frame. For example, the processor (320) can acquire a frame for preview composed of a Bayer pattern. For example, the processor (320) can acquire a frame for capture composed of a non-Bayer pattern.

[0097] According to one embodiment, at least one processor (320) can convert a first image signal composed of a first color pattern (e.g., a non-Bayer pattern) into a second image signal composed of a second color pattern (e.g., a Bayer pattern). For example, the processor (320) can convert the first image signal into the second image signal by performing remosaicing on the first image signal. For example, the processor (320) can convert the first image signal into the second image signal by performing binning on the first image signal. For example, the processor (320) can control a computation unit (e.g., a computation unit (417) of FIG. 4) of the image sensor (383) to convert the first image signal into the second image signal.

[0098] In one embodiment, at least one processor (320) can identify a received user input. For example, the processor (320) can identify a user input for changing a shooting mode (e.g., high-resolution mode, crop mode, low-light mode, multi-frame synthesis mode, still image shooting mode, video shooting mode). For example, the processor (320) can identify a user input related to capturing still images and / or videos. For example, the processor (320) can identify a user input for capturing a multi-frame synthesised image.

[0099] According to one embodiment, at least one processor (320) can obtain preview image data. For example, the processor (320) can generate the preview image data by applying an image signal composed of a Bayer pattern output from an image sensor (383) including a color filter of a Bayer pattern to the image signal processor (260). For example, the processor (320) can control an operation unit (e.g., operation unit (417) of FIG. 4) of the image sensor (383) to output an image signal composed of a Bayer pattern by binning an output value of a light-receiving element of the image sensor (383) including a color filter of a non-Bayer pattern (e.g., tetra, nona, hexadeca). For example, a tetra pattern can be a non-Bayer pattern in which a 2x2 array of sensor pixels corresponds to color filters of the same color. For example, a nona pattern can be a non-Bayer pattern in which a 3x3 array of sensor pixels corresponds to color filters of the same color. For example, a hexadecimal pattern could be a non-Bayer pattern in which a 4x4 array of sensor pixels corresponds to color filters of the same color.

[0100] According to one embodiment, at least one processor (320) can obtain a capture image. For example, the processor (320) can generate a capture image by performing at least one operation (e.g., demosaicing, white balance, contrast, operation for adjusting saturation value, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on a first image signal composed of a non-Bayer pattern output from an image sensor (383). For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a high-resolution mode. For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a crop mode.

[0101] According to one embodiment, at least one processor (320) may display acquired image data using a display (360). For example, the processor (320) may display at least one of preview image data and captured image data in at least a portion of a camera application.

[0102] According to one embodiment, the display (360) can display one or more pieces of information under the control of at least one processor (320). For example, the display (360) can display a user interface (UI) of an electronic device. For example, the display (360) can display an execution screen of an application running on the electronic device. For example, the display (360) can display preview image data and captured image data.

[0103] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment of the present disclosure. The image sensor of FIG. 4 may correspond to the image sensor (230, 383) described above with reference to FIGS. 2 and 3.

[0104] Referring to FIG. 4, an image sensor according to one embodiment may include a micro lens array (MLA) (411), a color filter array (CFA) (413), a light receiving unit (415), and a computation unit (417).

[0105] In one embodiment, the micro lens array (411) may be arranged so that a light bundle (421) passing through the lens assembly (e.g., lens assembly (210) of FIG. 2) is focused on a light receiving element of the light receiving unit (415). The light bundle (423) passing through the micro lens array (411) may have at least some wavelengths outside of a band corresponding to a specific color blocked as it passes through the color filter array (413).

[0106] According to one embodiment, the color filter array (413) may be positioned at a position corresponding to a sensor pixel of the image sensor (230). For example, the sensor pixel may correspond to a light-receiving element as a component unit of the image sensor (230). Light bundles (425) passing through the color filter array (413) may be detected by a light-receiving element (e.g., a photodiode) of the light-receiving unit (415).

[0107] In one embodiment, the light-receiving elements of the light-receiving unit (415) may include photodiodes arranged in an array having M rows and N columns to correspond to one micro lens. Here, M and N may each be a natural number greater than or equal to 1. The light-receiving unit (415) may include a light-receiving element (e.g., including a light receiving circuit) that generates a charge when receiving light and converts it into an electrical signal, and a circuit that selectively reads out the charge of the light-receiving element. A circuit for digitizing a signal read out from the light-receiving unit (415) or reducing noise may be further arranged between the light-receiving unit (415) and the calculation unit (417).

[0108] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light bundle (421) is incident, an area in which a single microlens included in the microlens array (411) is arranged may at least partially overlap an area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter array (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.

[0109] In one embodiment, the color pattern of the color filter array (413) is illustrated based on a Bayer pattern, but the color pattern of the color filter array (413) is not limited to that illustrated in FIG. 4. Areas within the color filter array (413) corresponding to a plurality of adjacent micro lenses may also be configured to include the same color channel.

[0110] In one embodiment, the operation unit (417) can perform an operation to process electrical data (or signal) (427) output from the light receiving unit (415). The operation unit (417) can output data obtained based on the operation result. The output of the operation unit (417) can be a sensor output (429) of the image sensor (230).

[0111] FIG. 5 is a diagram illustrating the configuration of an image sensor that acquires images using a rolling shutter method according to one embodiment of the present disclosure. The image sensor of FIG. 5 may correspond to the image sensor (230, 383) described above with reference to FIGS. 2, 3, and 4.

[0112] Referring to FIG. 5, an image sensor according to one embodiment may include an image pixel array (510) and a readout circuit (520). For example, the image pixel array (510) may include a micro lens array (MLA) (411), a color filter array (CFA) (413), and a light receiving unit (415), as described above with reference to FIG. 4.

[0113] In one embodiment, the image processing engine (530) may be positioned externally to the image sensor. For example, the image processing engine (530) may correspond to the processor (320) described above with reference to FIG. 3. For example, at least some of the operations performed by the image processing engine (530) may correspond to at least some of the operations performed by the processor (320) described above with reference to FIG. 3.

[0114] According to one embodiment, the image processing engine (530) may be included in the image sensor. For example, the image processing engine (530) may correspond to the operation unit (417) described above with reference to FIG. 4. For example, at least some of the operations performed by the image processing engine (530) may correspond to at least some of the operations performed by the operation unit (417) described above with reference to FIG. 4.

[0115] According to one embodiment, the image pixel array (510) may include sensor pixels arranged in an array having M rows and N columns (wherein M and N may each be a natural number greater than or equal to 1). For example, the image pixel array (510) may include M pixel lines. Each of the pixel lines may include sensor pixels included in the M rows. Each of the sensor pixels may include a microlens, a color filter, and a light receiving element.

[0116] In one embodiment, the sensor pixels may be exposed. For example, the sensor pixels may be exposed to a light bundle (501) for a set period of time in response to a control signal from a processor (e.g., processor (320) of FIG. 3 ). For example, the sensor pixels may convert light received during the exposure period into an electrical signal. For example, the sensor pixels may determine a pixel value based on the photons received during the exposure period.

[0117] In one embodiment, a processor (e.g., processor (320) of FIG. 3) can control exposure on a pixel line-by-pixel basis. For example, the processor (e.g., processor (320) of FIG. 3) can control exposure start time and exposure time on a pixel line-by-pixel basis. For example, the processor (e.g., processor (320) of FIG. 3) can control pixel lines such that pixel lines arranged from a first position (e.g., top or bottom of the image pixel array (510)) to a second position (e.g., bottom or top of the image pixel array (510)) of the image pixel array (510) are sequentially exposed.

[0118] In one embodiment, the sensor pixels may be read out as pixel values ​​(502). For example, the sensor pixels may be read out as pixel values ​​(502) and provided to a readout circuit (520). For example, the readout circuit (520) may temporarily store the pixel values ​​(502). For example, the pixel values ​​(502) read out from the sensor pixels may be input to an image processing engine (530) via the readout circuit (520).

[0119] According to one embodiment, the sensor pixels may be read out pixel values ​​(502) for each pixel line. For example, a processor (e.g., processor (320) of FIG. 3) may control the pixel lines so that the pixel values ​​(502) are read out for each pixel line. For example, the processor (e.g., processor (320) of FIG. 3) may control the pixel lines so that the pixel values ​​(502) are sequentially read out for pixel lines arranged from a first position (e.g., top or bottom of the image pixel array (510)) to a second position (e.g., bottom or top of the image pixel array (510)). For example, the pixel values ​​(503) sequentially read out from the pixel lines may be sequentially input to the image processing engine (530) through the readout circuit (520).

[0120] According to one embodiment, the image processing engine (530) may perform operations to process pixel values ​​(503). For example, the image processing engine (530) may perform a focus control operation (e.g., an auto focus (AF) operation). For example, the image processing engine (530) may perform an exposure control operation (e.g., an auto exposure (AE) operation). For example, the image processing engine (530) may perform a color control operation (e.g., an auto white balance (AWB) operation). For example, the image processing engine (530) may perform an operation to generate an image.

[0121] According to one embodiment, the image processing engine (530) can perform an auto focus operation based on the input pixel values. For example, the image processing engine (530) can perform an operation to obtain information about the distance between a camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) and an object of interest based on the input pixel values. For example, the image processing engine (530) can perform an operation to obtain information about the distance between a camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) and an object of interest based on a phase difference identified from the pixel values. For example, the image processing engine (530) can generate a control signal that controls a lens unit (e.g., a lens assembly (210) of FIG. 2, a lens unit (381) of FIG. 3) of a camera module (e.g., a camera module (180) of FIG. 2, a camera module (380) of FIG. 3)) to reduce a phase difference identified from pixel values ​​corresponding to a region of interest.

[0122] According to one embodiment, the image processing engine (530) may perform an exposure control operation (e.g., an auto exposure (AE) operation) based on input pixel values. For example, the image processing engine (530) may perform an operation to identify a difference between a set first exposure value and a second exposure value identified from the pixel values. For example, the image processing engine (530) may generate a control signal to change at least one value among an aperture, an exposure time, and a sensitivity of a camera module (e.g., the camera module (180) of FIG. 2 and the camera module (380) of FIG. 3) based on the difference between the first exposure value and the second exposure value.

[0123] According to one embodiment, the image processing engine (530) may perform a color control operation (e.g., an auto white balance (AWB) operation) based on input pixel values. For example, the image processing engine (530) may identify an appropriate color temperature based on color information identified from the pixel values. For example, the image processing engine (530) may perform an operation to adjust the values ​​of channels of defined colors (e.g., red, green, blue) based on the identified appropriate color temperature. For example, the image processing engine (530) may perform operations such as a gray will algorithm, a white patch search algorithm, and a color histogram analysis algorithm.

[0124] According to one embodiment, the image processing engine (530) can generate an image based on input pixel values. For example, the image processing engine (530) can generate an image by performing at least one operation (e.g., demosaicing, operations for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement) on input raw image data.

[0125] FIG. 6 is a diagram for explaining an operation of an image sensor that acquires an image using a rolling shutter method according to one embodiment of the present disclosure. The electronic device (600) of FIG. 6 may correspond to the electronic device (200) of FIG. 2 and the electronic device (101) of FIG. 3. The operation of the electronic device (600) described with reference to FIG. 6 may be performed under the control of a processor (for example, the processor (320) of FIG. 3).

[0126] According to one embodiment, the electronic device (600) can obtain an image (610) using the camera module (601). For example, the electronic device (600) can obtain an image (610) including at least one object.

[0127] According to one embodiment, the electronic device (600) can set an area of ​​interest (611).

[0128] For example, the electronic device (600) can set a region of interest (611) based on a user's input. For example, the electronic device (600) can receive a user's input for setting a region of interest (611). For example, the electronic device (600) can receive a user's input for setting a first region of a preview image as a region of interest (611). For example, in response to receiving the user's input, the electronic device (600) can set the first region as a region of interest (611).

[0129] For example, the electronic device (600) can set a region of interest (611) by analyzing a preview image. For example, the electronic device (600) can identify at least one object from the preview image. For example, the electronic device (600) can identify a main object and a sub-object from the preview image. For example, the electronic device (600) can set a region of interest (611) based on identifying the main object. For example, the electronic device (600) can set a first region corresponding to the main object as the region of interest (611). For example, the electronic device (600) can set a first region corresponding to a part of the main object as the region of interest (611).

[0130] According to one embodiment, the electronic device (600) can obtain pixel values ​​for a region of interest (611). For example, the electronic device (600) can identify sensor pixels of an image sensor corresponding to a first region set as the region of interest (611). For example, the electronic device (600) can identify second pixel lines (622) including sensor pixels corresponding to the first region. For example, the electronic device (600) can obtain pixel values ​​for the region of interest (611) by reading pixel values ​​of sensor pixels corresponding to the first region from the second pixel lines (622).

[0131] According to one embodiment, the electronic device (600) can obtain an image (610). For example, the electronic device (600) can obtain the image (610) by converting light received by an image sensor into an electrical signal.

[0132] According to one embodiment, the camera module (601) may include an image sensor that acquires images in a rolling shutter manner. For example, the image sensor may include a plurality of pixel lines (620). For example, the image sensor may sequentially expose the plurality of pixel lines (620) and read out pixel values. For example, the image sensor may sequentially expose pixel lines (620) arranged from a first position (e.g., the top or bottom of the image sensor) to a second position (e.g., the bottom or top of the image sensor) and read out pixel values.

[0133] According to one embodiment, the electronic device (600) can obtain image data from first sensor pixels included in a first pixel line (621) of an image sensor. For example, the electronic device (600) can initialize the first sensor pixels. For example, the electronic device (600) can expose the first sensor pixels. For example, the electronic device (600) can read pixel values ​​from the first sensor pixels.

[0134] For example, the first pixel line (621) may have sensor pixels initialized (621a) during a first time (e.g., from t0 to t1). The initialization (621a) of the sensor pixels may include an operation of initializing charges accumulated in the sensor pixels. For example, the operation of the electronic device (600) to initialize the sensor pixels may include an operation of releasing charges accumulated in the sensor pixels and initializing voltages.

[0135] For example, the first pixel line (621) may have sensor pixels exposed (621b) during a second time (e.g., t1 to t4). For example, the first pixel line (621) may begin exposing the sensor pixels (621b) at t1. For example, the first pixel line (621) may end exposing the sensor pixels (621b) at t4.

[0136] For example, the first pixel line (621) may have pixel values ​​read out (621c) during a third time period (e.g., t4 to t5). For example, the first pixel line (621) may have voltage values ​​read out corresponding to charges accumulated in the sensor pixels during the third time period.

[0137] According to one embodiment, the electronic device (600) can obtain image data from second sensor pixels included in a second pixel line (622) of the image sensor. The second pixel line (622) can include sensor pixels corresponding to the region of interest (611). For example, the electronic device (600) can initialize the second sensor pixels. For example, the electronic device (600) can expose the second sensor pixels. For example, the electronic device (600) can read pixel values ​​from the second sensor pixels.

[0138] For example, the second pixel line (622) may have sensor pixels initialized (622a) during a fourth time (e.g., t9 to t10). Initializing the sensor pixels may include an operation of initializing charges accumulated in the sensor pixels. For example, the operation of initializing the sensor pixels may include an operation of releasing charges accumulated in the sensor pixels and initializing voltages.

[0139] For example, the second pixel line (622) may have sensor pixels exposed (622b) during a fifth time period (e.g., t10 to t13). For example, the second pixel line (622) may begin exposing the sensor pixels (622b) at t10. For example, the second pixel line (622) may end exposing the sensor pixels (622b) at t13.

[0140] For example, the second pixel line (622) may have pixel values ​​read out (622c) during a sixth time period (e.g., t13 to t14). For example, the second pixel line (622) may have voltage values ​​read out corresponding to charges accumulated in the sensor pixels during the sixth time period.

[0141] Referring to FIG. 6, in the image sensor of the rolling shutter method, the pixel values ​​of the second pixel line (622) may be read later than the pixel values ​​of the first pixel line (621). Accordingly, at least one operation among the focus control operation (e.g., auto focus (AF) operation), exposure control operation (e.g., auto exposure (AE) operation), or color control operation (e.g., auto white balance (AWB) operation) performed using the pixel values ​​of the sensor pixels corresponding to the region of interest (611) may be delayed.

[0142] FIG. 7 is a diagram for explaining an operation of controlling a camera module using an image acquired by a rolling shutter method by an electronic device according to an embodiment of the present disclosure. FIG. 7 shows a time at which the electronic device (600) of FIG. 6 reflects pixel values ​​acquired through a pixel line (620) of the camera module (601) in an AF operation. The electronic device (700) of FIG. 7 may correspond to the electronic device (600) of FIG. 6. The operation of the electronic device (700) described with reference to FIG. 7 may be performed under the control of a processor (for example, the processor (320) of FIG. 3).

[0143] Hereinafter, with reference to FIG. 7, an embodiment in which an electronic device (700) acquires a first image using pixel values ​​(710) will be described, but the present invention can also be applied to an embodiment in which an electronic device (700) acquires a second image using pixel values ​​(720). Duplicate details will be omitted.

[0144] According to one embodiment, the electronic device (700) can sequentially read pixel values ​​(710, 720) from pixel lines of an image sensor (e.g., pixel lines (620) of FIG. 6). For example, the image sensor can read pixel values ​​(710, 720) from pixel lines (e.g., pixel lines (620) of FIG. 6) based on a control signal (701). For example, the electronic device (700) can obtain pixel values ​​constituting a first image in response to the control signal (701). For example, the electronic device (700) can obtain pixel values ​​constituting a second image in response to the control signal (701).

[0145] For example, pixel lines (e.g., pixel lines (620) of FIG. 6) can be sequentially exposed, and pixel values ​​(710, 720) can be read out. For example, a first pixel value (712, 722) can be read out from a first pixel line (e.g., a second pixel line (622) of FIG. 6) that includes sensor pixels corresponding to a region of interest (e.g., a region of interest (611) of FIG. 6). For example, a second pixel value (711, 721) can be read out from a pixel line (e.g., a first pixel line (621) of FIG. 6) that does not include sensor pixels corresponding to a region of interest (e.g., a region of interest (611) of FIG. 6). For example, depending on the location of the region of interest (e.g., region of interest (611) of FIG. 6), the first pixel values ​​(712, 722) may be read later than at least some of the second pixel values ​​(711, 721).

[0146] According to one embodiment, the electronic device (700) can obtain an image using the extracted pixel values. For example, the electronic device (700) can obtain a first image (790) by reconstructing the pixel values ​​(710).

[0147] For example, the electronic device (700) can image-process pixel values ​​(710) to generate a first image (790). The electronic device (700) can output (704) the first image (790) through a display (e.g., the display (360) of FIG. 3).

[0148] For example, the electronic device (700) can perform at least one operation for image processing (e.g., demosaicing, operations for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement). For example, the electronic device (700) can display a first image (790) generated through image processing through a display (e.g., the display (360) of FIG. 3).

[0149] According to one embodiment, the electronic device (700) may perform a color control operation (e.g., an auto white balance (AWB) operation) based on information about the color of the region of interest obtained from pixel values ​​corresponding to the region of interest. The electronic device (700) may obtain a first image whose color is adjusted through the color control operation. For example, the processor (e.g., the processor (320) of FIG. 3) may perform an operation to define an appropriate color temperature and / or color tone based on the color information identified from the pixel values. For example, the processor (e.g., the processor (320) of FIG. 3) may perform an operation to adjust the values ​​of channels of defined colors (e.g., red, green, blue) based on the defined appropriate color temperature and / or color tone. For example, the processor (e.g., the processor (320 of FIG. 3)) may perform an operation such as a gray will algorithm, a white patch search algorithm, or a color histogram analysis algorithm.

[0150] According to one embodiment, the electronic device (700) can perform an operation to control the camera module using the read pixel values ​​(710). For example, the electronic device (700) can perform a focus control operation (e.g., an auto focus (AF) operation) and an exposure control operation (e.g., an auto exposure (AE) operation).

[0151] According to one embodiment, the electronic device (700) may perform a focus control operation (702) using a first pixel value (712) acquired from sensor pixels corresponding to a region of interest (e.g., region of interest (611) of FIG. 6). For example, the electronic device (700) may perform the focus control operation (702) through a processor (e.g., processor (320) of FIG. 3). For example, the processor (e.g., processor (320) of FIG. 3) may perform an operation to acquire information about a distance between a camera module (730) (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) and an object of interest based on input pixel values. For example, the processor (e.g., processor (320) of FIG. 3) may perform an operation to acquire information about a distance between the camera module (730) and the object of interest based on a phase difference identified from the pixel values. For example, a processor (e.g., processor (320) of FIG. 3) can generate an AF control signal (703) that controls a lens unit (e.g., lens assembly (210) of FIG. 2, lens unit (381) of FIG. 3) of a camera module (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) to reduce a phase difference identified from pixel values ​​corresponding to a region of interest.

[0152] According to one embodiment, the electronic device (700) can control the camera module (730) based on the control signal (701). For example, the electronic device (700) can perform focus control (e.g., auto focus (AF) control) and exposure control (e.g., auto exposure (AE) control). For example, the electronic device (700) can control a camera AF operation (731) in which at least one lens of a lens assembly (e.g., lens assembly (210) of FIG. 2) moves.

[0153] According to one embodiment, a processor (e.g., processor (320) of FIG. 3) may perform a focus control operation of a camera module (e.g., camera module (601) of FIG. 6) using pixel values ​​acquired from sensor pixels corresponding to a region of interest (e.g., region of interest (611) of FIG. 6). For example, the camera module (e.g., camera module (601) of FIG. 6) may move at least one lens of a lens assembly (e.g., lens assembly (210) of FIG. 2) so that the region of interest (e.g., region of interest (611) of FIG. 6)) is maintained in focus. For example, the processor (e.g., processor (320) of FIG. 3) may perform an operation to acquire information regarding a distance between a camera module (730) (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3)) and an object of interest.

[0154] For example, a processor (e.g., processor (320) of FIG. 3) may perform an operation to obtain information about a distance between a camera module (730) (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) and an object of interest based on a phase difference identified from pixel values. For example, the processor (e.g., processor (320) of FIG. 3) may generate an AF control signal (703) to control a lens unit (e.g., lens assembly (210) of FIG. 2, lens unit (381) of FIG. 3) of the camera module (730) (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3)) so that a phase difference identified from pixel values ​​corresponding to a region of interest is reduced.

[0155] According to one embodiment, a processor (e.g., processor (320) of FIG. 3) can perform an exposure control operation (e.g., auto exposure (AE) operation) using first pixel values ​​(712) acquired from sensor pixels corresponding to a region of interest (e.g., region of interest (611) of FIG. 6). For example, the processor (e.g., processor (320) of FIG. 3) can perform an operation to identify a first exposure value (e.g., brightness value) based on the input pixel values. For example, the processor (e.g., processor (320) of FIG. 3) can perform an operation to compare the identified first exposure value with a defined second exposure value. For example, a processor (e.g., processor (320) of FIG. 3) may generate a control signal to change at least one value among aperture, exposure time, and sensitivity of a camera module (730) (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) based on a difference between a first exposure value and a second exposure value.

[0156] Referring to FIG. 7, the electronic device (700) may acquire pixel values ​​(710) using a rolling shutter type image sensor. Since the first pixel value (712) is read later than at least some of the second pixel values ​​(711), the electronic device (700) may not have enough time to perform an operation to control the camera module (e.g., the camera module (601) of FIG. 6) using the first pixel values ​​(712, 722). When the camera module (e.g., the camera module (601) of FIG. 6) acquires a second image, the control signal acquired using the first pixel values ​​(712, 722) may not be used. For example, the AF control signal (703) acquired based on the first pixel value (712) may not be used when the camera module (e.g., the camera module (601) of FIG. 6) acquires the second image.

[0157] In this case, the camera module (e.g., the camera module (601) of FIG. 6) can acquire a second image out of focus in a region of interest (e.g., the region of interest (611) of FIG. 6). For example, the exposure control signal acquired based on the first pixel value (712) may not be used when the camera module (e.g., the camera module (601) of FIG. 6) acquires the second image. In this case, the camera module (e.g., the camera module (601) of FIG. 6) can acquire a second image having a first exposure value different from the set second exposure value.

[0158] FIG. 8 is a diagram illustrating the configuration of an image sensor that acquires images using a global shutter method according to one embodiment of the present disclosure. The image sensor of FIG. 8 may correspond to the image sensor (230, 383) described above with reference to FIGS. 2, 3, and 4.

[0159] Referring to FIG. 8, an image sensor according to one embodiment may include a pixel array (810) and a readout circuit (820). For example, the pixel array (810) may include a micro lens array (MLA) (411), a color filter array (CFA) (413), and a light receiving unit (415), as described above with reference to FIG. 4.

[0160] In one embodiment, the image processing engine (830) may be positioned externally to the image sensor. For example, the image processing engine (830) may correspond to the processor (320) described above with reference to FIG. 3. For example, at least some of the operations performed by the image processing engine (830) may correspond to at least some of the operations performed by the processor (320) described above with reference to FIG. 3.

[0161] According to one embodiment, the image processing engine (830) may be included in the image sensor. For example, the image processing engine (830) may correspond to the operation unit (417) described above with reference to FIG. 4. For example, at least some of the operations performed by the image processing engine (830) may correspond to at least some of the operations performed by the operation unit (417) described above with reference to FIG. 4.

[0162] According to one embodiment, all sensor pixels included in the pixel array (810) can be exposed simultaneously. This method may be referred to as a global shutter method. An image sensor that generates an image using a global shutter method can sequentially read pixel values ​​of pixel lines arranged from a first position (e.g., the top or bottom of the image sensor) to a second position (e.g., the bottom or top of the image sensor).

[0163] According to one embodiment, the image pixel array (810) may include sensor pixels arranged in an array having M rows and N columns (wherein M and N may each be a natural number greater than or equal to 1). For example, the image pixel array (810) may include M pixel lines. Each of the pixel lines may include sensor pixels included in the M rows. Each of the sensor pixels may include a microlens, a color filter, and a light receiving element.

[0164] In one embodiment, the sensor pixels may be exposed. For example, the sensor pixels may be exposed to the light bundle (801) for a set period of time in response to a control signal from a processor (e.g., processor (320) of FIG. 3). For example, the sensor pixels may convert light received during the exposure period into an electrical signal. For example, the sensor pixels may determine a pixel value based on the photons received during the exposure period.

[0165] In one embodiment, the sensor pixels may have pixel values ​​read out. For example, the sensor pixels may have pixel values ​​read out and provided to the readout circuit (820). For example, the readout circuit (820) may temporarily store the pixel values. For example, the pixel values ​​read out from the sensor pixels may be input to the image processing engine (830) via the readout circuit (820).

[0166] In one embodiment, each of the sensor pixels may have a pixel value (802) read out. For example, a processor (e.g., processor (320) of FIG. 3) may control the sensor pixels and / or pixel lines such that pixel values ​​(802) are read out from each of the sensor pixels.

[0167] According to one embodiment, the sensor pixels may be read out as pixel values ​​(802) for each pixel line. For example, a processor (e.g., processor (320) of FIG. 3) may control the pixel lines so that the pixel values ​​(802) are read out for each pixel line. For example, the processor (e.g., processor (320) of FIG. 3) may control the pixel lines so that the pixel values ​​(802) are sequentially read out for pixel lines arranged from a first position (e.g., top or bottom of the image pixel array (810)) to a second position (e.g., bottom or top of the image pixel array (810). For example, the pixel values ​​(803) sequentially read out from the pixel lines may be sequentially input to the image processing engine (830) through the readout circuit (820).

[0168] According to one embodiment, a processor (e.g., processor (320) of FIG. 3) can control the order in which pixel values ​​(802) of pixel lines are read out. For example, the processor (e.g., processor (320) of FIG. 3) can read pixel values ​​of a pixel line including a region of interest before pixel values ​​of other pixel lines. For example, the processor (e.g., processor (320) of FIG. 3) can read first pixel values ​​of a first pixel line including a region of interest (e.g., first pixel line (1121) of FIG. 11) before second pixel values ​​of a second pixel line not including a region of interest.

[0169] According to one embodiment, the image processing engine (830) may perform operations that process pixel values. For example, the image processing engine (830) may perform a focus control operation (e.g., an auto focus (AF) operation). For example, the image processing engine (830) may perform an exposure control operation (e.g., an auto exposure (AE) operation). For example, the image processing engine (830) may perform a color control operation (e.g., an auto white balance (AWB) operation). For example, the image processing engine (830) may perform an operation that generates an image.

[0170] According to one embodiment, the image processing engine (830) can perform a focus control operation based on the first pixel values ​​of the first pixel line (e.g., the first pixel line (1121) of FIG. 11). For example, the image processing engine (830) can perform an operation to obtain information about the distance between the camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) and the object of interest based on the first pixel values. For example, the image processing engine (830) can perform an operation to obtain information about the distance between the camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) and the object of interest based on the phase difference identified from the first pixel values. For example, the image processing engine (830) can adjust the lens unit (e.g., the lens unit) of the camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3)) so that the phase difference identified from the first pixel values ​​is reduced. A control signal can be generated to control the lens assembly (210) of Fig. 2 and the lens unit (381) of Fig. 3.

[0171] According to one embodiment, the image processing engine (830) can perform an exposure control operation (e.g., an auto exposure (AE) operation) based on first pixel values ​​of a first pixel line (e.g., the first pixel line (1121) of FIG. 11). For example, the image processing engine (830) can perform an operation to identify a difference between a set first exposure value and a second exposure value identified from the first pixel values. For example, the image processing engine (830) can generate a control signal to change at least one value among an aperture, an exposure time, and a sensitivity of a camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) based on the difference between the first exposure value and the second exposure value.

[0172] According to one embodiment, the image processing engine (830) can perform a color control operation (e.g., an auto white balance (AWB) operation) based on the first pixel values ​​of the first pixel line (e.g., the first pixel line (1121) of FIG. 11). For example, the image processing engine (830) can identify an appropriate color temperature based on the color information identified from the first pixel values. For example, the image processing engine (830) can perform an operation to adjust the values ​​of channels of defined colors (e.g., red, green, blue) based on the identified appropriate color temperature. For example, the image processing engine (830) can perform an operation such as a gray will algorithm, a white patch search algorithm, or a color histogram analysis algorithm.

[0173] According to one embodiment, the image processing engine (830) may generate an image based on the first pixel values ​​and the second pixel values. For example, the image processing engine (830) may generate an image by performing at least one operation (e.g., demosaicing, white balance, contrast, saturation value adjustment operation, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on input raw image data.

[0174] FIG. 9 is a diagram illustrating sensor pixels of an image sensor that acquires images using a global shutter method according to one embodiment of the present disclosure. The sensor pixels (900) of FIG. 9 may correspond to the sensor pixels included in the pixel array (810) described above with reference to FIG. 8. The photodiode (910) of FIG. 9 may correspond to the light-receiving element of the light-receiving unit (415) described above with reference to FIG. 4.

[0175] According to one embodiment, the sensor pixel (900) may include a photodiode (910), a charge domain capacitor (920), a floating diffusion (FD) (930), and a source follower (SF) amplifier (940). However, the present invention is not limited thereto, and the sensor pixel (900) may include more or fewer components than those illustrated in FIG. 9. For example, some of the components illustrated in FIG. 9 may be included in the readout circuit (820) described above with reference to FIG. 8. For example, at least one of the charge domain capacitor (920), the floating diffusion (FD) (930), or the source follower (SF) amplifier (940) may be included in the readout circuit (820).

[0176] In one embodiment, the photodiode (910) can generate charges based on photons detected by exposure of the sensor pixel. For example, the photodiode (910) can generate electrons and holes based on the detected photons. For example, the photodiode (910) can accumulate charges by electrons and holes.

[0177] According to one embodiment, the charge domain capacitor (920) can accumulate charge by electrons and holes generated from the photodiode (910). For example, the charge domain capacitor (920) can store the accumulated charge. For example, the charge domain capacitor (920) can temporarily store the charge until the charge is read out. By temporarily storing the charge by the charge domain capacitor (920), all sensor pixels of the image sensor can be exposed simultaneously.

[0178] According to one embodiment, a floating diffusion (FD) (930) can convert a charge generated by a photodiode (910) into a voltage. For example, the FD (930) can be connected to a charge domain capacitor (920), so that a voltage corresponding to the charge stored in the charge domain capacitor (920) can be formed. For example, the voltage formed in the FD (930) can be converted into an electrical signal representing the intensity of light received by a sensor pixel.

[0179] According to one embodiment, the source follower (SF) amplifier (940) can amplify and / or transmit an electrical signal generated in the FD (930). For example, the source follower (SF) amplifier (940) can receive a voltage generated in the FD (930). For example, the source follower (SF) amplifier (940) can amplify the input voltage. For example, the SF amplifier (940) can perform the function of a buffer that transmits or temporarily stores an electrical signal. For example, the SF amplifier (940) can transmit an electrical signal to an operational circuit (for example, the operational unit (417) of FIG. 4). For example, the SF amplifier (940) can transmit a voltage generated in the FD (930) to the operational circuit. For example, the SF amplifier (940) can transmit an electrical signal obtained by amplifying a voltage input from the FD (930) to the operational circuit.

[0180] FIG. 10 is a diagram illustrating sensor pixels of an image sensor that acquires images using a global shutter method according to one embodiment of the present disclosure. The sensor pixels (1000) of FIG. 10 may correspond to the sensor pixels included in the pixel array (810) described above with reference to FIG. 8. The photodiode (1010) of FIG. 10 may correspond to the light-receiving element of the light-receiving unit (415) described above with reference to FIG. 4.

[0181] According to one embodiment, the sensor pixel (1000) may include a photodiode (1010), a floating diffusion (FD) (1020), a source follower (SF) amplifier (1030), and a voltage domain capacitor (1040). However, the present invention is not limited thereto, and the sensor pixel (1000) may include more or fewer components than those illustrated in FIG. 10. For example, some of the components illustrated in FIG. 10 may be included in the readout circuit (820) described above with reference to FIG. 8. For example, one of the floating diffusion (FD) (1020), the source follower (SF) amplifier (1030), or the voltage domain capacitor (1040) may be included in the readout circuit (820).

[0182] In one embodiment, the photodiode (1010) can generate charges based on photons detected by exposure of the sensor pixel (1000). For example, the photodiode (1010) can generate electrons and holes based on the detected photons. For example, the photodiode (1010) can accumulate charges by electrons and holes.

[0183] According to one embodiment, a floating diffusion (FD) (1020) can convert charges generated in a photodiode (1010) into voltage. For example, the FD (1020) can be connected to the photodiode (1010), so that a voltage corresponding to the charges accumulated in the photodiode (1010) can be formed. For example, the voltage formed in the FD (1020) can be converted into an electrical signal representing the intensity of light received by a sensor pixel.

[0184] According to one embodiment, the source follower (SF) amplifier (1030) can amplify and / or transmit an electrical signal generated by the FD (1020). For example, the source follower (SF) amplifier (1030) can receive a voltage generated by the FD (1020). For example, the source follower (SF) amplifier (1030) can amplify the input voltage. For example, the SF amplifier (1030) can perform the function of a buffer that transmits or temporarily stores an electrical signal.

[0185] According to one embodiment, a voltage domain capacitor (1040) can store a voltage value. For example, the voltage domain capacitor (1040) can store an electrical signal output from the SF amplifier (1030). For example, the voltage domain capacitor (1040) can be connected to the SF amplifier (1030) to store the voltage value output from the SF amplifier (1030). For example, the voltage domain capacitor (1040) can transmit an electrical signal of the stored voltage value to an operational circuit. For example, by temporarily storing the voltage value, the voltage domain capacitor (1040) can expose all sensor pixels of the image sensor simultaneously.

[0186] FIG. 11 is a diagram illustrating an operation of an image sensor that acquires an image using a global shutter method according to an embodiment of the present disclosure. The electronic device (1100) of FIG. 11 may correspond to the electronic device (200) of FIG. 2 and the electronic device (101) of FIG. 3. The operation of the electronic device (1100) described with reference to FIG. 11 may be performed under the control of a processor (for example, the processor (320) of FIG. 3).

[0187] According to one embodiment, the electronic device (1100) can acquire an image (1110) using the camera module (1101). For example, the electronic device (1100) can acquire an image (1110) including at least one object.

[0188] According to one embodiment, the electronic device (1100) can set an area of ​​interest (1111).

[0189] For example, the electronic device (1100) can set a region of interest (1111) based on a user's input. For example, the electronic device (1100) can receive a user's input for setting a region of interest (1111). For example, in response to receiving the user's input, the electronic device (1100) can set a first region of the preview image as the region of interest (1111).

[0190] For example, the electronic device (1100) can set a region of interest (1111) by analyzing a preview image. For example, the electronic device (1100) can identify at least one object from the preview image. For example, the electronic device (1100) can identify a main object and a sub-object from the preview image. For example, the electronic device (1100) can set a region of interest (1111) based on identifying the main object. For example, the electronic device (1100) can set a first region corresponding to the main object as the region of interest (1111). For example, the electronic device (1100) can set a first region corresponding to a part of the main object as the region of interest (1111).

[0191] According to one embodiment, the electronic device (1100) can obtain pixel values ​​for a region of interest (1111). For example, the electronic device (1100) can identify sensor pixels of an image sensor corresponding to a first region set as the region of interest (1111). For example, the electronic device (1100) can identify first pixel lines (1121) including sensor pixels corresponding to the first region. For example, the electronic device (1100) can obtain pixel values ​​for the region of interest (1111) by reading pixel values ​​of sensor pixels corresponding to the first region from the first pixel lines (1121).

[0192] According to one embodiment, the electronic device (1100) can acquire an image (1110). For example, the electronic device (1100) can acquire an image (1110) by converting light received by an image sensor into an electrical signal.

[0193] According to one embodiment, the camera module (1101) may include an image sensor that acquires an image using a global shutter method. For example, the image sensor may include a plurality of pixel lines (1120). The image sensor may sequentially read pixel values ​​from the plurality of pixel lines. For example, the image sensor may sequentially read pixel values ​​of pixel lines arranged from a first position (e.g., the top or bottom of the image sensor) to a second position (e.g., the bottom or top of the image sensor).

[0194] According to one embodiment, sensor pixels included in a plurality of pixel lines of an image sensor may be initialized (1120a) simultaneously. For example, the electronic device (1100) may initialize (1120a) all sensor pixels included in the image sensor. For example, the electronic device (1100) may initialize (1120a) the sensor pixels during a first time (e.g., t0 to t1). Initializing the sensor pixels may include an operation of initializing charges accumulated in the sensor pixels. For example, the operation of initializing the sensor pixels may include an operation of releasing charges accumulated in the sensor pixels and initializing a voltage.

[0195] According to one embodiment, sensor pixels included in a plurality of pixel lines (1120) of an image sensor may be exposed (1120b) simultaneously. For example, all pixel lines (1120) of the image sensor may have their sensor pixels exposed (1120b) during a second time (e.g., t1 to t4). For example, the exposure (1120b) of the sensor pixels of the plurality of pixel lines (1120) may begin at t1. For example, the exposure (1120b) of the sensor pixels of the plurality of pixel lines (1120) may end at t4.

[0196] According to one embodiment, the electronic device (1100) can control the order in which pixel values ​​of pixel lines are read out. For example, the electronic device (1100) can read the pixel value of a pixel line including a region of interest before pixel values ​​of other pixel lines. For example, the pixel value of a first pixel line (1121) including a region of interest can be read out before pixel values ​​of a second pixel line that does not include the region of interest. For example, pixel values ​​of the first pixel line (1121) can be read out (1121c) between t4 and t5. For example, pixel values ​​of the second pixel lines can be read out sequentially after t5. For example, pixel values ​​can be read out sequentially from a second pixel line disposed at a first location (e.g., the top or bottom of the image sensor) to a second pixel line disposed at a second location (e.g., the bottom or top of the image sensor).

[0197] Referring to FIG. 11, in the image sensor of the global shutter method, the pixel values ​​of the first pixel line (1121) can be read out faster than the pixel values ​​of the second pixel line. Accordingly, at least one operation among a focus control operation (e.g., an auto focus (AF) operation), an exposure control operation (e.g., an auto exposure (AE) operation), or a color control operation (e.g., an auto white balance (AWB) operation) can be performed without delay using the pixel values ​​of the sensor pixels corresponding to the area of ​​interest.

[0198] FIG. 12 is a diagram for explaining an operation of an electronic device controlling a camera module using an image acquired using a global shutter method, according to one embodiment of the present disclosure.

[0199] Fig. 12 shows the time taken by the electronic device (1100) of Fig. 11 to reflect pixel values ​​acquired through the pixel line (1120) of the camera module (1101) in the AF operation. The electronic device (1200) of Fig. 12 may correspond to the electronic device (1100) of Fig. 11. The operation of the electronic device (1200) described with reference to Fig. 12 may be performed under the control of a processor (for example, the processor (320) of Fig. 3).

[0200] Hereinafter, with reference to FIG. 12, an embodiment in which an electronic device (1200) acquires a first image (1290) using pixel values ​​(1210) will be described, but the present invention can also be applied to an embodiment in which an electronic device (1200) acquires a second image using pixel values ​​(1220). Duplicate details are omitted.

[0201] According to one embodiment, an electronic device (e.g., electronic device (1100) of FIG. 11) can read pixel values ​​(1210, 1220) from pixel lines of an image sensor. For example, the image sensor can read pixel values ​​(1210, 1220) based on a control signal (1201). For example, the image sensor can obtain pixel values ​​constituting a first image (1290) in response to the control signal (1201). For example, the image sensor can obtain pixel values ​​constituting a second image in response to the control signal (1201).

[0202] According to one embodiment, the electronic device (1200) can control the order in which pixel values ​​of pixel lines are read out. For example, the electronic device (1200) can read out the first pixel value (1212) of a first pixel line (e.g., the first pixel line (1121) of FIG. 11) that includes a region of interest before the second pixel values ​​(1211) of other pixel lines. For example, the first pixel value (1212) of a first pixel line (e.g., the first pixel line (1121) of FIG. 11) that includes a region of interest can be read out before the second pixel value (1211) of a second pixel line that does not include a region of interest. For example, the second pixel lines can have their second pixel values ​​(1211) read out sequentially. For example, pixel values ​​can be read out sequentially from a second pixel line positioned at a first location (e.g., the top or bottom of the image sensor) to the second pixel lines positioned at a second location (e.g., the bottom or top of the image sensor).

[0203] According to one embodiment, the electronic device (1200) may obtain line data including a tag. The line data may include pixel values ​​obtained from pixel lines. The tag may include information related to the line data. For example, the tag may include information about a region of interest (e.g., coordinates of the region of interest). For example, the tag may include information about a position of a pixel line corresponding to the line data including the tag among a plurality of pixel lines. For example, a tag of a first pixel line (e.g., the first pixel line (1121) of FIG. 11) may include information about an order in which the first pixel line is arranged among the plurality of pixel lines (e.g., the pixel lines (1120) of FIG. 11) from a first position (e.g., the top or bottom of the image sensor) to a second position (e.g., the bottom or top of the image sensor). For example, a tag of a first pixel line (e.g., the first pixel line (1121) of FIG. 11) may include information to identify that it is the 11th pixel line from the top of the image sensor.

[0204] According to one embodiment, the electronic device (1200) can obtain an image using the extracted pixel values. For example, the electronic device (1200) can obtain a first image (1290) by reconstructing the pixel values ​​(1210).

[0205] According to one embodiment, the electronic device (1200) can obtain the first image (1290) by reconstructing the positions of pixel values ​​based on the tag. For example, the electronic device (1200) can identify the position of the first pixel line (e.g., the first pixel line (1121) of FIG. 11) among a plurality of pixel lines based on information about the position of the first pixel line (e.g., the first pixel line (1121) of FIG. 11) included in the tag. For example, the electronic device (1200) can sequentially arrange the pixel values ​​to correspond to the arrangement order of the pixel lines according to the position of the first pixel line (e.g., the first pixel line (1121) of FIG. 11). The electronic device (1200) can obtain the first image (1290) using the sequentially arranged pixel values.

[0206] According to one embodiment, the electronic device (1200) can image process pixel values ​​(1210) to generate a first image (1290). The electronic device (1200) can output (1204) the first image (1290) through a display (e.g., the display (360) of FIG. 3).

[0207] For example, the electronic device (1200) can perform at least one operation for image processing (e.g., demosaicing, operations for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement). For example, the electronic device (1200) can display a first image (1290) generated through image processing through a display (e.g., the display (360) of FIG. 3).

[0208] According to one embodiment, the electronic device (1200) may perform a color control operation (e.g., an auto white balance (AWB) operation) based on information about the color of the region of interest obtained from pixel values ​​corresponding to the region of interest. The electronic device (1200) may obtain a first image whose color is adjusted through the color control operation. For example, the processor (e.g., the processor (320) of FIG. 3) may perform an operation to define an appropriate color temperature and / or color tone based on the color information identified from the pixel values. For example, the processor (e.g., the processor (320) of FIG. 3) may perform an operation to adjust the values ​​of channels of defined colors (e.g., red, green, blue) based on the defined appropriate color temperature and / or color tone. For example, the processor (e.g., the processor (320 of FIG. 3) may perform an operation such as a gray will algorithm, a white patch search algorithm, or a color histogram analysis algorithm.

[0209] According to one embodiment, the electronic device (1200) can control the camera module (1230) based on the control signal (1201). For example, the electronic device (1200) can perform focus control (e.g., auto focus (AF) control) and exposure control (e.g., auto exposure (AE) control). For example, the electronic device (1200) can control a camera AF operation (1231) in which at least one lens of a lens assembly (e.g., lens assembly (210) of FIG. 2) moves.

[0210] According to one embodiment, a processor (e.g., processor (320) of FIG. 3) may perform a focus control operation (1202) of a camera module (e.g., camera module (1101) of FIG. 11) using pixel values ​​acquired from sensor pixels corresponding to a region of interest (e.g., region of interest (1111) of FIG. 11). For example, the camera module (e.g., camera module (1101) of FIG. 11) may move at least one lens of a lens assembly (e.g., lens assembly (210) of FIG. 2) so that the region of interest (e.g., region of interest (1111) of FIG. 11)) is maintained in focus. For example, the processor (e.g., processor (320) of FIG. 3) may perform an operation to acquire information regarding a distance between a camera module (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) and an object of interest. For example, a processor (e.g., processor (320) of FIG. 3) may perform an operation to obtain information about a distance between a camera module (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) and an object of interest based on a phase difference identified from pixel values. For example, the processor (e.g., processor (320) of FIG. 3) may generate an AF control signal (1203) that controls a lens unit (e.g., lens assembly (210) of FIG. 2, lens unit (381) of FIG. 3) of a camera module (e.g., camera module (180) of FIG. 2, camera module (380) of FIG. 3) so that there is no phase difference identified from pixel values.

[0211] According to one embodiment, a processor (e.g., processor 320 of FIG. 3) may perform an exposure control operation (e.g., auto exposure (AE) operation) using pixel values ​​acquired from sensor pixels corresponding to a region of interest (e.g., region of interest 1111 of FIG. 11). For example, the processor (e.g., processor 320 of FIG. 3) may perform an operation to identify a first exposure value (e.g., brightness value) based on input pixel values. For example, the processor (e.g., processor 320 of FIG. 3) may perform an operation to compare the identified first exposure value with a defined second exposure value. For example, the processor (e.g., processor 320 of FIG. 3) may generate a control signal to change at least one value of an aperture, an exposure time, and a sensitivity of a camera module (e.g., camera module 180 of FIG. 2, camera module 380 of FIG. 3) based on a difference between the first exposure value and the second exposure value.

[0212] According to one embodiment, the electronic device (1200) can perform in parallel the operation of generating a first image (1290) and the operation of controlling a camera module (e.g., the camera module (1101) of FIG. 11). For example, the processor (e.g., the processor (320) of FIG. 3) may first obtain a pixel value (1212) of a first pixel line (e.g., the first pixel line (1121) of FIG. 11), perform the operation of controlling the camera module (e.g., the camera module (1101) of FIG. 11), and sequentially perform the operation of generating the first image (1290) by obtaining pixel values ​​(1210) of pixel lines (e.g., the pixel lines (1120) of FIG. 11).

[0213] Referring to FIG. 12, the electronic device (1200) can obtain pixel values ​​(1210) using a global shutter type image sensor. For example, the electronic device (1200) can obtain pixel values ​​(1212) of a first pixel line including sensor pixels corresponding to a region of interest (e.g., a region of interest (1111) of FIG. 11) and pixel values ​​(1211) of a second pixel line not including sensor pixels corresponding to the region of interest (e.g., a region of interest (1111) of FIG. 11). For example, the electronic device (1200) can read the pixel values ​​(1212) of the first pixel line before the pixel values ​​(1211) of the second pixel line. Since the pixel value (1212) of the first pixel line is read out with priority over the pixel values ​​(1211) of other pixel lines, time can be secured for performing an operation for controlling the camera module (e.g., the camera module (1101) of FIG. 11) using the pixel value (1212) of the first pixel line. Accordingly, when the camera module (e.g., the camera module (1101) of FIG. 11) acquires a second image, the AF control signal (1203) acquired based on the pixel value (1212) of the first pixel line can be used. In addition, when the camera module (e.g., the camera module (1101) of FIG. 11) acquires a second image, the exposure control signal acquired based on the pixel value (1212) of the first pixel line can be used. The second image may be an image acquired after the first image is acquired. The second image may include an image obtained using pixel values ​​(1220) obtained after pixel values ​​(1210) are obtained.

[0214] The operations of the electronic device (1200) to acquire a second image can be analogously applied to the operations of the electronic device (1200) to acquire a first image. For example, the electronic device (1200) can first acquire a pixel value (1222) of a first pixel line (e.g., the first pixel line (1121) of FIG. 11), perform an operation to control a camera module (e.g., the camera module (1101) of FIG. 11), and sequentially perform an operation to acquire pixel values ​​(1221) of pixel lines (e.g., the pixel lines (1120) of FIG. 11) to generate a second image.

[0215] FIG. 13 is a diagram illustrating data output from an image sensor that acquires an image using a global shutter method according to one embodiment. FIG. 13 may represent frame data including pixel values ​​(1210, 1220) described above with reference to FIG. 12. FIG. 13 may correspond to frame data output from the operation unit (417) described above with reference to FIG. 4.

[0216] Referring to FIG. 13, according to one embodiment of the present disclosure, frame data may include, but is not limited to, a frame start (FS) (1310), a pixel head (PH) (1320), an embedded data line (1330), pixel data lines (1340, 1350), a pixel frame (PF) (1360), a frame end (FE) (1380), a frame blanking (1390a), and a line blanking (1390b). The frame data may include more components than those shown in FIG. 13, or the frame data may be composed of only a smaller number of components. The frame data may be composed differently depending on the type and settings of the image sensor.

[0217] In one embodiment, the frame data may include a FS (1310). For example, the FS (1310) may indicate the start of a new frame. For example, the FS (1310) may include an identifier to indicate the starting point of the frame data.

[0218] According to one embodiment, the frame data may include a pixel head (PH) (1320). For example, the PH (1320) may include an identifier for indicating the starting point of pixel values. For example, the PH (1320) may include an identifier for indicating the start of data lines.

[0219] According to one embodiment, the frame data may include an embedded data line (1330). For example, the embedded data line (1330) may include additional data related to pixel values. For example, the embedded data line (1330) may include data regarding a shooting environment (context). For example, the embedded data line (1330) may include metadata, a timestamp, or status information of the frame. For example, the embedded data line (1330) may include information regarding the start and end of coordinates of sensor pixels corresponding to a region of interest and / or information regarding the size of the region of interest. For example, the embedded data line (1330) may include information regarding whether phase data is included in the data lines. For example, the embedded data line (1330) may include information regarding the order in which a pixel line corresponding to a first data line (1340) among a plurality of pixel lines constituting an image sensor is arranged.

[0220] In one embodiment, the frame data may include data lines (1340, 1350). For example, the data lines (1340, 1350) may include pixel values ​​obtained from sensor pixels. For example, the data lines (1340, 1350) may include pixel values ​​obtained from each pixel line, line by line.

[0221] According to one embodiment, the frame data may include a first data line (1340) that includes pixel values ​​obtained from a first pixel line (e.g., the first pixel line (1121) of FIG. 11). For example, the first data line (1340) may include pixel values ​​obtained from a first pixel line (e.g., the first pixel line (1121) of FIG. 11) that includes sensor pixels corresponding to a region of interest. For example, the first data line (1340) may include information regarding a distance between a camera module (e.g., the camera module (180) of FIG. 2, the camera module (380) of FIG. 3) associated with the region of interest (e.g., the region of interest (1111) of FIG. 11)) and an object of interest. For example, the first data line (1340) may include phase data of the region of interest (e.g., the region of interest (1111) of FIG. 11). In one embodiment, the frame data may include a second data line (1350) that includes pixel values ​​obtained from second pixel lines. For example, the second data line (1350) may include pixel values ​​obtained from second pixel lines that do not include sensor pixels corresponding to the region of interest. For example, the second data line (1350) may include phase data obtained from the sensor pixels.

[0222] In one embodiment, the frame data may be arranged such that the first data line (1340) takes precedence over the second data line (1350). For example, the first data line (1340) may be arranged prior to the second data line (1350) so that an electronic device (e.g., the electronic device (1100) of FIG. 11) may perform an operation to control a camera module using pixel values ​​acquired from sensor pixels corresponding to an area of ​​interest.

[0223] In one embodiment, the frame data may include a pixel frame (PF) (1360). For example, the PF (1360) may include an identifier to indicate that a particular pixel value is included in a portion of the frame data. For example, the PF (1360) may include an identifier to indicate the end of a data line (1340, 1350).

[0224] According to one embodiment, the frame data may include a frame end (FE) (1380). For example, the FE (1380) may include an identifier to indicate the last point of the frame data.

[0225] In one embodiment, the frame data may include frame blanking (1390a). For example, the frame data may include frame blanking (1390a), which is an inactive area between frame data.

[0226] In one embodiment, the frame data may include line blanking (1390b). For example, the frame data may include line blanking (1390b), which is an inactive area between data lines.

[0227] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can control a camera module (e.g., the camera module (1101) of FIG. 11) using frame data. For example, the electronic device (e.g., the electronic device (1100) of FIG. 11) can control focus and / or exposure of the camera module (e.g., the camera module (1101) of FIG. 11) using pixel values ​​acquired from a first data line.

[0228] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can acquire an image using frame data. For example, the electronic device (e.g., the electronic device (1100) of FIG. 11) can reconstruct an image using pixel values ​​from data lines included in the frame data. For example, the electronic device (e.g., the electronic device (1100) of FIG. 11) can reconstruct an image by combining pixel values ​​acquired from the first data line (1340) and pixel values ​​acquired from the second data line (1350) using information about the position of the first data line (1340).

[0229] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can perform in parallel an operation for acquiring an image and an operation for controlling a camera module (e.g., the camera module (1101) of FIG. 11). For example, the electronic device can perform an operation for controlling a camera module (e.g., the camera module (1101) of FIG. 11) using a first data line (1340) arranged prior to a second data line (1350), and can perform in parallel an operation for generating an image using pixel values ​​of sequentially arranged data lines (1340, 1350).

[0230] FIG. 14 is a diagram for explaining an artificial intelligence model used by an electronic device according to an embodiment of the present disclosure. The artificial intelligence model (1430) of FIG. 14 may be built in the electronic device described above with reference to FIGS. 2, 3, and 11 (for example, the electronic device (200) of FIG. 2, the electronic device (101) of FIG. 3, and the electronic device (1100) of FIG. 11). The artificial intelligence model of FIG. 14 may be built in the electronic device (102, 104) and / or the server (108) described above with reference to FIG. 1. The artificial intelligence model (1430) may be trained to output output data (1440) in response to input data (1420) being applied.

[0231] In one embodiment, the artificial intelligence model (1430) may be trained to acquire information about an object. For example, the artificial intelligence model (1430) may be trained to identify an object included in an image, which is input data (1420), and output information about the object as output data (1440). For example, the artificial intelligence model (1430) may be trained to output information about the type of object as output data (1440). For example, the artificial intelligence model (1430) may be trained to output a result of identifying a main object among objects included in an image. For example, the artificial intelligence model (1430) may be trained to output information about the location of a region of interest corresponding to the main object in the image. For example, the artificial intelligence model (1430) may be trained to track a region of interest and output information about the location of the region of interest.

[0232] In one embodiment, the artificial intelligence model (1430) may be trained to identify objects. For example, the artificial intelligence model (1430) may identify features of objects from images by analyzing images of multiple objects input as training data (1410). The training data (1410) may include images with labels relating to the objects. For example, the artificial intelligence model (1430) may identify features of objects using multiple layers. For example, the artificial intelligence model (1430) may identify features of objects using filters relating to at least one of the edges, shapes, or colors of the objects within the images. For example, the artificial intelligence model (1430) may update its parameters in a direction that minimizes a loss function representing the difference between the output data (1440) and the labels included in the training data (1410). For example, the artificial intelligence model (1430) may output the result of identifying an object included in the input data (1420) as output data (1440) based on the characteristics of the identified object. For example, the artificial intelligence model (1430) may output the result of identifying the type of object as output data (1440).

[0233] In one embodiment, the artificial intelligence model (1430) may be trained to identify primary and secondary objects. For example, the artificial intelligence model (1430) may identify primary and secondary objects based on the results of identifying the type of object. For example, the artificial intelligence model (1430) may identify primary and secondary objects among the identified objects based on the defined priorities of the objects. For example, the artificial intelligence model (1430) may identify a person as the primary object and identify animals and vehicles as secondary objects based on the priorities defined in the order of people, animals, and cars. For example, the artificial intelligence model (1430) may identify a face as the primary object and identify the head and upper body as secondary objects based on the priorities defined in the order of face, head, and upper body. For example, the artificial intelligence model (1430) may output information regarding the identified primary and secondary objects as output data (1440).

[0234] In one embodiment, the AI ​​model (1430) may be trained to establish a region of interest. For example, the AI ​​model (1430) may be trained to establish a region of interest around an identified main object. For example, the AI ​​model (1430) may be trained to output information (e.g., coordinates) regarding the location of the region of interest as output data (1440).

[0235] In one embodiment, the artificial intelligence model (1430) may be trained to track a region of interest. For example, the artificial intelligence model (1430) may be trained to output information about the location of a region of interest on a second image as output data (1440) when information about the location of a region of interest on a first image is applied as input data (1420). For example, the artificial intelligence model (1430) may be trained using training data (1410) that includes information about the location of a region of interest in each of a plurality of images (e.g., coordinates of the center, coordinates of a corner). For example, the artificial intelligence model (1430) may perform forward propagation using the input training data (1410) to update weights.

[0236] In one embodiment, the AI ​​model (1430) may be validated by data containing information regarding the location of a region of interest. For example, the AI ​​model (1430) may be validated by validation data containing information regarding the location of a region of interest (e.g., coordinates of the center, coordinates of a corner) in each of a plurality of images. For example, the AI ​​model (1430) may have its weights updated by performing backpropagation using the validation data.

[0237] In one embodiment, the electronic device may input information (e.g., coordinates) regarding the location of a region of interest on a first image to the artificial intelligence model (1430). For example, the electronic device may input coordinates of the center and / or the corners of the region of interest on the first image.

[0238] According to one embodiment, the electronic device may obtain information (e.g., coordinates) regarding the location of a region of interest on a second image from the artificial intelligence model (1430). For example, the electronic device may obtain the coordinates of the center and / or the coordinates of the corners of the region of interest on the second image output from the artificial intelligence model (1430).

[0239] According to one embodiment, the electronic device may set a region of interest on the second image based on information (e.g., coordinates) regarding the location of the region of interest on the second image obtained from the artificial intelligence model (1430). For example, the electronic device may set a region of interest on the second image by identifying the coordinates of the region of interest based on the coordinates of the center and / or the coordinates of the corners of the region of interest on the second image output from the artificial intelligence model (1430).

[0240] According to one embodiment, at least one processor (e.g., processor (320) of FIG. 3) may execute a neural network model as an artificial intelligence model (1430). For example, the neural network model may include a deep learning model that performs a specific purpose operation based on the result of learning learning data. For example, the neural network model may include at least one of various types of neural network models, such as a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted boltzmann machine (RBM), a fully convolutional network, a long short-term memory (LSTM) network, or a classification network. The artificial intelligence model (1430) may additionally or alternatively include a software structure in addition to a hardware structure.

[0241] FIG. 15 is a diagram for explaining an operation of an electronic device using an artificial intelligence model to identify an object in a region of interest according to an embodiment of the present disclosure. The artificial intelligence model (1520) of FIG. 15 may correspond to the artificial intelligence model (1430) described above with reference to FIG. 14. The electronic device (1500) of FIG. 15 may correspond to the electronic devices (101, 200, 1100) described above with reference to FIGS. 2, 3, and 11. The artificial intelligence model (1520) of FIG. 15 may be built in the electronic devices (101, 200, 1100) described above with reference to FIGS. 2, 3, and 11. The artificial intelligence model of FIG. 15 may be built in the electronic devices (102, 104) and / or the server (108) described above with reference to FIG. 1.

[0242] In one embodiment, the electronic device (1500) can acquire a first image (1510) using a camera module (1501). For example, the electronic device (1500) can acquire a first image (1510) including a person (1511), a window, and a building outside the window.

[0243] In one embodiment, the electronic device (1500) may apply the first image (1510) to an artificial intelligence model (1520). For example, the electronic device (1500) may apply the first image (1510) to an artificial intelligence model (1520) trained to identify objects from images. For example, the electronic device (1500) may apply the first image (1510) to an artificial intelligence model (1520) trained to identify a primary object. For example, the electronic device (1500) may apply the first image (1510) to an artificial intelligence model (1520) trained to set a region of interest in the primary object. For example, the electronic device (1500) may apply the first image (1510) to an artificial intelligence model (1520) trained to output coordinates of the region of interest.

[0244] In one embodiment, the artificial intelligence model (1520) can identify objects from the first image (1510). For example, the artificial intelligence model (1520) can identify a person (1511), a window, and buildings outside the window from the input first image (1510).

[0245] In one embodiment, the AI ​​model (1520) can identify a primary object from the first image (1510). For example, the AI ​​model (1520) can identify the primary object among the primary object (1511), the window, and the buildings outside the window. For example, the AI ​​model (1520) can identify the face of the primary object (1511).

[0246] In one embodiment, the AI ​​model (1520) may set a region of interest for the main object. For example, the AI ​​model (1520) may set a region corresponding to a person (1531) identified as the main object as the region of interest. For example, the AI ​​model (1520) may set a region corresponding to the face (1532) of the person (1531) as the region of interest.

[0247] In one embodiment, the artificial intelligence model (1520) can output coordinates of a region of interest. For example, the artificial intelligence model (1520) can output coordinates of a region of interest set in an image (1530). For example, the artificial intelligence model (1520) can output coordinates of corners and / or coordinates of a center of the region of interest. For example, the artificial intelligence model (1520) can output coordinates of corners and / or coordinates of a center of a region corresponding to a person (1531). For example, the artificial intelligence model (1520) can output coordinates of corners and / or coordinates of a center of a region corresponding to a face (1532).

[0248] In one embodiment, the electronic device (1500) can control the camera module (1501) based on the region of interest. For example, the electronic device (1500) can control the focus and / or exposure of the camera module (1501) based on pixel values ​​obtained from the region of interest.

[0249] FIG. 16 is a flowchart of an operation method of an electronic device for acquiring an image according to an embodiment of the present disclosure. The operation of the electronic device (e.g., the electronic device (200) of FIG. 2) illustrated in FIG. 16 may be performed by a processor (e.g., the processor (220) of FIG. 2) performing calculations or controlling components of the electronic device (e.g., the electronic device (200) of FIG. 2). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0250] Referring to operation 1610, an electronic device according to an embodiment may identify first coordinates of a region of interest on a first image sensor of a first camera module. For example, the electronic device may identify coordinates of an edge of the region of interest and / or coordinates of a center of the region of interest. For example, the electronic device may identify coordinates of the region of interest based on a user input received on a preview image. For example, the electronic device may identify coordinates of the region of interest based on an algorithm for setting the region of interest. For example, the electronic device may identify coordinates of the region of interest from output data including coordinates of the region of interest output from an artificial intelligence model. For example, the electronic device may identify sensor pixels corresponding to the region of interest on the first image sensor. For example, the first camera module may acquire an image using the image sensor described above with reference to FIGS. 8 to 10 .

[0251] Referring to operation 1620, an electronic device according to one embodiment may obtain a first control signal for acquiring image data including a region of interest. For example, the processor may obtain a first control signal for the camera module to acquire a first frame. For example, the first control signal may include at least one of a reset time, an exposure time, or a readout time of the image sensor.

[0252] Referring to operation 1630, an electronic device according to one embodiment may obtain first line data from a first pixel line including at least a portion of an area of ​​interest based on a first coordinate among pixel lines of a first image sensor. For example, the electronic device may obtain the first line data described above with reference to FIG. 13 (e.g., first line data (1340) of FIG. 13).

[0253] For example, the electronic device may read out pixel values ​​(1212) of a first pixel line (1121) including first sensor pixels corresponding to a region of interest (e.g., region of interest (1111) of FIG. 11) as described above with reference to FIGS. 11 and 12, with priority over second pixel values ​​(1211) of another pixel line not including first sensor pixels corresponding to a region of interest (e.g., region of interest (1111) of FIG. 11). For example, the electronic device may obtain first line data including pixel values ​​of first sensor pixels.

[0254] For example, the electronic device may obtain first line data including a first tag relating to an area of ​​interest. For example, the first tag may include coordinates of the area of ​​interest. For example, the first tag may include coordinates of a corner of the area of ​​interest and / or coordinates of the center of the area of ​​interest.

[0255] For example, the electronic device may obtain the first line data including a second tag relating to the position of the first pixel line. For example, the second tag may include information relating to the position of the pixel line corresponding to the first line data. For example, the second tag may include information relating to the position of the first pixel line among a plurality of pixel lines.

[0256] Referring to operation 1640, an electronic device according to one embodiment can perform first image processing using first line data.

[0257] For example, the electronic device may obtain first pixel values ​​corresponding to the region of interest based on the first tag. For example, the electronic device may obtain pixel values ​​corresponding to the region of interest based on the coordinates of the region of interest included in the first tag.

[0258] For example, the electronic device may control the focus of the camera module using the first pixel values. For example, the electronic device may obtain distance data regarding the distance between an object included in the region of interest and the camera module using the first pixel values, and control the focus of the camera module based on the distance data.

[0259] For example, the electronic device may control the exposure of the camera module using the first pixel values. For example, the electronic device may control at least one of the aperture, exposure time, or sensitivity of the camera module based on identifying the brightness of the region of interest using the first pixel values.

[0260] For example, the electronic device can control the camera module based on information about the object output from the AI ​​model by applying the first pixel values ​​to the AI ​​model. For example, the electronic device can control the focus of the camera module using the coordinates of the region of interest output from the AI ​​model. For example, the electronic device can control at least one of the aperture, exposure time, or sensitivity of the camera module based on the exposure adjustment value of the region of interest output from the AI ​​model.

[0261] Referring to operation 1650, an electronic device according to one embodiment may obtain second line data from a second pixel line different from the first pixel line. For example, the electronic device may obtain the second line data described above with reference to FIG. 13 (e.g., second line data (1350) of FIG. 13).

[0262] For example, the electronic device can sequentially read out line data containing pixel values ​​of pixel lines of the image sensor, as described above with reference to FIGS. 11 and 12. For example, the electronic device can sequentially read out pixel values ​​of pixel lines other than the first pixel line.

[0263] Referring to operation 1660, an electronic device according to one embodiment can obtain a first frame using first line data and second line data. For example, the electronic device can obtain the first frame by combining pixel values ​​of a first pixel line included in the first line data and pixel values ​​of a second pixel line included in the second line data.

[0264] For example, the electronic device can obtain the first frame by performing second image processing based on the second tag. For example, the electronic device can generate the first frame by combining pixel values ​​of the first pixel line and pixel values ​​of the second pixel line based on the second tag relating to the position of the first pixel line.

[0265] For example, the electronic device can obtain a first frame by performing a second image processing based on a first tag. For example, the electronic device can identify sensor pixels corresponding to a region of interest based on the first tag. For example, the electronic device can identify a location of a first pixel line including sensor pixels based on the first tag. For example, the electronic device can generate a first frame by combining pixel values ​​of the first pixel line and pixel values ​​of the second pixel line according to the location of the first pixel line identified based on the first tag.

[0266] For example, the electronic device can adjust the color of an image acquired from a camera module using the first pixel values. For example, the electronic device can obtain information about the color of a region of interest using the first pixel values. For example, the electronic device can adjust the values ​​of channels of defined colors (e.g., red, green, blue) based on the defined color temperature and hue of the region of interest.

[0267] In one embodiment, the electronic device may acquire a second frame based on the result of performing the first image processing. For example, the electronic device may acquire the second frame based on a camera module controlled using the first pixel values.

[0268] For example, the electronic device may acquire first distance data using first pixel values ​​and control the focus of the camera module based on the first distance data. For example, the electronic device may acquire a second frame based on the controlled focus of the camera module.

[0269] For example, the electronic device may identify the brightness of the region of interest using the first pixel values. For example, the electronic device may control one of the aperture, exposure time, or sensitivity of the camera module based on the result of comparing the exposure value of the region of interest with a defined exposure value. For example, the electronic device may acquire a second frame using the controlled camera module.

[0270] For example, the electronic device can obtain a second frame based on information about the object identified using the first pixel values. For example, the electronic device can obtain the coordinates of the region of interest based on the result of identifying the object. For example, the electronic device can control the focus of the camera module based on the coordinates of the region of interest. For example, the electronic device can obtain the second frame using the camera module with the controlled focus.

[0271] FIG. 17 is a flowchart illustrating an operation method for an electronic device to acquire an image using multiple camera modules according to an embodiment of the present disclosure. FIG. 18 is a diagram illustrating an operation for an electronic device to acquire an image using multiple camera modules according to an embodiment of the present disclosure.

[0272] The operation of the electronic device illustrated in FIG. 17 (e.g., the electronic device (200) of FIG. 2) may be performed by a processor (e.g., the processor (220) of FIG. 2) performing calculations or controlling components of the electronic device (e.g., the electronic device (200) of FIG. 2). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0273] The electronic device (1800) of FIG. 18 may correspond to the electronic device (200) of FIG. 2 and the electronic device (101) of FIG. 3. The operation of the electronic device (1800) described with reference to FIG. 18 may be performed under the control of a processor (for example, the processor (320) of FIG. 3). The image sensor included in the first camera module (1801) of FIG. 18 may correspond to the image sensor described above with reference to FIG. 11. The image sensor included in the second camera module (1802) of FIG. 18 may correspond to the image sensor described above with reference to FIG. 11.

[0274] Referring to FIG. 18, an electronic device (1800) according to one embodiment may include a first camera module (1801) and a second camera module (1802). The first camera module (1801) may acquire a first image (1810a) in a first field of view. That is, the first camera module (1801) may include a first lens module formed to have a first focal length. The second camera module (1802) may acquire a second image (1820a) in a second field of view. That is, the second camera module (1802) may include a second lens module formed to have a second focal length.

[0275] According to one embodiment, the electronic device (1800) can match image pixels of a first image acquired through a first image sensor and image pixels of a second image acquired through a second image sensor. For example, the electronic device (1800) can match coordinates of image pixels acquired by image signals output from the first sensor pixels and the second sensor pixels that are mapped to each other. For example, the electronic device (1800) can match the first coordinate of the first image pixel corresponding to the first sensor pixel and the second coordinate of the second image pixel corresponding to the second sensor pixel.

[0276] According to one embodiment, the electronic device (1800) may set a first region of interest (1811) on a first image (1810a) acquired through a first camera module (1801) and set a second region of interest (1821) on a second image (1820a) acquired through a second camera module (1802). For example, the electronic device (1800) may set the selected first region as the first region of interest (1811) in response to receiving a user input touching the first region of the first image (1810a). For example, the electronic device (1800) may identify information about the location of the first region of interest (1811) (e.g., coordinates of a corner, coordinates of a center). For example, the electronic device (1800) may identify coordinates of a first sensor pixel of a sensor of the first image (1810a) corresponding to the first region of interest (1811). For example, the electronic device (1800) can obtain information about the location of the second region of interest (1821) corresponding to the second sensor pixel from the coordinates of the second sensor pixel (e.g., coordinates of the corner, coordinates of the center).

[0277] According to one embodiment, the electronic device (1800) can acquire at least one frame using at least one of the first camera module (1801) and the second camera module (1802). For example, the electronic device (1800) can acquire the first frame (1810b) using the first camera module (1801). For example, the electronic device (1800) can acquire the second frame (1810c) using the first camera module (1801). For example, the electronic device (1800) can acquire the third frame (1820b) using the second camera module (1802). For example, the electronic device (1800) can acquire the first frame (1810b) and the second frame (1810c) using line data (1840, 1860) acquired using the first camera module (1801). For example, the electronic device (1800) can acquire a third frame (1820b) using line data (1890) acquired using the second camera module (1802).

[0278] Referring to operation 1710, an electronic device (1800) according to one embodiment may obtain a control signal that causes a first camera module (1801) to enter a first state and a second camera module (1802) to enter a second state. For example, the first state may include a state in which an image is acquired and stored. For example, the second state may include a standby state or a state in which only pixel values ​​of some sensor pixels are acquired.

[0279] For example, the electronic device (1800) can obtain control signals (1830a, 1830b) for driving the first camera module (1801) and the second camera module (1802) by driving a camera application. For example, the electronic device (1800) can obtain a control signal including a camera open command. For example, the electronic device (1800) can provide the control signals (1830a, 1830b) to the first camera module (1801) and the second camera module (1802), thereby allowing the first camera module (1801) and the second camera module (1802) to generate image data.

[0280] Referring to operation 1720, an electronic device (1800) according to an embodiment can identify a first coordinate of a first region of interest (1811) on a first image (1810a) and a second coordinate of a second region of interest (1821) on a second image (1820a). The first image (1810a) and the second image (1820a) may have coordinates that match. For example, the electronic device (1800) can identify a first coordinate of the first region of interest (1811) in response to receiving a user input on the first image (1810a). For example, the electronic device (1800) can identify a first coordinate of the first region of interest (1811) corresponding to a main object identified from the first image (1810a). For example, the electronic device (1800) can identify the second coordinates of the second region of interest (1821) based on the identification of the first coordinates of the first region of interest (1811).

[0281] Referring to operation 1730, an electronic device (1800) according to one embodiment may obtain a control signal for obtaining a first frame (1810b).

[0282] For example, the electronic device (1800) may obtain a control signal for controlling the focus of the first camera module (1801) so that the first region of interest (1811) is focused. For example, the electronic device (1800) may obtain a control signal for controlling at least one of an aperture, exposure time, and sensitivity of the first camera module (1801) so that the exposure value of the first region of interest (1811) corresponds to a defined exposure value.

[0283] For example, the electronic device (1800) may obtain a control signal for controlling the focus of the second camera module (1802) so that the second region of interest (1821) is focused. For example, the electronic device (1800) may obtain a control signal for controlling at least one of an aperture, exposure time, and sensitivity of the second camera module (1802) so that the exposure value of the second region of interest (1821) corresponds to a defined exposure value.

[0284] Referring to operation 1740, an electronic device (1800) according to one embodiment may obtain first line data (1841, 1861) from a first pixel line of a first camera module (1801) and obtain third line data (1851, 1871) from a third pixel line of a second camera module (1802). The first pixel line may include a pixel line including sensor pixels corresponding to a first region of interest (1811). The third pixel line may include a pixel line including sensor pixels corresponding to a second region of interest (1821).

[0285] For example, the electronic device (1800) can control the first camera module (1801) to obtain the first line data (1841, 1861) with priority over the second line data (1842, 1862). For example, the electronic device (1800) can obtain the first line data (1841, 1861) including a tag regarding the position of the first pixel line.

[0286] For example, the electronic device (1800) can control the second camera module (1802) to obtain only the third line data (1851, 1871) from among the line data (1850, 1870). For example, the electronic device (1800) can obtain the third line data (1851, 1871) including a tag regarding the location of the second region of interest from among the line data (1850, 1870). For example, the electronic device (1800) can obtain only the pixel values ​​of the third line data (1851, 1871) from among the line data (1850, 1870). For example, the electronic device (1800) can control not to obtain the fourth line data (1852, 1872) from among the line data (1850, 1870). For example, the electronic device (1800) can obtain only pixel values ​​corresponding to the second area of ​​interest (1821) from among the third line data (1851, 1871).

[0287] Referring to operation 1750, an electronic device (1800) according to one embodiment can perform first image processing using first line data (1841, 1861) and perform third image processing using third line data (1851, 1871).

[0288] For example, the electronic device (1800) can control the focus and / or exposure of the first camera module (1801) using the pixel values ​​corresponding to the first region of interest (1811) included in the first line data (1841, 1861). For example, the electronic device (1800) can control the lens unit of the first camera module (1801) using the phase difference identified from the pixel values ​​corresponding to the first region of interest (1811). For example, the electronic device (1800) can control at least one of the aperture, exposure time, or sensitivity of the first camera module (1801) using the exposure value identified from the pixel values ​​corresponding to the first region of interest (1811).

[0289] For example, the electronic device (1800) can control the focus and / or exposure of the second camera module (1802) using the pixel values ​​corresponding to the second region of interest (1821) included in the third line data (1851, 1871). For example, the electronic device (1800) can control the lens unit of the second camera module (1802) using the phase difference identified from the pixel values ​​corresponding to the second region of interest (1821). For example, the electronic device (1800) can control at least one of the aperture, exposure time, or sensitivity of the second camera module (1802) using the exposure value identified from the pixel values ​​corresponding to the second region of interest (1821).

[0290] Referring to operation 1760, an electronic device (1800) according to an embodiment may obtain second line data (1842, 1862) from a second pixel line of a first image sensor included in a first camera module (1801). The second pixel line may include a pixel line that does not include a sensor pixel corresponding to the first region of interest (1811). For example, the electronic device may obtain the second line data (1842, 1862) by sequentially reading pixel values ​​of the pixel lines of the first image sensor.

[0291] Referring to operation 1770, an electronic device (1800) according to an embodiment can obtain a first frame using first line data (1841) and second line data (1842). For example, the electronic device can obtain the first frame (1810b) by combining pixel values ​​of a first pixel line included in the first line data (1841) and pixel values ​​of a second pixel line included in the second line data (1842). For example, the electronic device can generate the first frame (1810b) by combining pixel values ​​of the first pixel line and pixel values ​​of the second pixel line based on a tag regarding the position of the first pixel line.

[0292] For example, the electronic device can adjust the color of the first frame (1810b) using the first pixel values. For example, the electronic device can obtain information about the color of the first region of interest (1811) using the first pixel values. For example, the electronic device can adjust the values ​​of the channels of defined colors (e.g., red, green, blue) based on the defined color temperature and hue of the first region of interest (1811).

[0293] According to one embodiment, the electronic device (1800) can acquire the second frame (1810c) after the first frame (1810b) is acquired. For example, the electronic device (1800) can acquire the second frame (1810c) based on the first image processing result of operation 1750. For example, the electronic device (1800) can acquire the second frame (1810c) by using the focus of the first camera module (1801) controlled based on the first image processing. For example, the electronic device (1800) can acquire the second frame (1810c) by using at least one of the aperture, exposure time, or sensitivity of the first camera module (1801) controlled based on the first image processing.

[0294] According to one embodiment, the electronic device (1800) may obtain pixel values ​​of sensor pixels corresponding to the second region of interest (1821) from the second camera module (1802) based on the third image processing result. For example, the electronic device (1800) may obtain third line data (1871) that includes only pixel values ​​corresponding to the second region of interest (1821).

[0295] According to one embodiment, the electronic device (1800) can obtain a second frame by repeating at least one operation from operations 1720 to 1770. Duplicate details are omitted.

[0296] Referring to operation 1780, an electronic device (1800) according to an embodiment may obtain a control signal that causes a first camera module (1801) to enter a second state and a second camera module (1802) to enter a first state. For example, the first state may include a state of acquiring and storing an image. For example, the second state may include a standby state or a state of acquiring only pixel values ​​of some sensor pixels. The electronic device (1800) may switch the operations of the first camera module (1801) and the second camera module (1802).

[0297] Referring to operation 1790, an electronic device (1800) according to an embodiment can obtain a third frame (1820b) using line data (1890) obtained through a second camera module (1802). For example, the electronic device (1800) can obtain the third frame (1820b) using third line data (1891) and fourth line data (1892). For example, the electronic device can obtain the fourth line data (1892) by sequentially reading pixel values ​​of pixel lines of a second image sensor. The fourth line data (1892) can be obtained from a fourth pixel line that does not include a sensor pixel corresponding to a second region of interest (1821).

[0298] For example, the electronic device can obtain the third frame (1820b) by combining the pixel values ​​of the third pixel line included in the third line data (1891) and the pixel values ​​of the fourth pixel line included in the fourth line data (1892). For example, the electronic device can generate the third frame (1820b) by combining the pixel values ​​of the third pixel line and the pixel values ​​of the fourth pixel line based on a tag regarding the position of the third pixel line.

[0299] For example, the electronic device (1800) can adjust the color of the third frame (1820b) using the third pixel values. For example, the electronic device can obtain information about the color of the second region of interest (1821) using the third pixel values. For example, the electronic device (1800) can adjust the values ​​of the channels of defined colors (e.g., red, green, blue) based on the defined color temperature and color tone of the region of interest (1821).

[0300] According to one embodiment, the electronic device (1800) can display a third frame (1820b). For example, the electronic device (1800) can display the third frame (1820b) after the first frame (1810b) and the second frame (1810c) are displayed.

[0301] According to one embodiment, the electronic device (1800) can control the first camera module (1801) to acquire line data (1880). For example, the electronic device (1800) can control the first camera module (1801) to acquire only the first line data (1881). For example, the electronic device (1800) can control the first camera module (1801) to not acquire the second line data (1882). For example, the electronic device (1800) can acquire the first line data (1881) including a tag regarding the location of the first region of interest.

[0302] Referring to FIGS. 17 and 18, the electronic device (1800) performs an operation to control the camera module for each frame using pixel values ​​acquired from the region of interest, so that even when the camera module that acquires the image is switched, a frame with maintained focus, exposure, and color can be acquired. In addition, even when the camera module is switched, the electronic device may not initiate operation of the switched camera, so that smooth images can be provided without interruption.

[0303] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description below.

[0304] An electronic device (e.g., the electronic device (1100) of FIG. 11) according to one embodiment may include a first camera module (e.g., the camera module (1101) of FIG. 11) including a first image sensor (e.g., the image sensor (383) of FIG. 3). The electronic device (e.g., the electronic device (1100) of FIG. 11) may include at least one processor (e.g., the processor (320) of FIG. 3) including a processing circuit. The electronic device (e.g., the electronic device (1100) of FIG. 11) may include a memory (e.g., the memory (330) of FIG. 3) for storing instructions. The electronic device (e.g., the electronic device (1100) of FIG. 11) may identify first coordinates of a region of interest (e.g., the region of interest (1111) of FIG. 11)) on the first image sensor (e.g., the image sensor (383) of FIG. 3). An electronic device (e.g., an electronic device (1100) of FIG. 11) can obtain a first control signal (e.g., a control signal (1201) of FIG. 12) for obtaining image data including a region of interest (e.g., a region of interest (1111) of FIG. 11). An electronic device (e.g., an electronic device (1100) of FIG. 11) may, in response to a first control signal (e.g., a control signal (1201) of FIG. 12), obtain first line data (e.g., a pixel value (1212) of FIG. 12) from a first pixel line (e.g., a first pixel line (1121) of FIG. 11) including at least a portion of a region of interest (e.g., a region of interest (1111) of FIG. 11)) based on a first coordinate among pixel lines (e.g., pixel lines (!120) of FIG. 11) of a first image sensor (e.g., an image sensor (383) of FIG. 3). An electronic device (e.g., an electronic device (1100) of FIG. 11) can perform first image processing (e.g., pixel values ​​(1212) of FIG. 12) using first line data (e.g., pixel values ​​(1212) of FIG. 12).An electronic device (e.g., an electronic device (1100) of FIG. 11) can obtain second line data (e.g., a second pixel value (1211) of FIG. 12) from a second pixel line that is different from a first pixel line among pixel lines of a first image sensor. The electronic device (e.g., an electronic device (1100) of FIG. 11) can perform second image processing using the first line data (e.g., a pixel value (1212) of FIG. 12) and the second line data (e.g., a second pixel value (1211) of FIG. 12), thereby obtaining a first frame (e.g., a first image (1290) of FIG. 12).

[0305] According to one embodiment, an electronic device (e.g., an electronic device (1100) of FIG. 11) may cause all pixel lines (e.g., pixel lines (1120) of FIG. 11) of a first image sensor to be exposed for a first time and for a first period of time in response to a first control signal (e.g., a control signal (1201) of FIG. 12). The electronic device (e.g., an electronic device (1100) of FIG. 11) may sequentially obtain line data (e.g., pixel values ​​(1210) of FIG. 12) from the pixel lines (e.g., pixel lines (1120) of FIG. 11) of the first image sensor.

[0306] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) may obtain first line data (e.g., the first line data (1340)) including a first tag relating to a region of interest (e.g., the region of interest (1111) of FIG. 11). The electronic device (e.g., the electronic device (1100) of FIG. 11) may obtain first pixel values ​​corresponding to the region of interest (e.g., the region of interest (1111) of FIG. 11)) based on the first tag. The electronic device (e.g., the electronic device (1100) of FIG. 11) may perform first image processing (e.g., the focus control operation (1202) of FIG. 12) using the first pixel values. An electronic device (e.g., an electronic device (1100) of FIG. 11) can obtain a second frame based on a result of performing a first image processing (e.g., a focus control operation (1202) of FIG. 12).

[0307] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain first line data (e.g., the first line data (1340) of FIG. 13) including a second tag regarding a position of a first pixel line (e.g., the first pixel line (1121) of FIG. 11) among pixel lines (e.g., the pixel lines (1120) of FIG. 11) of a first image sensor. The electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain a first frame (e.g., the first image (1290) of FIG. 12) by performing second image processing based on the second tag.

[0308] According to one embodiment, an electronic device (e.g., an electronic device (1100) of FIG. 11) may perform an auto-focus function by controlling a lens module (e.g., a lens unit (381) of FIG. 3) of a first camera module (e.g., a camera module (1101) of FIG. 11) for a region of interest using first line data (e.g., a pixel value (1212) of FIG. 12).

[0309] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain first pixel values ​​of a region of interest (e.g., the region of interest (1111) of FIG. 11) from first line data (e.g., the pixel values ​​(1212) of FIG. 12). The electronic device (e.g., the electronic device (1100) of FIG. 11) can identify brightness of the region of interest (e.g., the region of interest (1111) of FIG. 11)) based on the first pixel values. The electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain a second frame by controlling an exposure time of a first image sensor based on the brightness of the region of interest (e.g., the region of interest (1111) of FIG. 11).

[0310] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain first pixel values ​​of a region of interest (e.g., the region of interest (1111) of FIG. 11) from first line data (e.g., the pixel values ​​(1212) of FIG. 12). The electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain information about a color of the region of interest (e.g., the region of interest (1111) of FIG. 11)) based on the first pixel values. The electronic device (e.g., the electronic device (1100) of FIG. 11) can adjust a color of a first frame (e.g., the first image (1290) of FIG. 12) based on the information about the color of the region of interest (e.g., the region of interest (1111) of FIG. 11).

[0311] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can define a first color temperature and a first hue based on information about a color of a region of interest (e.g., the region of interest (1111) of FIG. 11). The electronic device (e.g., the electronic device (1100) of FIG. 11) can adjust a color of a first frame (e.g., the first image (1290) of FIG. 12) based on the first color temperature and the first hue.

[0312] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain first pixel values ​​of a region of interest (e.g., the region of interest (1111) of FIG. 11) from first line data (e.g., the pixel values ​​(1212) of FIG. 12). The electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain information about an object (e.g., output data (1440) of FIG. 14) output from the artificial intelligence model (e.g., the artificial intelligence model (1430) of FIG. 14)) by applying the first pixel values ​​to an artificial intelligence model (e.g., the artificial intelligence model (1430) of FIG. 14) that identifies an object from an image. The electronic device (e.g., the electronic device (1100) of FIG. 11) can obtain a second frame based on the information about the object (e.g., the output data (1440) of FIG. 14).

[0313] According to one embodiment, an electronic device (e.g., the electronic device (1100) of FIG. 11) may further include a second camera module (e.g., the second camera module (1802) of FIG. 18) including a second image sensor. The electronic device (e.g., the electronic device (1100) of FIG. 11) may obtain a second control signal that causes the first camera module (e.g., the first camera module (1801) of FIG. 18) to be in a first state and the second camera module (e.g., the second camera module (1802) of FIG. 18) to be in a second state. The electronic device (e.g., the electronic device (1100) of FIG. 11) may identify second coordinates of a region of interest (e.g., the second region of interest (1821) of FIG. 18) on the second image sensor. An electronic device (e.g., an electronic device (1100) of FIG. 11) may, in response to a second control signal, obtain third line data (e.g., third line data (1851) of FIG. 18) from a third pixel line including at least a portion of a region of interest (e.g., a second region of interest (1821) of FIG. 18) based on second coordinates among pixel lines of a second image sensor. The electronic device (e.g., an electronic device (1100) of FIG. 11) may perform third image processing (e.g., a focus control operation (1202) of FIG. 12) using the third line data (e.g., the third line data (1851) of FIG. 18). An electronic device (e.g., an electronic device (1100) of FIG. 11) can obtain a third control signal that causes a second camera module (e.g., a second camera module (1802) of FIG. 18) to be in a first state and a first camera module (e.g., a first camera module (1801) of FIG. 18) to be in a second state.An electronic device (e.g., an electronic device (1100) of FIG. 11) may perform fourth image processing in response to a third control signal by using third line data (e.g., the third line data (1891) of FIG. 18) and fourth line data (e.g., the fourth line data (1892) of FIG. 18) obtained from a fourth pixel line different from the third pixel line among the pixel lines of the second image sensor, thereby obtaining a third frame (e.g., the third frame (1820b) of FIG. 18). The electronic device (e.g., the electronic device (1100) of FIG. 11) may display the third frame (e.g., the third frame (1820b) of FIG. 18).

[0314] An operating method of an electronic device according to one embodiment may include an operation of identifying first coordinates of a region of interest on a first image sensor of a first camera module. The operating method may include an operation of obtaining a first control signal for obtaining image data including the region of interest. The operating method may include an operation of obtaining, in response to the first control signal, first line data from a first pixel line including at least a portion of the region of interest based on the first coordinates among pixel lines of the first image sensor. The operating method may include an operation of performing first image processing using the first line data. The operating method may include an operation of obtaining second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor. The operating method may include an operation of performing second image processing using the first line data and the second line data to obtain a first frame.

[0315] According to one embodiment, the operation of obtaining the first line data may include the operation of obtaining the first line data including the first tag relating to the region of interest. The operation of performing the first image processing may include the operation of obtaining first pixel values ​​corresponding to the region of interest based on the first tag. The operation of performing the first image processing may include the operation of performing the first image processing using the first pixel values. The operating method may include the operation of obtaining the second frame based on the result of performing the first image processing.

[0316] According to one embodiment, the operation of acquiring the first line data may include the operation of acquiring the first line data including a second tag relating to a position of a first pixel line among pixel lines of the first image sensor. The operation of acquiring the first frame may include the operation of acquiring the first frame by performing second image processing based on the second tag.

[0317] According to one embodiment, the operating method may include an operation of performing an auto-focus function by controlling a lens module of a first camera module for a region of interest using first line data.

[0318] According to one embodiment, the operation of performing the first image processing may include an operation of obtaining first pixel values ​​of a region of interest from the first line data. The operation of performing the first image processing may include an operation of identifying brightness of the region of interest based on the first pixel values. The method of operation may include an operation of obtaining a second frame by controlling an exposure time of the first image sensor based on the brightness of the region of interest.

[0319] According to one embodiment, the operation of performing the first image processing may include an operation of obtaining first pixel values ​​of a region of interest from the first line data. The operation of performing the first image processing may include an operation of obtaining information about a color of the region of interest based on the first pixel values. The operation of obtaining the first frame may include an operation of adjusting a color of the first frame based on the information about the color of the region of interest.

[0320] In one embodiment, the act of performing the first image processing may include the act of defining a first color temperature and a first hue based on information about the color of the region of interest. The act of acquiring the first frame may include the act of adjusting the color of the first frame based on the first color temperature and the first hue.

[0321] According to one embodiment, the operation of performing the first image processing may include obtaining first pixel values ​​of a region of interest from the first line data. The operation of performing the first image processing may include obtaining information about the object output from the artificial intelligence model by applying the first pixel values ​​to an artificial intelligence model that identifies the object from the image. The method of operation may include obtaining a second frame based on the information about the object.

[0322] In one embodiment, the operation of identifying the first coordinate may include obtaining a second control signal that causes the first camera module to be in a first state and the second camera module including the second image sensor to be in a second state. The operation of identifying the first coordinate may include identifying a second coordinate of a region of interest on the second image sensor. The operating method may include obtaining, in response to the second control signal, third line data from a third pixel line including at least a portion of the region of interest, based on the second coordinate, among pixel lines of the second image sensor. The operating method may include performing third image processing using the third line data. The operating method may include obtaining a third control signal that causes the second camera module to be in the first state and the first camera module to be in the second state. The operating method may include performing, in response to the third control signal, fourth image processing using the third line data and fourth line data obtained from a fourth pixel line different from the third pixel line among the pixel lines of the second image sensor, thereby obtaining a third frame. The method of operation may include an operation of displaying a third frame.

[0323] A non-transitory computer-readable recording medium having recorded thereon instructions for controlling an electronic device according to one embodiment may include instructions for identifying first coordinates of a region of interest on a first image sensor of a first camera module. The recording medium may include instructions for obtaining a first control signal for obtaining a first frame including the region of interest. The recording medium may include instructions for obtaining, in response to the first control signal, first line data from a first pixel line including at least a portion of the region of interest based on the first coordinates among pixel lines of the first image sensor. The recording medium may include instructions for performing first image processing using the first line data. The recording medium may include instructions for obtaining second line data from a second pixel line different from the first pixel line among pixel lines of the first image sensor. The recording medium may include instructions for performing second image processing using the first line data and the second line data to obtain a first frame.

[0324] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0325] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together with the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0326] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0327] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0328] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0329] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0330] It will be appreciated that the various embodiments according to the claims and description of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0331] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), which include computer-executable instructions that, when executed alone or in combination by one or more processors of the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0332] Such software may be stored in a volatile or non-volatile storage device (e.g., a storage device such as read-only memory (ROM), whether erasable or rewritable), or in a memory form such as a random access memory (RAM), a memory chip, device, or integrated circuit, or in an optically or magnetically readable medium such as a compact disc (CD), a digital versatile disc (DVD), a magnetic disk, or a magnetic tape. It will be appreciated that the storage device and the storage medium are various embodiments of non-transitory machine-readable storage devices suitable for storing a computer program or computer programs that, when executed, include instructions for implementing various embodiments of the present disclosure. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as claimed in one of the claims of the present disclosure, and a non-transitory machine-readable storage device storing such a program.

[0333] While the present disclosure has been illustrated and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In electronic devices, A first camera comprising a first image sensor; At least one processor comprising a processing circuit; Memory that stores instructions; The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Identifying a first coordinate of an area of ​​interest on the first image sensor, Obtain a first control signal for obtaining image data including the region of interest, In response to the first control signal, first line data is obtained from a first pixel line including at least a portion of the region of interest based on the first coordinate among pixel lines of the first image sensor, Using the first line data, first image processing is performed, Among the pixel lines of the first image sensor, second line data is acquired from a second pixel line different from the first pixel line, By performing second image processing using the first line data and the second line data, a first frame is obtained. Electronic devices.

2. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: In response to the first control signal, all pixel lines of the first image sensor are exposed for a first time at a first time, To sequentially acquire line data from pixel lines of the first image sensor, Electronic devices.

3. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtaining the first line data including the first tag regarding the area of ​​interest, Based on the first tag, first pixel values ​​corresponding to the region of interest are obtained, Using the first pixel values, the first image processing is performed, Based on the result of performing the first image processing, a second frame is obtained. Electronic devices.

4. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtaining the first line data including a second tag regarding the position of the first pixel line among the pixel lines of the first image sensor, By performing the second image processing based on the second tag, the first frame is obtained. Electronic devices.

5. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: By controlling the lens of the first camera module for the region of interest using the first line data, an auto focus function is performed. Electronic devices.

6. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtaining the first pixel values ​​of the region of interest from the first line data, Based on the first pixel values, identify the brightness of the region of interest, By controlling the exposure time of the first image sensor based on the brightness of the region of interest, a second frame is acquired. Electronic devices.

7. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtaining the first pixel values ​​of the region of interest from the first line data, Based on the first pixel values, information about the color of the region of interest is obtained, Adjusting the color of the first frame based on information about the color of the region of interest; Electronic devices.

8. In paragraph 7, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Define a first color temperature and a first tint based on information about the color of the above area of ​​interest, Adjusting the color of the first frame based on the first color temperature and the first tone, Electronic devices.

9. In paragraph 1, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtaining the first pixel values ​​of the region of interest from the first line data, By applying the above first pixel values ​​to an artificial intelligence model that identifies an object from an image, information about the object output from the artificial intelligence model is obtained, To obtain a second frame based on information about the above object, Electronic devices.

10. In paragraph 1, The electronic device further comprises a second camera comprising a second image sensor, The at least one processor, individually or collectively, executes the instructions, thereby causing the electronic device to: Obtain a second control signal that causes the first camera to be in a first state and the second camera to be in a second state; Identifying the second coordinates of the region of interest on the second image sensor, In response to the second control signal, third line data is acquired from a third pixel line including at least a portion of the region of interest based on the second coordinate among the pixel lines of the second image sensor, Using the above third line data, third image processing is performed, Obtain a third control signal that causes the second camera to be in the first state and the first camera to be in the second state; In response to the third control signal, a fourth image processing is performed using the third line data and fourth line data obtained from a fourth pixel line different from the third pixel line among the pixel lines of the second image sensor, thereby obtaining a third frame. To display the third frame above, Electronic devices.

11. In a method performed by an electronic device, An operation of identifying a first coordinate of an area of ​​interest on a first image sensor of a first camera; An operation of obtaining a first control signal for obtaining image data including the region of interest; In response to the first control signal, an operation of acquiring first line data from a first pixel line including at least a portion of the region of interest based on the first coordinate among the pixel lines of the first image sensor; An operation of performing first image processing using the first line data; An operation of acquiring second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor; and An operation of obtaining a first frame by performing second image processing using the first line data and the second line data, method.

12. In paragraph 11, The operation of acquiring the above first line data is as follows: An operation of obtaining the first line data including the first tag regarding the area of ​​interest, The operation of performing the above first image processing is: An operation of obtaining first pixel values ​​corresponding to the region of interest based on the first tag; and An operation of performing the first image processing using the first pixel values ​​is included, The above method An operation of obtaining a second frame based on the result of performing the first image processing, method.

13. In paragraph 11, The operation of acquiring the above first line data is An operation of obtaining the first line data including a second tag regarding the position of the first pixel line among the pixel lines of the first image sensor, The operation of obtaining the above first frame is An operation of obtaining the first frame by performing the second image processing based on the second tag, method.

14. In paragraph 11, The above method, An operation of performing an auto-focus function by controlling a lens of the first camera for the region of interest using the first line data, method.

15. A computer-readable, non-transitory recording medium having recorded thereon a command for controlling an electronic device, wherein the command, when individually or collectively executed by one or more processors of the electronic device, causes the electronic device to perform operations, the operations being: An operation of identifying a first coordinate of an area of ​​interest on a first image sensor of a first camera; An operation of obtaining a first control signal for obtaining a first frame including the region of interest; In response to the first control signal, an operation of acquiring first line data from a first pixel line including at least a portion of the region of interest based on the first coordinate among the pixel lines of the first image sensor; An operation of performing first image processing using the first line data; An operation of acquiring second line data from a second pixel line different from the first pixel line among the pixel lines of the first image sensor; and An operation of obtaining the first frame by performing second image processing using the first line data and the second line data, Recording medium.

Images