Image sensor, electronic device comprising image sensor, and operation method thereof

Abstract

According to an embodiment, an electronic device may comprise a camera, at least one processor, and a memory. The camera may comprise an image sensor comprising a plurality of sensor pixels arranged along a plurality of pixel lines. The at least one processor may acquire, from the image sensor, sensor driving information related to an operation of the image sensor. The at least one processor may determine at least one of area of interest information and exposure control information on the basis of the sensor driving information. The at least one processor may transmit, to the image sensor through control communication, a control signal comprising at least one of the determined area of interest information and the determined exposure control information. The image sensor may be configured to read pixel values from at least one pixel line corresponding to the area of interest information from among the plurality of pixel lines, on the basis of the exposure control information. The image sensor may be configured to output, to the at least one processor, a second image frame obtained by cropping an area corresponding to the area of interest information from among the read pixel values.

Description

Image sensor, electronic device including image sensor, and method of operating same

[0001] The present disclosure relates to an image sensor, an electronic device including the image sensor, and an operating method thereof.

[0002] An electronic device can obtain an image frame from the output of an image sensor that converts an optical signal into an electrical signal. The image sensor can include a color filter and a photodetector. The photodetector can receive light that has passed through a color filter corresponding to a specific color and output an electrical signal corresponding to the received light. For example, the photodetector can include a photodiode. The electronic device can obtain image data including a set of pixel values ​​determined based on the values ​​output from the image sensor. The image quality of the image data can be affected by the number of pixels included in the image data. Devices and methods for obtaining image data with a high pixel count are being developed to improve the image quality.

[0003] Since the photodetector receives light that has passed through a color filter of a specific color, information detected through one photodetector can correspond to one color. The color filter plays a role in allowing the final acquired image to express color, but it can also cause the final acquired image to be expressed with a lower resolution than the total number of photodetectors arranged in the image sensor. Color reproduction and resolution can be in a conflicting relationship. The Bayer pattern has been widely used as a pattern to accurately reproduce colors while reducing the loss of resolution. For example, U.S. Patent Publication No. US 3,971,065 discloses a color filter array (CFA) having a Bayer pattern.

[0004] A camera including an image sensor can capture images based on the field of view (FOV) that the camera can support. The FOV may refer to the range that the camera can capture through a lens included in the camera. For example, a lens may be referred to as an ultra-wide angle lens, a wide angle lens, a standard lens, or a telephoto lens, in descending order of the angle that constitutes the FOV based on the position of the lens. A camera equipped with each of these lenses may be referred to as an ultra-wide angle camera, a wide angle camera, a standard camera, or a telephoto camera.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0006] In one embodiment, an electronic device may include a camera, at least one processor, and a memory. The camera may include an image sensor including a plurality of sensor pixels arranged along a plurality of pixel lines. The at least one processor may include processing circuitry. The memory may store instructions. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to obtain a first image frame from the image sensor. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to obtain sensor operating information related to the operation of the image sensor from the image sensor. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to determine at least one of region-of-interest information or exposure control information based on the sensor operating information. The above instructions may be individually or collectively executed by the at least one processor to cause the electronic device to transmit a control signal including at least one of the determined region of interest information or the determined exposure control information from the at least one processor to the image sensor via control communication. The image sensor may be configured to read pixel values ​​from at least one pixel line corresponding to the region of interest information among the plurality of pixel lines based on the exposure control information.The image sensor may be configured to output a second image frame, which is a cropped area corresponding to the region of interest information among the read pixel values, to the at least one processor.

[0007] A method of operating an electronic device including an image sensor including a plurality of sensor pixels arranged along a plurality of pixel lines and at least one processor according to one embodiment may include obtaining a first image frame from the image sensor. The method may include obtaining, by the at least one processor, sensor driving information associated with the first image frame from the image sensor. The method may include determining, based on the sensor driving information, at least one of region of interest information or exposure control information. The method may include transmitting, by the at least one processor, a control signal including at least one of the determined region of interest information or the determined exposure control information to the image sensor via control communication. The method may include reading, by the image sensor, pixel values ​​from at least one pixel line corresponding to the region of interest information among the plurality of pixel lines. The method may include outputting, by the image sensor, a second image frame obtained by cropping an area corresponding to the region of interest information among the read pixel values ​​to the at least one processor.

[0008] In one embodiment, a computer-readable non-transitory recording medium may have recorded thereon a computer program for performing the above-described method.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0012] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment.

[0013] FIG. 5 is a diagram illustrating examples of patterns of light-receiving elements, micro lenses, and color filters arranged in an image sensor according to one embodiment.

[0014] FIG. 6 is a circuit diagram illustrating an example of a circuit for outputting values ​​of pixels included in an image sensor according to one embodiment.

[0015] FIG. 7 is a diagram illustrating an example of a color pattern of pixels included in an image acquired through an image sensor according to one embodiment.

[0016] FIG. 8 is a diagram illustrating an example of a color pattern of pixels included in an image acquired through an image sensor according to one embodiment.

[0017] FIG. 9 is a diagram illustrating an example of a color pattern of pixels included in an image acquired through an image sensor according to one embodiment.

[0018] FIG. 10 is a diagram illustrating an example of an image provided by an electronic device according to one embodiment based on an ROI according to object tracking.

[0019] FIG. 11 is a diagram illustrating an example of pixel lines and a region of interest for an image sensor according to one embodiment.

[0020] FIG. 12 illustrates an example of an operation of an image sensor of an electronic device according to one embodiment to read pixel lines according to an operation mode.

[0021] Figure 13 illustrates an example of an operation in which an image sensor reads pixel lines to acquire image frames according to a region of interest.

[0022] FIG. 14 is a block diagram illustrating the structure of an electronic device that displays an image acquired using two or more image sensors, in one embodiment.

[0023] FIG. 15 is a flowchart illustrating a process by which an electronic device acquires an image frame based on sensor actuation information, in one embodiment.

[0024] FIG. 16 illustrates an example of a restricted area determined by an electronic device in one embodiment.

[0025] FIG. 17 illustrates an example of a maximum exposure value determined when the starting position of the ROI to be changed is within the restricted area in one embodiment.

[0026] FIG. 18 is a flowchart illustrating a process by which an electronic device obtains sensor driving information from a packet containing an image frame and controls an image sensor, in one embodiment.

[0027] FIG. 19 illustrates an example of a packet output by an image sensor of an electronic device to at least one processor, in one embodiment.

[0028] FIG. 20 illustrates an example of sensor drive information included in at least one of a header or footer of a packet, in one embodiment.

[0029] FIG. 21 is a flowchart illustrating a process in which an electronic device obtains sensor driving information through control communication and controls an image sensor in one embodiment.

[0030] FIG. 22 illustrates an example of stored sensor drive information that an electronic device can obtain via control communication, in one embodiment.

[0031] FIG. 23 illustrates an example of an electronic device adjusting an ROI in one embodiment.

[0032] FIG. 24 illustrates an example of an electronic device adjusting an ROI in one embodiment.

[0033] FIG. 25 is a flowchart illustrating a process by which an image sensor controls an ROI or exposure value in one embodiment.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0035] In the present disclosure, the term 'pixel' may refer to the smallest unit constituting a digital image. The resolution of an image may be expressed by the number of pixels included in the image. For example, if an image is composed of axb pixels arranged in a rows and b columns, the resolution of the image may be indicated as axb. In the present disclosure, the term 'sensor pixel' may refer to a component that becomes a unit by which an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, and the second image sensor (384) of FIG. 3) can obtain a pixel value through a light-receiving element. The pixel value mentioned in the present disclosure may also include a pixel address value indicating the location of the pixel.

[0036] An image sensor, an electronic device including the image sensor, and an operating method thereof according to one embodiment may be configured to prevent a phenomenon in which at least some image frames are dropped when regions of interest (ROI) for image frames output from the image sensor are different. In the present disclosure, "dropping of image frames" may mean a phenomenon in which image frames are not output or the output of image frames is slowed down due to overlapping exposure sections or readout sections of the image sensor.

[0037] An image sensor, an electronic device including the image sensor, and an operating method thereof according to one embodiment may be for improving the quality of an image acquired for a region of interest.

[0038] An image sensor, an electronic device including the image sensor, and an operating method thereof according to one embodiment may be for improving a frame rate, which means the number of image frames output per unit time.

[0039] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure pertains.

[0040] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0041] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0042] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0043] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0044] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0045] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0046] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0047] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0048] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0049] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0050] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0051] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0052] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0053] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0054] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0055] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0056] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0057] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0058] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0059] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0060] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0061] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0062] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0063] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0064] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0065] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0066] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0067] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0068] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0069] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0070] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0071] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0072] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0073] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment.

[0074] In one embodiment, the electronic device (101) may include at least one processor (32) (e.g., processor 120 of FIG. 1, image signal processor 260 of FIG. 2), memory (330) (e.g., memory 130 of FIG. 1, memory 250 of FIG. 2) and at least one camera (e.g., camera module (180) of FIGS. 1 and 2). The at least one camera may include a first camera (381) including a first image sensor (383). The at least one camera may further include a second camera (382) including a second image sensor (384), but the at least one camera may be configured as a single camera. The at least one camera may include three or more cameras. The memory may store instructions that the at least one processor individually or collectively executes to operate the electronic device. In the present disclosure, the operation performed by the electronic device (101) may be understood as being performed by at least one processor executing instructions stored in a memory to perform a calculation or controlling a component of the electronic device (101). In one embodiment, the at least one processor (320) may include at least one of a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, the at least one processor (320) may include an application processor configured to perform a plurality of functions.At least one processor (320) can acquire image data based on an image frame including information read from at least one of the first image sensor (383) or the second image sensor (384).

[0075] In one embodiment, at least one processor (320) may control the operation of at least one of the first image sensor (383) or the second image sensor (384). The electronic device (101) may acquire an image frame based on pixel values ​​acquired from pixels arranged in a certain area among a plurality of sensor pixels included in the first image sensor (383). The certain area where the sensor pixels for acquiring the image frame are arranged may be referred to as a region of interest (ROI). The at least one processor (320) may determine a region of interest for the image sensor to read out the image frame. For example, the at least one processor (320) may determine a region including a subject in an image acquired through the first camera (381) or the second camera (382) as a region of interest. The at least one processor (320) may control the first image sensor (383) to read out a pixel value from at least one sensor pixel arranged in an area corresponding to the region of interest. For example, at least one processor (320) can transmit a control signal related to a region of interest to the first image sensor (383) based on control communication (e.g., I2C (inter-integrated circuit) communication, I3C (improved inter-integrated circuit) communication). The first image sensor (383) can output an image frame read from a sensor pixel arranged in an area corresponding to the region of interest within a pixel array included in the first image sensor (383). For example, the first image sensor (383) can transmit image data through an interface (e.g., MIPI (mobile industry processor interface)) connected to at least one processor (320). The at least one processor (320) can obtain an image that enlarges the region of interest from the image frame output from the first image sensor (383).

[0076] In one embodiment, the electronic device (101) may further include a display (360) (e.g., the display module (160) of FIG. 1). At least one processor (320) may control the display (360) to display the acquired image. If the display (360) includes a touchscreen, the electronic device (101) may receive a touch input corresponding to a position of the displayed screen. The at least one camera may further include a second camera (382) including a second image sensor (384). The at least one processor (320) may acquire an image frame through the first camera (381) and may also acquire an image frame through the second camera (382). The at least one processor (320) may control the display (360) to display a preview screen generated based on the image acquired through the first camera (381) and the image acquired through the second camera (382). For example, the electronic device (101) may display a preview screen including an image acquired through the first camera (381) displayed in a portion of the screen together with an image acquired through the second camera (382). In one embodiment, the first camera (381) may support a first field of view (FOV), and the second camera (382) may support a second FOV that is different from the first FOV. For example, the first camera may include a wide angle camera, and the second camera may include an ultra-wide angle camera that supports a wider FOV than the first camera. The second FOV may be wider than the first FOV, but is not limited thereto.

[0077] In one embodiment, the first image sensor (383) may include a pixel array including a plurality of sensor pixels. The first image sensor (383) may perform an operation of reading out a pixel value in units of pixel lines, which represent a group of sensor pixels arranged in one line among the plurality of sensor pixels. The operation of reading out a pixel value from a sensor pixel may include a reset operation (or shutter operation) of resetting the sensor pixel, an exposure operation of accumulating a charge generated by light received by a light-receiving element of the sensor pixel, and a readout operation of obtaining the accumulated charge. The first image sensor (383) may sequentially read out at least some of the plurality of pixel lines included in the first image sensor (383) in units of pixel lines. However, the present invention is not limited thereto. For example, the first image sensor (383) may also perform the readout operation in units of some of the pixel lines.

[0078] In one embodiment, the first image sensor (383) may output an image sensor having a higher resolution than the second image sensor (384). For example, the first image sensor (383) may output an image tracking an area where an object is displayed by outputting an image sensor having a higher resolution based on a region of interest including an area where an object is identified within an image frame acquired through the second image sensor (384). However, the resolution of the image output from the first image sensor (383) and the resolution of the image output from the second image sensor (384) are not limited thereto.

[0079] FIG. 4 is a diagram illustrating a configuration of an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment.

[0080] In one embodiment, the image sensor may include a micro lens array (MLA) (411), a color filter (413), a light receiving unit (415), and a computation unit (417).

[0081] In one embodiment, the microlens array (411) may be configured such that a light bundle (421) that passes through the lens unit (e.g., the lens assembly (210) of FIG. 2) and forms an image on an image sensor is focused on a light-receiving element of the light-receiving unit (415). The light bundles (423) that pass through the microlens array (411) may have at least a portion of wavelengths other than a band corresponding to a specific color blocked as they pass through the color filter (413). The light bundles (425) that pass through the color filter (413) may be detected by a light-receiving element (e.g., a photodiode) of the light-receiving unit (415). The light-receiving unit (415) may include a light-receiving element that generates a charge when receiving light and converts it into an electrical signal, and a circuit that selectively reads out the charge of the light-receiving element. A circuit for digitizing the signal read from the light receiving unit (415) or reducing noise may be placed between the light receiving unit (415) and the calculation unit (417).

[0082] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light bundle (421) is incident, an area in which a single microlens included in the microlens array (411) is arranged may at least partially overlap an area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.

[0083] In one embodiment, the calculation unit (417) can perform an operation to process electrical data (427) output from the light receiving unit (415). The calculation unit (417) can output data acquired based on the calculation result. The output of the calculation unit (417) can be an output of an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3). In one embodiment, the calculation unit (417) can perform an operation to calibrate the read data as an operation to process the electrical data (427). For example, the operation performed by the operation unit (417) may include at least one of an operation for reducing deviation between pixels due to optical asymmetry or relative positions of sensors, an operation for reducing noise generated in an analog signal, an operation for removing defects, an operation for performing re-mosaic, or an operation according to a specific application field (e.g., proximity sensor function, timing adjustment function, HDR (high dynamic range) tone mapping function). The sensor output (429) output from the operation unit (417) may be input to at least one processor (e.g., application processor) through an interface.

[0084] In FIG. 4, the color pattern of the color filter (413) is illustrated based on the Bayer pattern, but the color pattern of the color filter (413) according to one embodiment is not limited to that illustrated in FIG. 4. Areas within the color filter (413) corresponding to a plurality of adjacent micro lenses may be configured to include the same color channel. For example, an image sensor including a color filter (413) may include a structure in which the pattern illustrated in FIG. 5 is repeated.

[0085] FIG. 5 is a diagram illustrating examples of patterns of a light-receiving element (e.g., a light-receiving element included in a light-receiving unit (415) of FIG. 4)), a microlens (e.g., a microlens included in a microlens array (411) of FIG. 4), and a color filter (e.g., a color filter (413) of FIG. 4) disposed in an image sensor (e.g., an image sensor (230) of FIG. 2, a first image sensor (383) of FIG. 3, a second image sensor (384) of FIG. 3) according to one embodiment.

[0086] In one embodiment, an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may be configured to output images with different resolutions depending on the operating mode in which the image sensor operates. The pattern in which the microlenses, light-receiving elements, and color filters included in the image sensor are arranged may be configured in various ways. For example, the image sensor may include a pattern in which any one of the first sensor pattern (501), the second sensor pattern (502), or the third sensor pattern (503) is repeated. However, the patterns illustrated in FIG. 5 are only examples for describing one embodiment and are not limited thereto. The image sensor may output an image with a resolution depending on the operating mode based on the pattern. Each light-receiving element may include an element (e.g., a photodiode) that can detect light and output an electrical signal. At least one photodetector included in the image sensor may be arranged in an area corresponding to one microlens. In Fig. 5, the area corresponding to each photodetector may be referred to as a sensor pixel.

[0087] According to one embodiment, in the first sensor pattern (501), four light-receiving elements (511, 512, 513, 514) arranged in a 2 x 2 array may be arranged to correspond to one micro lens (520). The micro lens (520) within the first sensor pattern (501) may be arranged in an area corresponding to one color channel (e.g., a green channel) of a color filter. Within the first sensor pattern (501), another micro lens arranged adjacent to the micro lens (520) may be arranged in an area corresponding to a color of the color filter corresponding to the area where the micro lens (520) is arranged and another color channel (e.g., a red channel, a blue channel) of the color filter.

[0088] According to one embodiment, in the second sensor pattern (502), nine light-receiving elements (551, 552, 553, 554, 555, 556, 557, 558, 559) arranged in a 3 x 3 array may be arranged to correspond to one micro lens (560). The micro lens (560) within the second sensor pattern (502) may be arranged in an area corresponding to one color channel (e.g., a green channel) of a color filter. Another micro lens arranged adjacent to the micro lens (560) within the second sensor pattern (502) may be arranged in an area corresponding to a color channel (e.g., a red channel, a blue channel) different from the color channel of the color filter corresponding to the area where the micro lens (560) is arranged.

[0089] According to one embodiment, in the third sensor pattern (503), four micro lenses (591, 592, 593, 594) arranged in a 2 x 2 array may be arranged in the same color channel (e.g., green channel) of the color filter. The light-receiving elements may include four light-receiving elements arranged in a 2 x 2 array and arranged to correspond to one micro lens. For example, in the third sensor pattern (503), four light-receiving elements (571, 572, 573, 574) may be arranged to correspond to the micro lens (591). Four light-receiving elements (575, 576, 577, 578) may be arranged to correspond to the micro lens (592). Four light-receiving elements (579, 580, 581, 582) may be arranged to correspond to the micro lens (593). Four photodetectors (583, 584, 585, 586) can be arranged to correspond to the micro lens (594).

[0090] In one embodiment, the pattern of the image sensor is not limited to the patterns illustrated in FIG. 5. For example, the pattern of the image sensor may be replaced with another pattern that can operate by changing the resolution.

[0091] FIG. 6 is a circuit diagram illustrating an example of a circuit for outputting values ​​of pixels included in an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment.

[0092] In one embodiment, a pixel circuit constituting sensor pixels included in an image sensor may include a plurality of at least one photodiode (611, 612, 613, 614, 615, 616, 617, 618). For example, four photodiodes (611, 612, 613, 614) having a 2 x 2 array arranged to correspond to one micro lens may correspond to the light-receiving elements (511, 512, 513, 514) included in the first pattern (501) of FIG. 5. For example, four photodiodes (611, 612, 613, 614) having a 2 x 2 array arranged to correspond to one micro lens may correspond to light-receiving elements (571, 572, 573, 574) included in the third pattern (503) of FIG. 5. For example, four photodiodes (615, 616, 617, 618) having a 2 x 2 array arranged to correspond to one micro lens may correspond to light-receiving elements (575, 576, 577, 578) included in the third pattern (503) of FIG. 5. However, FIG. 6 merely describes an example of a pixel circuit for explaining one embodiment, and the configuration of the pixel circuit is not limited to that illustrated in FIG. 6. In one embodiment, five or more photodiodes may be arranged to correspond to one micro lens. For example, nine photodiodes in a 3x3 array may be arranged to correspond to one microlens.

[0093] Referring to FIG. 6, four photodiodes (e.g., a first photodiode (611), a second photodiode (612), a third photodiode (613), and a fourth photodiode (614)) arranged to correspond to one micro lens and four photodiodes (e.g., a fifth photodiode (615), a sixth photodiode (616), a seventh photodiode (17), and an eighth photodiode (618)) arranged to correspond to another micro lens may be connected to one floating diffusion node (630). Switches (621, 622, 623, 624, 625, 626, 627, and 628) may be connected between each of the photodiodes (611, 612, 613, 614, 615, 616, 617, and 618) and the floating node (630). For example, the switches (621, 622, 623, 624, 625, 626, 627, 628) may include transistors (e.g., transfer gates).

[0094] According to one embodiment, charge may be accumulated in the photodiodes (611, 612, 613, 614, 615, 616, 617, 618) during the exposure time. While the charge is accumulated, the switch may be kept in an off state so that the photodiode and the floating node (630) are not connected. When the switch is turned on, the photodiode is connected to the floating node (630), and the accumulated charge may move to the floating node (630). For example, when the first switch (621) is turned on, the charge accumulated in the first photodiode (611) may move to the floating node (630). The charge stored in the floating node (630) may be read out through the source follower (SF) (650) and output as an electrical signal. An image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) can acquire analog data corresponding to the charge transferred to the floating node (630). For example, the analog data can include information about the amount of charge accumulated in at least one photodiode during the exposure time.

[0095] In one embodiment, a line selector (SEL) (660) can be controlled to be on or off to output analog data for a selected line. The line selector (660) can be controlled to be on to read out the values ​​of sensor pixels connected to the floating node (630) according to the readout order.

[0096] In one embodiment, the image sensor can obtain analog data corresponding to pixel values ​​of raw image data. For example, the image sensor can obtain analog data corresponding to light quantity data obtained through at least one of the photodiodes (611, 612, 613, 614, 615, 616, 617, 618) by controlling the switches (621, 622, 623, 624, 625, 626, 627, 628). When the image sensor turns on the first switch (621), the image sensor can obtain analog data based on the light quantity data obtained through the first photodiode (611). The image sensor can convert the obtained analog data into digital data through an analog-digital converter (ADC) to obtain a pixel value.

[0097] In one embodiment, the image sensor may perform a reset operation to remove charges accumulated in the floating node (630) by turning on a reset switch (e.g., a reset transistor (RST) (670)) after acquiring analog data. The image sensor may perform the reset operation by sequentially turning on the switches (621, 622, 623, 624, 625, 626, 627, 628) one by one with a time difference, thereby reading out the output of each of the photodiodes (611, 612, 613, 614, 615, 616, 617, 618) as a single pixel value. After performing the reset operation by turning on the reset transistor (670), the image sensor may turn on the first switch (621) to transfer the charges accumulated by the first photodiode (611) to the floating node (630). When the charge output by the first photodiode (611) is accumulated in the floating node (630), if the line selector (660) is turned on, the image sensor can read out a voltage value corresponding to the charge accumulated by the first photodiode (611) through the source follower (650). The image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can transfer the charge accumulated by the second photodiode (612) to the floating node (630) by turning on the second switch (622). When the charge output by the second photodiode (612) is accumulated in the floating node (630), if the line selector (660) is turned on, the image sensor can read out a voltage value corresponding to the charge accumulated by the second photodiode (612) through the source follower (650). Afterwards, the image sensor can perform a reset operation to remove the charge accumulated in the floating node by turning on the reset transistor (670).Thereafter, the image sensor can turn on the third switch (623) to transfer the charge accumulated by the third photodiode (613) to the floating node (630). In a state where the charge output by the third photodiode (613) is accumulated in the floating node (630), if the line selector (660) is on, the image sensor can read out a voltage value corresponding to the charge accumulated by the third photodiode (613) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can turn on the fourth switch (624) to transfer the charge accumulated by the fourth photodiode (614) to the floating node (630). When the charge output by the fourth photodiode (614) is accumulated in the floating node (630), and the line selector (660) is in the on state, the image sensor can read out a voltage value corresponding to the charge accumulated by the fourth photodiode (614) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can turn on the fifth switch (625) to transfer the charge accumulated by the fifth photodiode (615) to the floating node (630). When the charge output by the fifth photodiode (615) is accumulated in the floating node (630), and the line selector (660) is in the on state, the image sensor can read out a voltage value corresponding to the charge accumulated by the fifth photodiode (615) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can turn on the sixth switch (626) to transfer the charge accumulated by the sixth photodiode (616) to the floating node (630).When the charge output by the sixth photodiode (616) is accumulated in the floating node (630), and the line selector (660) is in the on state, the image sensor can read out a voltage value corresponding to the charge accumulated by the sixth photodiode (616) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can turn on the seventh switch (627) to transfer the charge accumulated by the seventh photodiode (617) to the floating node (630). When the charge output by the seventh photodiode (617) is accumulated in the floating node (630), if the line selector (660) is in the on state, the image sensor can read out a voltage value corresponding to the charge accumulated by the seventh photodiode (617) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). Thereafter, the image sensor can turn on the eighth switch (628) to transfer the charge accumulated by the eighth photodiode (618) to the floating node (630). When the charge output by the eighth photodiode (618) is accumulated in the floating node (630), if the line selector (660) is in the on state, the image sensor can read out a voltage value corresponding to the charge accumulated by the eighth photodiode (618) through the source follower (650). Thereafter, the image sensor can perform a reset operation to remove the charge accumulated in the floating node (630) by turning on the reset transistor (670). For example, when operating based on the first shooting mode that generates a high-pixel image, the image sensor can read one pixel value from each of the photodiodes (611, 612, 613, 614, 615, 616, 617, 618).

[0098] In one embodiment, when the first switch (621), the second switch (622), the third switch (623), and the fourth switch (624) are turned on together, the charges accumulated in the first photodiode (611), the second photodiode (612), the third photodiode (613), and the fourth photodiode (614) move to the floating node (630). In this case, the image sensor may analogically add up the light amounts corresponding to the first photodiode (611), the second photodiode (612), the third photodiode (613), and the fourth photodiode (614), thereby obtaining light amount data corresponding to one micro lens. For example, when the image sensor operates based on a second mode that generates an image with a lower pixel count than the first mode, the image sensor may read out a sum of the charges accumulated in the first photodiode (611), the second photodiode (612), the third photodiode (613), and the fourth photodiode (614) as one pixel value.

[0099] In one embodiment, when the switches (621, 622, 623, 624, 625, 626, 627, 628) are turned on together, the charge accumulated in the photodiodes (611, 612, 613, 614, 615, 616, 617, 618) can move to the floating node (630). In this case, by analogically adding up the light amounts corresponding to the eight photodiodes (611, 612, 613, 614, 615, 616, 617, 618), the image sensor can obtain light amount data corresponding to two micro lenses (e.g., the first micro lens (591) and the second micro lens (592) of FIG. 5). The image sensor may similarly acquire light quantity data corresponding to adjacent microlenses (e.g., the third microlens (593) and the fourth microlens (594) of FIG. 5), and then sum the light quantity data corresponding to two microlenses (e.g., the first microlens (591) and the second microlens (592) of FIG. 5) to read out a value corresponding to one color channel as one pixel value. For example, when operating based on the third shooting mode that generates an image with lower pixels than the second mode, the image sensor may read out a value obtained by summing the charges accumulated in the photodiodes corresponding to four microlenses (e.g., the first microlens (591), the second microlens (592), the third microlens (593), and the fourth microlens (594) of FIG. 5) as one pixel value. Here, the read-out pixel values ​​may have a color order according to the Bayer pattern.

[0100] In one embodiment, when the first switch (621), the third switch (623), the fifth switch (625), and the seventh switch (627) are turned on while the second switch (622), the fourth switch (624), the sixth switch (626), and the eighth switch (628) are turned off, the image sensor can obtain pixel values ​​for pixels arranged on the first side among the pixels corresponding to the micro lens (e.g., the first photodiode (611), the third photodiode (613), the fifth photodiode (615), and the seventh photodiode (617) of FIG. 6) (hereinafter referred to as “left pixels”). When the first switch (621), the third switch (623), the fifth switch (625), and the seventh switch (627) are turned off and the second switch (622), the fourth switch (624), the sixth switch (626), and the eighth switch (628) are turned on, the image sensor can obtain pixel values ​​for pixels (e.g., the second photodiode (612), the fourth photodiode (614), the sixth photodiode (616), and the eighth photodiode (618) of FIG. 6) (hereinafter referred to as “right pixels”) arranged on the second side, which is different from the first side, among the pixels corresponding to the microlens. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain phase difference information based on the correlation between the left pixel value and the right pixel value. However, this is for explaining an example, and the method of obtaining the left pixel value and the right pixel value is not limited thereto. For example, an electronic device according to one embodiment may output luminance values ​​corresponding to the first photodiode (611) and the third photodiode (613) and luminance values ​​corresponding to the fifth photodiode (615) and the seventh photodiode (617) as individual left pixel values.

[0101] The first mode, which outputs sixteen pixel values ​​from an active pixel sensor array (APS) and the second mode, which outputs one pixel value from a 2 x 2 array of sensor pixels, have a color pattern of an image frame output by Bayer that is different from the Bayer pattern, so that the electronic device can perform a remosaic operation that converts image data output by the first mode or the second mode into a Bayer pattern that can be processed by an image signal processor.

[0102] FIG. 7 is a drawing showing an example of a color pattern of pixels included in an image acquired through a first sensor pattern (501) of an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment.

[0103] In one embodiment, an image sensor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform a binning operation to merge the outputs of a 2 x 2 array of sensor pixels (e.g., the photodetectors (511, 512, 513, 514) of FIG. 5) that include information for the same color channel within a first sensor pattern (501) to output an image having a first raw image pattern (710) corresponding to a Bayer pattern. In this case, the image having the first raw image pattern (710) may have a resolution of 1 / 4 compared to the number of sensor pixels of the image sensor. For example, an image having a first raw image pattern (710) having a resolution of 4 x 4 may be output from an 8 x 8 array of photodetectors included in the first sensor pattern (501) illustrated in FIG. 5.

[0104] In one embodiment, an image sensor of an electronic device may acquire an image in which each sensor pixel has a second raw image pattern (720) corresponding to pixels of the image. The electronic device may perform a remosaic operation to convert a color order on the image having the second raw image pattern (720) to acquire an image having a Bayer pattern (725). In the present disclosure, the remosaic operation may be performed on an image acquired from the image sensor by at least one processor (e.g., the processor (120) of FIG. 1 , the image signal processor (260) of FIG. 2 , and at least one processor (320) of FIG. 3 ), but it may be understood that the image sensor may also output a result of performing the remosaic operation. An image acquired based on the second raw image pattern (720) (e.g., an image having the Bayer pattern (725)) may have a resolution four times higher than an image acquired based on the first raw image pattern (710).

[0105] FIG. 8 is a drawing showing an example of a color pattern of pixels included in an image acquired through a second sensor pattern (502) of an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment.

[0106] In one embodiment, an image sensor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform a binning operation to merge outputs of a 3 x 3 array of sensor pixels (e.g., the photodetectors (551, 552, 553, 554, 555, 556, 557, 558, 559) of FIG. 5) that include information for the same color channel within a second sensor pattern (502) to output an image having a first raw image pattern (810) corresponding to a Bayer pattern. In this case, the image having the first raw image pattern (810) may have a resolution of 1 / 9 compared to the number of sensor pixels of the image sensor. For example, from the 6 x 6 array of light-receiving elements included in the second sensor pattern (502) illustrated in FIG. 5, an image having a first raw image pattern (810) with a resolution of 2 x 2 can be output.

[0107] In one embodiment, an image sensor of an electronic device can acquire an image having a second raw image pattern (820). The electronic device can perform a remosaic operation on the image having the second raw image pattern (820) to acquire an image having a Bayer pattern (825). An image acquired based on the second raw image pattern (820) (e.g., an image having the Bayer pattern (825)) can have a resolution nine times higher than an image acquired based on the first raw image pattern (810).

[0108] FIG. 9 is a diagram illustrating an example of a configuration of a color pattern of an image obtained based on a third sensor pattern (503) of an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment.

[0109] In one embodiment, an image sensor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform a binning operation to merge a 4 x 4 array of sensor pixels containing information for the same color channel within a third sensor pattern (503) to output an image having a first raw image pattern (910) corresponding to a Bayer pattern. The image having the first raw image pattern (910) may have a resolution of 1 / 16 relative to the number of sensor pixels of the image sensor.

[0110] In one embodiment, the electronic device can perform a binning operation that merges outputs of sensor pixels in a 2 x 2 array to obtain an image having a second raw image pattern (920). The electronic device can perform a re-mosaic operation on the image having the second raw image pattern (920) to obtain an image having a Bayer pattern (925). An image obtained based on the second raw image pattern (920) (e.g., an image having the Bayer pattern (925)) can have a resolution four times higher than an image obtained based on the first raw image pattern (910).

[0111] In one embodiment, the electronic device can acquire an image in which each sensor pixel has a third raw image pattern (930) corresponding to pixels of the image. The electronic device can perform a re-mosaic operation on the image having the third raw image pattern (630) to acquire an image having a Bayer pattern (935). An image acquired based on the third raw image pattern (930) (e.g., an image having the Bayer pattern (935)) can have a resolution 16 times higher than an image acquired based on the first raw image pattern (910).

[0112] FIG. 10 is a diagram illustrating an example of an image provided by an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment based on an ROI according to object tracking.

[0113] In one embodiment, the electronic device can identify an object (subject) captured in the acquired image (1000). For example, the electronic device can display the acquired image (1000) as a preview image through a display (e.g., the display module (160) of FIG. 1, the display (360) of FIG. 3). The electronic device (101) can receive a touch input for a location corresponding to an area where an object (1015) included in the displayed preview image is displayed. The electronic device (101) can execute an image recognition algorithm to identify the object (1015) in the image (1000) based on the location of the touch input. The electronic device (101) can determine a first region of interest (1010) based on the object (1015). For example, the electronic device can determine a first region of interest (1010) including an area where the object (1015) appears based on at least one of a location or a size of the object (1015) in the image (1000).

[0114] In one embodiment, the electronic device (101) can obtain an image (1030) corresponding to the first region of interest (1010). For example, the electronic device (101) can obtain the image (1030) by cropping pixel values ​​obtained from sensor pixels located within the first region of interest (1010) among sensor pixels included in an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) and the second image sensor (384) of FIG. 3). The electronic device (101) can display the image (1030) as a preview image through a display. The electronic device (101) can store image data (e.g., a video file) including the image (1030) in a memory (e.g., the memory (130) of FIG. 1, the memory (330) of FIG. 3). The electronic device (101) can reduce the degree of image quality deterioration due to the digital zoom operation by acquiring an image (1030) through an operation mode capable of acquiring high-resolution image data (for example, a mode for acquiring an image having a Bayer pattern (725) of FIG. 7, a Bayer pattern (825) of FIG. 8, a Bayer pattern (925) of FIG. 9, or a Bayer pattern (935) of FIG. 9 through a re-mosaic operation).

[0115] In one embodiment, the electronic device (101) can track the location of the object (1015). The electronic device (101) can identify an object that is determined to be identical to the object (1015) in a subsequent image captured after the image (1000). For example, if the location of the object (1015) moves (1025) in the subsequent image, the electronic device (1011) can determine an area corresponding to the moved location as a second region of interest (1020). By acquiring an image based on the second region of interest (1020), the electronic device (1011) can prevent the object (1015) from disappearing from the image even if the object (1015) moves within the field of view of the camera.

[0116] FIG. 11 is a diagram illustrating examples of pixel lines and regions of interest for an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) according to one embodiment. Although FIG. 11 is described with respect to the configuration of the first image sensor (383), the second image sensor (384) may also include the same configuration.

[0117] In one embodiment, the first image sensor (383) may include a plurality of sensor pixels (PX). The sensor pixels (PX) may be arranged in a first axis direction and a second axis direction. Each of the sensor pixels may include at least one light-receiving element (e.g., a photodiode). For example, the first axis direction may mean a +y-axis direction. For example, the second axis direction may mean a +x-axis direction that is perpendicular to the first axis direction. In the present disclosure, the resolution of the image sensor may correspond to the number of sensor pixels included in the image sensor. For example, when a (8 in the case of FIG. 11) sensor pixels are arranged in the x-axis direction and b (8 in the case of FIG. 11) sensor pixels are arranged in the y-axis direction, the resolution of the image sensor may be referred to as axb.

[0118] An image sensor according to one embodiment may include a plurality of pixel lines (1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118). In the present disclosure, a pixel line may mean a group of pixels arranged in one direction. Referring to FIG. 11, a pixel line may mean pixels arranged in a second axial direction. An electronic device may read pixel values ​​for at least some of the sensor pixels included in a pixel line in parallel. In the present disclosure, reading a "pixel line" may mean reading information about pixel values ​​(or the pixel values ​​themselves) from the sensor pixels included in the pixel line. In the present disclosure, data obtained by reading the pixel lines may be referred to as image data included in an image frame. In the present disclosure, "reading an image frame" may mean including an operation of obtaining image data by reading the pixel lines.

[0119] In one embodiment, the image sensor can read pixel lines, at least some of which are included in a region of interest (1120). In the present disclosure, “at least one pixel line associated with a region of interest” may mean at least one pixel line, at least some of which pixels included in the pixel line are located within the region of interest. The region of interest (1120) may include at least a portion of an area in which sensor pixels (PX) are arranged within the image sensor. For example, referring to FIG. 10 , the region of interest (1120) may include an area in which sensor pixels for capturing an image corresponding to the region of interest (1010) of FIG. 10 are arranged. Referring to FIG. 11 , when a region of interest (1120) determined to acquire an image frame is determined, the image sensor can read pixel lines (1114, 1115, 1116). For example, when the reading direction is forward, the image sensor can sequentially read out the pixel line (1114) and the pixel line (1115) and the pixel line (1116) arranged in the -y-axis direction. For example, when the reading direction is reverse, the image sensor can sequentially read out the pixel line (1115) and the pixel line (1114) arranged in the +y-axis direction after reading out the pixel line (1116).

[0120] FIG. 12 illustrates an example of an operation of an image sensor (e.g., an image sensor (230) of FIG. 2, a first image sensor (383), a second image sensor (384) of FIG. 3) of an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 3) according to one embodiment of the present invention to read out a pixel line according to an operation mode.

[0121] In one embodiment, an image sensor (e.g., the image sensor 230 of FIG. 2, the first image sensor 383, the second image sensor 384 of FIG. 3) may perform an operation of reading pixel lines based on a plurality of operating modes. The plurality of operating modes may include a first mode (1201) and a second mode (1202). The first mode (1201) may refer to an operating mode in which the image sensor outputs a raw image having a first resolution. For example, referring to FIG. 9, the electronic device may acquire a raw image having a first raw image pattern (910) corresponding to a Bayer pattern by merging outputs of sensor pixels from the image sensor in the first mode. The raw image having the first raw image pattern (910) may be expressed at a first resolution. The second mode (1202) may refer to an operating mode in which the image sensor outputs a raw image having a second resolution higher than the first resolution. The electronic device can obtain a color image through image signal processing including a remosaic process that changes the color pattern of the raw image into a designated pattern (e.g., Bayer pattern) in the second mode (1202). For example, referring to FIG. 9, the electronic device can obtain a raw image having a second raw image pattern (920) or a third raw image pattern (930) in the second mode (1202), and convert the raw image to have a pattern corresponding to the Bayer pattern (925 or 935).

[0122] In one embodiment, the second single pixel line readout time (1223) in the second mode (1202) may be longer than the first single pixel line readout time (1213) in the first mode (1201). In the present disclosure, the term "single pixel line readout time" may refer to the time required for a readout operation to read one line. In one embodiment, when the electronic device operates based on the first mode (1201), the image sensor may perform an operation of reading out pixel lines (1211) during a readout period (1215). The pixel lines (1211) may include all pixel lines included in the image sensor. When the electronic device operates based on the second mode (1202), the image sensor may perform an operation of reading out at least one pixel line (1221) associated with a region of interest during a readout period (1225). In the present disclosure, a “readout period” may mean a period of time during which a readout operation for reading one image frame is performed.

[0123] FIG. 13 illustrates an example of an operation in which an image sensor (1310) (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) reads out pixel lines to acquire image frames according to a region of interest.

[0124] In one embodiment, the image sensor (1310) can perform a first operation (1321) and a second operation (1322). The first operation (1321) can include a reset operation, an exposure operation, and a readout operation to output a first image frame by reading at least one pixel line associated with a first region of interest (1311). The reset operation may also be referred to as a shutter operation. The readout operation of the first operation (1321) can include a first section (1 stMIPI Active) can be performed during the second operation (1322). The second operation (1322) can include a reset operation, an exposure operation, and a readout operation for outputting a second image frame by reading at least one pixel line associated with the second region of interest (1312). The readout operation of the second operation (1322) can be performed during the first period (1 st After the end of the blank period (Vertical Black) of MIPI Active, the second period (2) begins. nd It can be performed during MIPI Active).

[0125] Referring to FIG. 13, an overlapping section (1330) may occur depending on the positions of the first region of interest (1311) and the second region of interest (1312) and the length of the exposure period (1340) for acquiring the second image frame. The overlapping section (1330) may include a section in which at least one of a reset operation or an exposure operation for performing the second operation (1322) is initiated before at least one of the exposure operation or the readout operation included in the first operation (1321) is completed for at least a portion of at least one pixel line associated with the first region of interest. The overlapping section (1330) may cause frame drops or camera errors for some image frames (e.g., the second image frame). To prevent frame drops from occurring, the exposure time (1340) included in the second operation (1322) may be shortened. However, shortening the exposure time (1340) may result in insufficient exposure for acquiring the second image frame, which may deteriorate the image quality.

[0126] FIG. 14 is a block diagram illustrating the structure of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) that displays an image acquired using two or more image sensors (e.g., the first image sensor (383) of FIG. 3, the second image sensor (384) of FIG. 3), in one embodiment.

[0127] In one embodiment, at least one processor (320) of the electronic device (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and at least one processor (320) of FIG. 3) may acquire an image frame (1401) from a second image sensor (384). However, FIG. 14 is intended to illustrate an example of displaying an image acquired using two or more image sensors, and the electronic device according to one embodiment may display an image acquired using one image sensor. For example, the image frame (1401) may be replaced with an image frame acquired through the first image sensor (383). The at least one processor (320) may determine a region of interest including an area in which an object of interest is identified within the image frame (1401).

[0128] In one embodiment, at least one processor (320) may transmit control information to the first image sensor (383) via control communication (1402) so that the first image sensor (383) reads at least one pixel line associated with the region of interest based on the determined region of interest. The control communication (1402) may be performed based on a communication method that transmits and receives signals based on, for example, an inter-integrated circuit (I2C) communication method or an improved inter-integrated circuit (I3C) communication method. The control information may include region of interest information about the region of interest. The region of interest information may include information indicating a location corresponding to the region of interest within the image sensor. For example, the information about the region of interest may include a region of interest address that includes at least one of the coordinates of a start point or an end point of the region of interest. The first image sensor (383) may transmit an image frame (1403) read from pixels included in the region of interest to the at least one processor (320).

[0129] In one embodiment, at least one processor (320) can display a preview image through a display (360). At least one processor (320) can generate a preview image based on an image included in an acquired image frame (1403). At least one processor (320) can also generate a preview image based on an image frame (1403) acquired from a first image sensor (383) and an image frame (1401) acquired from a second image sensor (384).

[0130] FIG. 15 is a flowchart (1500) illustrating a process by which an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) acquires an image frame based on sensor driving information, in one embodiment.

[0131] In one embodiment, the electronic device may obtain sensor driving information in operation 1510. In operation 1510, at least one processor (e.g., the processor 120 of FIG. 1, the image signal processor 260 of FIG. 2, and at least one processor 320 of FIG. 3) may obtain sensor driving information from an image sensor. The sensor driving information may include information obtained by an operation performed by an image sensor of the electronic device (e.g., the image sensor 230 of FIG. 2, the first image sensor 383 of FIG. 3, and the second image sensor 384 of FIG. 3). When the image sensor outputs a first image frame and subsequently outputs a second image frame, the sensor driving information may include information for controlling an operation of the second image frame based on a region of interest of the first image frame. For example, the sensor driving information may include information about a restricted region associated with a location of the region of interest of the second image frame or an offset value associated with a maximum exposure time of the second image frame. The restricted area may refer to an area indicating a location of a second image frame where an overlapping section (e.g., an overlapping section (1330) of FIG. 13) is likely to occur. For example, the restricted area may indicate an area where an overlapping section is likely to occur when a start position of the second image frame (e.g., a start position (1713) of FIG. 17) is positioned. The offset value may include a value that can be used by at least one processor to determine a maximum exposure time.

[0132] In one embodiment, the electronic device may determine at least one of region of interest information or exposure control information based on sensor driving information in operation 1520. The at least one processor may determine whether the image sensor operates without an error based on the sensor driving information. For example, if the start position of the second image frame is included within the restricted area and the exposure time set for the second image frame exceeds the maximum exposure time, the at least one processor may determine that an error may occur in the operation of the image sensor. If it is determined that an error may occur in the operation of the image sensor, the at least one processor may determine at least one of region of interest information or exposure control information to prevent the error from occurring. For example, the electronic device may determine region of interest information that moves the position of the region of interest to prevent an overlapping section (e.g., the overlapping section 1330 of FIG. 13) from occurring. For example, referring to FIG. 13, the electronic device may adjust the position of the region of interest for acquiring an image frame so that the y-axis direction position of the start point of the region of interest is located outside the restricted area based on the information about the restricted area included in the sensor driving information. For example, the electronic device may reduce the exposure time for reading the image frame based on sensor actuation information so that an overlapping region (e.g., overlapping region (1330) of FIG. 13) does not occur. In one embodiment, when the electronic device reduces the exposure time, the electronic device may adjust (increase) a gain value applied to pixel values ​​of the image frame based on the reduced exposure time.

[0133] In one embodiment, the electronic device may transmit a control signal from at least one processor to the image sensor based on the region of interest information or exposure control information determined in operation 1530. For example, the at least one processor may transmit the control signal to the image sensor via control communication. The electronic device may acquire an image frame read by the image sensor based on the control signal transmitted in operation 1540.

[0134] FIG. 16 illustrates an example of a restricted area determined by an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) in one embodiment.

[0135] In one embodiment, an image sensor of an electronic device (e.g., an image sensor (230) of FIG. 2, a first image sensor (383) of FIG. 3, a second image sensor (384) of FIG. 3) may perform an operation of reading out a first image frame during a first image frame readout period (1610). The image sensor may perform an operation of reading out a second image frame during a second image frame readout period (1620). The image sensor may sequentially perform the readout operation for at least one pixel line corresponding to a region of interest according to a readout direction. A readout method that sequentially performs a shutter operation, an exposure operation, and a readout operation for pixel lines as illustrated in FIG. 16 may be referred to as a rolling shutter method.

[0136] In one embodiment, the image sensor can determine a limit region (1650) for the starting point of the region of interest of the second image frame. For example, information about the limit region (1650) may include an axial coordinate value y corresponding to the reading direction. o or may include at least one of y1, y1, y2, y3, y4, y5, y6, y7, y8, y9, y11, y12, y13, y14, y15, y16, y17, y18, y19, y20, y21, y22, y23, y24, y25, y26, ocan indicate a position of one end in the opposite direction of the reading direction of the restricted area (1650). The coordinate value y1 can indicate a position of one end in the reading direction of the restricted area (1650). For example, the image sensor may indicate a position where the interval (1630) between the reading operation of the first image frame and the shutter operation of the second image frame becomes smaller than a threshold when the starting position of the region of interest for the second image frame is located at the coordinate value y o can be determined by the coordinate value y o may vary depending on the exposure time (1640) of the second image frame. For example, the image sensor may determine the position of the last pixel line in the readout direction among the pixel lines located within the region of interest for the first image frame as the coordinate value y1.

[0137] FIG. 17 illustrates an example of a maximum exposure value determined when the starting position of the ROI to be changed is within the restricted area in one embodiment.

[0138] In one embodiment, an image sensor (1710) (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can read at least one pixel line including pixels within a first ROI (1711). The image sensor can output a first image frame in which values ​​included in an area corresponding to the first ROI (1711) among values ​​read from the at least one pixel line including pixels within the first ROI (1711) are cropped. The image sensor can perform a read operation on the first image frame during a first image frame readout period (1730). The image sensor can read a second image frame subsequent to the first image frame. The image sensor can perform a readout operation on the second image frame during the second image frame readout section (1740). The image sensor can read out at least one pixel line including pixels within the second ROI (1712). The image sensor can output a second image frame that crops values ​​included in a corresponding area within the second ROI (1712) among values ​​read out from at least one pixel line including pixels within the second ROI (1712).

[0139] In one embodiment, the image sensor can determine a restricted region (1714) based on the location of the first ROI (1711) and an exposure value input for a readout operation of a second image frame. At least one processor of the electronic device (e.g., the processor 120 of FIG. 1 , the image signal processor 260 of FIG. 2 , and at least one processor 320 of FIG. 3 ) can determine whether a start position (1713) of the second ROI (1712) is included within the restricted region (1714) based on information about the restricted region (1714). The at least one processor can identify that an error may occur in the operation of the image sensor if the start position (1713) is included within the restricted region (1714) and the exposure time for the second image frame exceeds a maximum exposure time. The at least one processor can adjust the location of the region of interest so that the start position of the region of interest is not included within the restricted region of a previous frame. The at least one processor can adjust an exposure value for reading the image frame so that the exposure time does not exceed the maximum exposure time. At least one processor can input at least one of information about the adjusted region of interest or exposure value to the image sensor.

[0140] In one embodiment, if the starting position (1713) is within the restricted area (1714), the electronic device may perform an operation to adjust the region of interest or exposure time based on the sensor driving information to avoid overlapping the operations of the image sensor. If the starting position (1713) is located outside the restricted area (1714), the electronic device may determine exposure control information based on the frame rate at which the image sensor operates. The frame rate may indicate the frequency at which the image sensor outputs image frames.

[0141] In one embodiment, the image sensor can adjust the position of the second ROI (1712) input to the image sensor by at least one processor so that the starting position (1713) is not included within the restricted region (1714). For example, the image sensor can adjust the vertical (e.g., y-axis direction of FIG. 13) coordinate value of the starting position of the second ROI (1712) so that it is located outside the restricted region (1714). The image sensor can read out the second image frame based on an ROI other than the second ROI (1712) transmitted by the at least one processor through a control signal. By reading out pixel lines corresponding to the adjusted ROI and outputting the second image frame, the image sensor can prevent a section in which a shutter operation or an exposure operation is performed on a pixel line for reading out the second image frame from overlapping with a reading operation on a pixel line for reading out the first image frame.

[0142] In one embodiment, the image sensor can adjust the exposure time for reading the second image frame so as to be less than or equal to the maximum exposure value (1750). For example, the image sensor can perform an exposure operation for the second image frame based on a control value different from a control value input by at least one processor to control the operation of reading the second image frame. By adjusting the exposure time for acquiring the second image frame, the image sensor can prevent a section in which it performs a shutter operation or an exposure operation for a pixel line for reading the second image frame from overlapping with a readout operation for a pixel line for reading the first image frame.

[0143] In one embodiment, at least one processor may determine a maximum exposure value (1750) based on sensor driving information. The at least one processor may determine the maximum exposure value (1750) based on a difference and an offset value between a y-axis coordinate value y1 (hereinafter referred to as a 'first coordinate value') of a y-axis end of a restricted area (1714) and a y-axis coordinate value y (hereinafter referred to as a 'second coordinate value') of a y-axis end of a second ROI (1712). The maximum exposure value (1750) may be determined as a sum of a first section (1751) and a second section (1753). The first section (1751) may be determined as a value obtained by multiplying the difference between the first coordinate value and the second coordinate value by a single pixel line readout time (e.g., the second single pixel line readout period (1223) of FIG. 12). The second section (1753) can be determined by multiplying the offset value included in the sensor drive information by the single pixel line readout time. This can be expressed mathematically as shown in Equation 1 below.

[0144]

[0145] In mathematical expression 1, y1 may mean a first coordinate value. In mathematical expression 1, y may mean a second coordinate value. In mathematical expression 1, SM_Forbidden_MIN_LINE_TIME may mean an offset value included in sensor driving information. In mathematical expression 1, 1H_Time may mean a single pixel line readout time. The single pixel line readout time may be calculated based on a register value. For example, the single pixel line readout time may be calculated by dividing the number of pixel clocks for operation of one line (e.g., line length pck (1950) of FIG. 19) by one pixel processing frequency.

[0146] FIG. 18 is a flowchart (1800) illustrating a process in which, in one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) obtains sensor driving information from a packet containing an image frame and controls an image sensor.

[0147] In one embodiment, the electronic device may identify a change in a region of interest set to acquire an image frame through an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) at operation 1810. For example, the electronic device may identify a change in a region of interest for acquiring an image frame based on a wide-angle camera as a position of an object of interest moves within an image captured by an ultra-wide-angle camera. For example, the electronic device may also identify a change in a region of interest based on a user input to change the region of interest.

[0148] In one embodiment, the electronic device may obtain sensor driving information from a packet output from an image sensor in operation 1820. The packet output from the image sensor may include an image frame. The image sensor may output a packet including the image frame through an interface connected to at least one processor (e.g., the processor 120 of FIG. 1 , the image signal processor 260 of FIG. 2 , or at least one processor 320 of FIG. 3 ). The at least one processor of the electronic device (e.g., the processor 120 of FIG. 1 , the image signal processor 260 of FIG. 2 , or at least one processor 320 of FIG. 3 ) may obtain sensor driving information included in a designated location within the packet. The sensor driving information may be included in a header or footer embedded in the packet. The sensor driving information may include at least one of information about a restricted area and information about an offset value.

[0149] In one embodiment, at least one processor of the electronic device may determine whether a change restriction condition of the region of interest is satisfied based on the sensor driving information acquired in operation 1830. For example, the electronic device may determine whether a start position of an image frame subsequent to an image frame included in a packet acquired in operation 1820 is included within the restricted region. For example, if the start position of the image frame is included within the restricted region, the electronic device may determine whether an exposure time of the subsequent image frame is greater than a maximum exposure value determined based on at least an offset value.

[0150] In one embodiment, based on determining that the change constraint is satisfied, the electronic device may adjust at least one of the region of interest information or the exposure control information for controlling the image sensor in operation 1840. In operation 1850, the at least one processor may transmit a control signal including at least one of the adjusted region of interest information or the exposure control information to the image sensor. Based on determining that the change constraint is not satisfied, the electronic device may skip operation 1840 and transmit the control signal from the at least one processor to the image sensor in operation 1850.

[0151] FIG. 19 illustrates an example of a packet (1900) output by an image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3), in one embodiment.

[0152] In one embodiment, a packet (1900) output by an image sensor may include an image frame including visible data (1910) including information for visually representing an image and phase difference data (1920) for performing an auto-focus function, etc. The packet (1900) output by the image sensor may further include at least one of a header (1930) or a footer (1940). The header (1930) or the footer (1940) may include information associated with the image frame.

[0153] FIG. 20 illustrates, in one embodiment, an example of sensor drive information (2000) included in at least one of a header (e.g., header (1930) of FIG. 19) or a footer (e.g., footer (1940) of FIG. 19) of a packet (e.g., packet (1900) of FIG. 19).

[0154] In one embodiment, an image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3) may perform an operation to obtain sensor driving information and output the sensor driving information (2000) by including it in a packet including an image frame.

[0155] In one embodiment, the sensor operation information (2000) included in the packet may include at least one of information about the restricted area or information about the offset value. Referring to FIG. 20, the sensor operation information (2000) may include a coordinate value (2010) indicating the location of one end of the restricted area. For example, the coordinate value (2010) SM_Forbidden_TOP may be the coordinate value y of FIG. 16. oThe sensor driving information (2000) may include a coordinate value (2020) indicating the location of the other end of the restricted area. For example, the coordinate value (2020) SM_Forbiddem_BOTTOM may include the coordinate value y1 of FIG. 16. The sensor driving information (2000) may include an offset value (2030). For example, the offset value (2030) SM_Forbidden_MIN_LINE_TIME may include SM_Forbidden_MIN_LINE_TIME of Mathematical Expression 1.

[0156] FIG. 21 is a flowchart (2100) illustrating a process in which, in one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) obtains sensor driving information through control communication and controls an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)).

[0157] In one embodiment, the electronic device may identify a change in a region of interest set to acquire an image frame through an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) at operation 2110. For example, the electronic device may identify a change in a region of interest for acquiring an image frame based on a wide-angle camera as a position of an object of interest moves within an image captured by an ultra-wide-angle camera. For example, the electronic device may also identify a change in a region of interest based on a user input to change the region of interest.

[0158] In one embodiment, the electronic device may transmit at least one of region of interest information or exposure control information for obtaining an image frame in operation 2120 from at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to an image sensor. The at least one processor may transmit a control signal including at least one of region of interest information or exposure control information to the image sensor via control communication.

[0159] In one embodiment, the image sensor can obtain restricted region information for a region of interest set to read an image frame. The image sensor can store restricted region information and an offset value in a designated control address. The image sensor can determine whether an ROI change restriction condition is satisfied based on an exposure time according to exposure control information input to the image sensor and a position of the ROI according to the region of interest information. While performing an operation to determine whether the ROI change restriction condition is satisfied, the image sensor can store a first flag value as a first value (e.g., 0) in the designated control address. Based on completion of the operation to determine whether the ROI change restriction condition is satisfied, the image sensor can change the first flag value to a second value (e.g., 1). Based on the operation result indicating that the ROI restriction condition is satisfied, the image sensor can store the second flag value as a third value (e.g., 1) in the designated control address. Based on the determination that the ROI restriction condition is not satisfied, the image sensor can store the second flag value as a fourth value (e.g., 0).

[0160] In one embodiment, at least one processor can read information stored in a designated control address. For example, at least one processor can read at least one of sensor driving information, a first flag value, or a second flag value stored by an image sensor through control communication. In operation 2130, at least one processor can determine whether the first flag value is a second value (e.g., 1). If the first flag value is not the second value, the at least one processor can perform operation 2130 until the first flag value becomes the second value. Based on whether the first flag value is the second value, the electronic device can perform operation 2140.

[0161] In one embodiment, at least one processor may determine whether the second flag value is a third value (e.g., 1) in operation 2140. Based on the second flag value having the third value, at least one processor may adjust at least one of the region of interest information or the exposure control information based on the sensor driving information in operation 2150. In operation 2150, the at least one processor may determine at least one of the ROI or the exposure value that prevents an overlapping section (e.g., the overlapping section (1330) of FIG. 13) from occurring. In operation 2120, the at least one processor may transmit a control signal including at least one of the region of interest information or the exposure control information determined in operation 2150 to the image sensor via control communication. If the second flag value is not the third value, operation 2150 may be omitted.

[0162] FIG. 22 illustrates an example of stored sensor drive information that an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) can obtain through control communication, in one embodiment.

[0163] In one embodiment, an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) may store sensor driving information or at least one flag value in a designated control address (2210). At least one processor may obtain the sensor driving information or at least one flag value stored in the designated control address (2210) through control communication (e.g., the control communication (1402) of FIG. 14).

[0164] In one embodiment, the sensor operation information that at least one processor can obtain through control communication may include at least one of information about a restricted area or information about an offset value. Referring to FIG. 22, the sensor operation information may include a coordinate value (2220) indicating a position of one end of the restricted area. For example, the coordinate value (2220) SM_Forbidden_TOP may be the coordinate value y of FIG. 16. o The sensor operation information may include a coordinate value (2230) indicating the location of the other end of the restricted area. For example, the coordinate value (2230) SM_Forbiddem_BOTTOM may include the coordinate value y1 of FIG. 16. The sensor operation information may include an offset value (2240). For example, the offset value (2240) SM_Forbidden_MIN_LINE_TIME may include SM_Forbidden_MIN_LINE_TIME of Mathematical Expression 1.

[0165] In one embodiment, the at least one flag value may include at least one of a first flag value (2250) or a second flag value (2260). The first flag value (2250) may indicate whether the image sensor has completed an operation related to the ROI change constraint condition. The second flag value (2260) may indicate whether an image frame according to the changed ROI satisfies the ROI change constraint condition.

[0166] FIG. 23 illustrates an example of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) adjusting an ROI in one embodiment.

[0167] In one embodiment, an image sensor (2300) of an electronic device (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) can output a second image frame subsequent to the output of the first image frame. The electronic device can output at least one pixel line including pixels within a first ROI (2311) within the image sensor (2300). For the second image frame, a second ROI (2312) can be set within the image sensor, the location of which is shifted by a first movement amount (2331) from the first ROI (2311). The starting position of the second ROI (2312) can be a coordinate in an axial direction corresponding to the reading direction shifted in the reading direction with respect to the starting position of the first ROI (2311). If the location of the second ROI (2312) is a location included in the restricted area, at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, or at least one processor (320) of FIG. 3) or an image sensor can determine a second' ROI (2342) that has moved from the first ROI (2311) by a second movement amount (2332) that is smaller than the first movement amount (2331). The image sensor can read a second image frame based on the second' ROI (2342). After reading the second image frame, the image sensor can read a third image frame based on the third ROI (2343). The third ROI (2343) can correspond to the second ROI (2312).

[0168] In one embodiment, the electronic device may perform an operation to avoid the changed position of the ROI from being included in the restricted area by reducing the amount of movement (2332, 2333) of the ROI.

[0169] FIG. 24 illustrates an example of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) adjusting an ROI in one embodiment.

[0170] In one embodiment, an image sensor (2400) of an electronic device (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3) can output a second image frame subsequent to the output of a first image frame. The electronic device can output at least one pixel line including pixels within a first ROI (2411) within the image sensor (2400). For the second image frame, a second ROI (2412) can be set within the image sensor, the location of which is shifted by a first movement amount (2431) from the first ROI (2411). The starting position of the second ROI (2412) can be a coordinate in an axial direction corresponding to the reading direction shifted in the reading direction with respect to the starting position of the first ROI (2411). If the location of the second ROI (2412) is a location included in the restricted area, at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, or at least one processor (320) of FIG. 3) or the image sensor can determine a second' ROI (2413) moved in the reading direction from the location of the second ROI (2412) so that an area (2401) overlapping with the first ROI (2411) does not occur. The image sensor can read the second image frame based on the second' ROI (2413).

[0171] In one embodiment, the electronic device may perform an operation to avoid the changed position of the ROI from being included in the restricted area by adjusting the position of the ROI so that an overlapping area (2401) does not occur.

[0172] FIG. 25 is a flowchart (2500) illustrating a process by which an image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384) of FIG. 3) controls an ROI or exposure value in one embodiment.

[0173] In one embodiment, the electronic device may identify a change in a region of interest set for acquiring an image frame through an image sensor at operation 2510. For example, the electronic device may identify a change in a region of interest for acquiring an image frame based on a wide-angle camera as the position of an object of interest moves within an image captured by an ultra-wide-angle camera. For example, the electronic device may also identify a change in a region of interest based on a user input that causes the region of interest to change.

[0174] In one embodiment, the electronic device may input at least one of region of interest information or exposure control information for the region of interest changed in operation 2520 to the image sensor. For example, at least one processor (e.g., processor (120) of FIG. 1 , image signal processor (260) of FIG. 2 , at least one processor (320) of FIG. 3 ) may transmit at least one of region of interest information or exposure control information to the image sensor via control communication (e.g., control communication (1402) of FIG. 14 ).

[0175] In one embodiment, at operation 2530, the image sensor may determine whether a ROI change restriction condition is satisfied based on region of interest information or exposure control information input to the image sensor. The image sensor may determine whether the ROI change restriction condition is satisfied before performing an operation of reading an image frame based on the input region of interest information or exposure control information. For example, the image sensor may determine whether a start position of a region of interest corresponding to the input region of interest information is included within a restricted region. For example, if the start position is included within the restricted region, the image sensor may determine whether an exposure time corresponding to the input exposure control information exceeds a maximum exposure time.

[0176] In one embodiment, based on determining that the ROI change constraint condition is satisfied, the image sensor may adjust at least one of the region of interest corresponding to the input region of interest information in operation 2540 or the exposure value corresponding to the input exposure control information. For example, if the ROI can be changed, the image sensor may adjust the position of the ROI so that the starting position is not included in the constraint area. For example, if the ROI cannot be changed, the image sensor may adjust the exposure value so that the exposure time is less than or equal to the maximum exposure time. For example, the image sensor may determine based on a threshold (minimum processing time required) for an interval (e.g., interval (1630) of FIG. 16) between a readout operation of a previous image frame and a shutter operation of a subsequent image frame. For example, the image sensor may determine the adjusted exposure time based on Equation 2 below.

[0177]

[0178] In mathematical expression 2, E' may refer to the changed exposure time. In mathematical expression 2, E0 may refer to the exposure time input to the image sensor. In mathematical expression 2, minProcessingTime may refer to the minimum processing required time. In mathematical expression 2, y may refer to the y-axis coordinate value of the subsequent image frame. In mathematical expression 2, y o may refer to a y-axis coordinate value indicating the position of a part of the restricted area. However, E' may be set to a value greater than minProcessingTime.

[0179] In one embodiment, when adjusting the exposure value in operation 2540, the image sensor may adjust the gain applied to the pixel values ​​to read the image frame. For example, the image sensor may determine the adjusted gain value based on Equation 3 below.

[0180]

[0181] In mathematical expression 3, G' can mean the changed gain value. In mathematical expression 3, G o may refer to a gain value input to the image sensor through control communication. In mathematical expression 3, E' may refer to a changed exposure time. In mathematical expression 3, E0 may refer to an exposure time input to the image sensor.

[0182] In one embodiment, the image sensor can determine whether to adjust the region of interest or the exposure value based on the result of calculating the changed exposure time. For example, the image sensor can determine whether to adjust the region of interest or the exposure value based on the result of calculating the changed exposure time E'. o If the ratio specified for is greater than or equal to (e.g., E' is E o If the area of ​​interest is greater than 50% of the area of ​​interest), the image sensor can maintain the area of ​​interest and adjust the exposure time and gain values ​​to E' and G'.

[0183] For example, the image sensor receives an input exposure time E' with a changed exposure time E o If it is less than the specified ratio (e.g., E' is E o If less than 50% of the image area is visible), the image sensor can adjust the position of the area of ​​interest.

[0184] For example, when adjusting the position of the area of ​​interest, the upper part of the restricted area (e.g., within the restricted area (1650) of FIG. 16) o Based on the fact that the starting position of the region of interest is contained within the 50% area of ​​the side, the image sensor determines that E' is E o where the ratio is specified for (e.g. E'=E o The y-axis coordinate value of the region of interest can be adjusted by the position of the region of interest. The image sensor can re-determine the exposure time and gain value based on the position of the adjusted region of interest.

[0185] For example, when adjusting the position of the region of interest, based on the fact that the start position of the region of interest is included in the lower part of the restricted area (e.g., the y1-side 50% area within the restricted area (1650) of FIG. 16), the image sensor may adjust the y-axis coordinate of the start position of the region of interest to correspond to the next pixel line at the bottom of the restricted area (e.g., y1+1 of FIG. 16). Since the start position of the region of interest is included in the lower part of the restricted area, the start position of the region of interest may be outside the restricted area. Since the start position of the region of interest is located outside the restricted area, the image sensor may adjust the exposure time E input to the image sensor. o and gain value G o The image frame can be read based on this.

[0186] For example, when adjusting the position of a region of interest, the image sensor can adjust the x-axis coordinate value of the region of interest to an average value of the x-axis coordinate value of the previous frame and the x-axis coordinate value included in the input region of interest information.

[0187] In operation 2550, the image sensor according to one embodiment may perform an operation of reading out an image frame based on at least one of the region of interest or exposure value adjusted in operation 2540. If the ROI change constraint is not satisfied in operation 2530, the image sensor may skip operation 2540 and, in operation 2550, read out an image frame based on at least one of the region of interest or exposure value input from at least one processor.

[0188] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may include a camera (e.g., camera module (180) of FIGS. 1 and 2, first camera (383), third camera (384) of FIG. 3), at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3)) and a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3). A camera (e.g., a camera module (180) of FIGS. 1 and 2, a first camera (383), a third camera (384) of FIG. 3) may include an image sensor (e.g., an image sensor (230) of FIG. 2, a first image sensor (383), a second image sensor (384) of FIG. 3) that includes a plurality of sensor pixels (e.g., sensor pixels (PX) of FIG. 11) arranged along a plurality of pixel lines (e.g., pixel lines (1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118) of FIG. 11). At least one processor (e.g., a processor (120) of FIG. 1, an image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3)) may include processing circuitry. Memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3) can store instructions.The above commands may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to obtain a first image frame from the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3). The above instructions may be individually or collectively executed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3) so that the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) obtains sensor driving information related to the operation of the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) from the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3)). The above commands may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine at least one of region of interest information or exposure control information based on the sensor driving information.The above instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to transmit a control signal including at least one of the determined region of interest information or the determined exposure control information from the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3)) to the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) through control communication (e.g., the control communication (1402) of FIG. 14). The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) may be configured to read pixel values ​​from at least one pixel line (e.g., at least one pixel line (1114, 1115, 1116) of FIG. 11) corresponding to the region of interest information among the plurality of pixel lines (e.g., the pixel lines (1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118) of FIG. 11) based on the exposure control information. The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) may be configured to output a second image frame obtained by cropping an area corresponding to the area of ​​interest information (e.g., the area of ​​interest (1120) of FIG. 11) among the read pixel values ​​to at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3).

[0189] In one embodiment, the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) may be configured to output a packet (e.g., the packet (1900) of FIG. 19) including the first image frame to the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3). The packet (e.g., the packet (1900) of FIG. 19) may include the sensor driving information in at least one of a header (e.g., the header (1930) of FIG. 19) or a footer (e.g., the footer (1940) of FIG. 19) embedded in the packet.

[0190] In one embodiment, the instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to obtain the sensor driving information from the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3)) through the control communication (e.g., the control communication (1402) of FIG. 14) by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3).

[0191] In one embodiment, the instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to read the sensor driving information stored in a control communication address (e.g., the control communication address (2210) of FIG. 22) defined in the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3)) by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3)).

[0192] In one embodiment, the constant sensor operation information may include at least one flag value. The instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to determine at least one of the region of interest information or the exposure control information based on the at least one flag value.

[0193] In one embodiment, the at least one flag value may include a first flag value and a second flag value. The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) may be configured to store the first flag value as a first value while calculating the sensor driving information. The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) may be configured to store the first flag value as a second value based on the completion of calculating the sensor driving information. The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) may be configured to store the second flag value as a third value when it is determined that at least one of the region of interest information or exposure control information input to the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) satisfies a defined condition based on the calculation result. The image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) may be configured to store the second flag value as a fourth value when it is determined that at least one of the region of interest information or exposure control information input to the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384)) does not satisfy a defined condition based on the calculation result.The above instructions may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine whether the first flag value is the second value until the first flag value becomes the second value from the first value by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3). The above instructions may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) determines whether the second flag value is the third value based on the first flag value being the second value by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3).The above commands may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine at least one of the region of interest information or the exposure control information using the sensor driving information based on determining that the second flag value is the third value.

[0194] In one embodiment, the sensor driving information may include information about a restricted area (e.g., restricted area 1650 of FIG. 16, restricted area 1714 of FIG. 17) associated with a location of the region of interest of the second image frame. The instructions may be individually or collectively executed by the at least one processor (e.g., the processor 120 of FIG. 1, the image signal processor 260 of FIG. 2, and the at least one processor 320 of FIG. 3) to cause the electronic device (e.g., the electronic device 101 of FIG. 1, the electronic device 101 of FIG. 3) to determine whether a start position of the region of interest for the second image frame (e.g., the start position 1713 of FIG. 17) is included within the restricted area (e.g., restricted area 1650 of FIG. 16, restricted area 1714 of FIG. 17). The above commands are individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) can adjust at least one of the region of interest information or the exposure control information based on the sensor driving information when the starting position is included within the restricted area.The above instructions may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine exposure control information based on a frame rate indicating a frequency at which the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3) outputs image frames when the starting position is positioned outside the restricted area.

[0195] In one embodiment, the information about the restricted area may include a first coordinate value (e.g., a y-axis coordinate value) in an axial direction corresponding to a reading direction in which the pixel lines are read. The instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and the at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to determine a maximum exposure value based on a difference between the first coordinate value and a second coordinate value in the axial direction of the starting position and an offset value, when the starting position is included within the restricted area. The above commands may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine the exposure control information such that the exposure time is less than or equal to the maximum exposure value.

[0196] In one embodiment, the instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to determine the maximum exposure value based on a value obtained by multiplying a single pixel line readout time by a value obtained by adding a difference between the first coordinate value and the second coordinate value and the offset value.

[0197] In one embodiment, the instructions may be individually or collectively executed by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, and at least one processor (320) of FIG. 3) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to determine a first region of interest for the image frame. The above commands may be individually or collectively executed by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to cause the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to adjust the region of interest from the second region of interest of the previous frame to a third region of interest, the distance moved from the first region of interest being smaller than the distance moved from the first region of interest, if the starting position is included within the restricted region.

[0198] An operating method of an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 3) including an image sensor (e.g., an image sensor (230) of FIG. 2, a first image sensor (383) of FIG. 3, a second image sensor (384) of FIG. 3) and at least one processor (e.g., a processor (120) of FIG. 1, an image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3)) comprising a plurality of sensor pixels (e.g., sensor pixels (PX) of FIG. 11) arranged along a plurality of pixel lines (e.g., pixel lines (1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118) of FIG. 11) according to one embodiment, comprises: The method may include an operation of obtaining a first image frame from a sensor (384). The method may include an operation of obtaining sensor driving information associated with the first image frame from the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3). The method may include an operation of determining at least one of region of interest information or exposure control information based on the sensor driving information. The method may include an operation in which at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) transmits a control signal including at least one of the determined region of interest information or the determined exposure control information to the image sensor (e.g., image sensor (230) of FIG. 2, first image sensor (383), second image sensor (384) of FIG. 3) via control communication (e.g., control communication (1402) of FIG. 14).The method may include an operation in which the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) reads pixel values ​​from at least one pixel line (e.g., at least one pixel line (1114, 1115, 1116) of FIG. 11) corresponding to the region of interest information among the plurality of pixel lines (e.g., the pixel lines (1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118) of FIG. 11). The method may include an operation in which the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) outputs a second image frame obtained by cropping an area corresponding to the region of interest information among the read pixel values ​​to at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3).

[0199] In one embodiment, the operation of acquiring the first image frame may include an operation of the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383) of FIG. 3, the second image sensor (384) of FIG. 3) outputting a packet including the first image frame to the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3). The packet may include the sensor operating information in at least one of a header or a footer embedded in the packet.

[0200] In one embodiment, the operation of obtaining the sensor driving information may include an operation of obtaining the sensor driving information from the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) through the control communication (e.g., the control communication (1402) of FIG. 14) by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3).

[0201] In one embodiment, the operation of obtaining the sensor driving information from the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) through the control communication (e.g., the control communication (1402) of FIG. 14) may include an operation of the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3) reading the sensor driving information stored in a control communication address (e.g., the control communication address (2210) of FIG. 22) defined in the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3)).

[0202] In one embodiment, the sensor driving information may include at least one flag value. The operation of determining at least one of the region of interest information or exposure control information may include an operation of determining at least one of the region of interest information or exposure control information based on the at least one flag value.

[0203] In one embodiment, the at least one flag value may include a first flag value and a second flag value. The method may include an operation of storing the first flag value as a first value while the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) calculates the sensor driving information. The method may include an operation of storing the first flag value as a second value based on the completion of the calculation of the sensor driving information. The method may include an operation of storing the second flag value as a third value when it is determined that at least one of the region of interest information or exposure control information input to the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) satisfies a defined condition based on the operation result, and an operation of storing the second flag value as a fourth value when it is determined that at least one of the region of interest information or exposure control information input to the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) does not satisfy a defined condition based on the operation result. The operation of determining at least one of the region of interest information or the exposure control information based on the sensor driving information may include an operation of determining whether the first flag value is the second value until the first flag value becomes the second value from the first value.The operation of determining at least one of the region of interest information or the exposure control information based on the sensor driving information may include an operation of determining, based on the first flag value being the second value, whether the second flag value is the third value by the at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, or the at least one processor (320) of FIG. 3). The operation of determining at least one of the region of interest information or the exposure control information based on the sensor driving information may include an operation of determining at least one of the region of interest information or the exposure control information using the sensor driving information based on the second flag value being determined to be the third value.

[0204] In one embodiment, the sensor driving information may include information about a restricted region associated with a location of the region of interest of the second image frame. The operation of determining at least one of the region of interest information or the exposure control information based on the sensor driving information may include, when a start position of the region of interest for the second image frame is included within the restricted region, adjusting at least one of the region of interest information or the exposure control information based on the sensor driving information. The operation of determining at least one of the region of interest information or the exposure control information based on the sensor driving information may include, when the start position is positioned outside the restricted region, determining exposure control information based on a frame rate indicating a frequency at which the image sensor (e.g., the image sensor (230) of FIG. 2, the first image sensor (383), the second image sensor (384) of FIG. 3) outputs image frames.

[0205] In one embodiment, the information about the restricted area may include a first coordinate value in the axial direction corresponding to the reading direction in which the pixel lines are read. The operation of adjusting at least one of the region of interest information or the exposure control information may include an operation of determining a maximum exposure value based on a difference between the first coordinate value and a second coordinate value in the axial direction of the starting position and an offset value. The operation of adjusting at least one of the region of interest information or the exposure control information may include an operation of determining the exposure control information such that an exposure time is less than or equal to the maximum exposure value.

[0206] In one embodiment, the operation of determining the maximum exposure value may include an operation of determining the maximum exposure value based on a value obtained by multiplying a single pixel line readout time by a sum of a difference between the first coordinate value and the second coordinate value and the offset value.

[0207] In one embodiment, the operation of adjusting at least one of the region of interest information or the exposure control information may include an operation of determining a first region of interest. The operation of adjusting at least one of the region of interest information or the exposure control information may include an operation of adjusting the region of interest to a third region of interest, the distance traveled from the second region of interest of the previous frame being less than the distance traveled from the first region of interest, if the starting position is included within the restricted region.

[0208] In one embodiment, a computer-readable non-transitory recording medium may have recorded thereon a computer program for performing at least one of the above-described methods.

[0209] An image sensor, an electronic device including an image sensor, and an operating method thereof according to various embodiments can prevent a phenomenon in which an image frame is dropped when an ROI for obtaining an image frame output from the image sensor is changed.

[0210] An image sensor, an electronic device including an image sensor, and an operating method thereof according to various embodiments can improve the quality of an image acquired for a region of interest.

[0211] In various embodiments, an image sensor, an electronic device including an image sensor, and a method of operating the same can improve the frame rate of image frames output by the image sensor.

[0212] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the description of the present disclosure.

[0213] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0214] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

[0215] In the present disclosure, the functions or operations performed by the electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The functions or operations of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include circuitry for performing calculations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operations of the electronic device described above.

[0216] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0217] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0218] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0219] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0220] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0221] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0222] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, or including a combination of two or more (including a and b, including b and c, including a and c, or including all of a, b, and c).

[0223] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

[0224] In this disclosure, the term "if" will be understood to mean "when, upon," "in response to determining," or "in response to detecting," as the context requires. Similarly, "if it is determined to do," or "if [a stated condition or event] is detected," will optionally be understood to mean "upon determining," or "in response to determining," "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]."

[0225] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. A processing device (or processing circuit) may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0226] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0227] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0228] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

In electronic devices, A camera comprising an image sensor comprising a plurality of sensor pixels arranged along a plurality of pixel lines; At least one processor comprising processing circuitry; and Contains memory that stores instructions, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: Acquire a first image frame from the image sensor, The at least one processor obtains sensor driving information related to the operation of the image sensor from the image sensor, Determine at least one of region of interest information or exposure control information based on the above sensor driving information, Transmitting a control signal including at least one of the determined region of interest information or the determined exposure control information from the at least one processor to the image sensor through control communication, The above image sensor, Based on the above exposure control information, pixel values ​​are read from at least one pixel line corresponding to the region of interest information among the plurality of pixel lines, An electronic device configured to output a second image frame obtained by cropping an area corresponding to the region of interest information among the extracted pixel values ​​to the at least one processor. In claim 1, The image sensor outputs a packet including the first image frame to the at least one processor, An electronic device wherein the packet includes the sensor driving information in at least one of a header or a footer embedded in the packet. In claim 1, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: An electronic device that obtains sensor driving information from the image sensor through the control communication by the at least one processor. In claim 3, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: An electronic device that reads the sensor driving information stored in a control communication address defined within the image sensor by at least one processor. In claim 3, The constant sensor operation information includes at least one flag value, An electronic device, wherein the commands are individually or collectively executed by the at least one processor to cause the electronic device to determine at least one of the region of interest information or the exposure control information based on the at least one flag value. In claim 5, wherein at least one flag value comprises a first flag value and a second flag value, The above image sensor: While calculating the above sensor driving information, store the first flag value as the first value, Based on the completion of the operation of the above sensor driving information, the first flag value is stored as a second value, If it is determined that at least one of the region of interest information or exposure control information input to the image sensor satisfies a defined condition based on the above operation result, the second flag value is stored as a third value, If it is determined that at least one of the region of interest information or exposure control information input to the image sensor does not satisfy the defined condition based on the above operation result, the second flag value is stored as the fourth value, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: determining whether the first flag value is the second value by at least one processor until the first flag value becomes the second value from the first value; Based on the first flag value being the second value, determining whether the second flag value is the third value by the at least one processor, An electronic device that determines at least one of the region of interest information or the exposure control information using the sensor driving information based on the determination that the second flag value is the third value. In claim 1, The sensor driving information includes information about a restricted area associated with the location of the area of ​​interest of the second image frame, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: Determine whether the starting position of the region of interest for the second image frame is included within the restricted region, If the starting position is included within the restricted area, at least one of the region of interest information or the exposure control information is adjusted based on the sensor driving information, An electronic device that determines exposure control information based on a frame rate indicating a frequency at which the image sensor outputs image frames when the starting position is located outside the restricted area. In claim 7, Information about the above restricted area includes a first coordinate value in the axial direction corresponding to the reading direction in which the pixel lines are read, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: If the starting position is included within the restricted area, the maximum exposure value is determined based on the difference and offset value of the first coordinate value and the second coordinate value in the axial direction of the starting position, An electronic device that determines the exposure control information so that the exposure time is less than or equal to the maximum exposure value. In claim 8, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: An electronic device that determines the maximum exposure value based on a value obtained by multiplying a single pixel line reading time by a value obtained by adding the difference between the first coordinate value and the second coordinate value and the offset value. In claim 7, The instructions are individually or collectively executed by the at least one processor, such that the electronic device: Determine a first region of interest for the above image frame, An electronic device that adjusts the region of interest to a third region of interest having a distance less than the first region of interest from the second region of interest of the previous frame when the starting position is included within the restricted region. A method of operating an electronic device comprising an image sensor including a plurality of sensor pixels arranged along a plurality of pixel lines and at least one processor, An operation of acquiring a first image frame from the image sensor; An operation in which at least one processor obtains sensor driving information associated with the first image frame from the image sensor; An operation of determining at least one of region of interest information or exposure control information based on the sensor driving information; An operation in which the at least one processor transmits a control signal including at least one of the determined region of interest information or the determined exposure control information to the image sensor via control communication; An operation in which the image sensor reads pixel values ​​from at least one pixel line corresponding to the region of interest information among the plurality of pixel lines; and A method comprising an operation in which the image sensor outputs a second image frame, which is a cropped area corresponding to the region of interest information among the read pixel values, to the at least one processor. In claim 11, The operation of obtaining the above first image frame is: The image sensor comprises an operation of outputting a packet including the first image frame to the at least one processor, A method wherein the packet includes the sensor driving information in at least one of a header or a footer embedded in the packet. In claim 11, A method wherein the operation of obtaining the sensor driving information includes an operation of obtaining the sensor driving information from the image sensor through the control communication by the at least one processor. In claim 13, The operation of obtaining the sensor driving information from the image sensor through the above control communication is as follows: A method comprising the operation of at least one processor reading sensor driving information stored in a control communication address defined within the image sensor. In claim 13, The above sensor driving information includes at least one flag value, The operation of determining at least one of the above-mentioned area of ​​interest information or exposure control information is: A method comprising an operation of determining at least one of the region of interest information or exposure control information based on the at least one flag value.

Images