Electronic device for adjusting amount of light received by image sensor

Abstract

This electronic device comprises: a first processor (302); and an image sensor (301) configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor. The image sensor (301) may perform: an operation of calculating a first shutter value and a first gain value for allowing the brightness of image data obtained from the image sensor to converge to a designated target brightness value; and a streaming operation of obtaining image data by using the first shutter value and the first gain value and outputting the image data to the first processor (302).

Description

An electronic device for controlling the amount of light received by an image sensor

[0001] The present disclosure relates to an electronic device capable of controlling the amount of light received by an image sensor of a camera.

[0002] The shutter acts like the eyelids of the human eye, and shutter speed refers to the speed at which the shutter opens and closes. A slower shutter speed allows the image sensor to receive light for a longer period of time, resulting in a brighter image. A faster shutter speed allows the image sensor to receive light for a shorter period of time, resulting in a darker image. Shutters can be categorized into mechanical shutters, which operate by moving the shutter, and electronic shutters, which have no mechanical parts. Portable electronic devices (e.g., smartphones) typically use electronic shutters. Therefore, shutter speed can be expressed as exposure time (the amount of time the pixel array on the image sensor receives light).

[0003] Overexposure can make an image too bright, resulting in severe saturation, while relatively underexposure can make it too dark, making it difficult to discern. Electronic devices can adjust the amount of light entering the image sensor to produce an image with appropriate brightness. For example, an electronic device can perform auto exposure control (AEC), which uses an electronic shutter to adjust the exposure time.

[0004] The above information is provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] When a camera is turned on, AEC can be performed. For example, an image sensor can generate image data by converting light received through a lens into an electrical signal and converting the electrical signal into a digital signal. In an electronic device, a processor (e.g., an application processor (AP) and / or an image signal processor (ISP)) can use the image data received from the image sensor to determine the duration of exposure of the image sensor to light and the gain (the amplification rate of the electrical signal) so that the image data has a specified brightness value.

[0006] An image sensor can generate image data based on an exposure time and gain determined by a processor. The image sensor can provide the generated image data to the processor. The processor can process the image data received from the image sensor into preview data for display on a display and display the preview data on the display.

[0007] In an embodiment of the present disclosure, the image sensor directly performs calculations of shutter and gain values ​​for brightness conversion, thereby improving processing speed, reducing frame loss and / or power consumption, and enabling efficient use of resources.

[0008] In embodiments of the present disclosure, an electronic device can quickly enter a preview mode for displaying an image on a display by performing AEC on an image sensor without intervention of a processor.

[0009] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] According to one embodiment, an electronic device includes a processor; an image sensor configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor; and a memory storing instructions. The instructions, when individually or collectively executed by the image sensor, may cause the image sensor to perform an operation of calculating a first shutter value and a first gain value so that brightness of image data acquired by the image sensor converges to a designated target brightness value. The instructions may cause the image sensor to perform a streaming operation of acquiring image data using the first shutter value and the first gain value and outputting the image data to the processor.

[0011] According to one embodiment, a method of operating an electronic device is provided. The electronic device includes a processor; and an image sensor configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor. The method may include an operation in which the image sensor calculates a first shutter value and a first gain value so that brightness of image data acquired by the image sensor converges to a designated target brightness value; and a streaming operation in which the image sensor acquires image data using the first shutter value and the first gain value and outputs the image data to the processor.

[0012] According to one embodiment, a recording medium is provided for storing instructions readable by an image sensor in an electronic device. The image sensor can convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor. The instructions, when individually or collectively executed by the image sensor, can cause the image sensor to perform an operation of calculating a first shutter value and a first gain value for causing the brightness of image data acquired by the image sensor to converge to a designated target brightness value. The instructions can cause the image sensor to perform a streaming operation of acquiring image data using the first shutter value and the first gain value and outputting the image data to the processor.

[0013] According to embodiments of the present disclosure, the image sensor directly calculates shutter and gain values ​​for brightness conversion, thereby improving processing speed, reducing frame loss and / or power consumption, and enabling efficient use of resources. According to embodiments of the present disclosure, an electronic device can quickly enter a preview mode for displaying an image on a display. In addition, various effects, directly or indirectly understood through this document, may be provided.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0016] FIG. 3 is a block diagram of an electronic device according to various embodiments.

[0017] FIG. 4 is a drawing for explaining examples of streaming modes performed in an electronic device.

[0018] FIG. 5 is a flowchart illustrating an AEC mode performed in an image sensor according to one embodiment.

[0019] Figure 6 illustrates an example of an exposure chart available from an image sensor to ensure that the brightness of the image converges to the target brightness.

[0020] FIG. 7 is a flowchart illustrating operations performed in an electronic device to quickly obtain an image having appropriate brightness from an image sensor, according to one embodiment.

[0021] FIG. 8 is a flowchart illustrating operations performed in an electronic device to quickly obtain an image having appropriate brightness from an image sensor, according to one embodiment.

[0022] FIGS. 9a, 9b, and 9c are diagrams illustrating exposure values ​​calculated while performing AEC on an image sensor (301) according to one embodiment.

[0023] FIG. 10 is a flowchart illustrating operations performed in an electronic device to quickly obtain an image having appropriate brightness from an image sensor, according to one embodiment.

[0024] FIG. 11 is a diagram for explaining an AEC mode performed in an image sensor according to one embodiment.

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0026] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0049] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0050] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. According to one embodiment, the image stabilizer (240) can detect the movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer.

[0051] The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0052] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations can include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) can perform control (e.g., exposure time control, readout timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least a part of the processor (120) or may be configured as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) by the processor (120) as is or after undergoing additional image processing.

[0053] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0054] FIG. 3 is a block diagram of an electronic device (300) according to various embodiments. FIG. 4 is a drawing for explaining examples of streaming modes performed in the electronic device (300).

[0055] Referring to FIG. 3, an electronic device (300) (e.g., electronic device (101)) may include an image sensor (301), a first processor (302), a display (303), and a first memory (304).

[0056] The image sensor (301) can convert light received through the lens into an electrical signal. Furthermore, the image sensor (301) can convert the electrical signal into image data and provide or output the converted image data to the first processor (302). According to various embodiments of the present disclosure, this will be described in more detail below.

[0057] The image sensor (301) may be a component of a camera (e.g., the image sensor (230) of FIG. 2) and may generate image data and provide it to a first processor (e.g., an application processor) (302). The first processor (302) may process the image data (e.g., compress it for storage in a first memory (304) and / or resize it for previewing it on a display (303)) and display the processed image data on a display (303) (e.g., the display module (160) of FIG. 1) or store it in the first memory (304).

[0058] The first memory (304) (e.g., memory (130) of FIG. 1) can store instructions. When executed by the first processor (302), the instructions can cause the first processor (302) to perform a given function (e.g., auto exposure control (AEC), white balance).

[0059] The image sensor (301) may include an analog block (310) and a digital block (320). The analog block (310) may convert light received through a lens (e.g., the lens assembly (210) of FIG. 2) into an electrical signal and output the signal to the digital block (320). The analog block (310) may include a driver (311), a pixel array (312), and a readout circuit (313). The digital block (320) may control the analog block (310) based on the control of the first processor (302) and provide image data received from the analog block (310) to the first processor (302). According to one embodiment, the digital block (320) may include a second memory (321), a controller (322), a second processor (330), and a second interface (340).

[0060] The driver (311) can output a control signal (e.g., a selection signal, a reset signal, and a transmission signal) to the pixel array (512) based on the control of the controller (322).

[0061] A pixel array (312) may include a plurality of pixels (P(i, j); where i and j represent pixel positions, i is a row number, and j is a column number) arranged in two dimensions. For example, each of the pixels may include a photoelectric transformation element (or a position sensitive detector (PSD)) and a plurality of transistors (e.g., a reset transistor, a transfer transistor, a selection transistor, and a driver transistor).

[0062] The pixel array (312) may include a plurality of column lines. The column lines may be electrically connected to pixels arranged in a column direction, respectively. The pixels of the ith row in the pixel array (312) may receive a control signal from the driver (311), perform photoelectric conversion (convert an optical signal into an electrical signal) by the control signal, and output an electrical signal to the readout circuit (313) through the jth column line.

[0063] For example, the reset transistor can reset the floating diffusion (FD) region of the corresponding pixel in response to a reset signal (RS) received from the driver (311). The transfer transistor can transfer the charge accumulated in the photoelectric conversion element to the floating diffusion region in response to a transfer signal received from the driver (311). The drive transistor can amplify the electrical potential of the charge accumulated in the floating diffusion region. The select transistor can output the electrical potential amplified by the drive transistor to the column line in response to a select signal received from the driver (311).

[0064] The readout circuit (313) can sequentially select column lines one by one, receive an electrical signal (electrical potential) from the selected column lines, and output it to a digital block (320).

[0065] The readout circuit (313) may include an analog-to-digital converter (313a; ADC) that converts an electrical signal received from a selected column line into pixel data (digital signal) and outputs it. According to one embodiment, the analog-to-digital converter (313a) may be included in the digital block (320) or may be included as a separate component.

[0066] The digital block (320) (e.g., controller (322)) sequentially receives pixel data from the readout circuit (313) and stores it in a buffer (321a) (e.g., memory area allocated for pixel data storage in the second memory (321)). When pixel data is collected from designated pixels (i.e., one image data (in other words, a frame) is obtained), the image data can be output to another component of the electronic device (300) (e.g., first processor (302)) through the second interface (340).

[0067] In the digital block (320), the controller (322) can control the analog block (310) and the analog-to-digital converter (313a; ADC). The controller (322) can control the operation of the driver (311) (e.g., a reset operation (e.g., an operation of outputting a reset signal to pixels in the ith column among pixels), a transmission operation (e.g., an operation of outputting a transmission signal to pixels in the ith column), and a row line selection operation (e.g., an operation of outputting a selection signal to pixels in the ith column)). In addition, the controller (322) can control the column line selection operation of the readout circuit (313) (e.g., an operation of enabling the jth column line and disabling the remaining column lines).

[0068] The controller (322) can control the input / output of the buffer (321a). For example, the controller (322) can integrate pixel data sequentially received from the readout circuit (313) to generate one frame and store it in the buffer (321a). The controller (322) can output the frame stored in the buffer (321a) to the first processor (302) via the second interface (340). The controller (322) can also output the frame stored in the buffer (321a) to the second processor (330).

[0069] A series of operations (e.g., a reset operation, a transmission operation, a low line selection operation, an operation of generating a frame and storing it in a buffer (321a), and an operation of outputting the frame stored in the buffer (321a) to the first processor (302) and / or the second processor (330)) performed to generate a frame in the image sensor (301) as described above and output it to the first processor (302) and / or the second processor (330) may be referred to as a streaming operation (in other words, a streaming mode).

[0070] The streaming operation may be performed based on the configuration information stored in the second memory (321). For example, the controller (322) may perform the streaming operation based on the configuration information stored in the second memory (321) in response to a “streaming command” received from the first processor (302) via the second interface (340). According to one embodiment, when streaming is not required (e.g., when the streaming mode is set to streaming off), the image sensor (301) may be operated in a standby mode. For example, when the second memory (321) is a volatile memory, the second memory (321) may be supplied with battery power, for example, through a PMIC (e.g., a power management module (188) of FIG. 1), so that the configuration information is not erased while in the standby mode, and the power supply to other components (e.g., the analog block (310), the controller (322), and the second processor (330)) may be cut off.

[0071] The second memory (321) can receive and store setting information for streaming operation from the first memory (304) through the second interface (340). For example, the second interface (340) can transfer the setting information stored in the first memory (304) to the second memory (321) under the control of the first processor (302) or the controller (322). The setting information stored in the second memory (321) can include information on parameters for driving the camera (e.g., light exposure time, frame interval indicating the time between the exposure start time of a frame and the exposure start time of the next frame, so-called frame readout time required for the readout circuit (313) to read electrical signals from all pixels, frame rate (e.g., fps (frames per second)), white balance, shutter speed). In addition, the setting information stored in the second memory (321) may include information about a designated streaming mode (e.g., AEC (auto exposure control) mode, preview mode, video shooting mode, still shooting mode, panorama mode, or scene mode, etc.). For example, when the image sensor (301) is maintaining information related to parameters for operating the camera in the standby mode or the streaming mode, and the image sensor (301) is changed from the standby mode to the streaming mode, the image sensor (301) may generate a frame using the information and output it to the first processor (302) and / or the second processor (330).

[0072] The second interface (340) can connect the digital block (320) to the first processor (302) via at least one data line, and transmit a command of the first processor (302) to the digital block (320) (e.g., the controller (322) and / or the second processor (330)). For example, the second interface (340) can receive a command for setting the operating mode of the image sensor (301) to a streaming mode or a standby mode from the first processor (302) and transmit the command to the digital block (320). The second interface (340) can output a frame generated in the digital block (320) to the first processor (302).

[0073] In the image sensor (301), the digital block (320) can communicate data with the first processor (302) through a CCI (Camera Control Interface) interface (e.g., an I2C (inter integrated circuit) interface) (341) in the second interface (340). According to one embodiment, the controller (322) can receive setting information for performing AEC (auto exposure control) in the digital block (320) (e.g., an exposure chart (see FIG. 6) for obtaining a shutter value and a gain value (in other words, a sensor sensitivity value) as parameters required for performing the AEC mode) from the first processor (302) through the CCI interface (341). The controller (322) can provide the result performed in the AEC mode (e.g., information on the brightness of the frame) to the first processor (302) through the CCI interface (341). The shutter value described above can be a value indicating a shutter speed or an exposure time. The above-described gain value may be a value representing the amplification rate of an electric signal in the readout circuit (313).

[0074] In the image sensor (301), the digital block (320) can communicate data with the first processor (302) through the MIPI (mobile industry processor interface) interface (342) in the second interface (340). For example, the controller (322) can output image data (frame) generated by the image sensor (301) to the first processor (302) through the MIPI interface (342). According to one embodiment, the controller (322) can also provide the result (e.g., information regarding the brightness of the frame) performed in the AEC mode to the first processor (302) through the MIPI interface (342).

[0075] When the AEC mode is performed under the leadership of the first processor (302), data communication via an interface between the image sensor (301) and the first processor (302) is essential, and thus power consumption may occur due to the data communication. In addition, while the first processor (302) processes the AE (auto exposure) procedure, the image sensor (301) may perform streaming. Such streaming may result in frame loss (e.g., the first processor (302) does not process a received frame while performing the AE operation) and / or unnecessary power consumption. According to various embodiments of the present disclosure, by performing the AE procedure in the image sensor (301), frame loss and / or power consumption may be reduced, and resources may be used efficiently.

[0076] In the image sensor (301), the second processor (330) can perform the AEC mode (410; see FIG. 4). The operating mode of the camera can be switched from a power-off mode in which power supply to the image sensor (301) is cut off, or a standby mode in which power supply to the second memory (321) is maintained to prevent the setting information stored in the second memory (321) from being erased, to the AEC mode (410) based on the occurrence of a camera on event. The camera on event can include a user input to execute a camera application. For example, the first processor (302) can receive a user touch input for a camera icon from the display (303). In response to receiving the camera on event, the first processor (302) can control the PMIC to supply power to the image sensor (301). In addition, when a camera on event occurs in the power off mode, the first processor (302) can transmit setting information (e.g., exposure line) required for the image sensor (301) to operate in a streaming mode (e.g., AEC mode (410)) to the image sensor (301) through the first interface (370). Here, the first interface (370) is an element configured in the first processor (302) and can include a CCI interface (371) and a MIPI interface (372). For example, the first processor (302) can transmit setting information to the image sensor (301) through the CCI interface (371) and receive image data generated by the image sensor (301) from the image sensor (301) through the MIPI interface (342).

[0077] According to one embodiment, when a command stored in the first memory (304) and / or the second memory (321) is executed by the second processor (330), the image sensor (301) may perform an AEC mode (410). The AEC mode (410) may include a reset operation, a transfer operation, a low line selection operation, an operation of generating a frame and storing it in a buffer (321a), and an operation of outputting the frame stored in the buffer (321a) to the second processor (330). The second processor (330) may be a block (or logic) in charge of command processing (i.e., AEC) in the digital block (320).

[0078] The second processor (330) can perform the AEC mode (410) at a fast speed (or high speed). According to one embodiment, the second processor (330) can control the driver (311) and the readout circuit (313) through the controller (322) to output pixel data at a high speed (e.g., 240 fps). While the image sensor (301) performs the AEC mode (410) at a high speed, the second processor (330) can control the digital block (320) to generate frames at a low resolution. For example, the second processor (330) can control the readout circuit (313) through the controller (322) to perform a binning mode (or, binning operation). The binning mode may include, for example, a process in which the readout circuit (313) adds electrical signals (electrical potentials) received from adjacent pixels (e.g., four) and outputs the combined electrical signals to the controller (322) by performing an ADC. By performing this binning mode, a relatively small number of pixel data may be output to the controller (322), and as a result, the resolution of the frame generated from the digital block (320) may be lowered. The reason for lowering the resolution may be to allow the image sensor (301) to perform AE processing quickly. Here, the AE processing may be defined as a task of determining the time for which the pixel array (312) is exposed to light and the amplification rate (hereinafter, a gain value) of the electrical signal in the readout circuit (313) so that the image (frame) has a designated target brightness value. The second processor (330) can perform fast AE processing by processing low-resolution frames.

[0079] According to one embodiment of the AE procedure, the second processor (330) can obtain first image data from the controller (322) through the buffer (321a) and check the brightness value of the first image data (hereinafter, referred to as the first brightness value). The second processor (330) can calculate a shutter value and a gain value for converging the brightness of the image data to be provided from the image sensor (301) to the first processor (302) from the first brightness value to a target brightness value (hereinafter, referred to as the second brightness value). The second processor (330) can control the driver (311) and the readout circuit (313) through the controller (322) so that the pixel array (312) is exposed to light based on the calculated shutter value and an electric signal is output based on the calculated gain value. The AE procedure performed in the image sensor (301) will be described in detail below with reference to FIG. 5.

[0080] When the AE procedure is completed by the second processor (330) (e.g., the brightness of the image data converges to the target brightness value), the image sensor (301) may perform a preview mode (420; see FIG. 4). For example, the controller (322) may control the driver (311) and the readout circuit (313) to output pixel data at a low speed (e.g., 60 fps). The controller (322) may terminate the binning mode, generate a frame of a specified high resolution, and provide the frame to the first processor (302) via the second interface (340) (e.g., the MIPI interface (342)). The first processor (302) may include an image signal processor (ISP; 360) (e.g., the image signal processor (260) of FIG. 2). The ISP (360) can process (e.g., resize) a frame received through the first interface (370) (e.g., MIPI interface (372)) into a preview image and output it to the display (303). The display (303) can display the preview image received from the first processor (302). Thereafter, the first processor (302) can command the image sensor (301) through the first interface (370) (e.g., CCI interface (371)) to switch the operating mode of the camera from the preview mode (420) to another streaming mode (e.g., the still shooting mode (431), the video shooting mode (432), or the video call mode (433) of FIG. 4) based on a user input. The controller (322) can receive the mode switching command from the second interface (340) (e.g., CCI interface (341)). The controller (322) can check the setting information corresponding to the mode requested by the first processor (302) in the second memory (321) and generate a frame based on the checked setting information. The controller (322) can transmit the generated frame to the first processor (302) through the second interface (340) (e.g., MIPI interface (342)).

[0081] According to one embodiment, the AEC mode may be classified into an automatic mode, a manual mode, and a delay mode, as exemplified in Table 1 below, with respect to the performing entity and whether the mode is automatically switched. The first processor (302) may transmit a command to start streaming (e.g., a stream on command) to the image sensor (301) through the first interface (370) (e.g., a CCI interface (371)) based on the occurrence of a camera on event. When the image sensor (301) is in a standby mode, setting information regarding one of the automatic mode, the manual mode, and the delay mode may already be stored in the second memory (321). When the image sensor (301) is in a power-off state, the first processor (302) may transmit configuration information including information regarding one of an automatic mode, a manual mode, and a delay mode to the image sensor (301) through the first interface (370) (e.g., a CCI interface (371)) based on the occurrence of a camera-on event. The second processor (330) may perform the AEC mode based on the configuration information.

[0082] AEC's performer: Automatic switching to preview mode. Automatic mode: Image sensor. Automatic switching. Manual mode: Image sensor. Manual switching. Delay mode: Image sensor. Manual switching (standby).

[0083] If the setup information is for automatic mode, the second processor (330) can control the controller (322) to perform AEC mode and, when the AE procedure is completed as a result of the performance, automatically switch the mode and perform preview mode on the image sensor (301). Accordingly, in automatic mode, the time until streaming is performed can be shortened without requiring additional data communication.

[0084] If the setup information corresponds to the manual mode, the second processor (330) may perform the AEC mode to complete the AE procedure and, based on the control of the first processor (302), may switch the mode from the AEC mode to the preview mode. According to one embodiment, the second processor (330) may transmit a message indicating that the AE procedure is completed to the first processor (302) through the second interface (340) (e.g., the CCI interface (341)). For example, the second processor (330) may transmit a flag to the first processor (302) that triggers the first processor (302) to issue a command to start streaming (e.g., a stream on command). The first processor (302), in response to receiving the flag, may transmit a streaming command to the image sensor (301) through the first interface (370) (e.g., the CCI interface (371)). The image sensor (301) operates in standby mode after completing the AE procedure, and can then perform a preview mode in response to a streaming command received through the second interface (340) (e.g., CCI interface (341)).

[0085] If the setting information corresponds to the delay mode, the second processor (330) may perform the AEC mode to complete the AE procedure. After the AE procedure is completed, the second processor (330) may wait for a command from the first processor (302) without switching to the preview mode. If a designated waiting time (e.g., 66 ms) has elapsed after transmitting the first command (e.g., stream on command) to start streaming, the first processor (330) may transmit a second command to start streaming to the image sensor (301) through the first interface (370) (e.g., CCI interface (371)). The image sensor (301) may operate in the standby mode after the AE procedure is completed, and then perform the preview mode in response to the second command received through the second interface (340) (e.g., CCI interface (341)).

[0086] FIG. 5 is a flowchart illustrating an AEC mode performed in an image sensor (301) according to one embodiment. FIG. 6 illustrates an example of an exposure chart available in the image sensor (301) to converge the brightness of an image to a target brightness. In FIG. 6, the x-axis (horizontal axis) represents an exposure value (Ev) and the y-axis (vertical axis) represents a shutter value (Tv) or a gain value (Sv). Instructions may be stored in the first memory (304) and / or the second memory (321). When the instructions are executed by the image sensor (301) (e.g., the second processor (330)), the instructions may cause the image sensor (301) to perform the operations of FIG. 5. Information corresponding to the exposure chart of FIG. 6 may be stored in the second memory (321). The image sensor (301) can utilize information regarding exposure leads stored in the second memory (321) when performing AEC mode. According to one embodiment, the image sensor (301) can perform the operations of FIG. 5 in response to a streaming command (e.g., stream on command) of the first processor (302).

[0087] In operation 510, the image sensor (301) can check the current brightness value (hereinafter, the first brightness value) in the current frame (e.g., the frame stored in the buffer (321a)) (hereinafter, the first image data). The image sensor (301) can calculate the brightness value by calculating the average for the entire area of ​​the current frame. Alternatively, the image sensor (301) can also calculate the average for a designated region of interest (ROI) in the current frame. For example, the image sensor (301) can divide the current frame into 16 (4*4), 64 (8*8), 256 (16*16), or 4096 (64*64) areas and calculate the average for an ROI (e.g., a central area) among the areas. The brightness value can mean a luminance value indicating how bright an image (subject) appears when viewed by an observer (user).

[0088] In operation 520, the image sensor (301) can obtain a shutter value (Tv; see FIG. 6) and a gain value (Sv; see FIG. 6) to converge the brightness of the frame from a first brightness value to a target brightness value (hereinafter, a second brightness value). The shutter value (Tv; see FIG. 6) may be a value indicating a shutter speed or exposure time. The gain value (Sv; see FIG. 6) may be a value indicating an amplification rate of an electric signal in the readout circuit (313).

[0089] According to one embodiment, the image sensor (301) can obtain a target exposure value (target Ev; see FIG. 6) for causing the frame to have a target brightness value, for example, using the exposure diagram of FIG. 6. Here, the exposure value may mean an illuminance value indicating the amount of light reaching the image sensor (301) (subject). Referring to FIG. 6, the image sensor (301) can calculate delta Ev using a given mathematical formula. For example, if the given mathematical formula for calculating delta Ev is log2(Current Y / Target Y)(= log(Current Y / Target Y) / log 2) and the current brightness value (Current Y) is half of the target brightness value (Target Y), delta Ev is calculated as -1(=log2(0.5)). When the current exposure value (Current Ev) corresponding to the current brightness value is 11, the target exposure value (target Ev) (= Current Ev + delta Ev) is 10. The image sensor (301) can confirm that the shutter value (Tv) corresponding to the target exposure value (target Ev) of 10 is 6.7 and the gain value (Sv) is 4 in the exposure diagram of Fig. 6.

[0090] In operation 530, the image sensor (301) can generate a frame (hereinafter, second image data) using the acquired shutter value (Tv) and gain value (Sv).

[0091] In operation 540, the image sensor (301) can determine whether the second image data has a target brightness value. If the second image data does not converge to the target brightness value, the image sensor (301) can perform operations 520 and 530 again and selectively perform operation 540 based on the result. For example, if the number of repetitions reaches a predetermined threshold number (e.g., 4 times) or the elapsed time exceeds a threshold time set for the AE procedure (e.g., several ms (milliseconds)), the following operation 550 can be performed.

[0092] When the AE procedure is completed in the image sensor (301) (e.g., when the second image data has a target brightness value (e.g., within the target brightness range or within the brightness range for the target brightness value) or when the above-described condition (e.g., when a threshold number of times is reached or a threshold time is exceeded) is satisfied), the image sensor (301) may perform streaming to output a frame (e.g., image data) to the first processor (302) in operation 550. For example, the image sensor (301) may generate a frame based on the shutter value (Tv) and the gain value (Sv) previously acquired through the AE procedure (operations 510 to 530) and the setting information corresponding to the preview mode (e.g., frame rate, white balance, resolution). The image sensor (301) may store the generated frame in the buffer (321a) and output it to the first processor (302) through the second interface (340) (e.g., the MIPI interface (342)).

[0093] According to one embodiment, the exposure curve used to obtain the target exposure value (Target Ev) may vary depending on the properties of the image to be obtained. For example, if the image to be obtained is set to motion blur (e.g., an image having a visual effect in which the movement of the subject remains as an afterimage), the image sensor (301) may use the exposure curve for obtaining the motion blur during the AE procedure. For example, the shutter value in the corresponding exposure curve may be set higher than the shutter value in a general exposure curve. Additionally, the gain value in the corresponding exposure curve may be set relatively lower than the gain value in a general exposure curve. As another example, the gain value may vary depending on the type of the set streaming mode. If the streaming mode is a video shooting mode, the image sensor (301) may use an exposure curve optimized for video digital image stabilizer (Vdis).

[0094] According to the embodiment described with reference to the above-described FIG. 5, the AE procedure can be performed quickly (e.g., 240 fps) on the image sensor (301). Consequently, resources (e.g., communication interface, memory, battery power) can be used efficiently since the AE procedure is performed primarily by the image sensor (301) that creates the frame (image data).

[0095] FIG. 7 is a flowchart illustrating operations performed in an electronic device (300) to quickly obtain an image having an appropriate brightness from an image sensor (301), according to one embodiment. Instructions may be stored in a first memory (304) and / or a second memory (321). When the instructions are executed by the image sensor (301) (e.g., a second processor (330)), the instructions may cause the image sensor (301) to perform the operations of FIG. 7. Some of the operations of FIG. 7 may be performed by the first processor (302).

[0096] In operation 710, the image sensor (301) may receive a streaming command (e.g., a command to operate in preview mode) from the first processor (302). According to one embodiment, the image sensor (301) may receive setting information (e.g., frame rate, resolution, white balance) for setting the preview mode from the first memory (304) through the first processor (302) and store the setting information in the second memory (321). In addition, the image sensor (301) may receive a command from the first processor (302) to perform AEC in an automatic mode (e.g., Table 1). For example, the image sensor (301) may receive a mode switching command from the first processor (302) to automatically switch to the preview mode after performing the AEC mode. The image sensor (301) may receive information regarding a target brightness value (and / or a target exposure value) from the first processor (302). The image sensor (301) can receive a streaming command from the first processor (302) after receiving the mode switching command and information about the AEC mode.

[0097] In operation 720, the image sensor (301) may perform the AEC mode in response to the streaming command. According to one embodiment, the operations of FIG. 5 described above for AE processing may be performed. The operations of FIG. 5 are, as an example, AE processing, and various AE algorithms may be used for the AE processing. For example, the APEX System (additive system of photographic exposure) may be used as an algorithm for AE processing in operation 720. Therefore, in this embodiment, the time until streaming is performed may be shortened without additionally requiring data communication.

[0098] In operation 730, the image sensor (301) may perform a designated streaming mode (e.g., preview mode) based on the completion of the AE procedure as a result of performing operation 720 (e.g., when the brightness of the frame converges to a first target brightness value (e.g., within a target brightness range or within a brightness range for the target brightness value) or when a condition described with reference to FIG. 5 (e.g., ...

[0099] According to one embodiment, operations 740 and 750 may be performed as a verification procedure for the AE procedure performed by the image sensor (301) (or a procedure for a more accurate AE procedure).

[0100] In operation 740, the first processor (302) can check the brightness value of the frame received from the image sensor (301). The first processor (302) can perform operation 750 based on the checked brightness value. For example, the first processor (302) can check whether the checked brightness value is different from a second target brightness value (e.g., may be the same as or different from the first target brightness value) and perform operation 750 for the secondary AE procedure accordingly.

[0101] In operation 750, the first processor (302) may perform a secondary AEC mode based on the verified brightness value. The AE algorithm used for the secondary AE procedure may be an algorithm according to the embodiment of FIG. 5 or may be an AE algorithm of another type (e.g., APEX System). The first processor (302) may transmit the shutter value and / or gain value calculated as a result of performing the secondary AEC mode to the image sensor (301). The image sensor (301) may perform a streaming operation to acquire image data using the shutter value and / or gain value received from the first processor (302) and output the image data to the first processor (302). Through operation 750, a verification procedure and / or calculation of more accurate shutter values ​​and gain values ​​using the first processor (302) may be performed. Operations 740 and 750 are optional and may be omitted. The secondary AE procedure requires data communication via an interface between the image sensor (301) and the first processor (302), and thus may take relatively longer than the primary AE procedure performed by the image sensor (301) itself. If verification of the AE procedure and / or a more accurate AE procedure is prioritized, operation 750 may be performed.

[0102] In operation 760, the first processor (302) may process the frame received from the image sensor (301). Operation 760 may optimize the properties of the acquired image data before displaying it on the display. For example, the ISP (360) in the first processor (302) may adjust the size of the frame to fit the size of the display area of ​​the screen. As another example, the ISP (360) may perform color correction and / or backlight correction on the frame. As another example, the ISP (360) may also perform white balance for balancing white colors in the frame. Operation 760 is optional and may be omitted.

[0103] In operation 770, the first processor (302) can display a frame received (or processed) from the image sensor (301) on the display (303).

[0104] According to the embodiment described with reference to the above-described FIG. 7, the first AE procedure can be performed quickly (e.g., 240 fps) on the image sensor (301), and optionally, the second AE procedure can be performed under the leadership of the first processor (302). Consequently, a preview image can be displayed quickly on the display (303). In addition, by performing the first AE procedure under the leadership of the image sensor (301), resources (e.g., communication interface, memory, battery power) can be used efficiently.

[0105] FIG. 8 is a flowchart illustrating operations performed in an electronic device (300) to quickly obtain an image having an appropriate brightness from an image sensor (301) according to one embodiment. In the description of FIG. 8, any content overlapping with FIG. 7 may be omitted or briefly described. Instructions may be stored in a first memory (304) and / or a second memory (321). When the instructions are executed by the image sensor (301) (e.g., the second processor (330)), the image sensor (301) may perform the operations of FIG. 8. Some of the operations of FIG. 8 may be performed by the first processor (302). FIGS. 9A, 9B, and 9C are diagrams illustrating exposure values ​​calculated while performing AEC in the image sensor (301) according to one embodiment.

[0106] In operation 810, the image sensor (301) may receive a streaming command (e.g., a command to operate in preview mode) from the first processor (302). According to one embodiment, the image sensor (301) may receive setting information (e.g., frame rate, resolution, white balance) for setting the preview mode from the first memory (304) through the first processor (302) and store the setting information in the second memory (321). In addition, the image sensor (301) may receive a command from the first processor (302) to perform AEC in a manual mode (e.g., Table 1). For example, the image sensor (301) may notify the first processor (302) of the completion of the AE procedure and receive a mode switching command from the first processor (302) to switch to the preview mode based on a secondary command of the first processor (302). The image sensor (301) can receive information about a target brightness value (and / or a target exposure value) from the first processor (302). After receiving the mode switching command and the information about the AEC mode, the image sensor (301) can receive a streaming command from the first processor (302).

[0107] In operation 820, the image sensor (301) may perform AEC mode in response to a streaming command. According to one embodiment, the operations of FIG. 5 described above for AE processing may be performed. The operations of FIG. 5 are one embodiment of AE processing, and various AE algorithms may be used for AE processing.

[0108] At operation 830, the image sensor (301) may transmit a message (e.g., “flag” in the description related to Table 1) to the first processor (302) indicating that the AE procedure has been completed.

[0109] According to one embodiment, operations 840 and 850 may be performed as a verification procedure for the AE procedure performed by the image sensor (301) (or a procedure for a more accurate AE procedure).

[0110] In operation 840, the first processor (302) may obtain the result of the AE procedure from the image sensor (301). According to one embodiment, the first processor (302) may access a memory area (e.g., a register) allocated to store the result of the AEC procedure (e.g., a brightness value, an exposure value, a gain value) in the second memory (321) of the image sensor (301) and read the result.

[0111] In operation 850, the first processor (302) may perform a secondary AEC mode based on the read AEC result. The AE algorithm used for the secondary AE procedure may be an algorithm according to the embodiment of FIG. 5 or may be an AE algorithm of another type (e.g., APEX System). The first processor (302) may transmit the second shutter value and / or the second gain value calculated as a result of performing the secondary AEC mode to the image sensor (302). Operations 840 and 850 are optional and may be omitted.

[0112] At operation 860, the first processor (302) may transmit a secondary streaming command to the image sensor (301) based on the AE procedure completion message being received from the image sensor (301). Accordingly, a manual switching mode may be provided, wherein the message indicates that the image sensor (301) may receive a streaming command to perform streaming when the AE procedure is completed.

[0113] In operation 870, the image sensor (301) may perform a designated streaming mode (e.g., preview mode) based on the secondary streaming command. According to one embodiment, the image sensor (302) may perform a streaming operation to acquire image data using the first shutter value and the first gain value calculated as a result of performing the primary AEC mode and output the image data to the first processor (302). According to one embodiment, the image sensor (302) may also perform a streaming operation to acquire image data using the second shutter value and / or the second gain value received from the first processor (302) and output the image data to the first processor (302).

[0114] In operation 880, the first processor (302) may process the frame received from the image sensor (301). For example, the ISP (360) in the first processor (302) may adjust the size of the frame to fit the size of the display area of ​​the screen. As another example, the ISP (360) may perform color correction and / or backlight correction on the frame. As another example, the ISP (360) may also perform white balance for balancing white colors in the frame. Operation 880 is optional and may be omitted.

[0115] According to one embodiment, the first processor (302) may include exposure information indicating an exposure value for each area in the frame in the result of the AE procedure acquired from the image sensor (301). For example, the frame may be divided into a total of 16 areas, and exposure information indicating the exposure value (Ev) for each area may be included in the result of the AE procedure. The first processor (302) may perform operation 880 (e.g., color correction and / or backlight correction) using exposure information of a scene captured in AEC mode (e.g., a scene in which the entire frame area is oversaturated with 20Ev as in FIG. 9a, a scene in which the entire upper area (910) of the frame is oversaturated with 20Ev as in FIG. 9b (e.g., including the sky) or a scene in which at least a portion of the upper area (920) of the frame has an exposure value less than 20Ev (e.g., including the subject) as in FIG. 9c).

[0116] In operation 890, the first processor (302) can display a frame received (or processed) from the image sensor (301) on the display (303).

[0117] According to the embodiment described with reference to the above-described FIG. 8, the first AE procedure can be performed quickly (e.g., 240 fps) on the image sensor (301), and optionally, the second AE procedure can be performed under the leadership of the first processor (302). Consequently, a preview image can be displayed quickly on the display (303). In addition, by performing the first AE procedure under the leadership of the image sensor (301), resources (e.g., communication interface, memory, battery power) can be used efficiently.

[0118] FIG. 10 is a flowchart illustrating operations performed in an electronic device (300) to quickly obtain an image having an appropriate brightness from an image sensor (301) according to one embodiment. In the description of FIG. 10, any content overlapping with FIGS. 7 and 8 may be omitted or briefly described. Instructions may be stored in a first memory (304) and / or a second memory (321). When the instructions are executed by the image sensor (301) (e.g., the second processor (330)), the image sensor (301) may perform the operations of FIG. 10. Some of the operations of FIG. 10 may be performed by the first processor (302).

[0119] In operation 1010, the image sensor (301) may receive a streaming command (e.g., a command to operate in preview mode) from the first processor (302). According to one embodiment, the image sensor (301) may receive setting information for setting the preview mode (e.g., frame rate, resolution, white balance) from the first memory (304) through the first processor (302) and store it in the second memory (321). In addition, the image sensor (301) may receive a command from the first processor (302) to perform AEC in a delayed mode (e.g., Table 1). For example, the image sensor (301) may receive a mode switching command from the first processor (302) to switch to the preview mode based on a secondary command of the first processor (302) after completion of the AE procedure. The image sensor (301) may receive information regarding a target brightness value (and / or a target exposure value) from the first processor (302). The image sensor (301) can receive a streaming command from the first processor (302) after receiving the mode switching command and information about the AEC mode.

[0120] In operation 1020, the image sensor (301) may perform AEC mode in response to a streaming command. According to one embodiment, the operations of FIG. 5 described above for AE processing may be performed. The operations of FIG. 5 are, as one embodiment, for AE processing, and various AE algorithms may be used for AE processing.

[0121] In operation 1030, the first processor (302) may wait for a specified time after transmitting the streaming command to allow the image sensor (301) to complete the AE procedure. Thus, a delay mode is provided that does not necessarily require additional data communication (message) to receive the streaming command when time elapses.

[0122] According to one embodiment, while waiting, operations 1040 and 1050 may be performed as a verification procedure of the AE procedure performed by the image sensor (301) (or a procedure for a more accurate AE procedure). For example, operations 1040 and 1050 may be identical to operations 840 and 850. The first processor (302) may transmit the second shutter value and / or the second gain value calculated as a result of performing the second AEC mode to the image sensor (302). Operations 1040 and 1050 are optional and may be omitted.

[0123] At operation 1060, the first processor (302) may transmit a secondary streaming command to the image sensor (301) based on the elapsed waiting time.

[0124] In operation 1070, the image sensor (301) may perform a designated streaming mode (e.g., preview mode) based on the secondary streaming command. According to one embodiment, the image sensor (301) may perform a streaming operation to acquire image data using the first shutter value and the first gain value calculated as a result of performing the primary AEC mode and output the image data to the first processor (302). According to one embodiment, the image sensor (301) may also perform a streaming operation to acquire image data using the second shutter value and / or the second gain value received from the first processor (302) and output the image data to the first processor (302).

[0125] In operation 1080, the first processor (302) may process the frame received from the image sensor (301). For example, the ISP (360) in the first processor (302) may adjust the size of the frame to fit the size of the display area of ​​the screen. As another example, the ISP (360) may perform color correction and / or backlight correction on the frame. As another example, the ISP (360) may also perform white balance for balancing white colors in the frame. Operation 1080 is optional and may be omitted.

[0126] According to one embodiment, the first processor (302) may include exposure information indicating an exposure value for each area in the frame in the result of the AE convergence acquired from the image sensor (301). For example, the frame may be divided into a total of 16 areas, and exposure information indicating the exposure value (Ev) for each area may be included in the result of the AE convergence. For example, the first processor (302) may determine, based on the result of the AE convergence, that the scene in the oversaturated state of FIG. 9A, the scene including the sky of FIG. 9B, or the scene including the subject of FIG. 9C was captured in the AEC mode, and may perform operation 1080 (e.g., color correction and / or backlight correction) based on the exposure information of the scene.

[0127] In operation 1090, the first processor (302) can display a frame received (or processed) from the image sensor (301) on the display (303).

[0128] According to the embodiment described with reference to the above-described FIG. 10, the first AE procedure can be performed quickly (e.g., 240 fps) on the image sensor (301), and optionally, the second AE procedure can be performed under the leadership of the first processor (302). As a result, a preview image can be displayed quickly on the display (303). In addition, by performing the first AE procedure under the leadership of the image sensor (301), resources (e.g., communication interface, memory, battery power) can be used efficiently.

[0129] According to one embodiment, the image sensor (301) can perform AEC using lookup tables such as Tables 2 and 3 below. For example, Tables 2 and 3 can be used when performing operations 520, 720, 820, or 1020 described above. Information regarding Tables 2 and 3 can be stored in the second memory (321). For example, based on the occurrence of a camera-on event, the corresponding information can be loaded from the first memory (304) to the second memory (321).

[0130] Frame rate 240 fps Frame 1st frame 2nd frame 3rd frame 4th frame 5th frame 6th frame Duration 4.17 ms 4.17 ms 4.17 ms 4.17 ms 4.17 ms Gain value (Sv) 1 1 1 1.66 13 9 0 8 9 2 4.5 5 6 3 5 Shutter value (Tv) 0.10 4 1.64 4.17 4.17 Tv*Sv 0.10 4 1.66 4 2 5.6 1 0 2 4 ratio 1 4 1 6 6 4 2 5 6 1 0 2 4

[0131] Referring to Table 2, the image sensor (301) can set the frame rate to 240 fps. Accordingly, the duration per frame can be specified as 4.17 ms. The gain value (Sv) and shutter value (Tv) can be set as shown in Table 2 so that the brightness can increase at a specified ratio from the first frame to the sixth frame. In Table 2, the shutter value (Tv) is a value indicating the exposure time, and the unit can be milliseconds, which is the same as the duration. Therefore, the maximum value of the exposure time can be the same as the duration.

[0132] Exposure value (Ev) 1st frame brightness value 2nd frame brightness value 3rd frame brightness value 4th frame brightness value 5th frame brightness value 6th frame brightness value 20 10 24 19 5 12 18 25 6 10 24 17 12 8 5 12 16 6 4 25 6 10 24 15 32 12 8 5 12 14 16 6 4 25 6 10 24 13 8 32 12 8 5 12 12 4 16 6 4 25 6 10 24 11 2 8 32 12 8 5 12 10 14 16 6 4 25 6 10 24 9 28 32 12 8 5 12 8 14 16 6 4 25 6 7 28 32 12 8 6 14 16 6 4 5 28 32 4 14 16 32 8 21 4 12 0 1

[0133] The image sensor (301) can divide the frame into a specified number of areas (e.g., 16 (4*4), 64 (8*8), 256 (16*16), or 4096 (64*64)) and check the brightness value for each area.

[0134] Referring to Table 3, if the brightness value (e.g., average of the entire frame or average of the region(s) of interest)) from the first frame is saturated at 1024 (maximum value), the image sensor (301) can determine the exposure value of the corresponding region as 20Ev. For example, if the user takes a picture toward the sky in a bright daylight with strong sunlight, an exposure value of 20Ev can be derived from the first frame.

[0135] According to one embodiment, if the brightness value of the sixth frame (average of the entire frame) in the first attempt for AE processing is measured as 512, which is half of 1024, the image sensor (301) may determine the exposure value of the corresponding area to be 9. For example, if 10Ev is the target exposure value (target Ev), in the second attempt, the image sensor (301) may re-set the gain value (Sv) and / or the shutter value (Tv) for the corresponding area so that the brightness value can increase from the first frame to the sixth frame at a rate specified in Table 2. When setting the gain value (Sv) and / or the shutter value (Tv), the image sensor (301) may obtain a value obtained by multiplying Tv and Sv as the same as the numerical value described in Table 2. When the brightness value increases from the first frame to the sixth frame at a ratio specified in Table 2, the image sensor (301) determines that the exposure value corresponds to 10Ev, and accordingly completes the AE procedure and can perform another streaming mode (e.g., preview mode).

[0136] According to one embodiment, the image sensor (301) may perform AEC using Table 3 and Table 4 below. For example, Tables 3 and 4 may be used when performing operations 520, 720, 820, or 1020 described above. Information regarding Tables 3 and 4 may be stored in the second memory (321). For example, based on the occurrence of a camera-on event, the corresponding information may be loaded from the first memory (304) to the second memory (321). Compared to using Table 2, the speed until the AE procedure is completed is slow, but the gain value is relatively small to reduce noise components, so the calculation of the exposure value may be relatively accurate.

[0137] Frame rate 240fps 66fps 35fps Frame 1st frame 2nd frame 3rd frame 4th frame 5th frame 6th frame Frame duration 4.17ms 4.17ms 4.17ms 15ms 28.33ms Gain value (Sv) 1111.61.70 66673.614543 Shutter value (Tv) 0.10.41.64 1528.33 Tv*Sv 0.10.41.66.4 25.6102.4 ratio 1416642561024

[0138] FIG. 11 is a drawing for explaining the AEC mode performed in an image sensor (301) according to one embodiment.

[0139] The image sensor (301) can divide the frame into a specified number of regions (e.g., 16 (4*4) as in FIG. 11). The image sensor (301) can generate frames using the above-described Table 2 (or Table 4) and check the brightness value for each region in each frame. The image sensor (301) can determine the exposure value (Ev) for each region based on the checked brightness value and Table 3. When the ratio of regions having a target exposure value (e.g., 7 to 12Ev) in all regions is equal to or greater than a specified reference value (e.g., 0.4), the image sensor (301) can complete the AE procedure and perform another streaming mode (e.g., preview mode). Referring to FIG. 11, in the first attempt for AE procedure, if the brightness value in all areas of the first frame is 1024 (see Table 3), the image sensor (301) may determine the exposure value (Ev) as 20 for all areas and the ratio as '0'. In the second attempt, the image sensor (301) may re-set the gain value (Tv) and the shutter value (Sv) so that the ratio can be increased. When the image sensor (301) sets the values, the product of the two values ​​(Tv*Sv) may be equal to the values ​​described in Tables 2 and 3. The image sensor (301) may determine that the result of the second attempt is 0.3125 (=5 / 16), which is lower than the set reference value (e.g., 0.4), and may re-set the gain value (Sv) and the shutter value (Tv) for the third attempt. In this case, too, the product of the two values ​​may be equal to the values ​​described in Tables 2 and 3. The image sensor (301) can complete the AE procedure by confirming that the result of the third attempt is 0.625 (=10 / 16), which is higher than the set standard (e.g., 0.4).

[0140] According to one embodiment, an electronic device (e.g., the electronic device (300) of FIG. 3 or the electronic device (101) of FIG. 1) includes a processor; an image sensor configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor; and a memory storing instructions. The instructions, when individually or collectively executed by the image sensor, may cause the image sensor to perform an operation of calculating a first shutter value and a first gain value for causing the brightness of image data acquired by the image sensor to converge to a designated target brightness value. The instructions may cause the image sensor to perform a streaming operation of acquiring image data using the first shutter value and the first gain value and outputting the image data to the processor. Therefore, by performing an AE procedure in the image sensor, frame loss and / or power consumption can be reduced, and resources can be used efficiently. In addition, since the operation procedure is performed directly in the image sensor, it can be performed quickly. As a result, the AE process is primarily performed by the image sensor, which generates frames (image data) so that resources (e.g., communication interfaces, memory, battery power) can be used efficiently.

[0141] The above command may cause the image sensor to receive a streaming command from the processor (e.g., operation 710) and, based on the reception of the streaming command, perform the calculation operation and the streaming operation. Accordingly, in this automatic mode, when the calculation is completed, streaming is automatically performed, thereby reducing the time until streaming is performed without requiring additional data communication.

[0142] The above instructions may cause the image sensor to receive a streaming command from the first processor (e.g., operation 810), perform the calculation operation in response to receiving the streaming command, transmit a message to the processor indicating that the calculation operation is complete (e.g., operation 830), receive a second streaming command from the processor after the message is transmitted to the processor (e.g., operation 860), and perform the streaming operation based on the reception of the second streaming command. Accordingly, a passive mode may be provided, wherein when the calculation is complete, the message indicates that the image sensor may receive a streaming command to perform streaming. For example, the image sensor may operate in a standby mode after completing the calculation. As a result, the preview image may be displayed on the display quickly, and since the calculation is primarily performed by the image sensor, resources (e.g., communication interface, memory, battery power) may still be reduced.

[0143] The above instruction may cause the image sensor to receive a streaming command from the first processor (e.g., operation 1010), perform the calculation operation based on the reception of the streaming command, receive a second streaming command from the processor after the performance of the calculation operation is completed (e.g., operation 1060), and perform the streaming operation based on the reception of the second streaming command. The processor may transmit the streaming command and, after a specified time has elapsed, transmit the second streaming command to the image sensor (e.g., operation 1060). Therefore, a delay mode is provided in which no additional data communication (message) is required for receiving the streaming command when the time has elapsed, and thus, no additional data communication is necessarily required. In addition, since the calculation is mainly performed by the image sensor, resources (e.g., communication interface, memory, battery power) can be used efficiently.

[0144] The above command may cause the image sensor to transmit information regarding the first shutter value and the first gain value to the processor based on a request from the processor. The processor may use the information to process image data received from the image sensor through the streaming operation. This may optimize properties of the acquired image data before displaying it on a display. For example, the size may be adjusted to the size of the display area of ​​the screen, color correction and / or backlight compensation on the frame, and white balance to balance the white color in the frame. The command may cause the image sensor to transmit the information to the processor through a Camera Control Interface (CCI) interface.

[0145] The above command may cause the image sensor to perform the streaming operation via a mobile industry processor interface (MIPI) interface.

[0146] The processor may determine brightness in image data acquired using the first shutter value and the first gain value, calculate a second shutter value and / or a second gain value to converge the determined brightness to the target brightness value, and transmit the second shutter value and / or the second gain value to the image sensor. The instructions may cause the image sensor to perform a streaming operation of acquiring image data using the second shutter value and / or the second gain value received from the processor and outputting the image data to the processor. This may enable more accurate calculation of the shutter value and the gain value using a verification procedure and / or the processor. This may also be used for testing (verification). For example, if the identified brightness value is different from the target brightness value, the secondary calculation of the processor may include a result used by the image sensor for the streaming operation of acquiring image data using the shutter value and / or the gain value received from the processor and outputting the image data to the processor.

[0147] The above command may cause the image sensor to acquire image data at a first frame rate as part of the output operation and to acquire image data at a second frame rate, which is lower than the first frame rate, as part of the streaming operation. This allows the image sensor to perform calculations of shutter values ​​and gain values ​​much faster.

[0148] The processor can process image data received from the image sensor into a preview image through the streaming operation of the image sensor and display the preview image on the display. Accordingly, the electronic device can quickly enter a preview mode for displaying an image on the display by performing calculations on the image sensor without the intervention of the processor.

[0149] According to one embodiment, a method of operating an electronic device (e.g., the electronic device (300) of FIG. 3 or the electronic device (101) of FIG. 1) is provided. The electronic device includes a processor; and an image sensor configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the processor. The method may include an operation in which the image sensor calculates a first shutter value and a first gain value for causing brightness of image data acquired by the image sensor to converge to a specified target brightness value; and a streaming operation in which the image sensor acquires image data using the first shutter value and the first gain value and outputs the image data to the processor.

[0150] The method may further include an operation in which the image sensor receives a streaming command from the processor. The calculation operation and the streaming operation may be performed based on the reception of the streaming command.

[0151] The method may further include: an operation in which the image sensor receives a streaming command from the processor; an operation in which the image sensor transmits a message to the processor indicating that the calculation operation has been completed; and an operation in which, after the message is transmitted to the processor, the image sensor receives a second streaming command from the processor. The calculation operation may be performed based on reception of the streaming command. The streaming operation may be performed based on reception of the second streaming command.

[0152] The method may further include an operation in which the image sensor receives a streaming command from the processor; an operation in which the processor transmits the streaming command and, after a specified time has elapsed, transmits a second streaming command to the image sensor; and an operation in which the image sensor receives the second streaming command from the processor. The calculation operation may be performed based on reception of the streaming command. The streaming operation may be performed based on reception of the second streaming command.

[0153] The method may further include an operation of transmitting information about the first shutter value and the first gain value to the processor based on a request from the processor; and an operation of the processor processing image data received from the image sensor through the streaming operation using the information.

[0154] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0155] In the above explanation, the prefixes “first,” “second,” and “third” are only used to distinguish between the same names and do not have any special meaning in themselves, such as importance or order.

[0156] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0157] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0158] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. In one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0159] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0160] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0161] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, First processor (302); An image sensor (301) including a second processor (330) and configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the first processor (302); and Contains a memory (304, 321) for storing instructions, The above command, when executed individually or collectively by the second processor (330), causes the image sensor (301) to: An operation of calculating a first shutter value (Tv) and a first gain value (Sv) to cause the current brightness of image data acquired from the image sensor (301) to converge to a specified target brightness value; and An electronic device that performs a streaming operation to acquire image data using the first shutter value and the first gain value and output the image data to the first processor (302).

2. In the first paragraph, when the command is executed by the second processor (330), the image sensor (301): Receive a streaming command from the first processor (302), An electronic device that performs the above-described production operation and the above-described streaming operation based on receipt of the above-described streaming command.

3. In the first paragraph, when the command is executed by the second processor (330), the image sensor (301): Receive a streaming command from the above processor (301), Based on the reception of the above streaming command, the above output operation is performed, Transmitting a message indicating that the above-mentioned production operation has been completed to the first processor (302), After the above message is transmitted to the first processor (302), a second streaming command is received from the first processor (302), An electronic device that performs the streaming operation based on receipt of the second streaming command.

4. In the first paragraph, the first processor (302) is configured to transmit the streaming command to the image sensor (301) and transmit a second streaming command to the image sensor (301) after a specified time has elapsed. The above command, when executed by the second processor (330), causes the image sensor (301) to: Receive a streaming command from the above processor (302), Based on the reception of the above streaming command, the above output operation is performed, Receive the above second streaming command from the first processor (302), An electronic device that performs the streaming operation based on receipt of the second streaming command.

5. In one of the clauses 1 to 4, when the command is executed by the second processor (330), the image sensor (301) Based on a request from the first processor (302), transmit information about the first shutter value and the first gain value to the first processor (302), An electronic device configured to process image data received from the image sensor (301) through the streaming operation using information about the first shutter value and the first gain value, wherein the first processor (302) is configured to process image data received from the image sensor (301) through the streaming operation.

6. In the fifth paragraph, when the command is executed by the second processor (330), the image sensor (301) An electronic device that transmits information about the first shutter value and the first gain value to the first processor (302) through a CCI (Camera Control Interface) interface (341, 371).

7. In one of the clauses 1 to 4, when the command is executed by the second processor (330), the image sensor (301) An electronic device that performs the above streaming operation via a MIPI (mobile industry processor interface) interface (342, 372).

8. In one of the clauses 1 to 4, The above first processor (302) Check the brightness in the image data obtained using the first shutter value and the first gain value, Calculating a second shutter value and / or a second gain value to cause the above-determined brightness to converge to the target brightness value; configured to transmit the second shutter value and / or the second gain value to the image sensor, The above command, when executed by the second processor (330), causes the image sensor (301) to: An electronic device that performs a streaming operation to acquire image data using the second shutter value and / or the second gain value received from the first processor (302) and output the image data to the first processor (302).

9. In one of the clauses 1 to 4, when the command is executed by the second processor (330), the image sensor (301): As part of the above production operation, image data is acquired at a first frame rate, An electronic device that acquires image data at a second frame rate that is lower than the first frame rate as part of the above streaming operation.

10. In one of the clauses 1 to 4, the first processor (302) An electronic device configured to process image data received from the image sensor (301) into a preview image through the streaming operation of the image sensor (301) and display the preview image on a display (303).

11. In a method of operating an electronic device, The electronic device includes an image sensor (301) configured to convert light received through a lens into an electrical signal, convert the electrical signal into image data, and output the image data to the first processor (302); and a second processor (330). An operation of calculating a first shutter value and a first gain value so that the brightness of image data acquired from the image sensor (301) converges to a specified target brightness value; and A method including a streaming operation in which the image sensor acquires image data using the first shutter value and the first gain value and outputs the image data to the first processor (302).

12. In paragraph 11, The image sensor (301) further includes an operation of receiving a streaming command from the first processor (302), A method in which the above-mentioned production operation and the above-mentioned streaming operation are performed based on reception of the above-mentioned streaming command.

13. In paragraph 11, An operation in which the image sensor (301) receives a streaming command from the first processor (302); An operation in which the image sensor (301) transmits a message indicating that the calculation operation has been completed to the first processor (302); and After the message is transmitted to the first processor (302), the image sensor (301) further includes an operation of receiving a second streaming command from the first processor (302). The above output operation is performed based on the reception of the above streaming command, The above streaming operation is performed based on the reception of the second streaming command. method.

14. In paragraph 11, An operation in which the image sensor (301) receives a streaming command from the first processor (302); An operation in which the first processor (302) transmits the streaming command and, after a specified time has elapsed, transmits a second streaming command to the image sensor (301); and The image sensor (301) further includes an operation of receiving the second streaming command from the first processor (302), The above output operation is performed based on the reception of the above streaming command, The above streaming operation is performed based on the reception of the second streaming command. method.

15. In one of the clauses 11 to 14, An operation of transmitting information about the first shutter value and the first gain value to the first processor (302) based on a request from the first processor (302); and A method further comprising an operation in which the first processor (302) processes image data received from the image sensor (301) through the streaming operation using information about the first shutter value and the first gain value.

Images