Electronic device acquiring image through image sensor and method for operating the same

Abstract

An electronic device according to various embodiments includes: a camera including an image sensor; a memory storing instructions; and a processor operably connected to the communication circuit and the memory, in which the processor may be configured to control the image sensor to output image data according to a selected photographing mode, and to process the image data output according to the photographing mode based on a machine learning model, or to perform image signal processing to obtain an image. Various other embodiments are possible.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for acquiring an image through an image sensor and an operating method of the electronic device. Background Art

[0002] An electronic device may acquire image data from the output of an image sensor that converts an optical signal into an electrical signal. The image sensor may include a color filter and a light receiving element (e.g., a photodiode, a pinned-photodiode, a phototransistor, a photogate). The light receiving element may receive light that has passed through the color filter corresponding to a specific color and may output an electrical signal corresponding to the received light. The electronic device may acquire image data including a set of pixel values ​​determined based on the values ​​output from the image sensor. The image quality of the image data may depend on the number of pixels included in the image data, and devices and methods for acquiring image data having a larger number of pixels to obtain image data of higher image quality are being developed.

[0003] Because a light-receiving element receives light that passes through a color filter of a specific color, the information detected by a single light-receiving element corresponds to a single color. The color filter enables the final image to be represented in color, but this may result in the final image being represented at a lower resolution than the total number of light-receiving elements provided in the image sensor. Color reproduction and resolution can be in a conflicting trade-off. The Bayer pattern has been widely used as a pattern for accurately reproducing colors while minimizing resolution degradation.

[0004] The above information is provided as related art for ease of understanding of the present disclosure. No assertion or determination is made as to whether any of the above information applies as prior art related to the present disclosure. Summary of the Invention

[0005] Solution to the problem According to one aspect of the present disclosure, an electronic device may include: a camera including a lens and an image sensor, wherein the image sensor is configured to convert an optical signal passing through the lens into a digital signal; a memory configured to store instructions; and at least one processor, wherein the image sensor includes a plurality of microlenses, a plurality of light receiving elements, and a color filter, wherein the plurality of microlenses includes a first microlens, the plurality of light receiving elements corresponding to a plurality of photodiodes arranged in an M×N configuration corresponding to the first microlens, at least one of M or N being a natural number greater than or equal to 2, wherein the plurality of photodiodes includes a first photodiode and a group of second photodiodes, the color filter includes a plurality of color channels, and wherein the instructions executed by the at least one processor cause the electronic device to perform the following operations: determining and processing the image signal to be transmitted by using the a shooting mode corresponding to the resolution of the image captured by the camera; based on the determined shooting mode corresponding to the first resolution, outputting first original image data corresponding to the first data read out from the first photodiode by controlling the image sensor, wherein the output of the first photodiode corresponds to one pixel; based on the result of inputting the first original image data into the machine learning model, acquiring a first image corresponding to the first resolution; based on the determined shooting mode corresponding to a second resolution lower than the first resolution, outputting second original image data obtained by performing at least one image processing operation on the second data read out from the group of second photodiodes by controlling the image sensor, wherein the output of the group of second photodiodes corresponds to one pixel; and acquiring a second image corresponding to the second resolution by performing second image signal processing on the second original image data.

[0006] According to one aspect of the present disclosure, a method is performed by an electronic device including a camera, the camera including an image sensor, wherein the image sensor includes a plurality of photodiodes arranged in an M×N configuration corresponding to a first microlens, wherein at least one of M or N is a natural number greater than or equal to 2, the method including: determining a shooting mode corresponding to a resolution of an image to be captured by using the camera; outputting first original image data corresponding to first data read out from a first photodiode among a plurality of photodiodes by controlling the image sensor based on the determined shooting mode corresponding to the first resolution, wherein the output of the first photodiode corresponds to one pixel; acquiring a first image corresponding to the first resolution based on a result of inputting the first original image data into a machine learning model; outputting second original image data obtained by performing at least one image processing operation on second data read out from a group of second photodiodes among the plurality of photodiodes based on the determined shooting mode corresponding to a second resolution lower than the first resolution, wherein the output of the group of second photodiodes corresponds to one pixel; and acquiring a second image corresponding to the second resolution by performing second image signal processing on the second original image data.

[0007] According to one aspect of the present disclosure, an image sensor may include: a plurality of light receiving elements; a plurality of microlenses, including a first microlens; and a color filter, including a plurality of color channels, wherein the plurality of light receiving elements correspond to a plurality of photodiodes arranged in an M×N configuration corresponding to the first microlens, and at least one of M or N is a natural number greater than or equal to 2, wherein the image sensor is configured to: output first original image data corresponding to first data read out from a first photodiode among a plurality of photodiodes based on a first shooting mode corresponding to a first resolution, wherein the output of the first photodiode corresponds to one pixel; and output second original image data obtained by performing at least one image processing operation on second data read out from a group of second photodiodes among the plurality of photodiodes based on a second shooting mode corresponding to a second resolution lower than the first resolution, wherein the output of the group of second photodiodes corresponds to one pixel.

[0008] According to one aspect of the present disclosure, a computer-readable non-transitory recording medium may store a computer program recorded thereon for executing the above-mentioned method when an electronic device executes the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an electronic device in a network environment according to various embodiments; Figure 2 is a block diagram illustrating a camera module according to various embodiments; Figure 3 shows a configuration of an electronic device according to an embodiment; Figure 4 shows the configuration of an image sensor according to an embodiment; Figure 5 shows examples of patterns of light receiving elements, microlenses, and color filters arranged in an image sensor according to an embodiment; Figure 6 An example of a circuit for outputting a value of a pixel included in an image sensor according to an embodiment is shown; Figure 7 Another example of a circuit for outputting a value of a pixel included in an image sensor according to an embodiment is shown; Figure 8 A process of acquiring an image by an electronic device according to an embodiment is shown; Figure 9 shows lens pupils assigned to light receiving elements included in an image sensor of an electronic device according to an embodiment; Figure 10 1. A process of selecting a shooting mode by an electronic device according to an embodiment is shown; Figure 11 shows an example of a screen including a resolution selection user interface displayed by an electronic device according to an embodiment; Figure 12 A process of acquiring an image based on context information by an electronic device according to an embodiment is shown; Figure 13 A concept illustrating an image signal processing process performed by an electronic device to acquire an image according to an embodiment; and Figure 14 A process of acquiring an image by an electronic device based on three modes related to operations of an image sensor according to an embodiment is illustrated. DETAILED DESCRIPTION

[0010] In this disclosure, the term "pixel" refers to the smallest unit that constitutes a digital image. The resolution of an image can be represented by the number of pixels included in the image. For example, when an image is composed of a×b pixels arranged in a rows and b columns, the resolution of the image can be represented by a×b.

[0011] Figure 1 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Figure 1In the network environment 100, the electronic device 101 can communicate with the electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or can communicate with at least one of the electronic device 104 and the server 108 via a second network 199 (e.g., a long-range wireless communication network). Depending on an embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. Depending on an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) may be implemented as a single component (eg, display module 160 ).

[0012] The processor 120 may execute, for example, software (e.g., program 140) to control at least one other component of the electronic device 101 (e.g., a hardware component or a software component) in conjunction with the processor 120, and may perform various data processing or computations. According to one embodiment, as at least part of the data processing or computation, the processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the commands or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. Depending on the embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121 or be adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0013] When the main processor 121 is inactive (e.g., sleeping), the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states associated with at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190). Alternatively, when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 may work with the main processor 121 to control at least some of the functions or states associated with at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190). Depending on the embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component functionally related to the auxiliary processor 123 (e.g., the camera module 180 or the communication module 190). Depending on the embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware structures dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0014] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134.

[0015] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0016] The input module 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by another component of the electronic device 101 (e.g., the processor 120). The input module 150 may include, for example, a microphone, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus).

[0017] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0018] The display module 160 can visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. Depending on the embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of the force caused by the touch.

[0019] The audio module 170 can convert sound into an electrical signal, and vice versa. Depending on the embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly coupled to the electronic device 101.

[0020] The sensor module 176 can detect the operating state of the electronic device 101 (e.g., power or temperature) or the environmental state outside the electronic device 101 (e.g., the state of the user), and then generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0021] The interface 177 may support one or more specific protocols for directly (e.g., wired) or wirelessly connecting the electronic device 101 to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the interface 177 may include, for example, a High-Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital (SD) card interface, or an audio interface.

[0022] The connection end 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102). Depending on the embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0023] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0024] The camera module 180 may capture still images or moving images. Depending on the embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0025] The power management module 188 may manage power supply to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0026] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0027] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate via a first network 198 (e.g., a short-range communication network such as Bluetooth TM ), Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)), or a second network 199 (for example, a long-distance communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (for example, a LAN or a wide area network (WAN)))). These various types of communication modules may be implemented as a single component (for example, a single chip), or may be implemented as multiple components (for example, multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (for example, an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0028] The wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies (e.g., New Radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance in high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., the second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or 1 ms or less round trip).

[0029] Antenna module 197 can transmit or receive signals or power to or from an external device (e.g., an external electronic device) outside of electronic device 101. Depending on the embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). Depending on the embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (e.g., first network 198 or second network 199) may be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. Depending on the embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may also be formed as part of antenna module 197.

[0030] According to various embodiments, antenna module 197 may form a millimeter wave antenna module. According to embodiments, the millimeter wave antenna module may include a printed circuit board, an RFIC, and multiple antennas (e.g., array antennas), wherein the RFIC is disposed on or adjacent to a first surface (e.g., the bottom surface) of the printed circuit board and is capable of supporting a designated high frequency band (e.g., the millimeter wave band), and the multiple antennas are disposed on or adjacent to a second surface (e.g., the top surface or side surface) of the printed circuit board and are capable of transmitting or receiving signals in the designated high frequency band.

[0031] At least some of the above components may be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0032] According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type than the electronic device 101. According to an embodiment, all or some operations to be executed on the electronic device 101 may be executed on one or more of the external electronic devices 102, 104, or 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or execute another function or service related to the request, and transmit the results of the execution to the electronic device 101. The electronic device 101 may provide the results as at least a partial response to the request, either with or without further processing. To this end, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technologies may be used, for example. The electronic device 101 may use distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server that utilizes machine learning and / or neural networks. Depending on the embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to smart services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0033] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to embodiments of the present disclosure, the electronic device is not limited to those described above.

[0034] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but rather include various changes, equivalents, or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that, unless the relevant context clearly indicates otherwise, nouns in the singular form corresponding to a term may include one or more things. As used herein, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding phrase in the multiple phrases. As used herein, terms such as "first" and "second" or "first" and "second" may be used to simply distinguish a corresponding component from another component and do not limit the components in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., a second element)”, “coupled to another element (e.g., a second element)”, “connected with another element (e.g., a second element)”, or “connected to another element (e.g., a second element)”, with or without the terms “operably” or “communicatively” being used, it means that the element can be coupled with the other element directly (e.g., wired), wirelessly, or via a third element.

[0035] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or portion of the single integrated component. For example, depending on the embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0036] The various embodiments described herein can be implemented as software (e.g., program 140) comprising one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function in accordance with the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0037] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., the Play Store). TM ) The computer program product may be published online (e.g., downloaded or uploaded) or distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smartphones). If published online, at least part of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0038] According to various embodiments, each of the aforementioned components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately provided in different components. According to various embodiments, one or more of the aforementioned components may be omitted, or one or more additional components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In such a case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding one of the multiple components performed the one or more functions prior to integration. According to various embodiments, the operations performed by a module, program, or another component may be performed sequentially, in parallel, repeatedly, or in a heuristic manner, or one or more of the operations may be performed in a different order or omitted, or one or more additional operations may be added.

[0039] Figure 2 FIG2 is a block diagram 200 illustrating a camera module 180 according to various embodiments. Figure 2 The camera module 180 may include a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., a buffer memory), or an image signal processor 260. The lens assembly 210 may collect light emitted from or reflected by an object whose image is to be captured. The lens assembly 210 may include one or more lenses. Depending on the embodiment, the camera module 180 may include multiple lens assemblies 210. In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the multiple lens assemblies 210 may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that differ from those of another lens assembly. The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

[0040] Flash 220 can emit light, where the emitted light is used to enhance light reflected from an object. Depending on the embodiment, flash 220 may include one or more light-emitting diodes (LEDs) (e.g., red, green, and blue (RGB) LEDs, white LEDs, infrared (IR) LEDs, or ultraviolet (UV) LEDs) or a xenon lamp. Image sensor 230 can capture an image corresponding to an object by converting light emitted from or reflected from the object and transmitted through lens assembly 210 into an electrical signal. Depending on the embodiment, image sensor 230 may include one image sensor selected from a plurality of image sensors having different properties (e.g., an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor), multiple image sensors having the same properties, or multiple image sensors having different properties. Each image sensor included in image sensor 230 may be implemented using, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor.

[0041] Image stabilizer 240 can move image sensor 230 or at least one lens included in lens assembly 210 in a specific direction, or control operational properties of image sensor 230 (e.g., adjust readout timing) in response to movement of camera module 180 or electronic device 101 including camera module 180. This allows for compensating for at least a portion of adverse effects (e.g., image blur) resulting from movement of the image being captured. Depending on the embodiment, image stabilizer 240 can use a gyroscope sensor (not shown) or an accelerometer (not shown) disposed within or outside camera module 180 to sense such movement of camera module 180 or electronic device 101. Depending on the embodiment, image stabilizer 240 can be implemented as, for example, an optical image stabilizer. Memory 250 can at least temporarily store at least a portion of an image acquired via image sensor 230 for subsequent image processing tasks. For example, if multiple images are captured quickly or image capture is delayed due to shutter lag, the captured original image (e.g., a Bayer pattern image, a high-resolution image) may be stored in memory 250, and its corresponding duplicate image (e.g., a low-resolution image) may be previewed via display module 160. Then, if a specified condition is met (e.g., through user input or system command), at least a portion of the original image stored in memory 250 may be captured and processed by, for example, image signal processor 260. Depending on the embodiment, memory 250 may be configured as at least a portion of memory 130, or may be configured as a separate memory that operates independently of memory 130.

[0042] The image signal processor 260 may perform one or more image processing operations on images acquired via the image sensor 230 or stored in the memory 250. These one or more image processing operations may include, for example, depth map generation, three-dimensional (3D) modeling, panoramic image generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor 260 may control at least one of the components included in the camera module 180 (e.g., the image sensor 230) (e.g., exposure time control or readout timing control). Images processed by the image signal processor 260 may be stored back in the memory 250 for further processing, or may be provided to an external component outside the camera module 180 (e.g., the memory 130, the display module 160, the electronic device 102, the electronic device 104, or the server 108). Depending on the embodiment, the image signal processor 260 may be configured as at least a portion of the processor 120, or may be configured as a separate processor that operates independently of the processor 120. If the image signal processor 260 is configured as a separate processor from the processor 120 , the at least one image processed by the image signal processor 260 may be displayed as it is by the processor 120 via the display module 160 , or may be displayed after being further processed.

[0043] Depending on the embodiment, the electronic device 101 may include multiple camera modules 180 having different properties or functions. In this case, at least one of the multiple camera modules 180 may form, for example, a wide-angle camera, and at least another of the multiple camera modules 180 may form a telephoto camera. Similarly, at least one of the multiple camera modules 180 may form, for example, a front-facing camera, and at least another of the multiple camera modules 180 may form a rear-facing camera.

[0044] Figure 3 is a block diagram showing the configuration of the electronic device 101 according to the embodiment.

[0045] The electronic device 101 according to an embodiment may include at least one processor (eg, Figure 1 processor 120) 320, memory (e.g., Figure 1 memory 130) 330 or a camera (eg, Figure 1 and Figure 2 The electronic device 101 may further include a display (eg, Figure 1 The display module 160) 360. The display 360 may be replaced by an external display connected to the electronic device 101.

[0046] In an embodiment, the camera 380 may include a lens unit 381 and an image sensor 383. The lens unit 381 includes at least one lens for collecting light, and the image sensor 383 is configured to convert light signals passing through the lens unit 381 into digital signals. According to an embodiment, the image sensor 383 may include multiple light-receiving elements, multiple microlenses, and color filters including multiple color channels. The multiple light-receiving elements may include (or correspond to) photodiodes arranged in M ​​rows and N columns, corresponding to one microlens. Here, M and N are natural numbers greater than or equal to 2. In the present disclosure, the photodiode may be replaced with other light-receiving elements (e.g., a pinned photodiode, a phototransistor, or a photogate).

[0047] In an embodiment, the memory 330 may store instructions to be executed by the at least one processor 320. The at least one processor 320 may execute the instructions stored in the memory 330 to perform calculations or control elements of the electronic device 101, thereby operating the electronic device 101. In the present disclosure, when the at least one processor 320 executes the instructions, the operation of the electronic device 101 is performed. In an embodiment, the at least one processor 320 may include a central processing unit (CPU), an image signal processor (ISP) (e.g., Figure 2 The at least one processor 320 may include at least one of an image signal processor 260, a graphics processing unit (GPU), or a neural processing unit (NPU). For example, the at least one processor 320 may include an application processor.

[0048] In an embodiment, the at least one processor 320 may determine a shooting mode corresponding to the resolution of the image to be captured by the camera 380. For example, the at least one processor 320 may select either a first shooting mode (corresponding to a first resolution) or a second shooting mode (corresponding to a second resolution). The first shooting mode may generate images with a higher resolution than the second shooting mode. For example, the first resolution may be 200 mega pixels, and the second resolution may be 50 or 12.5 MP. The types of shooting modes are not limited to these and may vary depending on the configuration of the image sensor. For example, the at least one processor 320 may select either a first shooting mode corresponding to a first resolution, a second shooting mode corresponding to a second resolution, or a third shooting mode corresponding to a third resolution. Here, the first resolution may be 200 MP, the second resolution may be 50 MP, and the third resolution may be 12.5 MP. In an embodiment, the number of shooting modes selected by the at least one processor 320 may be higher than the above.

[0049] In an embodiment, at least one processor 320 may control the image sensor 383 to operate based on the selected shooting mode. For example, the image sensor 383 may output first raw image data corresponding to first data based on a first shooting mode, wherein the first data is read out (readout) in a manner such that outputs from a plurality of light receiving elements correspond to one pixel. The first raw image data corresponding to the first data may be understood as meaning that a pixel value of one pixel of the first raw image data is determined based on a value read out from a light receiving element corresponding to one photodiode. The image sensor 383 may be configured to capture the image of a pixel of a pixel based on a camera (e.g., Figure 1 Camera module 180, Figure 3 The zoom magnification of the camera 380) or the lens unit (e.g., Figure 3The image sensor 383 calibrates the first data based on at least one of the focus positions of the lens unit 381 to output first raw image data. Calibrating the first data by the image sensor 383 may include compensating pixels by calibrating the first data to remove deviations caused by factors other than deviations in the data attributable to the subject. The image sensor 383 may statically compensate the first data based on a predefined lookup table, or dynamically correct the first data based on information from peripheral pixels. For example, an image sensor 383 with a "hexadeca" pattern may require calibration due to its four photodiode (4PD) structure, a "tetra" pattern, and the difference (Gb-Gr diff) between the green channel arranged in the same row as the blue channel and the green channel arranged in the same row as the red channel. In the present disclosure, the hexadeca pattern may refer to a repeating pattern in which four microlenses corresponding to the same color channel are arranged adjacent to each other, with four photodiodes arranged to correspond to each microlens. The sixteen-grid pattern can be configured in an array form in which the microlenses and photodiode patterns are arranged in the same manner in areas corresponding to color channels of different colors. In the present disclosure, a 4PD structure refers to a pattern in which four photodiodes are arranged to correspond to one microlens. In the present disclosure, a quaternary pattern refers to a repetition of a pattern in which four microlenses (arranged to correspond to the same color channel) are arranged adjacent to each other. The quaternary pattern can be configured in an array form in which the microlenses and photodiode patterns are arranged in the same manner in areas corresponding to color channels of different colors. However, this is merely an example for explaining an example in which multiple light receiving elements are arranged to correspond to one microlens and should not be considered restrictive.

[0050] In an embodiment, the first raw image data output in the first capture mode may be data that has not been image processed by the at least one processor 320 (e.g., an ISP) that performs image signal processing. For example, the first raw image data may be data that has not undergone re-mosaicing of the values ​​read out by the light receiving elements. In other words, the first raw image data may be data that maintains the color pattern of the image sensor 383 by not changing the color order of the color pattern.

[0051] In an embodiment, when an image sensor having a pattern in which microlenses corresponding to the same color channel are arranged adjacent to each other and a plurality of photodiodes are arranged in one microlens outputs the output of one photodiode by matching it to one pixel, it may be difficult to obtain an image with desired image quality through re-mosaicing. Therefore, the electronic device 101 according to an embodiment may generate an image by using a machine learning model.

[0052] In an embodiment, at least one processor 320 may receive the first raw image data output from the image sensor 383 via an interface (e.g., an interface based on the MIPI protocol) connecting the image sensor 383 and the at least one processor 320. The at least one processor 320 may input the first raw image data into a prediction model. For example, the at least one processor 320 may use at least one of a CPU, GPU, NPU, ISP, or peripheral device included in the electronic device 101 to perform neural network calculations corresponding to a prediction model (e.g., a neural network model). For example, the prediction model may be generated by learning expected raw image data (input image) and an image (ground truth (GT) image) corresponding to a desired result from the raw image data (input image). In this case, the at least one processor 320 may input the first raw image data into the prediction model and perform calculations to obtain the first image as output from the prediction model. For example, the prediction model may be generated by learning the raw image data and image data in a Bayer pattern corresponding to the raw image data. In this case, the at least one processor 320 may input the first raw image data into the prediction model and obtain image data in a Bayer pattern corresponding to the re-mosaiced result. At least one processor (e.g., ISP) 320 can acquire a first image by performing image processing (e.g., demosaicing) on ​​Bayer-patterned image data. In the case of an image sensor in which multiple light-receiving elements (e.g., multiple photodiodes) are arranged to correspond to one microlens, performing demosaicing calculations on raw image data read out in high-resolution mode using an image processing circuit (e.g., an ISP chain) rather than a machine learning model may fail to remove artifacts and may result in a low modulation transfer function (MTF) value. However, acquiring an image from the raw image data using a machine-trained prediction model can reduce artifacts and acquire an image with a high MTF value.

[0053] In embodiments, the output of the prediction model may be implemented in various ways. For example, the output of the prediction model may include a first image in the RGB domain, resulting from post-processing processes that apply various filters or image enhancements. In this case, the prediction model may be generated by learning from the original image data and an RGB domain image corresponding to the original image data (or by being trained using the original image data and an RGB domain image corresponding to the original image data) through a machine learning algorithm. For example, the output of the prediction model may include a Y, Cb, Cr (YUV) image corresponding to the data before post-processing. In this case, the prediction model may be generated by learning from the original image data and an YUV domain image derived from the original image data through a machine learning algorithm. For example, the output of the prediction model may include the result of transforming the original image data into a Bayer pattern. Calculations corresponding to post-processing processes for demosaicing (e.g., white balance, gamma correction, color correction, noise reduction, sharpening) may not be included in the prediction model, but processes corresponding to demosaicing may need to be performed by the prediction model.

[0054] In an embodiment, the image sensor 383 may output second raw image data corresponding to second data based on a second shooting mode, wherein the second data is read out in a manner such that the output of a light receiving element corresponding to two or more photodiodes among a plurality of photodiodes corresponds to one pixel. For example, a pixel value of the second raw image data may be composed based on a value obtained by binning or summing the output values ​​of two or more light receiving elements. The second raw image data output in the second shooting mode may be data on which at least one image processing operation has been performed. For example, the second raw image data may correspond to the result of re-mosaicing the values ​​read out from the light receiving elements. For example, the second raw image data may be image data having a Bayer pattern.

[0055] In an embodiment, the at least one processor 320 may receive the second raw image data output from the image sensor 383 through an interface connecting the image sensor 383 and the at least one processor 320. The at least one processor (e.g., ISP) 320 may acquire a second image by performing at least one image signal processing operation (e.g., white balance, demosaicing, gamma correction, color correction, noise reduction, sharpening) on ​​the second raw image data.

[0056] Figure 4 is a diagram showing an image sensor (eg, Figure 2 Image sensor 230, Figure 3 A diagram of a conceptual configuration of an image sensor 383 ).

[0057] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may include a micro lens array (MLA) 411 , a color filter 413 , a light receiving unit 415 , and a calculation unit 417 .

[0058] In an embodiment, the micro lens array 411 may be configured to facilitate the transmission of light through a lens unit (e.g., Figure 3 lens unit 381) and on an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 Light beam 421 imaged on image sensor 383 of the image sensor 383 is collected at the light receiving element of light receiving unit 415. Light beam 423 passing through microlens array 411 passes through color filter 413, so that at least a portion of wavelengths other than the band corresponding to a specific color are blocked. Light beam 425 passing through color filter 413 can be detected by the light receiving element (e.g., a photodiode) of light receiving unit 415. Light receiving unit 415 may include a light receiving element and a circuit, wherein the light receiving element generates an electric charge when receiving light and converts the charge into an electrical signal, and the circuit is used to selectively read out the charge of the light receiving element. Circuitry may be provided between light receiving unit 415 and computing unit 417 to digitize the signal read out from light receiving unit 415 or to reduce noise.

[0059] In an embodiment, the calculation unit 417 may perform calculations to process the electrical data 427 output from the light receiving unit 415. The calculation unit 417 may output data obtained based on the calculation result. The output of the calculation unit 417 may be an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 383). In embodiments, the computation unit 417 may perform an operation to calibrate the readout data as a computation for processing the electrical data 427. For example, the computation performed by the computation unit 417 may include at least one of reducing deviations between pixels caused by optical asymmetry or the relative position of the sensor, reducing noise induced in analog signals, removing defects, performing re-mosaicing, or computations required for specific application areas (e.g., proximity sensor functionality, timing adjustment functionality, high dynamic range (HDR) tone mapping functionality). The sensor output 429 output from the computation unit 417 may be input to at least one processor (e.g., an application processor) via an interface.

[0060] Figure 5 38 is a diagram illustrating an example of patterns of light receiving elements, micro lenses, and color filters arranged in the image sensor 383 according to the embodiment.

[0061] In an embodiment, four microlenses 521, 522, 523, 524 (which are included in the image sensor 383 and arranged in a 2×2 configuration adjacent to each other) may be arranged in the same color channel of the color filter (e.g., the green channel 531). The light receiving elements included in the image sensor 383 may be arranged so that four photodiodes arranged in a 2×2 configuration correspond to one microlens. For example, referring to Figure 5 , the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 may be arranged to correspond to the first microlens 521. The fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518 may be arranged to correspond to the second microlens 522. Thus, sixteen (16) photodiodes (arranged in a 4×4 configuration adjacent to each other) may be arranged to correspond to one color channel.

[0062] In an embodiment, the image sensor 383 may be arranged in a repeated manner. Figure 5 The pattern shown in is used to form the Figure 5 The pattern shown in may be referred to as a "hexadeca" pattern. Figure 5 The pattern shown in is only an example, and the pattern of the image sensor 383 can be replaced by other patterns that can be operated by changing the resolution. For example, a 4PD pattern can be used, which is configured so that Figure 5Adjacent microlenses shown are arranged to correspond to different color channels, four light receiving elements are arranged to correspond to each microlens, and a Bayer pattern is output by performing binning calculation on the four light receiving elements corresponding to each microlens.

[0063] Figure 6 is a diagram showing a method for outputting a signal included in an image sensor (eg, Figure 3 and Figure 5 A circuit diagram of an example of a circuit for storing pixel values ​​in an image sensor 383).

[0064] In an embodiment, a method comprising: Figure 3 and Figure 5 The light receiving element in the image sensor 383 may include (or correspond to) four photodiodes (e.g., a first photodiode 511, a second photodiode 512, a third photodiode 513, and a fourth photodiode 514) arranged to correspond to one microlens and having a 2×2 configuration. In an embodiment, five or more photodiodes may be arranged to correspond to one microlens. For example, nine (9) photodiodes having a 3×3 configuration may be arranged to correspond to one microlens.

[0065] Reference Figure 6 Four photodiodes (e.g., first photodiode 511, second photodiode 512, third photodiode 513, and fourth photodiode 514) arranged to correspond to one microlens and a fourth photodiode (e.g., fifth photodiode 515, sixth photodiode 516, seventh photodiode 517, and eighth photodiode 518) arranged to correspond to another microlens may be connected to a floating diffusion node 630. Switches 621, 622, 623, 624, 625, 626, 627, and 628 may be connected between each of the photodiodes 511, 512, 513, 514, 515, 516, 517, and 518 and the floating diffusion node 630. For example, the switches 621, 622, 623, 624, 625, 626, 627, and 628 may include transistors (e.g., transistor gates).

[0066] According to an embodiment, charge may be accumulated in the photodiodes 511, 512, 513, 514, 515, 516, 517, and 518 during the exposure time. While the charge is being accumulated, the switch may disconnect the photodiode from the floating diffusion node 630 by maintaining an open state. When the switch is closed, the photodiode is connected to the floating diffusion node 630, so that the accumulated charge moves to the floating diffusion node 630. For example, when the first switch 621 is closed, the charge accumulated in the first photodiode 511 may move to the floating diffusion node 630. The charge stored in the floating diffusion node 630 may be read out by the source follower 650 and may be output as an electrical signal. An image sensor (e.g., Figure 3 and Figure 5 The image sensor 383 may acquire analog data corresponding to the charges moved to the floating diffusion node 630. For example, the analog data may include information on the amount of charges accumulated in the at least one photodiode during the exposure time.

[0067] In an embodiment, the line selector 660 may be controlled in an on or off state to output analog data on a selected line.

[0068] In an embodiment, an image sensor (e.g., Figure 3 and Figure 5 The image sensor 383 of the image sensor 383 can obtain analog data corresponding to the pixel values ​​of the original image data. For example, the image sensor (e.g., Figure 3 and Figure 5 The image sensor 383) can control switches 621, 622, 623, 624, 625, 626, 627, 628 to obtain analog data corresponding to light amount data obtained by at least one of the photodiodes 511, 512, 513, 514, 515, 516, 517, 518. Figure 3 and Figure 5 When the image sensor 383 of FIG. 1 turns on the first switch 621 to close the switch, the image sensor (eg, Figure 3 and Figure 5 The image sensor 383 may acquire analog data based on the light amount data acquired by the first photodiode 511 .

[0069] In an embodiment, after acquiring the analog data, the image sensor (e.g., Figure 3 and Figure 5The image sensor 383 of FIG. 383 may perform a reset operation of turning on a reset switch (eg, a reset transistor) 670 and removing charges accumulated in a floating diffusion node. Figure 3 and Figure 5 The image sensor 383 can read out the value detected from each of the photodiodes 511, 512, 513, 514, 515, 516, 517, and 518 as a pixel value by alternately performing a reset operation at time intervals while sequentially turning on the switches 621, 622, 623, 624, 625, 626, 627, and 628. For example, when operating based on the first shooting mode to generate a high-pixel (e.g., 200 Mp) image, the image sensor (e.g., Figure 3 and Figure 5 The image sensor 383 can read out the value detected from each of the photodiodes 511, 512, 513, 514, 515, 516, 517, and 518 as a pixel value.

[0070] In an embodiment, when the first switch 621, the second switch 622, the third switch 623, and the fourth switch 624 are simultaneously turned on, the charges accumulated in the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 may be moved to the floating diffusion node 630. In this case, by analog summing the amounts of light corresponding to the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514, the image sensor (e.g., Figure 3 and Figure 5 The image sensor 383) can acquire images with a micro lens (e.g., Figure 5 For example, when operating based on the second shooting mode to generate an image with lower pixels (e.g., 50 Mp) than in the first shooting mode, the image sensor (e.g., Figure 3 and Figure 5 The image sensor 383 can read out a value obtained by summing the charges accumulated in the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 as a pixel value.

[0071] In an embodiment, when switches 621, 622, 623, 624, 625, 626, 627, 628 are simultaneously turned on, charges accumulated in photodiodes 511, 512, 513, 514, 515, 516, 517, 518 may be moved to floating diffusion node 630. In this case, by analog summing the amounts of light corresponding to eight (8) photodiodes 511, 512, 513, 514, 515, 516, 517, 518, an image sensor (e.g., Figure 3 and Figure 5 The image sensor 383) can acquire images with two micro lenses (e.g., Figure 5 The image sensor (eg, Figure 3 and Figure 5 The image sensor 383) can acquire images with similarly adjacent microlenses (e.g., Figure 5 The light amount data corresponding to the third microlens 523 and the fourth microlens 524) can then be combined with the light amount data corresponding to similarly adjacent microlenses corresponding to the two microlenses (for example, Figure 5 The light amount data of the first microlens 521 and the second microlens 522) are summed and can be read out in association with one color channel (e.g., Figure 5 For example, when operating based on the third shooting mode to generate an image with lower pixels (e.g., 12.5 Mp) than in the second shooting mode, the image sensor (e.g., Figure 3 and Figure 5 The image sensor 383) can read out the image formed by the four micro lenses (e.g. Figure 5 The value obtained by summing the charges accumulated in the photodiodes corresponding to the first microlens 521, the second microlens 522, the third microlens 523, and the fourth microlens 524 is used as a pixel value. Here, the read pixel value may have a color order according to the Bayer pattern.

[0072] In an embodiment, when the first switch 621, the third switch 623, the fifth switch 625, and the seventh switch 627 are turned on while the second switch 622, the fourth switch 624, the sixth switch 626, and the eighth switch 628 are turned off, the image sensor (e.g., Figure 3 and Figure 5The image sensor 383 can obtain the brightness value of the left pixel of the microlens. When the second switch 622, the fourth switch 624, the sixth switch 626 and the eighth switch 628 are turned on while the first switch 621, the third switch 623, the fifth switch 625 and the seventh switch 627 are turned off, the image sensor can obtain the brightness value of the right pixel. Figure 1 and Figure 3 The electronic device 101 of the embodiment may obtain phase difference information based on the correlation between the left pixel value and the right pixel value. However, this is merely for explaining an example, and the method for obtaining the left pixel value and the right pixel value is not limited thereto. For example, the electronic device according to the embodiment (e.g., Figure 1 and Figure 3 The electronic device 101 may output luminance values ​​corresponding to the first photodiode 511 and the third photodiode 513 and luminance values ​​corresponding to the fifth photodiode 515 and the seventh photodiode 517 as separate left pixel values.

[0073] In the first shooting mode of outputting sixteen (16) pixels from the active pixel sensor array (APS), the highest resolution raw image data can be output, but many artifacts may appear even if the output data is re-mosaiced.

[0074] Figure 7 is a diagram showing a method for outputting a signal included in an image sensor (eg, Figure 3 and Figure 5 A circuit diagram of another example of a circuit for detecting pixel values ​​in an image sensor 383 .

[0075] In an embodiment, the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 may be connected to the first floating diffusion node 731 through the first switch 721, the second switch 722, the third switch 723, and the fourth switch 724, respectively. Figure 6 Compared to the photodiodes in the circuit diagram of FIG5 , the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518 may not share the first floating diffusion node 731 with the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514. The fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518 may be connected to the second floating diffusion node 732 via a fifth switch 725, a sixth switch 726, a seventh switch 727, and an eighth switch 728, respectively.

[0076] In an embodiment, in a state where the first selector 761 is turned on to read out data of a line including at least one of the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514, analog data may be output via the first source follower 751 based on the charge stored in the first floating diffusion node 731. After the analog data is output, the first reset switch 771 may be turned on so that the charge accumulated in the first floating diffusion node 731 may be removed.

[0077] In an embodiment, in a state where the second selector 762 is turned on to read out data of a line including at least one of the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518, analog data may be output through the second source follower 752 based on the charge stored in the second floating diffusion node 732. After the analog data is output, the second reset switch 772 may be turned on so that the charge accumulated in the second floating diffusion node 732 can be removed.

[0078] Figure 8 is a diagram showing a method of performing a process ... Figure 1 and Figure 3 Flowchart 800 of the operation of acquiring an image by the electronic device 101). In the present disclosure, the operation of executing the image stored in the memory (eg, Figure 1 Memory 130, Figure 3 at least one processor (e.g., Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 at least one processor 320) to execute the Figure 1 and Figure 3 The electronic device 101 performs operations.

[0079] In operation 810, an electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may determine a shooting mode for shooting an image. Figure 1 and Figure 3 The electronic device 101 of the embodiment may receive a user input for selecting a shooting mode, and may determine the shooting mode based on the received user input. Figure 1 and Figure 3 The electronic device 101 may determine a shooting mode for shooting an image based on a default setting. Figure 1 and Figure 3The electronic device 101 may determine a shooting mode for capturing an image based on a previously set shooting mode.

[0080] In operation 820, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may determine whether the determined shooting mode is the first shooting mode corresponding to the first resolution. When the determined shooting mode is the first shooting mode, in operation 831, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 of the electronic device outputs first original image data corresponding to the first resolution based on the first mode. Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 Here, the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The first data is read out from the light receiving unit of the image sensor 383 to obtain the first original image data without re-mosaicing. In operation 831, the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The first raw image data output by the image sensor 383 may be data configured in such a manner that the output of a light receiving element corresponding to one photodiode corresponds to one pixel.

[0081] In an embodiment, an electronic device (e.g., Figure 1 and Figure 3The electronic device 101 may enable the image sensor to output raw image data according to a determined resolution. For example, when the determined shooting mode is a first shooting mode corresponding to a first resolution, the image sensor may operate based on the first mode to output first raw image data, wherein the output of a first light receiving element corresponding to one photodiode among a plurality of photodiodes included in the image sensor corresponds to one pixel. When the determined shooting mode is a second shooting mode corresponding to a second resolution lower than the first resolution, the image sensor may operate based on a second mode to output second raw image data, wherein the output of a second light receiving element corresponding to at least two photodiodes among the plurality of photodiodes included in the image sensor (for example, the photodiodes 511, 512, 513, 514 arranged in an M×N configuration corresponding to one microlens 521, or the photodiodes arranged to correspond to a plurality of microlenses 521, 522, 523, 524) corresponds to one pixel.

[0082] In operation 832, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a first image of a first resolution based on the first original image data. The first image may be an image in an RGB domain having RGB values ​​for each pixel. In an embodiment, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 of the embodiment can obtain the first image by inputting the first original image data into a prediction model (machine learning model) generated based on machine learning. For example, the electronic device (e.g., Figure 1 and Figure 3 The electronic device 101) may output the first image by performing a calculation corresponding to the prediction model. Figure 1 and Figure 3 The electronic device 101) can obtain the information that can be obtained by the electronic device (for example, Figure 1 and Figure 3 The image signal processor of the electronic device 101 (eg, Figure 2 The image signal processor 260 processes the data. The electronic device (eg, Figure 1 and Figure 3 The electronic device 101) may be processed by an image signal processor (eg, Figure 2 The image signal processor 260 performs image processing on the data obtained from the output of the prediction model to obtain a first image.

[0083] The prediction model can be generated by inputting an input database into a machine learning model, wherein the input database includes artificially generated raw image data for a ground truth image, and the ground truth image is stored as a label for the raw image data. Alternatively, for example, the prediction model can be generated by inputting an input database into a machine learning model, wherein the ground truth image is generated by merging pixels of the raw image data, and the input database is configured by sampling each microlens position. However, the above method of generating the prediction model is merely an example and is not limiting.

[0084] When the determined photographing mode is the second photographing mode, at operation 841, the electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 of the electronic device outputs the second original image data corresponding to the second resolution based on the second mode. Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 of FIG. 8 outputs the second original image data corresponding to the resolution of 50 Mp. Here, the second original image data may be data to which re-mosaicing is applied. In operation 841, the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The second raw image data output by the image sensor 383 may be data configured such that the outputs of light receiving elements corresponding to a plurality of photodiodes correspond to one pixel. For example, the number of photodiodes may be the number of photodiodes arranged to correspond to one microlens (e.g., M×N photodiodes).

[0085] In operation 842, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a second image of a second resolution based on the second original image data. Figure 1 and Figure 3The electronic device 101 may acquire a second image by performing at least one image signal processing operation (e.g., white balance, demosaicing, gamma correction, color correction, noise reduction, sharpening) on ​​the second raw image data. Here, the second image may be, for example, an image in an RGB domain having RGB values ​​for each pixel.

[0086] Figure 9 shows the data allocated to the electronic device (e.g. Figure 1 and Figure 3 The image sensor of the electronic device 101 (eg, Figure 2 Image sensor 230, Figure 3 The light receiving element (eg, Figure 5 The lens pupil of the photodiodes 511, 512, 513, 514).

[0087] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383) can operate based on the first shooting mode by sequentially reading out the images accumulated in the light receiving elements (for example, Figures 5 to 7 The first raw image data having a high pixel count (e.g., 200 Mp) is outputted by collecting the charges in the first photodiode 511, the second photodiode 512, the third photodiode 513, the fourth photodiode 514, the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518. Figure 5 As shown, when multiple photodiodes are arranged to correspond to one microlens, light entering one microlens can be divided and received by multiple photodiodes. Since the main direction of light entering the lens can be determined according to the image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The position within the image sensor 383) varies, and thus the ratio at which light is divided may vary depending on the photodiode.

[0088] For example, refer to Figure 9 900, the light is divided by four photodiodes at equal ratios. In the present disclosure, the area where the light is divided by the photodiodes may be referred to as a pupil allocated area. Figure 9In 900, an upper left pupil allocation area 901, an upper right pupil allocation area 902, a lower left pupil allocation area 903, and a lower right pupil allocation area 904 may be divided to have substantially uniform widths.

[0089] Reference Figure 9 910, the lens pupil may be biased toward the upper left end. Due to the bias of the lens pupil, the upper left pupil allocation area 911 may be larger than the other pupil allocation areas 912, 913, and 914. Figure 9 920, the lens pupil may be biased toward the upper right end. Due to the bias of the lens pupil, the upper right end pupil allocation area 922 may be larger than the other pupil allocation areas 921, 923, and 924. Figure 9 930, the lens pupil may be offset toward the lower left end. Due to the offset of the lens pupil, the lower left end pupil allocation area 933 may be larger than the other pupil allocation areas 931, 932, and 934. Figure 9 In the lens 940, the lens pupil may be offset toward the lower right end. Due to the offset of the lens pupil, the lower right end pupil allocation area 944 may be larger than the other pupil allocation areas 941, 942, and 943.

[0090] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 of the embodiment may output the first raw image data based on the result of calibrating the pupil allocation areas of unequal sizes for the respective photodiodes. Since the main direction of light entering the lens may vary according to the zoom magnification of the camera or the focus position of the lens, the image sensor according to the embodiment (for example, Figure 2 Image sensor 230, Figure 3 The image sensor 383 of FIG. 3 may calibrate the first data read out from the photodiode based on the zoom magnification or the focus position. For example, the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 may include a processor that receives information about at least one of the zoom magnification or the focus position of the lens and calibrates the first data.

[0091] Figure 10 is a diagram showing a method of performing a process ... Figure 1 and Figure 3 1000 is a flowchart of an operation of selecting a shooting mode in an electronic device 101 ).

[0092] can be used by electronic devices (e.g. Figure 1 and Figure 3 The electronic device 101) selects the shooting mode operation Figure 10 For example, in the execution Figure 8The process of operation 810 is performed including Figure 10 The operations in flowchart 1000 are shown.

[0093] In operation 1010, an electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may run a camera application. The camera application may include a method for providing a camera to be used by the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) has a camera (eg, Figure 1 and Figure 2 Camera module 180, Figure 3 For example, the camera application may include a shooting application that shoots still images and / or moving images and stores the still images and / or moving images, a shooting application that sends images through a camera (e.g., Figure 1 and Figure 2 Camera module 180, Figure 3 380 ) to a video call application of another camera, or to send an image captured by a camera (e.g., Figure 1 and Figure 2 Camera module 180, Figure 3 The video captured by the camera 380) is streamed to an external broadcast application.

[0094] In operation 1020, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 can be displayed through a display (eg, Figure 1 Display module 160, Figure 3 The display 360) displays a user interface for selecting a resolution based on the running camera application.

[0095] For example, Figure 11 An electronic device (eg, Figure 1 and Figure 3 An example of a screen including a resolution selection user interface displayed by an electronic device 101 of FIG. 10A . In an embodiment, an electronic device (eg, Figure 1 and Figure 3 The electronic device 101) may display a screen including a user interface including at least one item for selecting a resolution. Figure 11 , the user interface for selecting a resolution may include at least one of a first icon 1101 corresponding to a first resolution, a second icon 1102 corresponding to a second resolution, or a third icon 1103 corresponding to a third resolution.

[0096] In an embodiment, an electronic device (e.g., Figure 1 and Figure 3The electronic device 101 may determine a shooting mode based on user input (e.g., touch input). For example, when a touch input corresponding to a position where the first icon 1101 is displayed is received, the electronic device may determine a first shooting mode that generates an image with a resolution of 200 Mp. When a touch input corresponding to a position where the second icon 1102 is displayed is received, the electronic device may determine a second shooting mode that generates an image with a resolution of 50 Mp. When a touch input corresponding to a position where the third icon 1103 is displayed is received, the electronic device may determine a third shooting mode that generates an image with a resolution of 12.5 Mp. However, the resolution values ​​presented in the present disclosure are merely examples and may vary.

[0097] In operation 1030, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 In operation 1030, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs raw image data for acquiring an image corresponding to the determined shooting mode.

[0098] For example, when the determined shooting mode is the first shooting mode, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383) individually and sequentially reads out the light accumulated in each light receiving element (for example, Figures 5 to 7 The charge values ​​in the first photodiode 511, the second photodiode 512, the third photodiode 513, the fourth photodiode 514, the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517 and the eighth photodiode 518 are calculated. Figure 2 Image sensor 230, Figure 3 The image sensor 383 may output first raw image data including pixels corresponding to the readout charge values. A pattern of pixels included in the first raw image data may be different from a Bayer pattern.

[0099] For example, when the determined shooting mode is the second shooting mode, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) can read out data simultaneously with the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383) has 2×2 photodiodes corresponding to the light receiving elements (eg, Figures 5 to 7 The charge accumulated in the first photodiode 511, the second photodiode 512, the third photodiode 513 and the fourth photodiode 514 is then stored. Figure 2 Image sensor 230, Figure 3 The image sensor 383) can simultaneously and concurrently read out the light receiving elements corresponding to the other photodiodes in the 2×2 configuration (e.g., Figures 5 to 7 The charge accumulated in the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517 and the eighth photodiode 518 of the image sensor (for example, Figure 2 Image sensor 230, Figure 3 The image sensor 383) may output second raw image data having a Bayer pattern by performing re-mosaicing on data including the readout charge values.

[0100] For example, when the determined shooting mode is the third shooting mode, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) can simultaneously read out the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383) has 4×2 photodiodes corresponding to the light receiving elements (eg, Figures 5 to 7 The charge value accumulated in the first photodiode 511, the second photodiode 512, the third photodiode 513, the fourth photodiode 514, the fifth photodiode 515, the sixth photodiode 516, the seventh photodiode 517, and the eighth photodiode 518 of the image sensor (for example, Figure 2 Image sensor 230, Figure 3 The image sensor 383) can convert the read result into a first digital value through an analog to digital converter (ADC). Figure 2 Image sensor 230, Figure 3 The image sensor 383) can be configured by comparing the first digital value and the second digital value obtained from the adjacent 4×2 configuration (e.g., from the image sensor corresponding to Figure 5The values ​​read out from the photodiodes of the third microlens 523 and the fourth microlens 524 are digitally merged to obtain the values ​​corresponding to one color channel (for example, Figure 5 That is, the image sensor (e.g., Figure 3 Image sensor 230, Figure 3 The image sensor 383) can obtain a pixel value based on the value detected by the light receiving element corresponding to the photodiode of the 4×4 configuration. Figure 5 In the case of the sixteen-square pattern shown in , when acquiring pixel values ​​for each color channel, the pixel values ​​may be arranged according to the Bayer pattern. In this case, the image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may output third raw image data having a Bayer pattern without undergoing re-mosaicing processing.

[0101] In an embodiment, an electronic device (e.g., Figure 1 and Figure 3 The electronic device 101 may display a preview screen 1110 corresponding to an image acquired based on the selected photographing mode through a display.

[0102] However, Figure 10 The method shown is only performed by electronic means (e.g. Figure 1 and Figure 3 An example of a method of selecting a shooting mode by the electronic device 101 ), and by the electronic device (eg, Figure 1 and Figure 3 The method of selecting the shooting mode of the electronic device 101 is not limited to Figure 10 1000. For example, an electronic device (e.g., Figure 1 and Figure 3 The electronic device 101) may be based on the Figure 1 and Figure 3 The sensor (eg, Figure 1 The shooting mode is selected based on whether the information obtained by the sensor module 176) meets the predefined conditions.

[0103] Figure 12 is a diagram showing a method of performing a process ... Figure 1 and Figure 3 1200 of an operation of the electronic device 101 ) acquiring an image based on context information. Figure 12 The operations in can be performed in conjunction with the operations of acquiring images based on the machine learning model. For example, when performing Figure 8 When the operation 832 is executed, Figure 12 The operation shown.

[0104] In operation 1210, an electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may analyze context information related to image capture. The context information related to image capture may include information for determining whether the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 can be connected to the computer by a camera (eg, Figure 1 and Figure 2 Camera module 180, Figure 3 camera 380) or a separate sensor (e.g. Figure 1 For example, an electronic device (e.g., a sensor module 176) may acquire image data to obtain context information. Figure 1 and Figure 3 The electronic device 101 of FIG. 101 can obtain the illuminance value through image data or an illuminance sensor. Figure 1 and Figure 3 The electronic device 101 of FIG. 1 may acquire information about whether the image included in the image data is a defocused image. For example, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 of FIG. 1 may acquire information about high frequency components included in image data. For example, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 may acquire information for classifying a shooting scene (eg, a night scene, a scene with backlight around an object). However, examples of context information are not limited thereto.

[0105] about Figure 8 In operation 832, when an image is captured based on the first shooting mode, a high-quality first image may be acquired by using a machine learning model. However, machine learning such as deep learning may require a large amount of computation. Therefore, in a scenario where high image quality is not required, the electronic device according to the embodiment (e.g., Figure 1 and Figure 3 The electronic device 101 of the embodiment may acquire the first image without performing calculation based on the machine learning model. In operation 1220, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101) can determine whether the context information corresponds to a specified condition in order to determine whether a machine learning model needs to be used for calculation.

[0106] For example, electronic devices (e.g. Figure 1 and Figure 3 The electronic device 101 may determine whether the illumination value is less than a threshold value. Figure 1 and Figure 3 The electronic device 101 of FIG. 101 may determine whether the captured image is a defocused image. For example, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 of the embodiment may determine whether the size of the high frequency component included in the image is less than a threshold value. Figure 1 and Figure 3 The electronic device 101 may determine whether the captured image includes a scene included in the designated category. However, the designated condition is not limited thereto.

[0107] In an embodiment, when the context information does not correspond to the specified condition, the electronic device (e.g., Figure 1 and Figure 3 The electronic device 101 may perform operation 1231 to obtain the image from the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The original image data output by the image sensor 383) is input to the machine training model. According to the electronic device of the embodiment (for example, Figure 1 and Figure 3 The electronic device 101 of the embodiment may perform operation 1233 of acquiring a first image based on an output acquired by performing calculation according to a model to which original image data is input.

[0108] In an embodiment, when the context information corresponds to a specified condition, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 may execute a Figure 2 Image sensor 230, Figure 3 The image signal processing operation 1241 is performed on the raw image data output by the image sensor 383. Here, the image signal processing operation 1241 may be performed by a processor included in an electronic device (eg, Figure 1 and Figure 3 The image signal processing chain (ISP chain) in the electronic device 101 of FIG. 101 performs image signal processing. The image signal processing performed in operation 1241 may be performed without using the machine-trained model used in operation 1231. For example, in operation 1241, the electronic device (e.g., FIG. 101 ) Figure 1 and Figure 3 The electronic device 101 may perform image processing on the raw image data based on an image signal chain for performing at least one of white balancing, demosaicing, gamma correction, color correction, noise reduction, or sharpening. In operation 1243, the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a first image based on a result of the image signal processing performed in operation 1241 .

[0109] In operation 1241, according to an embodiment, an electronic device (eg, Figure 1 and Figure 3 The image signal processing performed by the electronic device 101 may include converting the raw image data into data to be demosaiced. Converting the raw image data may include fusing or binning the pixels included in the raw image data. For example, when compared with a Figure 5 The image sensor of the sixteen-square grid pattern shown (for example, Figure 2 Image sensor 230, Figure 3 When the output of one light receiving element corresponding to one photodiode included in the image sensor 383 is configured as one pixel value, it may be difficult to obtain an image of desired quality while maintaining resolution. Figure 13 When pixels are processed using the fusion shown, the resolution may be reduced, but an image with a Bayer pattern can be obtained.

[0110] Figure 13 An electronic device (eg, Figure 1 and Figure 3 The electronic device 101 performs an image signal processing process to obtain an image.

[0111] In an embodiment, when operating based on the first shooting mode, a sixteen-square pattern (such as Figure 5 shown) of an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may output first original image data having a first pattern 1300. For example, Figure 12 The first original image data input to the machine-trained model in operation 1231 may have a first pattern 1300. However, in operation 1241, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101) may be configured to receive an image sensor (eg, Figure 2 Image sensor 230, Figure 3 The pattern of data output by the image sensor 383 is changed from the first pattern to a pattern (eg, pattern 1310 or pattern 1321 ) capable of performing demosaicing.

[0112] For example, electronic devices (e.g. Figure 1 and Figure 3The electronic device 101 may perform Bayer merging to merge pixels of a 4×4 configuration including information on the same color channel in the first pattern 1300 , thereby facilitating the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The first original image data output by the image sensor 383 of the image sensor 383 has a second pattern 1310 corresponding to the Bayer pattern. In this case, the first original image data having the changed pattern may have a resolution of 1 / 16 of the resolution before the pattern is changed. Therefore, the electronic device (e.g., Figure 1 and Figure 3 The electronic device 101 may upscale the first original image data having the pattern changed 16 times.

[0113] Alternatively, for example, an electronic device (e.g., Figure 1 and Figure 3 The electronic device 101) can facilitate the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The first original image data output by the image sensor 383 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 of FIG. 13 may perform re-mosaicing on the third pattern 1321 to convert the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The output of the image sensor 383) is changed to have a fourth pattern 1323 corresponding to the Bayer pattern. In this case, the first original image data having the changed pattern may have a resolution of 1 / 4 of the resolution before the pattern change. Therefore, the electronic device (for example, Figure 1 and Figure 3 The electronic device 101 may upscale the first original image data having the pattern changed 4 times.

[0114] Figure 14 is a diagram showing a method of performing a process ... Figure 1 and Figure 3 The electronic device 101) is based on an image sensor (eg, Figure 2 Image sensor 230, Figure 3 Flowchart 1400 of operations related to the three modes of operation of the image sensor 383) to acquire images.

[0115] In operation 1410, an electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may determine a shooting mode for shooting an image. Figure 1 and Figure 3 The electronic device 101 of the embodiment may receive a user input for selecting a shooting mode, and may determine the shooting mode based on the received user input. Figure 1 and Figure 3 The electronic device 101 may determine a shooting mode for shooting an image based on a default setting. Figure 1 and Figure 3 The electronic device 101 may determine a shooting mode for capturing an image based on a previously set shooting mode.

[0116] In operation 1420, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may determine whether the determined shooting mode is a first shooting mode corresponding to a first resolution. The first resolution may be a first shooting mode obtained by an image sensor (eg, Figure 2 Image sensor 230, Figure 3 However, this should not be considered as limiting. When the determined shooting mode is the first shooting mode, at operation 1431, the electronic device (e.g., Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs first raw image data corresponding to the first resolution. Here, the first raw image data may be a raw image of the first resolution having a pattern that does not correspond to the Bayer pattern.

[0117] In operation 1432, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a first image of a first resolution based on the first original image data. Figure 1 and Figure 3 The electronic device 101 of the embodiment may acquire the first image based on the result of inputting the first original image data into the prediction model as the machine learning model. Figure 1 and Figure 3 The electronic device 101 may store the first image in a memory (eg, Figure 1 Memory 130, Figure 3 The first image may be sent to an external device or displayed through a display (eg, Figure 1 Display module 160, Figure 3 The display 360) outputs the first image.

[0118] When the determined photographing mode is not the first photographing mode, at operation 1430, the electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may determine whether the determined shooting mode is the second shooting mode. The second shooting mode may correspond to a second resolution lower than the first resolution. Figure 14 Flowchart 1400 shows operations 1420 and 1430, respectively, but this should not be considered limiting. For example, operations 1420 and 1430 may be replaced by operations 1431 performed when the determined shooting mode is the first shooting mode, operations 1441 performed when the determined shooting mode is the second shooting mode, and operations 1451 performed when the determined shooting mode is the third shooting mode.

[0119] When the determined photographing mode is the second photographing mode, at operation 1441, the electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs second raw image data corresponding to the second resolution. The second raw image data may include data configured to have a Bayer pattern through re-mosaicing calculation.

[0120] In operation 1442, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a second image of a second resolution based on the second original image data. Figure 1 and Figure 3 The electronic device 101 of the embodiment can obtain the second image by performing at least one image processing operation on the second original image data. Figure 1 and Figure 3 The electronic device 101 may store the second image in a memory (eg, Figure 1 Memory 130, Figure 3 The second image may be sent to an external device or displayed through a display (eg, Figure 1 Display module 160, Figure 3 The display 360) outputs the second image.

[0121] When the determined photographing mode is the third photographing mode (or not the first photographing mode and the second photographing mode), at operation 1451, the electronic device according to an embodiment (eg, Figure 1 and Figure 3 The electronic device 101) can control the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs third raw image data corresponding to a third resolution. The third shooting mode may correspond to a third resolution lower than the second resolution. The third raw image data may include data configured to have a Bayer pattern by performing analog fusion and / or digital fusion on output values ​​of the light receiving elements.

[0122] In operation 1452, the electronic device according to the embodiment (eg, Figure 1 and Figure 3 The electronic device 101 may acquire a third image of a third resolution based on the third original image data. Figure 1 and Figure 3 The electronic device 101 of the embodiment may obtain a third image by performing at least one image processing operation on the third original image data. Figure 1 and Figure 3 The electronic device 101 may store the third image in a memory (eg, Figure 1 Memory 130, Figure 3 The third image may be sent to an external device or displayed through a display (eg, Figure 1 Display module 160, Figure 3 The display 360) outputs a third image.

[0123] As the integration of light receiving elements included in an image sensor increases, multiple light receiving elements adjacent to each other can be arranged to correspond to color channels of the same color within a color filter. An image sensor having such a structure can operate based on two or more modes. The image sensor can perform binning or summing processing so that the data output by detecting from each light receiving element has a Bayer pattern. The image sensor can transform the pattern of the output data to have a Bayer pattern. The image sensor can change the color order to transform the pattern and can predict the value of the changed color. Changing the color order to transform the pattern and predicting the value of the changed color can be called remosaicing.

[0124] However, with certain patterns (e.g., a sixteen-square grid pattern), it can be difficult to obtain an image without artifacts due to re-mosaicing. Furthermore, maintaining the modulation transfer function (MTF) value is crucial for obtaining high-quality images. However, with certain patterns, maintaining the MTF value can be difficult even when re-mosaicing is performed.

[0125] The technical objectives to be achieved by the present disclosure are not limited to those mentioned above, and those skilled in the art can clearly understand other technical objectives not mentioned above based on the description of the present disclosure.

[0126] In an embodiment, an electronic device (e.g., Figure 1 and Figure 3 The electronic device 101 may include: a camera (eg, Figure 1 Camera module 180, Figure 3 The camera 380 includes a lens unit (eg, Figure 3 lens unit 381) and an image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 is configured to pass through the lens unit (eg, Figure 3 The lens unit 381) converts the optical signal into a digital signal; a memory (eg, Figure 1 Memory 130, Figure 3 330 ), configured to store instructions; and at least one processor (eg, Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 at least one processor 320). Image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may include a plurality of light receiving elements (eg, Figure 4 light receiving unit 415), a plurality of micro lenses (eg, Figure 4 microlens array 411) and color filters including multiple color channels (e.g., Figure 4 Filter 413). Multiple light receiving elements (eg, Figure 4 The light receiving unit 415 may include a first micro lens (eg, Figure 5 a plurality of photodiodes (eg, Figure 5The first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514). At least one of M and N may be a natural number greater than or equal to 2. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) determines the Figure 1 Camera module 180, Figure 3 The resolution of the image captured by the camera 380) corresponds to the shooting mode. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 controls the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs first raw image data corresponding to the first data, wherein the first data is generated by a plurality of photodiodes (eg, Figure 5 The output of the first light receiving element corresponding to one of the first photodiodes 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 is read out in a manner corresponding to one pixel. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) obtains a first image corresponding to a first resolution based on a result of inputting the first original image data into the machine learning model. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 controls the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs second raw image data obtained by performing at least one image processing operation on the second data, wherein the second data is obtained by performing at least one image processing operation on the second data, wherein the second data is obtained by performing at least one image processing operation on the second data. Figure 5 The output of the second light receiving element corresponding to at least two or more photodiodes of the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 is read out in a manner corresponding to one pixel. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 acquires a second image corresponding to a second resolution by performing image signal processing on the second original image data.

[0127] In an embodiment, the first original image data may include data in which a color order of the read data is maintained.

[0128] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may be configured to output second raw image data by performing a re-mosaicing process to change a color sequence of the second data to have a Bayer pattern.

[0129] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383) can be configured to: Figure 1 Camera module 180, Figure 3 The zoom magnification of the camera 380) or the lens unit (e.g., Figure 3 The lens unit 381) calibrates at least one of the focus positions of the lens unit 381 to output first raw image data.

[0130] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383) can be configured to: compensate to four photodiodes (eg, Figure 5 The first data is calibrated based on the division ratio of light from the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514.

[0131] In an embodiment, an electronic device (e.g., Figure 1 and Figure 3 The electronic device 101 may further include a display (eg, Figure 1 Display module 160, Figure 3 display 360). By at least one processor (e.g., Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) operates for Figure 1 Camera module 180, Figure 3 The camera 380) captures the image using a camera application. The camera 380 is captured by at least one processor (e.g., Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) is based on the running camera application, through the display (eg, Figure 1 Display module 160, Figure 3 The display 360) displays a screen including a user interface for selecting a resolution. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 controls the image sensor (eg, Figure 2 Image sensor 230, Figure 3 ), wherein the shooting mode is determined based on user input received based on a user interface.

[0132] In an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The plurality of microlenses of the image sensor 383) may be arranged such that a first microlens (eg, Figure 5 four second microlenses (eg, the first microlens 521) Figure 5 The first microlens 521, the second microlens 522, the third microlens 523 and the fourth microlens 524) correspond to one color channel within the color filter (eg, Figure 5One pixel included in the second raw image data may correspond to four photodiodes (eg, Figure 5 The outputs of the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514 are processed by at least one processor (e.g., Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 controls the image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 outputs third raw image data by performing at least one image processing operation on the third data, wherein the third data is represented by a plurality of light receiving elements (eg, Figure 4 The light receiving units 415) are arranged to correspond to one color channel (eg, Figure 5 The output of the third light receiving element corresponding to the photodiode of the green channel 531) is read out in a manner corresponding to one pixel. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101 acquires a third image corresponding to a third resolution by performing image signal processing on the third original image data.

[0133] In an embodiment, at least one processor (e.g., Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) acquires information about a context related to image capturing when the determined capturing mode corresponds to the first resolution. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3The electronic device 101) outputs a first image by performing image signal processing on the first raw image data based on the information about the context corresponding to the specified context. Figure 1 processor 120, Figure 2 Image signal processor 260, Figure 3 The instructions executed by the at least one processor 320 of the electronic device (eg, Figure 1 and Figure 3 The electronic device 101) obtains a first image corresponding to a first resolution based on a result of inputting the first original image data into a machine learning model based on the information about the context not corresponding to a specified context.

[0134] In an embodiment, the designated scenario may include at least one of a first scenario in which illumination is less than a threshold, a second scenario in which the captured image is defocused, or a third scenario in which a high-frequency component in the image is less than a threshold. Image signal processing for the first raw image data may include binning pixels of the first raw image data or re-mosaicing to convert the pattern of the first raw image data into a Bayer pattern, and upscaling to enlarge the image.

[0135] According to an embodiment, a method for operating a system comprising an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 image sensor 383) of a camera (e.g., Figure 1 Camera module 180, Figure 3 The electronic device (e.g., camera 380) Figure 1 and Figure 3 The method of the electronic device 101 may include: determining a target device by using a camera (eg, Figure 1 Camera module 180, Figure 3 The method may include: based on the determination that the shooting mode corresponds to the first resolution, controlling the image sensor (for example, Figure 2 Image sensor 230, Figure 3 The image sensor 383 of the image sensor 383 outputs first original image data corresponding to the first data, wherein the first data is in a format corresponding to the image sensor (for example, Figure 2 Image sensor 230, Figure 3The output of a first light receiving element corresponding to one photodiode among a plurality of photodiodes (e.g., a first photodiode 511, a second photodiode 512, a third photodiode 513, a fourth photodiode 514) of an image sensor 383) is read out in a manner corresponding to one pixel. The method may include: acquiring a first image corresponding to a first resolution based on a result of inputting the first original image data into a machine learning model. The method may include: outputting second original image data acquired by performing at least one image processing operation on the second data based on a determined shooting mode corresponding to a second resolution lower than the first resolution, wherein the second data is read out in a manner corresponding to the output of a second light receiving element corresponding to at least two or more photodiodes among a plurality of photodiodes (e.g., a first photodiode 511, a second photodiode 512, a third photodiode 513, a fourth photodiode 514) corresponds to one pixel. The method may include: acquiring a second image corresponding to a second resolution by performing image signal processing on the second original image data. An image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may include a plurality of light receiving elements including a plurality of photodiodes (e.g., first photodiode 511, second photodiode 512, third photodiode 513, and fourth photodiode 514) arranged in an M×N configuration corresponding to one first microlens (e.g., first microlens 521). At least one of M or N may be a natural number greater than or equal to 2.

[0136] In an embodiment, the first original image data may include data in which a color order of the read data is maintained.

[0137] In an embodiment, the step of outputting the second original image data may include: outputting the second original image data by an image sensor (eg, Figure 2 Image sensor 230, Figure 3 The image sensor 383 acquires second raw image data by performing a re-mosaicing process to change the color sequence of the second data to have a Bayer pattern.

[0138] In an embodiment, the step of outputting the first original image data may include: outputting the first original image data by an image sensor (eg, Figure 2 Image sensor 230, Figure 3 Image sensor 383) based on a camera (e.g., Figure 1 Camera module 180, Figure 3 380) of the camera) or the zoom magnification of the camera (e.g., Figure 1 Camera module 180, Figure 3 The lens unit of the camera 380) (eg, Figure 3The first data is calibrated by at least one of the focus positions of the lens unit 381).

[0139] In an embodiment, calibrating the first data may include compensating for a division ratio of light to a plurality of photodiodes (eg, the first photodiode 511 , the second photodiode 512 , the third photodiode 513 , and the fourth photodiode 514 ).

[0140] In an embodiment, the step of determining the shooting mode may include: executing a method for shooting a video by using a camera (e.g., Figure 1 Camera module 180, Figure 3 The camera 380 may be used to capture images using a camera application. Determining the capture mode may include displaying a screen including a user interface for selecting a resolution based on the running camera application. Determining the capture mode may include determining the capture mode based on user input received based on the user interface.

[0141] In an embodiment, one pixel included in the second raw image data may correspond to four photodiodes (eg, Figure 5 The method may further include: outputting third raw image data based on the third data based on the determined shooting mode corresponding to a third resolution lower than the second resolution, wherein the third data is arranged to correspond to one color channel (for example, Figure 5 The output of the third light receiving element corresponding to the plurality of photodiodes of the green channel 531 is read out in a manner corresponding to one pixel. The method may include acquiring a third image corresponding to a third resolution by performing image signal processing on the third raw image data.

[0142] In an embodiment, acquiring the first image may include acquiring information about a context related to image capture when the determined shooting mode corresponds to the first resolution. Acquiring the first image may include acquiring the first image by performing image signal processing on the first raw image data based on the information about the context corresponding to a specified context. Acquiring the first image may include acquiring the first image corresponding to the first resolution based on a result of inputting the first raw image data into a machine learning model based on the information about the context not corresponding to the specified context.

[0143] In an embodiment, the designated scenario may include at least one of a first scenario in which illumination is less than a threshold, a second scenario in which the captured image is defocused, or a third scenario in which a high-frequency component in the image is less than a threshold. Image signal processing for the first raw image data may include binning pixels of the first raw image data or re-mosaicing to convert the pattern of the first raw image data into a Bayer pattern, and upscaling to enlarge the image.

[0144] According to an embodiment, an image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383 may include: a plurality of light receiving elements (eg, Figure 4 a light receiving unit 415); a plurality of micro lenses; and a color filter including a plurality of color channels. Figure 4 The light receiving unit 415 may include a first micro lens (eg, Figure 5 The image sensor (e.g., the first microlens 521) includes a plurality of photodiodes (e.g., the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514). At least one of M and N may be a natural number greater than or equal to 2. Figure 2 Image sensor 230, Figure 3 The image sensor 383 of the embodiment of the present invention may be configured to: when operating based on a first shooting mode corresponding to a first resolution, output first raw image data corresponding to first data, wherein the first data is read out in such a manner that an output of a first light receiving element corresponding to one photodiode among a plurality of photodiodes (e.g., a first photodiode 511, a second photodiode 512, a third photodiode 513, and a fourth photodiode 514) corresponds to one pixel. The image sensor (e.g., Figure 2 Image sensor 230, Figure 3 The image sensor 383) can be configured to: when operating based on a second shooting mode corresponding to a second resolution lower than the first resolution, output second original image data obtained by performing at least one image processing operation on the read second data, wherein the second data is read out in a manner that the output of the second light receiving element corresponding to at least two or more photodiodes among a plurality of photodiodes (for example, the first photodiode 511, the second photodiode 512, the third photodiode 513, and the fourth photodiode 514) corresponds to one pixel.

[0145] In an embodiment, the first raw image data may include data in which a color order of the read data is maintained.

[0146] In an embodiment, a computer-readable non-transitory recording medium may have recorded thereon a computer-readable non-transitory recording medium for use in an electronic device (e.g., Figure 1 and Figure 3 A computer program for executing the method when the electronic device 101 executes the above method.

[0147] According to one or more embodiments, an electronic device and / or an operating method thereof is provided, which can acquire an image without artifacts when an image sensor operates based on an operating mode for acquiring an image of a greater number of pixels.

[0148] According to one or more embodiments, there is provided an electronic device and / or an operating method thereof, which can obtain a high MTF value when an image sensor operates in an operating mode for acquiring an image of a greater number of pixels.

[0149] Effects achieved by the present disclosure are not limited to those mentioned above, and other effects not mentioned above may be clearly understood by those skilled in the art based on the description provided below.

[0150] The methods according to the embodiments disclosed in the claims or the present disclosure may be implemented in hardware, software, or a combination of both.

[0151] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions for enabling the electronic device to execute the methods according to the claims or embodiments disclosed in this disclosure.

[0152] Programs (software modules or software) can be stored in random access memory (RAM), non-volatile memory including flash memory, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic disc storage devices, compact disc-ROMs (CD-ROMs), digital versatile discs (DVDs), or other forms of optical storage devices, as well as magnetic cassettes. Alternatively, the program can be stored in a memory configured by combining all or some of these storage media. Furthermore, the configured memory can be plural in number.

[0153] Furthermore, the program may be stored in an attachable storage device that can access an electronic device via a communication network (such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a communication network configured as a combination of networks). The storage device may access the device executing the embodiments of the present disclosure via an external port. Furthermore, an attached storage device on the communication network may access the device executing the embodiments of the present disclosure.

[0154] In the above-mentioned specific embodiments of the present disclosure, the elements included in the present disclosure are expressed in singular or plural form according to the specific embodiment. However, for ease of explanation, the singular or plural form is appropriately selected according to the presented situation, and the present disclosure is not limited to a single element or multiple elements. An element expressed in plural form can be configured in singular form, or an element expressed in singular form can be configured in plural form.

[0155] The term “unit” or “module” used in the present disclosure refers to a hardware component such as a processor or a circuit and / or a software component executed by a hardware component such as a processor.

[0156] A "unit" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "unit" or "module" may be implemented by a component (such as a software component, an object-oriented software component, a class component, and a task component), a process, a function, a property, a program, a subroutine, a program code segment, a driver, firmware, microcode, a circuit, data, a database, a data structure, a table, an array, and a parameter.

[0157] The specific implementations explained in this disclosure are merely examples, and the scope of this disclosure is not limited in any way. For the sake of clarity of the specification, descriptions of electronic components, control systems, software, and other functional aspects of the related art are omitted.

[0158] In the present disclosure, phrases such as "including at least one of a, b or c" may mean "including only a", "including only b", "including only c", "including a and b", "including b and c", "including a and c", "including all of a, b, c".

[0159] Although specific embodiments have been described in the detailed description of the present disclosure, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is not limited by the illustrated embodiments but by the appended claims and the equivalents of the claims.

Claims

1. An electronic device comprising: A camera comprising a lens unit and an image sensor, wherein the image sensor is configured to convert an optical signal passing through the lens unit into a digital signal; a memory configured to store instructions; and at least one processor, Wherein, the image sensor includes: a plurality of microlenses, including a first microlens, a plurality of light receiving elements corresponding to a plurality of photodiodes arranged in an M×N configuration corresponding to the first microlens, at least one of M or N being a natural number greater than or equal to 2, wherein the plurality of photodiodes includes a first photodiode and a group of second photodiodes, and Color filters, including multiple color channels, and The instructions executed by the at least one processor cause the electronic device to perform the following operations: determining a capture mode associated with a resolution of an image to be captured by using the camera; Based on the determined shooting mode corresponding to the first resolution, outputting first raw image data corresponding to first data read out from the first photodiode by controlling the image sensor, wherein the output of the first photodiode corresponds to one pixel; Based on a result of inputting the first original image data into the machine learning model, obtaining a first image corresponding to the first resolution; outputting, by controlling the image sensor, second raw image data obtained by performing at least one image processing operation on second data read out from the group of second photodiodes, based on the determined shooting mode corresponding to a second resolution lower than the first resolution, wherein outputs of the group of second photodiodes correspond to one pixel; and A second image corresponding to a second resolution is acquired by performing image signal processing on the second original image data.

2. The electronic device according to claim 1, wherein The first original image data includes data in which the color order of the first data is maintained.

3. The electronic device according to claim 2, wherein: The image sensor is further configured to output second raw image data by performing a re-mosaicing process to change a color sequence of the second data to include a Bayer pattern.

4. The electronic device according to claim 2, wherein: The image sensor is further configured to output first raw image data by calibrating first data based on at least one of a zoom magnification of the camera or a focus position of the lens unit.

5. The electronic device according to claim 4, wherein: The image sensor is further configured to calibrate first data by compensating for a division ratio of light to the plurality of photodiodes.

6. The electronic device according to claim 1, further comprising a display, in, The instructions executed by the at least one processor further cause the electronic device to: Run the camera app; Based on the running camera application, displaying a user interface for selecting a resolution through the display; and The image sensor is controlled based on a shooting mode, wherein the shooting mode is determined based on user input received through the user interface.

7. The electronic device according to claim 1, wherein The plurality of microlenses of the image sensor are arranged such that four second microlenses including a first microlens correspond to one color channel among the plurality of color channels in the color filter, The plurality of photodiodes further includes a group of third photodiodes arranged to the one color channel, and The instructions executed by the at least one processor further cause the electronic device to perform the following operations: outputting third raw image data by performing at least one image processing operation on third data read out from the group of third photodiodes, based on the determined shooting mode corresponding to a third resolution lower than the second resolution, by controlling the image sensor, wherein outputs of the group of third photodiodes correspond to one pixel; and A third image corresponding to a third resolution is acquired by performing image signal processing on the third original image data.

8. The electronic device according to claim 1, wherein The instructions executed by the at least one processor further cause the electronic device to: acquiring information about a situation related to image capturing if the determined capturing mode corresponds to the first resolution; outputting a first image by performing image signal processing on first raw image data based on the information about the context corresponding to a specified context; as well as Based on the information about the context not corresponding to a specified context, a first image corresponding to a first resolution is acquired based on a result of inputting the first original image data into the machine learning model.

9. The electronic device according to claim 8, wherein: The designated scenario includes at least one of a first scenario in which illumination is less than a first threshold, a second scenario in which a captured image is defocused, or a third scenario in which a high-frequency component included in the image is less than a second threshold, and The image signal processing for the first original image data includes: a pixel merging process for the first original image data or a re-mosaicing process for converting a pattern of the first original image data into a Bayer pattern, and an upscaling process for enlarging the image.

10. A method performed by an electronic device comprising a camera, the camera comprising an image sensor, wherein: The image sensor includes a plurality of photodiodes arranged in an M×N configuration corresponding to one first microlens, wherein at least one of M or N is a natural number greater than or equal to 2, and the method includes: determining a capture mode associated with a resolution of an image to be captured by using the camera; Based on the determined shooting mode corresponding to the first resolution, outputting first raw image data corresponding to first data read out from a first photodiode among the plurality of photodiodes by controlling the image sensor, wherein the output of the first photodiode corresponds to one pixel; Based on a result of inputting the first original image data into the machine learning model, obtaining a first image corresponding to the first resolution; outputting second raw image data obtained by performing at least one image processing operation on second data read out from a group of second photodiodes among the plurality of photodiodes based on the determined shooting mode corresponding to a second resolution lower than the first resolution, wherein outputs of the group of second photodiodes correspond to one pixel; and A second image corresponding to a second resolution is acquired by performing image signal processing on the second original image data.

11. The method according to claim 10, wherein: The first original image data includes data in which the color order of the first data is maintained.

12. The method according to claim 11, wherein The step of outputting the second original image data includes acquiring the second original image data by performing a re-mosaicing process by the image sensor to change a color sequence of the second data to include a Bayer pattern.

13. The method according to claim 11, wherein Outputting the first raw image data includes calibrating, by the image sensor, the first data based on at least one of a zoom magnification or a focus position of the camera.

14. The method according to claim 13, wherein The step of calibrating the first data includes compensating for a division ratio of light to the plurality of photodiodes.

15. The method according to claim 10, wherein The step of determining the shooting mode includes: Run the camera app; Based on the running camera application, displaying a user interface for selecting a resolution; and A shooting mode is determined based on a user input received through the user interface.

Images