Electronic device comprising camera, and operating method therefor

Abstract

An electronic device of the disclosed embodiments may comprise a memory and a camera. The electronic device may be configured to process a non-Bayer pattern image signal using an image signal processor (ISP) designed to process a Bayer pattern image signal.

Description

Electronic device including a camera and method of operating the same

[0001] The present disclosure relates to an electronic device including a camera and a method of operating the same.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. Electronic devices can provide various functions. For example, electronic devices can run at least one application in the foreground and / or background to provide at least one function.

[0003] Electronic devices can provide various functions using a designated operating system (e.g., the Android™ operating system). For example, an electronic device can support multiple functions provided using a camera.

[0004] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0005] An electronic device according to one embodiment may include a color filter array including a first color pattern, an image sensor generating an image signal using light passing through the color filter array, an image signal processor processing an image signal composed of a second color pattern, a display, a memory storing instructions, and a processor executing the instructions. The electronic device may receive a user input regarding acquisition of a capture image. Based on the user input, the electronic device may acquire a first image signal of a first frame composed of the first color pattern through the image sensor. The electronic device may convert the first image signal of the first frame into a second image signal of the first frame composed of the second color pattern. The electronic device may acquire first draft image data by applying the second image signal of the first frame to the image signal processor. The electronic device may acquire first capture image data by demosaicing the first image signal of the first frame. The electronic device may replace the first draft image data with the first capture image data.

[0006] According to one embodiment, a method of operating an electronic device may include an operation method of an electronic device including a color filter array including a first color pattern, an image sensor generating an image signal using light passing through the color filter array, and an image signal processor processing an image signal composed of a second color pattern. The method of operating the electronic device may include an operation of receiving a user input regarding acquisition of a capture image. The method of operating the electronic device may include an operation of acquiring a first image signal of a first frame composed of a first color pattern through the image sensor based on the user input. The method of operating the electronic device may include an operation of converting the first image signal of the first frame into a second image signal of the first frame composed of a second color pattern. The method of operating the electronic device may include an operation of acquiring first draft image data by applying the second image signal of the first frame to the image signal processor. The method of operating the electronic device may include an operation of acquiring first capture image data by demosaicing the first image signal of the first frame. A method of operating an electronic device may include replacing first draft image data with first capture image data.

[0007] In one embodiment, a computer-readable non-transitory recording medium may be a recording medium having recorded thereon instructions for controlling an electronic device, the electronic device including a color filter array including a first color pattern, an image sensor for generating an image signal using light passing through the color filter array, and an image signal processor for processing an image signal composed of a second color pattern. The recording medium may store instructions for receiving a user input regarding acquisition of a capture image. The recording medium may store instructions for acquiring a first image signal of a first frame composed of a first color pattern through the image sensor based on the user input. The recording medium may store instructions for converting the first image signal of the first frame into a second image signal of the first frame composed of a second color pattern. The recording medium may store instructions for acquiring first draft image data by applying the second image signal of the first frame to the image signal processor. The recording medium may store instructions for acquiring first capture image data by demosaicing the first image signal of the first frame. The recording medium can store a command to replace the first draft image data with the first capture image data.

[0008] FIG. 1 is a diagram illustrating a method for an electronic device to acquire an image using a camera module according to one embodiment.

[0009] FIG. 2 is a block diagram illustrating an electronic device within a network environment according to various embodiments.

[0010] FIG. 3 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 4 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0012] FIG. 5 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment.

[0013] FIG. 6 is a diagram conceptually illustrating a pattern of a color filter layer according to one embodiment.

[0014] FIG. 7 is a drawing for explaining an operating method of an electronic device according to one embodiment.

[0015] Figure 8 is a flowchart of an operating method of an electronic device according to one embodiment.

[0016] FIG. 9 is a flowchart of an operating method of an electronic device according to one embodiment.

[0017] Fig. 10 is a flowchart of an operating method of an electronic device according to one embodiment.

[0018] FIG. 11 is a diagram conceptually illustrating conversion of a color pattern of an image signal according to one embodiment.

[0019] FIG. 12 is a diagram conceptually illustrating conversion of a color pattern of an image signal according to one embodiment.

[0020] FIG. 13 is a diagram conceptually illustrating upscaling of image data according to one embodiment.

[0021] FIG. 1 is a diagram illustrating a method for an electronic device to acquire an image using a camera module according to one embodiment.

[0022] Referring to FIG. 1, the electronic device (10) can acquire an image signal using an image sensor included in a camera module. For example, the electronic device (10) can acquire an image signal by controlling the camera module through a camera application. The camera application can generate a control signal including a request related to acquiring an image signal. The camera module can acquire an image signal in response to a request received from the camera application. The electronic device (10) can acquire still image and / or moving image data by processing the image signal output from the image sensor.

[0023] The electronic device (10) may include a hardware-implemented image signal processor (ISP) that consumes little power and processes image signals quickly. The image signal processor may be implemented as a hardware chip or as a hardware block included in a processor (e.g., an application processor (AP), a central processing unit (CPU), a graphical processing unit (GPU), or a neural processing unit (NPU)).

[0024] A hardware-implemented image signal processor can rapidly process image signals by utilizing a pipeline designed to process a predetermined type of image signal. For example, the image signal processor may be designed to process image signals composed of the commonly used Bayer pattern. For example, the image signal processor may not be able to process an image signal composed of a pattern different from the Bayer pattern (hereinafter referred to as a non-Bayer pattern).

[0025] An image acquired using a high-pixel image sensor including a color filter array of a non-Bayer pattern may have higher quality in a low-light environment than an image acquired using a high-pixel image sensor including a color filter array of a Bayer pattern. An image acquired using a high-pixel image sensor including a color filter array of a non-Bayer pattern may have higher quality in a high-light environment than an image acquired using a low-pixel image sensor including a color filter array of a Bayer pattern. Although image signals composed of a non-Bayer pattern are not commonly used, image sensors including a color filter array of a non-Bayer pattern are being mounted on electronic devices (10) to meet the demand for high-quality images.

[0026] In order for an electronic device (10) to process a second image signal (e.g., an image signal composed of a non-Bayer pattern) different from a first image signal of a preset type (e.g., an image signal composed of a Bayer pattern) using a hardware-implemented image processor, a new image signal processor with a redesigned pipeline capable of processing the second image signal must be installed. Developing and producing a new image signal processor requires a lot of time and cost. Therefore, methods are required in which the first image signal is processed using an existing image signal processor and the second image signal is processed using software.

[0027] According to one embodiment, the electronic device (10) can display a preview image (11) through a camera application. The preview image (11) can include an image provided by the electronic device (10) to enable the user to check the location, lighting, etc. of the subject to be photographed so as to obtain a desired image. The electronic device (10) can display the preview image (11) output by the camera module through the display in response to a preview request transmitted by the camera application to the camera module. The electronic device (10) can obtain an image signal of a frame through an image sensor in response to the preview request received from the camera application. The obtained image signal can be an image signal composed of a first color pattern (e.g., a Bayer pattern). The electronic device (10) can generate the preview image (11) by applying the image signal obtained from the image sensor to an image signal processor. The image signal processor can be a hardware-image signal processor designed to process an image signal composed of the first color pattern. An image signal processor can receive an image signal of a frame acquired in response to a preview image request from an image sensor and generate a preview image (11). An electronic device (10) can display the preview image (11) output from the image signal processor.

[0028] According to one embodiment, the electronic device (10) may acquire image data in response to a user input received through a camera application. For example, the electronic device (10) may acquire still images and / or moving images in response to a user input to acquire still images and / or moving images. The electronic device (10) may acquire image signals of frames for generating still images and / or moving images through an image sensor. The electronic device (10) may generate still images and / or moving images by processing the image signals. The electronic device (10) may generate draft image data (13) and capture image data in response to the user input. The draft image data (13) may include image data generated by applying only some of the operations that can be applied to image signals acquired by the image sensor so that the user can quickly check the image requested. The capture image data may include image data generated by applying one or more preset operations to the image signals. For example, the captured image data may include image data generated by applying at least one operation, such as demosaicing, operations for adjusting white balance, contrast, saturation values, gamma correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement. The captured image data may be an image of higher quality than the draft image data (13). For example, the captured image data may be an image composed of higher pixels than the draft image data (13). For example, the captured image data may be an image of higher quality than the draft image data (13) by removing noise.For example, the captured image data may be an HDR (high dynamic range) image generated by synthesizing multiple frames. The number of operations applied to generate the captured image data may be greater than the number of operations applied to generate the draft image data (13). The time required to generate the captured image data may be greater than the time required to generate the draft image data (13). The electronic device (10) may display the draft image data (13) before the captured image data is generated. For example, the electronic device (10) may display the draft image data (13) in at least a portion of a camera application. For example, the electronic device (10) may generate a thumbnail (12) by resizing the draft image data (13) and display the thumbnail (12) in a portion of the camera application. For example, the electronic device (10) may display the draft image data (13) using a gallery application. The electronic device (10) can replace the draft image data (13) with the captured image data generated after generating the draft image data (13). For example, the electronic device (10) can overwrite the draft image data (13) with the captured image data. For example, the electronic device (10) can display the draft image data (13) displayed on the display by replacing it with the captured image data.

[0029] According to one embodiment, the electronic device (10) may transmit a capture request to the camera module through the camera application in response to a user input requesting to acquire a capture image. The electronic device (10) may use the image signal of the first frame acquired for the first time after the camera module receives the capture request to generate capture image data. The image signal of the first frame may be an image signal composed of a second color pattern (e.g., a non-Bayer pattern). If the image signal processor is designed to process the image signal composed of the first color pattern (e.g., a Bayer pattern), the image signal processor may not be able to process the image signal of the first frame. The electronic device (10) may acquire the capture image data by performing signal processing on the image signal of the first frame using a software module. For example, the electronic device (10) may demosaic the image signal of the first frame using the software module. For example, the electronic device (10) can adjust the white balance value, contrast value, and saturation value for the image signal of the first frame using the software module. For example, the electronic device (10) can perform noise removal for the image signal of the first frame using the software module. The electronic device (10) can use the image signal of the second frame acquired before the first frame to generate draft image data (13). The image signal of the second frame may be a signal of an image acquired in response to a preview request. When the image signal of the second frame is a signal of an image acquired in response to a preview request, the image signal of the second frame may be an image signal composed of a first color pattern (e.g., a Bayer pattern). Therefore, the electronic device (10) can acquire draft image data (13) by applying the image signal of the second frame to the image signal processor.Due to the time difference between the acquisition of the first frame and the second frame by the electronic device (10), a difference in composition may occur between the captured image data acquired from the image signal of the first frame and the draft image data (13) acquired from the image signal of the second frame. The difference between the captured image data and the draft image data (13) may cause inconvenience to the user of the electronic device (10).

[0030] According to one embodiment, the electronic device (10) can use the image signal of the same frame to obtain the capture image data and the draft image data (13). For example, the electronic device (10) can use the image signal of the first frame that the camera module first obtains after receiving a capture request to generate the draft image data (13) and the capture image data. For example, the electronic device (10) can obtain the draft image data (13) through a hardware image signal processor by performing preprocessing on the image signal of the first frame. According to the disclosed embodiment, the electronic device (10) can obtain the capture image data without a difference in composition with respect to the draft image data (13).

[0031] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field belonging to the present disclosure from the description below.

[0032] FIG. 2 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 2, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0033] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0034] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0035] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0039] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0040] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0041] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0045] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0047] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0050] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0052] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] FIG. 3 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 3, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0055] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0056] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0057] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0058] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0059] FIG. 4 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0060] Referring to FIG. 4, an electronic device (101) according to one embodiment may include a camera module (380), a memory (330), and a processor (320). The electronic device (101) according to one embodiment may further include a display (360). The display (360) may be replaced with an external display connected to the electronic device (101). The electronic device (101), the processor (320), the memory (330), the camera module (380), and the display (360) may each correspond to the electronic device (101), the processor (120), the memory (130), the camera module (180), and the display module (160) described above with reference to FIGS. 2 and 3 . However, the components of the electronic device (101) illustrated in FIG. 4 are for describing one embodiment, and the electronic device (101) may include more components than the components illustrated in FIG. 4 or may include other components that may replace at least some of the components. For example, the memory (330) is not limited to a storage medium included in the electronic device (101), but may include a cloud storage external to the electronic device (101).

[0061] According to one embodiment, the camera module (380) may include a lens unit (381) including at least one lens for focusing light, and an image sensor (383) for converting an optical signal passing through the lens unit (381) into a digital signal. The image sensor (383) may correspond to the image sensor (230) described with reference to FIG. 3. The image sensor (383) may include a color filter array including a plurality of light-receiving elements, a plurality of micro-lenses, and a plurality of color channels. The plurality of light-receiving elements may include photodiodes arranged to correspond to one micro-lens in an array having M rows and N columns. Here, M and N may each be a natural number greater than or equal to 2. The color filter array included in the image sensor (383) may be configured with a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern). That is, each of the elements constituting the color filter array may be color-matched with a non-Bayer pattern. Among the elements that make up the color filter array, groups of elements that are grouped into elements that are matched with the same color and recognize each other can be color-matched to correspond to the Bayer pattern.

[0062] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a non-Bayer pattern. For example, the image sensor (383) including a color filter array composed of a non-Bayer pattern can output an image signal composed of a non-Bayer pattern by determining a pixel value using an output value of a light-receiving element corresponding to an element. The image signal output from the image sensor (383) can be data in which the color pattern is maintained by not changing the color order of the color pattern of the image sensor (383).

[0063] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) configured in a Bayer pattern. For example, an image sensor (383) including a color filter array configured in a non-Bayer pattern can output an image signal configured in a Bayer pattern by binning output values ​​of light-receiving elements corresponding to groups of elements constituting the color filter array.

[0064] According to one embodiment, the image sensor (383) may operate in a high-resolution shooting mode. The high-resolution shooting mode may include a mode in which each of the output values ​​of the light-receiving elements included in the image sensor (383) is used as the pixel value of the pixel corresponding to each of the light-receiving elements. For example, the electronic device (101) may acquire high-resolution image data by using the image sensor (383) composed of 50 Mp light-receiving elements and using each of the output values ​​of the 50 Mp light-receiving elements as the pixel value of the pixel corresponding to the light-receiving elements. In the present disclosure, the high-resolution shooting mode may be understood as a full-pixel shooting mode.

[0065] According to one embodiment, the image sensor (383) may operate in a crop shooting mode. The crop shooting mode may include a mode in which the output values ​​of a predetermined number of light-receiving elements (for example, a predetermined number of light-receiving elements located at the center of the image sensor) among the light-receiving elements are used as pixel values. The electronic device (101) may acquire an image with a narrowed angle of view through the crop shooting mode and provide the user with an experience similar to that of a zoom-in function. For example, when the electronic device (101) operates in a crop shooting mode in which the output values ​​of 12.5 Mp light-receiving elements located at the center of the image sensor (383) composed of 50 Mp light-receiving elements are used as pixel values, the electronic device may provide the user with an experience similar to that of a 2x zoom-in function. The electronic device (101) may control the image sensor (383) to operate in the crop shooting mode based on a user input that sets the zoom function to a preset magnification value.

[0066] According to one embodiment, the image sensor (383) may operate in a low-light shooting mode. The low-light shooting mode may include a mode in which the pixel values ​​of the first pixels corresponding to the first light-receiving elements are used based on the values ​​output from the first light-receiving elements corresponding to the elements of the color filter array matched with the same color of the light-receiving elements included in the image sensor (383).

[0067] According to one embodiment, the image sensor (383) may operate in a multi-frame synthesis mode. The multi-frame synthesis mode may include a mode in which the image sensor (383) acquires multiple frames in which differences in exposure values ​​exist.

[0068] According to one embodiment, the memory (330) can store instructions that can be executed by the processor (320). The processor (320) can perform operations or control components of the electronic device (101) by executing the instructions stored in the memory (330).

[0069] In the present disclosure, the operation of the electronic device (101) may be understood as being performed by at least one processor (320) executing instructions. According to one embodiment, the processor (320) may include at least one of an application processor (AP), a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 3), a graphical processing unit (GPU), or a neural processing unit (NPU).

[0070] According to one embodiment, the processor (320) can control the camera module (380) by executing a camera application. For example, the processor (320) can initiate the operation of the camera module (380) through the camera application. For example, the processor (320) can provide the camera module (380) with a request for acquisition of at least one frame (e.g., a frame for preview, a frame for capture) through the camera application. For example, the processor (320) can acquire one or more frames through the camera application. For example, the processor (320) can identify a user input received through the camera application (e.g., a user input for acquisition of a capture image). For example, the processor (320) can acquire image data (e.g., preview image data, draft image data, capture image data, video data) through the camera application. For example, the processor (320) can display the acquired image data using the display (360).

[0071] According to one embodiment, the processor (320) may control the image sensor to acquire an image corresponding to a user input. The processor (320) may control the image sensor (383) to operate based on a selected shooting mode. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of the light-receiving elements constituting the image sensor (383) based on a first shooting mode (e.g., high-resolution shooting mode or full-pixel shooting mode) so that the output corresponds to each pixel corresponding to each of the light-receiving elements. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of a predetermined number of first light-receiving elements among a plurality of light-receiving elements so that the output corresponds to each pixel corresponding to each of the first light-receiving elements based on a second shooting mode (e.g., crop shooting mode). For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out the output of a first group including a plurality of light-receiving elements based on a third shooting mode (e.g., low-light shooting mode) so that it corresponds to a first pixel corresponding to the first group.

[0072] According to one embodiment, the processor (320) may obtain image data by performing an operation on an image signal (e.g., raw image data) output from the image sensor (383). For example, the processor (320) may perform at least one operation (e.g., demosaicing, operations for adjusting white balance, contrast, saturation values, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on the image signal. For example, the processor (320) may perform remosaicking on a first image signal (e.g., raw image data composed of a non-Bayer pattern) output from the image sensor (383), thereby generating a second image signal (e.g., raw image data composed of a Bayer pattern). For example, the processor (320) can input an image signal output from the image sensor (383) to the image signal processor, thereby performing at least one operation (e.g., demosaicing, white balance, contrast, saturation value adjustment operation, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement). The processor (320) can store the generated image data in the memory (330).

[0073] In one embodiment, the processor (320) may initiate operation of the camera module (380). For example, the processor (320) may initiate operation of the camera module (380) by providing a control signal to a hardware abstraction layer (HAL) that includes an ID of the camera and a CameraOpen command that initiates operation of the camera.

[0074] In one embodiment, the processor (320) can acquire one or more frames. For example, the processor (320) can acquire one or more frames by controlling the camera module (380) through a request to acquire a frame. For example, the processor (320) can acquire a frame for preview composed of a Bayer pattern. For example, the processor (320) can acquire a frame for capture composed of a non-Bayer pattern.

[0075] According to one embodiment, the processor (320) can convert a first image signal composed of a first color pattern (e.g., a non-Bayer pattern) into a second image signal composed of a second color pattern (e.g., a Bayer pattern). For example, the processor (320) can convert the first image signal into the second image signal by performing remosaicing on the first image signal. For example, the processor (320) can convert the first image signal into the second image signal by performing binning on the first image signal. For example, the processor (320) can control the computation unit (e.g., 417 of FIG. 5) of the image sensor (383) to convert the first image signal into the second image signal.

[0076] In one embodiment, the processor (320) can identify a received user input. For example, the processor (320) can identify a user input that changes a shooting mode (e.g., a high-resolution shooting mode, a crop shooting mode, a low-light shooting mode, a multi-frame synthesis mode, a still image shooting mode, a video shooting mode). For example, the processor (320) can identify a user input related to capturing a still image and / or a video. For example, the processor (320) can identify a user input that requests capturing a multi-frame synthesised image.

[0077] According to one embodiment, the processor (320) can obtain preview image data. For example, the processor (320) can generate preview image data by applying an image signal composed of a Bayer pattern output from the image sensor (383) to an image signal processor. For example, the processor (320) can control the operation unit (e.g., 417 of FIG. 5) of the image sensor (383) to output an image signal composed of a Bayer pattern by binning the output values ​​of the light-receiving element of the image sensor (383).

[0078] According to one embodiment, the processor (320) can obtain draft image data. For example, the processor (320) can convert a first image signal composed of a non-Bayer pattern output from the image sensor (383) into a second image signal composed of a Bayer pattern, and apply the second image signal to an image signal processor, thereby generating draft image data. For example, the processor (320) can convert the pixel values ​​constituting the first image signal into the second image signal by binning. For example, the processor (320) can obtain draft image data by performing demosaicing and / or upscaling on the second image signal obtained by performing binning.

[0079] According to one embodiment, the processor (320) can obtain a capture image. For example, the processor (320) can generate a capture image by performing at least one operation (e.g., demosaicing, white balance, contrast, operation for adjusting saturation value, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on a first image signal composed of a non-Bayer pattern output from the image sensor (383). For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a high-resolution shooting mode. For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a crop shooting mode.

[0080] According to one embodiment, the processor (320) can obtain a multi-frame composite image. For example, the processor (320) can set a first frame among a plurality of frames obtained through the image sensor (383) as a reference frame. The processor (320) can synthesize an image signal of a second frame with an image signal of the first frame based on the first frame. The processor (320) can obtain a multi-frame composite image by performing demosaicing on the synthesized image signal.

[0081] According to one embodiment, the processor (320) may display the acquired image data using the display (360). For example, the processor (320) may display at least one of preview image data, draft image data, and captured image data in at least a portion of the camera application.

[0082] In one embodiment, the processor (320) can replace draft image data with captured image data. For example, the processor (320) can overwrite draft image data stored in the memory (330) with captured image data. For example, the processor (320) can control the display (360) to display the draft image data by replacing it with the captured image data.

[0083] According to one embodiment, the display (360) can display one or more pieces of information under the control of the processor (320). For example, the display (360) can display a user interface (UI) of the electronic device. For example, the display (360) can display an execution screen of an application running on the electronic device. For example, the display (360) can display at least one of preview image data, draft image data, and capture image data.

[0084] FIG. 5 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment. The image sensor of FIG. 5 may correspond to the image sensor (230, 383) described above with reference to FIGS. 3 and 4.

[0085] Referring to FIG. 5, in one embodiment, the image sensor (230) may include a micro lens array (MLA) (411), a color filter array (CFA) (413), a light receiving unit (415), and a computation unit (417).

[0086] In one embodiment, the microlens array (411) may be arranged so that a light bundle (421) that passes through the lens assembly and forms an image on the image sensor (230) is focused on a light-receiving element of the light-receiving unit (415). The light bundle (421) that passes through the microlens array (411) may have at least a portion of a wavelength other than a band corresponding to a specific color blocked as it passes through the color filter array (413). For example, the color filter array may be arranged at a position corresponding to a pixel of the image sensor (230). The light bundles (425) that pass through the color filter array (413) may be detected by a light-receiving element (e.g., a photodiode) of the light-receiving unit (415). The light-receiving unit (415) may include a light-receiving element (e.g., including a light receiving circuit) that generates a charge when receiving light and converts it into an electrical signal, and a circuit that selectively reads out the charge of the light-receiving element. A circuit may be further placed between the light receiving unit (415) and the calculation unit (417) to digitize the signal read from the light receiving unit (415) or reduce noise.

[0087] In one embodiment, the operation unit (417) can perform an operation to process electrical data (or signal) (427) output from the light receiving unit (415). The operation unit (417) can output data obtained based on the operation result. The output of the operation unit (417) can be a sensor output (429) of the image sensor (230).

[0088] In one embodiment, the calculation unit (417) may perform an operation to calibrate the read data as an operation to process the electrical data (427). For example, the operation performed by the calculation unit (417) may include at least one of an operation to reduce the deviation between pixels due to optical characteristics or the relative positions of sensors, an operation to reduce noise generated in an analog signal, an operation to remove a defect, an operation to perform remosaic, an operation to perform binning, or an operation to apply to a specific application (e.g., a proximity sensor function, a timing adjustment function, a HDR (high dynamic range) tone mapping function).

[0089] In one embodiment, the electronic device (101) may be configured so that the operations performed by the computation unit (417) are performed by another processor (e.g., an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP)). The sensor output (429) may be input to the processor (e.g., an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP)) through an interface.

[0090] Fig. 6 is a diagram conceptually illustrating a pattern of a color filter array according to one embodiment. The pattern applied to the color filter array (413) of Fig. 5 may be the pattern of the color filter array of Fig. 6.

[0091] Referring to FIG. 6, the color filter array may be configured with various patterns. For example, the color filter array may be configured with a Bayer pattern (510). For example, the color filter array may be configured with a non-Bayer pattern, which is a pattern different from the Bayer pattern. For example, the non-Bayer pattern may include a tetra pattern (521), a nona pattern (522), and a hexadeca pattern (523). The hexadeca pattern (523) is a tetra 2 (tetra square) pattern. The colors of groups of elements of a color filter array composed of a non-Bayer pattern can be matched to correspond to a Bayer pattern.

[0092] A color filter array composed of a Bayer pattern (510) may include a pattern in which each of the elements positioned above, below, left, and right of a first element among the elements constituting the color filter array has an element whose color is matched to a first color that matches the first element and a different color. Since the Bayer pattern (510) is a widely known pattern, further description thereof will be omitted.

[0093] A color filter array composed of a tetra pattern (521) may be arranged with an element group composed of four elements (2x2). The color filter array composed of a tetra pattern (521) may be arranged such that the elements constituting the first element group are matched with a first color, and each of the element groups located above, below, left, and right of the first element group is matched with a color different from the first color. The colors of the element groups of the color filter array composed of a tetra pattern may be matched to correspond to a Bayer pattern. Since the tetra pattern (521) is a known pattern, further description thereof will be omitted.

[0094] A color filter array composed of a nona pattern (522) may be arranged with an element group composed of nine elements (3x3). The color filter array composed of a nona pattern (522) may be configured such that the elements constituting the first element group are matched with a first color, and each of the element groups located above, below, left, and right of the first element group is matched with a color different from the first color. The colors of the element groups of the color filter array composed of a nona pattern (522) may be matched to correspond to a Bayer pattern. Since the nona pattern (522) is a known pattern, further description thereof will be omitted.

[0095] A color filter array composed of a hexadeca pattern (523) may include a pattern in which element groups composed of 16 elements (4x4) are arranged. The hexadeca pattern (523) may include a pattern in which element groups are arranged such that the elements constituting the first element group are matched with a first color, and each of the element groups located above, below, left, and right of the first element group is matched with a color different from the first color. The colors of the element groups of the color filter array composed of the hexadeca pattern (523) may be matched to correspond to a Bayer pattern. Since the hexadeca pattern (523) is a well-known pattern, further description thereof will be omitted.

[0096] Each of the elements of the color filter array can be arranged at a position corresponding to each of the light-receiving elements (e.g., 415 in FIG. 5) of the image sensor (e.g., 230 in FIG. 3, 383 in FIG. 4). Light passing through an element of the color filter array can have wavelengths other than a band corresponding to a specific color matched to the element filtered out, so that the light-receiving element corresponding to the element can detect light including a band corresponding to the specific color matched to the element.

[0097] The image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal by reading out charges from each of the light-receiving elements. By reading out charges from the light-receiving elements, the image sensor can generate an image signal including information about a color matched to an element corresponding to each of the light-receiving elements. For example, the image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal including color information of a color pattern by using pixel values ​​of pixels corresponding to each of the light-receiving elements obtained based on charge values ​​read out from each of the light-receiving elements. For example, the image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal including color information of a Bayer pattern from light-receiving elements that detect light passing through a color filter array of a Bayer pattern. For example, the image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal including color information of a tetra pattern from light-receiving elements that detect light passing through a color filter array of a tetra pattern. For example, the image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal including color information of a nona pattern from light-receiving elements that detect light passing through a color filter array of a nona pattern. For example, the image sensor (230 in FIG. 3, 383 in FIG. 4) can generate an image signal including color information of a hexadeca pattern from light-receiving elements that detect light passing through a color filter array of a hexadeca pattern.

[0098] According to one embodiment, the image sensor (230 of FIG. 3, 383 of FIG. 4) can generate an image signal in which pixels are configured to correspond to element groups. For example, the image sensor (230 of FIG. 3, 383 of FIG. 4) can generate an image signal including a pixel value of a first pixel corresponding to the first element group by calculating (e.g., binning, summing, or remosaicking) output values ​​of light-receiving elements corresponding to the first element group. For example, the image sensor (230 of FIG. 3, 383 of FIG. 4) including a color filter array configured in a tetra pattern can generate an image signal such that the first pixel corresponding to the first element group includes a pixel value for red by binning the output values ​​of four light-receiving elements corresponding to the first element group whose color is matched to red.

[0099] According to one embodiment, an image sensor (230 of FIG. 3, 383 of FIG. 4) including a color filter array configured with a non-Bayer pattern can generate an image signal including color information of a Bayer pattern based on a color pattern of an element group configured with a Bayer pattern. For example, based on the fact that each of the element groups of a color filter array configured with a tetra pattern, a nona pattern, or a hexadeca pattern is color-matched to correspond to a Bayer pattern, the image sensor (230 of FIG. 3, 383 of FIG. 4) can generate an image signal including color information of a Bayer pattern by binning output values ​​of light-receiving elements corresponding to the element groups, so that pixels corresponding to each of the element groups can generate an image signal including color information of a Bayer pattern. The image sensor (230 of FIG. 3, 383 of FIG. 4) can output an image signal including color information of a Bayer pattern. For example, an image sensor (230 in FIG. 3, 383 in FIG. 4) can output raw image data containing color information of a Bayer pattern.

[0100] According to one embodiment, an image sensor (230 in FIG. 3, 383 in FIG. 4) including a color filter array configured with a non-Bayer pattern can generate and output an image signal including color information of the non-Bayer pattern. For example, an image sensor (230 in FIG. 3, 383 in FIG. 4) including a color filter array configured with a tetra pattern, a nona pattern, or a hexadeca pattern can output raw image data including color information of the tetra pattern, the nona pattern, or the hexadeca pattern.

[0101] FIG. 7 is a drawing for explaining an operating method of an electronic device according to one embodiment.

[0102] FIG. 7 may include an operating method for an electronic device to acquire an image signal over time using a camera module including a color filter array configured with a non-Bayer pattern. FIG. 7 may include an operating method for an electronic device including a camera module (180, 380) described with reference to FIGS. 2 to 4.

[0103] According to one embodiment, the camera module can output an image signal in response to a received request. For example, the camera module can acquire and output an image corresponding to a request received from a camera application executed by the electronic device. The camera module can acquire and output frames at predetermined time intervals. The electronic device can output frames from the image sensor through a raw dum interface (RDI). The frames output from the image sensor can be transferred to a memory (e.g., 320 of FIG. 4). The electronic device can perform image processing on the frames output from the image sensor to generate image data (e.g., a preview image, a draft image, or a captured image) corresponding to the request.

[0104] Referring to FIG. 7, the camera module can output an image signal of a frame for preview in response to being in preview mode. For example, the camera module can acquire a frame for preview using an image sensor in response to a preview image request received from a camera application. For example, based on the fact that the default operating mode of the camera module is preview mode, the camera module can identify a request to initiate an operation as a request for a preview image. The camera module can acquire a frame for preview in response to receiving a request to initiate an operation from the camera application.

[0105] In one embodiment, the camera module may acquire a preview frame at predetermined time intervals. For example, the camera module may acquire a preview frame in response to receiving a preview image request from the camera application at predetermined time intervals. For example, the camera module may acquire a preview frame in response to receiving a request to initiate an operation until receiving an additional request (e.g., a capture image request) from the camera application.

[0106] According to one embodiment, a preview frame output from an image sensor may be a frame configured with a Bayer pattern. For example, a preview frame output from an image sensor may be a frame configured with a Bayer pattern. For example, the image sensor may output an image signal of a frame configured with a Bayer pattern by binning output values ​​of light-receiving elements. For example, the image sensor may output an image signal of a frame configured with a Bayer pattern by merging 2x2 arrays of pixels containing the same color information for an image signal of a frame configured with a Tetra pattern. For example, the image sensor may output an image signal of a frame configured with a Bayer pattern of 12.5 Mp (megapixel) by binning output values ​​of 50 million light-receiving elements. For example, the image sensor may output an image signal of a frame configured with a Bayer pattern by merging 3x3 arrays of pixels containing the same color information for an image signal of a frame configured with a Nona pattern. For example, an image sensor can output an image signal of a frame composed of a Bayer pattern by merging pixels in a 4x4 array containing the same color information for an image signal of a frame composed of a hexadecimal pattern.

[0107] According to one embodiment, an electronic device can generate preview image data based on an image signal of a frame output from a camera module. The electronic device can display the preview image data through a camera application. The electronic device can update the displayed preview image data at predetermined time intervals.

[0108] Referring to FIG. 7, the camera module can output an image signal of a capture frame in response to being in capture mode. Based on identifying that the camera module is in capture mode, the camera module can acquire capture frames at predetermined time intervals. For example, the camera module can identify that the camera module is in capture mode in response to receiving a capture image request from a camera application. For example, the camera module can receive a capture image request from the camera application based on a user input requesting to acquire a still image or a moving image. For example, the camera module can receive a capture image request from the camera application based on a user input requesting to acquire an image to which a function supported by the electronic device (e.g., multi-frame synthesis) is applied.

[0109] According to one embodiment, the camera module can obtain a number of capture frames corresponding to the number of capture image requests received from the camera application. For example, the camera module can obtain one capture frame corresponding to one capture image request received from the camera application based on a user input to obtain a still image. For example, the camera module can obtain a number of capture frames corresponding to the number of frames used for multi-frame synthesis from the camera application based on a user input to obtain an image to which multi-frame synthesis has been applied. For example, the camera module can obtain capture frames at predetermined time intervals corresponding to the number of frames used for multi-frame synthesis from the camera application based on a user input to obtain a video.

[0110] According to one embodiment, a capture frame output from an image sensor may be a frame configured with a non-Bayer pattern. For example, the image sensor may output an image signal (e.g., raw image data) of a frame configured with a non-Bayer pattern based on the high-resolution shooting mode. That is, the image sensor may output an image signal of a frame configured with a non-Bayer pattern by using each of the output values ​​of the photodetectors as the pixel value of the pixel corresponding to the photodetectors. For example, the image sensor may output an image signal (e.g., raw image data) of a frame configured with a non-Bayer pattern based on the crop shooting mode. For example, the image sensor may output an image signal of a frame configured with a non-Bayer pattern by using the output values ​​of 12.5 million photodetectors located at the center of the image sensor among 50 million photodetectors as the pixel value of the pixel corresponding to each of the photodetectors.

[0111] Referring to FIG. 7, the camera module can acquire a preview frame after acquisition of a capture frame is completed. For example, the camera module can change the frame acquisition mode from capture mode to preview mode after acquisition of a capture frame is completed. For example, based on the electronic device identifying that a capture image has been acquired through a camera application, the camera module can operate in preview mode by receiving a preview image request from the camera application. In response to receiving the preview image request, the camera module can reacquire a preview frame. The preview frame output after acquisition of a capture frame is completed may be a frame composed of a Bayer pattern.

[0112] According to one embodiment, an electronic device can process an image signal of a frame output from a camera module. For example, since a preview image requires fast signal processing, the electronic device can output raw image data composed of a 12.5 Mp Bayer pattern from an image sensor and apply it to an image signal processor implemented in hardware. For example, since a capture image must be a high-quality image, the electronic device can output a raw image composed of a tetra pattern from an image sensor and perform image processing. For example, since a draft image requires fast signal processing, the electronic device can convert first raw image data composed of a tetra pattern into second raw image data composed of a Bayer pattern and apply the second raw image data to an image signal processor implemented in hardware.

[0113] FIG. 8 is a flowchart illustrating an operating method of an electronic device according to one embodiment. The operation of the electronic device illustrated in FIG. 8 may be performed by a processor (e.g., processor 320 of FIG. 4) performing calculations or controlling components of the electronic device. The electronic device performing the operation of FIG. 8 may acquire an image signal using an image sensor including a color filter array configured in a non-Bayer pattern. The electronic device may generate a draft image and a captured image using the image signal of the same frame.

[0114] In one embodiment, at step 710, the electronic device may initiate operation of the camera module. For example, the electronic device may request the camera module to initiate operation by executing a camera application. The camera module may control the image sensor to acquire frames based on the request to initiate operation.

[0115] According to one embodiment, in operation 720, the electronic device may obtain a frame for preview. For example, the electronic device may obtain the preview image frame by controlling the camera module through a camera application. For example, the camera module may obtain the frame for preview based on the preview mode. For example, the camera module may obtain the frame for preview through the image sensor in response to a preview image request received from the camera application. The camera module may identify the request to initiate an operation received in operation 710 as a request for a preview image. The camera module may obtain the frame for preview in response to receiving the request to initiate an operation. The electronic device may obtain the frame for preview at predetermined time intervals.

[0116] According to one embodiment, an electronic device can obtain an image signal of a frame configured with a Bayer pattern. For example, the electronic device can obtain an image signal of a frame configured with a Bayer pattern by binning output values ​​of light-receiving elements of an image sensor. The electronic device can obtain preview image data by applying the obtained image signal to an image signal processor. The electronic device can display the preview image data through a camera application. The electronic device can update the displayed preview image data at predetermined time intervals.

[0117] According to one embodiment, at step 730, the electronic device may receive a user input regarding acquisition of a captured image. For example, the electronic device may receive a user input to acquire a still image and / or a moving image through a camera application. For example, the electronic device may receive a user input to acquire an image to which a function supported by the electronic device (e.g., multi-frame synthesis) is applied. The electronic device may operate in capture mode in response to the user input.

[0118] In one embodiment, an electronic device may acquire a capture frame in response to a user input. For example, the electronic device may acquire a capture frame based on operation in capture mode. For example, the electronic device may acquire a number of frames required for multi-frame synthesis and set a reference frame based on a capture image request received from a camera application.

[0119] According to one embodiment, the image signal of the capture frame may be an image signal configured with a non-Bayer pattern. For example, the electronic device may obtain the image signal of the capture frame configured with a non-Bayer pattern by using each of the output values ​​of the light-receiving elements as the pixel value of the pixel corresponding to each of the light-receiving elements. For example, the electronic device may output the image signal of the capture frame configured with a non-Bayer pattern by using the output values ​​of a predetermined number of light-receiving elements located at the center of the image sensor among the light-receiving elements as pixel values.

[0120] According to one embodiment, in operation 740, the electronic device may acquire first draft image data. The electronic device may display the acquired first draft image data so that the user can quickly check the subject captured. For example, the electronic device may display the first draft image data in at least a portion of a camera application. For example, the electronic device may display the first draft image data in at least a portion of a screen of the electronic device in response to a user input for checking the first draft image data. The electronic device may store the first draft image data in the electronic device.

[0121] For example, the electronic device can obtain draft image data using the image signal of the capture frame. For example, the electronic device can obtain draft image data using the image signal of the frame set as the reference frame. For example, since the draft image requires fast signal processing, the electronic device can obtain the draft image data by applying the image signal of the capture frame to an image signal processor. For example, the electronic device can perform an operation (e.g., binning, re-mosaicing) on ​​the first image signal of the capture frame composed of a non-Bayer pattern, thereby converting the first image signal into a second image signal composed of a Bayer pattern, and apply the second image signal to the image signal processor.

[0122] According to one embodiment, in operation 750, the electronic device may obtain first capture image data. For example, the electronic device may obtain the capture image data using an image signal of a capture frame. For example, since the capture image must be a high-quality image, the electronic device may output raw image data composed of a tetra pattern from an image sensor and perform image processing. For example, the electronic device may obtain the capture image data by performing operations such as demosaicing, white balance, contrast, and saturation value adjustment, noise removal, tone mapping, and edge enhancement on the image signal of the capture frame using a processor (e.g., 320 of FIG. 4). For example, the electronic device may obtain the capture image by performing multi-frame synthesis using a processor (e.g., 320 of FIG. 4). The processor may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). For example, an electronic device can align captured frames acquired for multi-frame synthesis using a processor (e.g., 320 of FIG. 4), generate a blending map, and synthesize them. The electronic device can obtain captured image data obtained by multi-frame synthesis by performing operations such as demosaicing, white balance, contrast, and saturation value adjustment, noise removal, tone mapping, and edge enhancement on the synthesized frames. For example, the electronic device can perform demosaicing on each of the captured frames acquired for multi-frame synthesis, and then align and synthesize them.The electronic device can obtain captured image data by performing operations such as noise removal, tone mapping, and edge enhancement on the synthesized frames.

[0123] In one embodiment, at operation 760, the electronic device may replace the first draft image data with the first captured image data. For example, the electronic device may overwrite the first draft image data stored in the electronic device with the first captured image data acquired at operation 750. For example, the electronic device may display the first draft image data displayed in at least a portion of the camera application by replacing it with the first captured image data.

[0124] According to one embodiment, a preview image frame may be acquired at operation 770. For example, the electronic device may operate in a preview mode in response to identifying that a capture frame has been acquired. For example, the electronic device may operate in a preview mode in response to identifying that first capture image data has been acquired. For example, the electronic device may operate in a preview mode in response to identifying that a number of frames required for multi-frame synthesis have been acquired. The electronic device may acquire a preview frame in response to being in the preview mode. The electronic device may acquire a preview frame until an additional request (e.g., a capture image request) is received from the camera application or until the camera application is terminated and no more requests are received. The electronic device may generate preview image data using the preview frame. The electronic device may display the preview image data. For example, the electronic device may display the preview image data in at least a portion of the camera application. The electronic device may update the displayed preview image data at predetermined time intervals. Action 770 may be analogized to the action of the electronic device described above with reference to action 720, and any duplicate content is omitted.

[0125] FIG. 9 is a flowchart illustrating an operating method of an electronic device according to an embodiment. The operations of the electronic device illustrated in FIG. 9 may be performed by a processor (e.g., processor 320 of FIG. 4) performing calculations or controlling components of the electronic device. The operations of FIG. 9 may be operations related to operations 730 to 760 of FIG. 8. The electronic device performing the operations of FIG. 9 may acquire an image signal using an image sensor including a color filter array configured with a non-Bayer pattern. The electronic device may generate a draft image and a captured image using image signals of the same frame.

[0126] According to one embodiment, in operation 810, the electronic device may receive a user input regarding acquisition of a capture image. For example, the electronic device may receive a user input to acquire a still image, a moving image, and / or an image to which a function supported by the electronic device (e.g., multi-frame synthesis) is applied through a camera application. The electronic device may operate in capture mode in response to the user input. Operation 810 may be analogized to the operation of the electronic device described above with reference to operation 730 of FIG. 8, and redundant details are omitted.

[0127] According to one embodiment, in operation 820, the electronic device may obtain a first image signal of a first frame composed of a first color pattern. For example, the electronic device may obtain a first image signal of a first frame composed of a non-Bayer color pattern (e.g., a tetra pattern, a nona pattern, a hexadecahedron pattern) output from an image sensor. For example, the electronic device may obtain an image signal (e.g., raw image data) of a frame composed of a non-Bayer pattern output by the image sensor by using output values ​​of light-receiving elements as pixel values ​​of pixels corresponding to the light-receiving elements. For example, the electronic device may obtain an image signal of a capture frame composed of a non-Bayer pattern by using each of the output values ​​of the light-receiving elements as pixel values ​​of pixels corresponding to each of the light-receiving elements. For example, the electronic device may obtain an image signal of a capture frame composed of a non-Bayer pattern by using output values ​​of a predetermined number of light-receiving elements located at the center of the image sensor among the light-receiving elements as pixel values.

[0128] According to one embodiment, in operation 830, the electronic device may convert a first image signal of a first frame into a second image signal of the first frame configured with a second color pattern. For example, the electronic device may convert a first image signal configured with a non-Bayer color pattern into a second image signal configured with a Bayer color pattern. For example, the electronic device may obtain the second image signal by performing binning or remosaicking on the first image signal. For example, the electronic device may obtain the second image signal by performing binning on the first image signal of the capture frame configured with a non-Bayer pattern by using each of the output values ​​of the light-receiving elements as the pixel value of the pixel corresponding to each of the light-receiving elements. For example, the electronic device can obtain a second image signal by performing remosaicing on a first image signal of a capture frame composed of a non-Bayer pattern by using the output values ​​of a predetermined number of light-receiving elements located at the center of the image sensor among the light-receiving elements as pixel values.

[0129] According to one embodiment, in operation 840, the electronic device may obtain first draft image data by applying the second image signal of the first frame to an image signal processor. For example, since the draft image requires fast signal processing, the electronic device may obtain the draft image data by applying the second image signal obtained in operation 830 to the image signal processor. The image signal processor may be a hardware processor designed to process an image signal configured with a Bayer pattern. According to one embodiment, obtaining the draft image data by applying the second image signal to the image signal processor may include generating the draft image data by the image signal processor using the second image signal. For example, obtaining the draft image data by applying the second image signal to the image signal processor may include controlling the image signal processor such that the processor (e.g., the processor 320 of FIG. 4) generates the draft image data by using the image signal processor.

[0130] According to one embodiment, in operation 850, the electronic device may obtain first captured image data by demosaicing the first image signal of the first frame. For example, the electronic device may obtain captured image data by performing operations such as demosaicing, white balance, contrast, and saturation value adjustment, noise removal, tone mapping, and edge enhancement on the image signal of the capture frame using a processor (e.g., 320 of FIG. 4). For example, the electronic device may obtain captured images by performing multi-frame synthesis using a processor (e.g., 320 of FIG. 4). The processor may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). Operation 850 may be analogized to and applied to the operation of the electronic device described above with reference to operation 750 of FIG. 8, and overlapping details will be omitted.

[0131] According to one embodiment, in operation 860, the electronic device may replace the first draft image data with the first captured image data. Operation 860 may be analogized to the operation of the electronic device described above with reference to operation 760 of FIG. 8, and any overlapping details are omitted.

[0132] Fig. 10 is a flowchart of an operating method of an electronic device according to one embodiment.

[0133] The operation of the electronic device illustrated in FIG. 10 may be performed by a processor (e.g., the processor (320) of FIG. 4) performing a calculation or controlling a component of the electronic device. The operations of FIG. 10 may be operations related to operations 730 and 740 of FIG. 8 and operations 820 to 840 of FIG. 9. The electronic device performing the operation of FIG. 10 may obtain an image signal using an image sensor including a color filter array configured with a non-Bayer pattern. The electronic device may generate a draft image and a captured image using an image signal of the same frame.

[0134] According to one embodiment, in operation 910, the electronic device may output a first image signal of a first frame from an image sensor. For example, the electronic device may obtain a first image signal of a first frame composed of a non-Bayer color pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern) output from the image sensor. Operation 910 may be analogized to the operation of the electronic device described above with reference to operation 820 of FIG. 9, and redundant content is omitted.

[0135] According to one embodiment, in operation 920, the electronic device may identify whether the first image signal requires re-mosaicing. For example, the electronic device may identify whether the first image signal requires re-mosaicing by identifying that the first frame is composed of a predetermined number or more of pixels. For example, the electronic device may identify that the first image signal requires re-mosaicing based on identifying that the first frame is composed of less than 12 megapixels. For example, the electronic device may identify that the first image signal does not require re-mosaicing based on identifying that the first frame is composed of 12 megapixels or more. For example, the electronic device may identify that the first image signal requires re-mosaicing based on identifying that the first frame was acquired using a crop shooting mode. For example, the electronic device may identify that the first image signal does not require re-mosaicing based on identifying that the first frame was acquired using a high-resolution shooting mode.

[0136] According to one embodiment, in operation 931, the electronic device may convert the first image signal into a second image signal by re-mosaicing. The electronic device may obtain a second image signal composed of a second color pattern (e.g., a Bayer pattern) by performing re-mosaicing on the first image signal. The number of pixels constituting the first image signal and the number of pixels constituting the second image signal may be the same. For example, the electronic device may perform re-mosaicing on the first image signal identified as requiring re-mosaicing in operation 930. For example, the electronic device may perform re-mosaicing on the first image signal based on identifying that the first frame is composed of pixels less than 12 Mp. For example, the electronic device may perform re-mosaicing on the first image signal based on identifying that the first frame was acquired by a crop shooting mode. A method by which the electronic device performs re-mosaicing is described in detail below with reference to FIG. 12.

[0137] According to one embodiment, in operation 932, the electronic device may convert the first image signal into a second image signal by binning the first image signal. The electronic device may obtain a second image signal composed of a second color pattern (e.g., a Bayer pattern) by performing binning on the first image signal. The number of pixels constituting the first image signal may be a multiple of the number of pixels constituting the second image signal. For example, the first image signal may be composed of pixels of 50 Mp, and the second image signal may be composed of pixels of 12.5 Mp. The electronic device may perform binning on the first image signal identified as not requiring re-mosaicing in operation 930. For example, the electronic device may perform binning on the first image signal based on identifying that the first image signal is composed of pixels of 50 Mp or more. For example, the electronic device may perform binning on the first image signal based on identifying that the first frame was acquired by a high-resolution shooting mode. How the electronic device performs binning is described in detail below with reference to FIG. 11.

[0138] According to one embodiment, in operation 940, the electronic device can obtain draft image data by applying a second image signal to an image signal processor. The image signal processor may be a hardware processor designed to process an image signal composed of a second color pattern (e.g., a Bayer pattern). The electronic device can quickly obtain draft image data by applying the second image signal composed of the second color pattern to the image signal processor. For example, the image signal processor can output the draft image data by demosaicing the second image signal.

[0139] According to one embodiment, in operation 950, the electronic device may identify whether upscaling of the draft image data is required. For example, the electronic device may identify whether upscaling of the draft image data is required based on identifying that the draft image data is composed of a predetermined number or more of pixels. For example, the electronic device may identify whether upscaling of the draft image data is required based on identifying that the draft image data is composed of less than 12 megapixels. For example, the electronic device may identify whether upscaling of the draft image data is required based on metadata of the draft image data. The metadata of the draft image data may include information regarding a capturing mode of an image signal used to generate the draft image data and / or an operation applied to the image signal. For example, the electronic device may identify whether upscaling of the draft image data is required based on information included in the metadata regarding whether a second image signal used to generate the draft image data was generated by a binning operation. For example, the electronic device may identify whether upscaling of the draft image data is necessary based on information that the first image signal used to generate the draft image data included in the metadata was acquired by a crop shooting mode.

[0140] According to one embodiment, in operation 951, the electronic device may perform upscaling on the draft image data. For example, the electronic device may perform upscaling on the draft image data based on identifying that the draft image data is comprised of fewer than a predetermined number of pixels. For example, the electronic device may perform upscaling on the draft image data based on identifying that the draft image data is comprised of fewer than 12 megapixels. For example, the electronic device may perform upscaling on the draft image data based on metadata of the draft image data. For example, the electronic device may perform upscaling on the draft image data based on information regarding whether a second image signal used to generate the draft image data, included in the metadata, was generated by a binning operation. For example, the electronic device may perform upscaling on the draft image data based on information regarding whether a first image signal used to generate the draft image data, included in the metadata, was acquired by a crop shooting mode. A method by which the electronic device performs upscaling is described in detail below with reference to FIG. 13.

[0141] According to one embodiment, in operation 960, the electronic device may output draft image data. For example, the electronic device may display the draft image data in at least a portion of a camera application. For example, the electronic device may display the draft image data in a gallery application.

[0142] According to one embodiment, the electronic device can replace the draft image data with captured image data corresponding to the draft image data.

[0143] FIG. 11 is a diagram conceptually illustrating conversion of a color pattern of an image signal according to one embodiment. Referring to FIG. 11, an electronic device can convert an image signal (1010) including color information of a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern) into an image signal (1020) including color information of a Bayer pattern. The electronic device can convert an image signal (1010) including color information of a non-Bayer pattern acquired through an image sensor including a color filter array composed of a non-Bayer pattern into an image signal (1020) including color information of a Bayer pattern. For example, the electronic device can generate and output an image signal (1020) including color information of a Bayer pattern from an image signal (1010) including color information of a non-Bayer pattern in response to a preview mode. For example, the electronic device can generate and output an image signal (1020) including color information of a Bayer pattern from an image signal (1010) including color information of a non-Bayer pattern to obtain a draft image.

[0144] According to one embodiment, the image sensor can generate an image signal (1020) including color information of a Bayer pattern by performing an operation on an image signal (1010) including color information of a non-Bayer pattern output from light-receiving elements corresponding to a color filter array configured with a non-Bayer pattern using a calculation unit (e.g., 417 of FIG. 5). For example, the calculation unit (e.g., 417 of FIG. 5) can generate an image signal (1020) including color information of a Bayer pattern by performing an operation of binning, summing, or remosaicking output values ​​output from the light-receiving elements.

[0145] According to one embodiment, the electronic device can generate an image signal (1020) including color information of a Bayer pattern by performing an operation on an image signal (1010) including color information of a non-Bayer pattern output from an image sensor using an image signal processor (e.g., 260 of FIG. 3). For example, the image signal processor (e.g., 260 of FIG. 3) can generate an image signal (1020) including color information of a Bayer pattern by performing an operation of binning, summing, or remosaicking on an image signal (1010) including color information of a non-Bayer pattern output from an image sensor.

[0146] According to one embodiment, the electronic device may generate an image signal (1020) including color information of a Bayer pattern by performing an operation on an image signal (1010) including color information of a non-Bayer pattern output from an image sensor using a processor (e.g., 320 of FIG. 4). For example, the processor (e.g., 320 of FIG. 4) may generate an image signal (1020) including color information of a Bayer pattern by performing an operation of binning, summing, or remosaicking on an image signal (1010) including color information of a non-Bayer pattern output from an image sensor. The processor may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU).

[0147] FIG. 12 is a diagram conceptually illustrating conversion of a color pattern of an image signal according to one embodiment.

[0148] Referring to FIG. 12, the electronic device can convert an image signal (1110) containing color information of a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern) into an image signal (1140) containing color information of a Bayer pattern.

[0149] For example, an electronic device may convert an image signal (1110) containing color information of a non-Bayer pattern into an image signal (1140) containing color information of a Bayer pattern by remosaicking the image signal (1110) using a processor (e.g., 320 of FIG. 4). Remosaicking may include an operation of changing the color order to convert the pattern and guessing the value of the changed color. The processor may include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and an image signal processor (ISP).

[0150] According to one embodiment, the electronic device can obtain a first image signal (1110) including color information of a non-Bayer pattern output from an image sensor. For example, the electronic device can obtain an image signal including color information of a non-Bayer pattern generated by an image sensor including a color filter array composed of a non-Bayer pattern using output values ​​of light-receiving elements.

[0151] According to one embodiment, the electronic device can identify an image signal including pixels including the same color information from the first image signal (1110). For example, the electronic device can identify pixels (1120a) including red color information from the first image signal (1110). The electronic device can identify pixels (1120b) including red color information from the first image signal (1110). The electronic device can identify pixels (1120c) including green color information from the first image signal (1110).

[0152] According to one embodiment, the electronic device can adjust pixel values ​​of pixels of the first image signal (1110) based on the respective positions of pixels (1120a, 1120b, 1120c) including color information. For example, the electronic device can adjust pixel values ​​of pixels of the first image signal (1110) to correspond to a Bayer pattern. For example, the electronic device can adjust pixel values ​​of pixels of the first image signal (1110) by changing the pixel value of a second pixel (1132) including green color information to a pixel value of a first pixel (1131) including red color information to correspond to a Bayer pattern. By adjusting the pixel values ​​of pixels of the first image signal (1110), the electronic device can obtain a second image signal (1140) including color information of the Bayer pattern.

[0153] According to one embodiment, the electronic device may convert a first image signal (1110) including color information of a non-Bayer pattern into a second image signal (1140) including color information of a Bayer pattern by using an artificial intelligence model learned to remosaic an image signal including color information of a non-Bayer pattern into color information of a Bayer pattern. For example, the electronic device (101) may perform a neural network operation corresponding to the artificial intelligence model by using at least one of an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), or a peripheral device. The artificial intelligence model may be generated by learning first raw image data including color information of a non-Bayer pattern, second raw image data including color information of a Bayer pattern obtained by converting the first raw image data, and / or an image (ground truth (GT) image) corresponding to the second raw image data.

[0154] FIG. 13 is a diagram conceptually illustrating upscaling of image data according to one embodiment.

[0155] Referring to FIG. 13, an electronic device can upscale first image data (1210) configured with a first resolution into second image data (1220) configured with a second resolution using a processor (e.g., 320 of FIG. 4). The processor can include an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or an image signal processor (ISP). For example, the electronic device can perform upscaling on the first image data (1210) using at least one algorithm from among a nearest neighbor algorithm, a bilinear algorithm, a bicubic algorithm, a Lanczos algorithm, a spline algorithm, and a NGU algorithm. For example, an electronic device can obtain a second image signal (1140) by upscaling a first image signal (1110) configured at a first resolution using an artificial intelligence model trained to upscale image data.

[0156] According to one embodiment, the electronic device can upscale first image data generated by binning values ​​output from a portion of light-receiving elements of the image sensor. For example, the electronic device can obtain a first image signal including color information of a 12.5-megapixel non-Bayer pattern from the image sensor through a crop shooting mode that uses only output values ​​of a portion of 50 million light-receiving elements. The electronic device can obtain a second image signal including color information of a 3.1-megapixel Bayer pattern by binning pixel values ​​constituting the first image signal including color information of a 12.5-megapixel non-Bayer pattern. The electronic device can obtain a third image signal including color information of a 12.5-megapixel Bayer pattern by performing upscaling on the second image signal including color information of a 3.1-megapixel Bayer pattern.

[0157] According to the disclosed embodiment, an electronic device can obtain a capture image without a difference in composition with respect to a draft image.

[0158] An electronic device (101) according to one embodiment may include a color filter array (413) including a first color pattern, an image sensor (230) generating an image signal using light passing through the color filter array, an image signal processor (260) processing an image signal composed of a second color pattern, a display (160, 360), a memory (330) storing instructions, and a processor (320) executing the instructions. The electronic device (101) may receive a user input regarding acquisition of a captured image. Based on the user input, the electronic device (101) may acquire a first image signal of a first frame composed of the first color pattern through the image sensor. The electronic device (101) may convert the first image signal of the first frame into a second image signal of the first frame composed of the second color pattern. The electronic device (101) may acquire first draft image data by applying the second image signal of the first frame to the image signal processor. The electronic device (101) can obtain first capture image data by demosaicing the first image signal of the first frame. The electronic device (101) can replace the first draft image data with the first capture image data.

[0159] In one embodiment, the instructions may be executed by the processor to cause the electronic device to display first draft image data in response to a user input. The instructions may be executed by the processor to cause the electronic device to display the first captured image data by replacing the first draft image data with the first captured image data.

[0160] According to one embodiment, the instructions may be executed by a processor, thereby causing the electronic device to set a first frame as a reference frame and acquire an image signal of a second frame based on a user input that sets a multi-frame synthesis function. The instructions may be executed by the processor, thereby causing the electronic device to generate first captured image data by demosaicing the image signal of the second frame with the image signal of the first frame based on the first frame.

[0161] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to convert pixel values ​​constituting the first image signal of the first frame into a second image signal of the first frame by binning them. The instructions, when executed by the processor, may cause the electronic device to obtain first draft image data by upscaling image data output from the image signal processor.

[0162] According to one embodiment, the instructions may be executed by the processor to cause the electronic device to acquire a first image signal of a first frame output from all of the photodetectors constituting the image sensor based on a user input setting a high-resolution shooting mode. The first image signal of the first frame may be composed of a first color pattern.

[0163] According to one embodiment, the instructions may be executed by the processor, thereby causing the electronic device to acquire a first image signal of a first frame output from preset light-receiving elements among light-receiving elements constituting the image sensor based on a user input that sets a zoom function to a preset magnification value. The first image signal of the first frame may be composed of a first color pattern.

[0164] According to one embodiment, the instructions, executed by the processor, may cause the electronic device to convert a first image signal of a first frame into a second image signal of the first frame by remosaicing the first image signal.

[0165] In one embodiment, the instructions may be executed by the processor to cause the electronic device to display preview image data before receiving user input regarding acquisition of a captured image. The preview image data may be acquired by applying a second image signal of a third frame composed of a second color pattern, which is a first image signal of a third frame composed of a first color pattern, to an image signal processor from an image sensor.

[0166] According to one embodiment, the instructions may be executed by the processor to cause the electronic device to display preview image data based on acquisition of first captured image data. The preview image data may be acquired by applying a second image signal of a second frame composed of a second color pattern, which is a first image signal of a second frame composed of a first color pattern, to an image signal processor from an image sensor.

[0167] According to one embodiment, a method of operating an electronic device may include an operation method of an electronic device including a color filter array including a first color pattern, an image sensor generating an image signal using light passing through the color filter array, and an image signal processor processing an image signal composed of a second color pattern. The method of operating the electronic device may include an operation of receiving a user input regarding acquisition of a capture image. The method of operating the electronic device may include an operation of acquiring a first image signal of a first frame composed of a first color pattern through the image sensor based on the user input. The method of operating the electronic device may include an operation of converting the first image signal of the first frame into a second image signal of the first frame composed of a second color pattern. The method of operating the electronic device may include an operation of acquiring first draft image data by applying the second image signal of the first frame to the image signal processor. The method of operating the electronic device may include an operation of acquiring first capture image data by demosaicing the first image signal of the first frame. A method of operating an electronic device may include replacing first draft image data with first capture image data.

[0168] According to one embodiment, the operation of the electronic device (101) acquiring the first draft image data may include an operation of displaying the first draft image data in response to a user input. The operation of the electronic device (101) replacing the first draft image data with the first capture image data may include an operation of displaying the first capture image data.

[0169] According to one embodiment, the operation of the electronic device (101) receiving a user input may include the operation of receiving an input from a user who has set a multi-frame synthesis function. The operation of the electronic device (101) obtaining a first image signal may include the operation of setting the first frame as a reference frame and the operation of obtaining an image signal of the second frame. The operation of the electronic device (101) obtaining first captured image data may include the operation of synthesizing the image signal of the second frame with the image signal of the first frame based on the first frame and the operation of demosaicing the synthesized image signal.

[0170] According to one embodiment, the operation of the electronic device (101) converting the first image signal into the second image signal may include the operation of converting the pixel values ​​constituting the first image signal of the first frame into the second image signal of the first frame by binning them. The operation of the electronic device (101) obtaining the first draft image data may include the operation of up-scaling the image data output from the image signal processor.

[0171] According to one embodiment, the operation of the electronic device (101) receiving a user input may include an operation of receiving a user input that sets a high-resolution shooting mode. The operation of the electronic device (101) acquiring a first image signal of a first frame may include an operation of acquiring a first image signal of the first frame output from all of the light-receiving elements constituting the image sensor. The first image signal of the first frame may be composed of a first color pattern.

[0172] According to one embodiment, the operation of the electronic device (101) receiving a user input may include an operation of receiving a user input that sets a zoom function to a preset magnification value. The operation of the electronic device (101) acquiring a first image signal of a first frame may include an operation of acquiring a first image signal of the first frame output from preset light-receiving elements among light-receiving elements constituting an image sensor. The first image signal of the first frame may be composed of a first color pattern.

[0173] According to one embodiment, the operation of the electronic device (101) converting the first image signal into the second image signal may include the operation of converting the first image signal of the first frame into the second image signal of the first frame by remosaicing the first image signal.

[0174] According to one embodiment, the method of operating the electronic device (101) may further include, before receiving a user input regarding acquisition of a capture image, displaying preview image data. The preview image data may be acquired by applying a second image signal of a third frame composed of a second color pattern, which is a first image signal of a third frame composed of a first color pattern, to an image signal processor from an image sensor.

[0175] According to one embodiment, the operating method of the electronic device (101) may further include an operation of displaying preview image data based on obtaining the first captured image data. The preview image data may be obtained by applying a second image signal of a second frame composed of a second color pattern, which is a first image signal of a second frame composed of a first color pattern, to an image signal processor from an image sensor.

[0176] In one embodiment, a computer-readable non-transitory recording medium may be a recording medium having recorded thereon instructions for controlling an electronic device, the electronic device including a color filter array including a first color pattern, an image sensor for generating an image signal using light passing through the color filter array, and an image signal processor for processing an image signal composed of a second color pattern. The recording medium may store instructions for receiving a user input regarding acquisition of a capture image. The recording medium may store instructions for acquiring a first image signal of a first frame composed of a first color pattern through the image sensor based on the user input. The recording medium may store instructions for converting the first image signal of the first frame into a second image signal of the first frame composed of a second color pattern. The recording medium may store instructions for acquiring first draft image data by applying the second image signal of the first frame to the image signal processor. The recording medium may store instructions for acquiring first capture image data by demosaicing the first image signal of the first frame. The recording medium can store a command to replace the first draft image data with the first capture image data.

[0177] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0178] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0179] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0180] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0181] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0182] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, A color filter array including a first color pattern; An image sensor that generates an image signal using light passing through the above color filter array; An image signal processor for processing an image signal composed of a second color pattern; display; At least one processor comprising a processing circuit; and a memory for storing instructions; The above instructions are executed by the processor, thereby causing the electronic device to: Receive user input regarding acquisition of a capture image, Based on the user input, a first image signal of a first frame composed of the first color pattern is acquired through the image sensor, Converting the first image signal of the first frame into a second image signal of the first frame composed of a second color pattern, By applying the second image signal of the first frame to the image signal processor, the first draft image data is obtained, By demosaicing the first image signal of the first frame, the first capture image data is obtained, To replace the first draft image data with the first capture image data, Electronic devices.

2. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: In response to the user's input, display the first draft image data, By replacing the first draft image data with the first capture image data, the first capture image data is displayed. Electronic devices.

3. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: Based on the user's input that sets the multi-frame synthesis function, the first frame is set as the reference frame, Acquire the image signal of the second frame, By demosaicing the image signal of the second frame by synthesizing it with the image signal of the first frame based on the first frame, the first captured image data is generated. Electronic devices.

4. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: By binning the pixel values ​​constituting the first image signal of the first frame, the image signal is converted into the second image signal of the first frame, By up-scaling the image data output from the image signal processor, the first draft image data is obtained. Electronic devices.

5. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: Based on a user's input that sets the high-resolution shooting mode, the first image signal of the first frame output from all of the light-receiving elements constituting the image sensor is acquired, The first image signal of the first frame is composed of the first color pattern, Electronic devices.

6. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: Based on a user's input that sets the zoom function to a preset magnification value, the first image signal of the first frame output from preset light-receiving elements among the light-receiving elements constituting the image sensor is acquired, The first image signal of the first frame is composed of the first color pattern, Electronic devices.

7. In paragraph 6, The above instructions are executed by the processor, thereby causing the electronic device to: By remosaicing the first image signal of the first frame, it is converted into the second image signal of the first frame. Electronic devices.

8. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: Before receiving user input regarding acquisition of the above captured image, display preview image data, The above preview image data is, A second image signal of a third frame composed of a second color pattern, which is converted from a first image signal of a third frame composed of a first color pattern, is applied from the image sensor to the image signal processor, thereby obtaining the image. Electronic devices.

9. In paragraph 1, The above instructions are executed by the processor, thereby causing the electronic device to: Based on obtaining the first captured image data, display the preview image data, The above preview image data is, A second image signal of a second frame composed of a second color pattern, which is converted from a first image signal of a second frame composed of a first color pattern, is applied from the image sensor to the image signal processor, thereby obtaining the image. Electronic devices.

10. A computer-readable, non-transitory recording medium having recorded thereon a command for controlling an electronic device including a color filter array including a first color pattern, an image sensor for generating an image signal using light passing through the color filter array, and an image signal processor for processing an image signal composed of a second color pattern, A command for receiving user input regarding acquisition of a capture image; A command for acquiring a first image signal of a first frame composed of the first color pattern through the image sensor based on the user input; A command for converting a first image signal of the first frame into a second image signal of the first frame composed of a second color pattern; A command for obtaining first draft image data by applying a second image signal of the first frame to the image signal processor; A command for obtaining first capture image data by demosaicing the first image signal of the first frame; and A command comprising: replacing the first draft image data with the first capture image data; Recording medium.

11. A method of operating an electronic device including a color filter array including a first color pattern, an image sensor generating an image signal using light passing through the color filter array, and an image signal processor processing an image signal composed of a second color pattern, An action to receive user input regarding acquisition of a capture image; An operation of acquiring a first image signal of a first frame composed of the first color pattern through the image sensor based on the user input; An operation of converting a first image signal of the first frame into a second image signal of the first frame composed of a second color pattern; An operation of obtaining first draft image data by applying a second image signal of the first frame to the image signal processor; An operation of obtaining first capture image data by demosaicing the first image signal of the first frame; and Comprising an action of replacing the first draft image data with the first captured image data, How it works.

12. In paragraph 11, The operation of obtaining the above first draft image data is as follows: In response to a user's input, comprising an action of displaying the first draft image data, The operation of replacing the first draft image data with the first capture image data is Including an operation of displaying the first captured image data, How it works.

13. In paragraph 11, The above operation of receiving user input includes an operation of receiving input from a user who has set a multi-frame synthesis function, The operation of acquiring the first image signal includes an operation of setting the first frame as a reference frame and an operation of acquiring an image signal of the second frame, The operation of obtaining the above first capture image data is An operation of synthesizing an image signal of the second frame to an image signal of the first frame based on the first frame; and Including an operation of demosaicing the synthesized image signal, How it works.

14. In paragraph 11, The operation of converting the above first image signal into the above second image signal is An operation of converting pixel values ​​constituting a first image signal of the first frame into a second image signal of the first frame by binning the pixel values, The operation of obtaining the above first draft image data is Comprising an operation of up-scaling image data output from the image signal processor, How it works.

15. In paragraph 11, The above operation of receiving user input includes an operation of receiving input from a user who has set a high-resolution shooting mode, The operation of obtaining the first image signal of the first frame is An operation of obtaining a first image signal of the first frame output from all of the light-receiving elements constituting the image sensor, The first image signal of the first frame is composed of the first color pattern, How it works.

Images