Electronic device for acquiring image through image sensor and method for operating same

Abstract

An electronic device according to various embodiments comprises: a display; a camera including an image sensor including a plurality of unit pixels; a memory storing instructions; and a processor, wherein the unit pixel has a first length in a horizontal direction and a second length smaller than the first length in a longitudinal direction, and includes a first number of sub-pixels, which is two or more, arranged in the horizontal direction and a second number of sub-pixels, which is smaller than the first number, arranged in the longitudinal direction. The electronic device may be configured, through the instructions executed by the processor, to: on the basis of the operation mode of the camera being a first operation mode, acquire a first raw image through the image sensor; acquire a first image including the first number of image pixels in the horizontal direction per unit pixel and having a first resolution through a remosaic operation on the first raw image; acquire a third image having a third resolution by performing a horizontal scale-down operation on the basis of the first image; on the basis of the operation mode of the camera being a second operation mode, acquire a second raw image through the image sensor; and acquire a second image having a second resolution lower than the third resolution by performing a longitudinal scale-down operation on the basis of the second raw image.

Description

Electronic device for acquiring an image through an image sensor and its operating method

[0001] The present disclosure relates to an electronic device for acquiring an image through an image sensor and an operating method thereof.

[0002] An electronic device can obtain image data from the output of an image sensor that converts an optical signal into an electrical signal. The image sensor can include a color filter and a photodetector. The photodetector can receive light that has passed through a color filter corresponding to a specific color and output an electrical signal corresponding to the received light. For example, a complementary metal-oxide semiconductor (CMOS) image sensor (CIS) can convert the electrical signal output from the photodetector into a digital signal through an analog-digital converter (ADC) and output the digital signal. The electronic device can obtain image data including a set of pixel values ​​determined based on the signal output from the image sensor. The image quality of the image data is affected by the number of pixels included in the image data, and devices and methods for obtaining image data with a high pixel count are being developed to obtain image data with higher image quality.

[0003] Since the photodetector receives light that has passed through a color filter of a specific color, the information detected through one photodetector corresponds to one color. The color filter plays a role in allowing the final acquired image to express color, but it can also cause the final acquired image to be expressed at a lower resolution than the total number of photodetectors arranged in the image sensor. Color reproduction and resolution can be in a trade-off relationship. The Bayer pattern has been widely used as a pattern to accurately reproduce color while reducing the loss of resolution.

[0004] A camera containing an image sensor can capture images based on the field of view (FOV) it can support. For example, a lens may be referred to as an ultra-wide angle lens, a wide angle lens, a standard lens, or a telephoto lens, in that order of their field of view. A camera equipped with each of these lenses may be referred to as an ultra-wide angle camera, a wide angle camera, a standard camera, or a telephoto camera.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0006] An electronic device according to one embodiment may include a display; a camera including an image sensor including a plurality of unit pixels; a memory storing commands; and one or more processors including a processing circuit. The unit pixel may have a first length in a horizontal direction and a second length in a vertical direction that is smaller than the first length, and a first number of sub-pixels, which are two or more, may be arranged in the horizontal direction, and a second number of sub-pixels, which are smaller than the first number, may be arranged in the vertical direction. The one or more processors may be configured to, when individually or collectively executing the instructions, acquire a first raw image through the image sensor based on the operating mode of the camera being a first operating mode, acquire a first image having a first resolution and including a first number of image pixels per unit pixel in a horizontal direction through a remosaic operation on the first raw image, and perform a horizontal scale-down operation based on the first image to acquire a third image having a third resolution. The one or more processors may be configured to, when individually or collectively executing the instructions, acquire a second raw image through the image sensor based on the operating mode of the camera being a second operating mode, and perform a vertical scale-down operation based on the second raw image to acquire a second image having a second resolution lower than the third resolution.

[0007] According to one embodiment, a method of operating an electronic device including an image sensor having a first length in a horizontal direction and a second length in a vertical direction smaller than the first length, a camera including the image sensor including a plurality of unit pixels in which a first number of sub-pixels are arranged in a horizontal direction and a second number of sub-pixels smaller than the first number are arranged in a vertical direction, and a camera including the image sensor, and one or more processors may include an operation of acquiring a first raw image through the image sensor based on an operation mode of the camera being a first operation mode. The method may include an operation of acquiring a first image including a first number of image pixels, two or more per unit pixel in a horizontal direction, and having a first resolution through a remosaic operation on the first raw image based on an operation mode of the camera being the first operation mode. The method may include an operation of performing a horizontal scale-down operation on the first image based on the operation mode of the camera being a first operation mode to obtain a third image having a third resolution. The method may include an operation of performing a vertical scale-down operation on the second raw image based on the operation mode of the camera being a second operation mode to obtain a second image having a second resolution lower than the third resolution.

[0008] A computer-readable, non-transitory recording medium according to one embodiment may record a computer program that is executed by one or more processors of an electronic device to cause the electronic device to perform the method described above.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments of the present disclosure.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments of the present disclosure.

[0011] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment of the present disclosure.

[0013] FIG. 5 is a diagram illustrating a process for an electronic device to acquire an image according to an embodiment of the present disclosure.

[0014] FIG. 6 is a diagram illustrating an example of pixels arranged in an image sensor according to one embodiment of the present disclosure.

[0015] FIG. 7 is a diagram illustrating an example of one unit pixel arranged in an image sensor according to one embodiment of the present disclosure.

[0016] FIG. 8A is a diagram conceptually illustrating a process in which an electronic device according to one embodiment of the present disclosure acquires a third image through a demosaicing operation.

[0017] FIG. 8b is a diagram conceptually illustrating a process of an electronic device acquiring a second image according to an embodiment of the present disclosure.

[0018] FIG. 9 is a flowchart illustrating a process for an electronic device to acquire an image based on an operation mode according to an embodiment of the present disclosure.

[0019] FIG. 10 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure acquiring an image based on a first operation mode.

[0020] FIG. 11 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure acquiring an image based on a second operation mode.

[0021] FIG. 12 is a flowchart illustrating a process for an electronic device to acquire an image based on a first operation mode according to an embodiment of the present disclosure.

[0022] FIG. 13 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure performing a lateral scale-up operation based on a first operation mode to obtain a fourth image.

[0023] FIG. 14 is a diagram illustrating an example of an electronic device according to an embodiment of the present disclosure obtaining an image cropped from an area remaining except for an area on one side of a short side of a third image.

[0024] FIG. 15 is a diagram showing the difference between a third image acquired using an image sensor of an electronic device according to one embodiment of the present disclosure and an image acquired using another image sensor.

[0025] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0026] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

[0027] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0028] Additionally, while terms such as first, second, etc. may be used to describe various components, the components are not limited by these terms. These terms are used to distinguish one component from another.

[0029] In this disclosure, the connection lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0030] In the present disclosure, an operating mode may refer to the manner in which an electronic device operates to acquire an image through a camera. For example, a first operating mode may indicate an operating mode for acquiring a high-resolution image through an image sensor included in a camera of the electronic device. For example, a second operating mode may indicate an operating mode for acquiring a low-resolution image through an image sensor included in a camera of the electronic device. The operating modes according to the present disclosure are not limited to the examples described above, and additional operating modes may exist.

[0031] In the present disclosure, a unit pixel may be a set of a plurality of sub-pixels arranged at a position corresponding to one micro lens in an image sensor of an electronic device. In one embodiment, the image sensor may read out one image pixel per unit pixel depending on the operating mode. For example, the electronic device may obtain the pixel value of the image pixel by binning the output of the sub-pixels included in the unit pixel.

[0032] In the present disclosure, a subpixel is a component of an image sensor corresponding to one light-receiving element of an electronic device, and may refer to the smallest unit from which the image sensor can acquire a pixel value. For example, in the case of a 2PD (2-phase detection) sensor, two subpixels may be arranged on one microlens. In one embodiment, the image sensor may read one image pixel per subpixel depending on the operating mode.

[0033] In the present disclosure, the color pattern may be a pattern according to a color channel (CFA, color filter array) in which a unit pixel or sub-pixel used to obtain a pixel value is located. For example, the color pattern may obtain a pixel value using RGB (red, green, blue) colors by using a Bayer Filter Array.

[0034] In the present disclosure, remosaic may be an operation or operation of obtaining image data corresponding to a specified pattern from a digital signal corresponding to a color pattern of an image sensor of an electronic device. For example, an electronic device may obtain image data having a Bayer pattern through a remosaic operation.

[0035] In the present disclosure, image pixel or pixel may mean the smallest unit that constitutes a digital image.

[0036] In the present disclosure, the resolution of an image may be expressed as the number of pixels contained in the image or video. For example, if an image is composed of axb pixels arranged in a rows and b columns, the resolution of the image may be indicated as axb.

[0037] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0038] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments of the present disclosure.

[0039] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0040] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0041] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0042] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0043] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0044] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0045] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0046] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0047] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0048] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0049] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0050] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0051] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0052] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0053] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0054] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0055] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0056] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0057] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0058] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0059] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0060] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0061] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0062] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another component (e.g., a second) with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., by wire), wirelessly, or through a third component. The term "module" as used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integral component or a smallest unit or part of a component that performs one or more functions.For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0063] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0064] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0065] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0066] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments.

[0067] Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0068] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0069] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0070] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0071] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0072] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment.

[0073] An electronic device (101) according to one embodiment may include at least one of a processor (e.g., a processor (120) of FIG. 1) (320), a memory (e.g., a memory (130) of FIG. 1) (330), or a camera (e.g., a camera module (180) of FIGS. 1 and 2) (380). The electronic device (101) may further include a display (e.g., a display module (160) of FIG. 1) (360). The display (360) may be replaced with an external display connected to the electronic device (101).

[0074] In one embodiment, the camera (380) may include a lens unit (381) including at least one lens for focusing light (e.g., lens assembly (210) of FIG. 2) and an image sensor (383) for converting an optical signal passing through the lens unit (381) into a digital signal (e.g., image sensor (230) of FIG. 2). The image sensor (383) according to one embodiment may include a plurality of light-receiving elements, a plurality of microlenses, and a color filter including a plurality of color channels.

[0075] In one embodiment, the memory (330) may store instructions that can be executed by one or more processors (320). The one or more processors (320) may execute the instructions stored in the memory (330) to perform operations or control components of the electronic device (101) to operate the electronic device (101). In the present disclosure, the operation of the electronic device (101) may be understood as being performed when the one or more processors (320) execute the instructions. In one embodiment, the one or more processors (320) may include at least one of a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, the one or more processors (320) may include an application processor (AP).

[0076] In one embodiment, one or more processors (320) may determine an operating mode corresponding to the resolution of an image or video to be captured using the camera (380). For example, one or more processors (320) may select either a first operating mode corresponding to high resolution or a second operating mode corresponding to low resolution. For example, high resolution may be 200 Mp (mega pixels), and low resolution may include either 50 Mp or 12.5 Mp. The types of operating modes are not limited thereto, and high resolution and / or low resolution may vary depending on the configuration of the image sensor or the electronic device. In one embodiment, the number of operating modes that one or more processors (320) may be more than this.

[0077] In one embodiment, one or more processors (320) may control the image sensor (383) to operate based on a selected operating mode. For example, the image sensor (383) may output a first raw image readout such that each output of a plurality of photodetectors corresponds to one pixel based on a first operating mode. For example, the image sensor (383) may output a first raw image having a first aspect ratio such that each subpixel has one image pixel. For example, the image sensor (383) may output a second raw image output through a binning operation on at least two or more of the subpixels (e.g., subpixels included in one unit pixel) such that the output corresponding to the unit pixel corresponds to one image pixel based on a second operating mode. For example, the image sensor (383) may output a second raw image having a second aspect ratio such that each of at least two or more subpixels has one image pixel through the binning operation. Alternatively, the processor (320) may obtain the second raw image through a binning operation that merges two or more pixels included in the first raw image. The image sensor (383) may output the first raw image and / or the second raw image based on at least one of a zoom magnification of a camera (e.g., the camera module (180) of FIG. 1, the camera (380) of FIG. 3) or a focus position of a lens unit (e.g., the lens unit (381) of FIG. 3).

[0078] In one embodiment, the first raw image and / or the second raw image output in the first operating mode and / or the second operating mode may include data prior to performing image signal processing (e.g., operations performed by the image signal processor (ISP) (260) of FIG. 2 or one or more processors (320). For example, the first raw image may include data for which re-mosaicing has not been performed on values ​​read from the light-receiving elements. That is, the first raw image and / or the second raw image may include data for which the color pattern is maintained by not changing the color order of the color pattern of the image sensor (383).

[0079] In one embodiment, one or more processors (320) can obtain an image by performing a remosaic operation on a first raw image obtained in a first operation mode. For example, the remosaic operation can be performed by an application processor (AP), an ISP, and / or an image sensor (383). For example, at least one processor (320) can obtain a first image of a Bayer pattern corresponding to a result of performing the remosaic operation on the first raw image. At least one processor (e.g., an ISP, an AP) (320) can perform image processing (e.g., demosaicing or Bayer interpolation) on image data of the Bayer pattern to obtain a color image (e.g., an RGB image, a YUV image).

[0080] In one embodiment, the image sensor (380) can acquire a second raw image having one image pixel per at least two sub-pixels through a binning operation in the first operating mode and / or the second operating mode. For example, the image sensor (380) can configure one pixel value of the second raw image based on a value obtained by binning output values ​​of two or more light-receiving elements or summing the output values. For example, the second raw image can include image data having a Bayer pattern.

[0081] In one embodiment, the electronic device (101) may perform a scale-up operation to scale up an image and / or a scale-down operation to scale down an image via the image sensor (383) and / or the processor (320). For example, the electronic device (101) may perform a scale-down operation in the horizontal direction based on a scale-down ratio on a first image or a color image acquired from the first image to obtain a third image. The third image may include an image corresponding to a high-resolution mode. For example, the electronic device (101) may perform a scale-down operation in the vertical direction based on a scale-down ratio on a second raw image or a color image acquired from the second raw image to obtain a second image. The second image may include an image corresponding to a low-resolution mode. For example, the electronic device (101) may perform a scale-up operation in the horizontal direction based on a scale-up ratio on the second raw image to obtain a fourth image. The fourth image may include an image corresponding to a high-resolution mode. The scale-up and / or scale-down operations may be performed by at least one of the image sensor (383) or the processor (320), but are not limited thereto.

[0082] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3) according to one embodiment.

[0083] In one embodiment, an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3) may include a micro lens array (MLA) (411), a color filter (413), a light receiving unit (415), and a computation unit (417).

[0084] In one embodiment, the microlens array (411) may be configured such that a light bundle (421) that passes through a lens unit (e.g., a lens unit (381) of FIG. 3) and forms an image on an image sensor (e.g., an image sensor (230) of FIG. 2, an image sensor (383) of FIG. 3) is focused on a light-receiving element of a light-receiving element (415). The light bundles (423) that pass through the microlens array (411) may have wavelengths outside a band corresponding to a specific color blocked at least partially as they pass through a color filter (413). The light bundles (425) that pass through the color filter (413) may be detected by a light-receiving element (e.g., a photodiode) of the light-receiving element (415). The light-receiving element (415) may include a light-receiving element that generates a charge when receiving light and converts it into an electrical signal, and a circuit that selectively reads out the charge of the light-receiving element. A circuit for digitizing the signal read from the light receiving unit (415) or reducing noise may be placed between the light receiving unit (415) and the calculation unit (417).

[0085] In one embodiment, the calculation unit (417) may perform an operation to process electrical data (427) output from the light receiving unit (415). The calculation unit (417) may output acquired data based on the calculation result. The output of the calculation unit (417) may be an output of an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3). In one embodiment, the calculation unit (417) may perform an operation to calibrate the read data as an operation to process the electrical data (427). For example, the operation performed by the calculation unit (417) may include at least one of an operation to reduce deviation between pixels due to optical asymmetry or relative positions of sensors, an operation to reduce noise generated in an analog signal, an operation to remove defects, an operation to perform re-mosaic, or an operation according to a specific application (e.g., a proximity sensor function, a timing adjustment function, an HDR tone mapping function). The sensor output (429) output from the operation unit (417) can be input to at least one processor (e.g., an application processor (AP)) through an interface. In one embodiment, the operation unit (417) can perform an operation of performing a scale-up and / or scale-down operation on the read data as an operation for processing the electrical data (427). For example, an operation of adjusting the number of pixels in the vertical and / or horizontal direction can be performed by performing a scale-down and / or scale-up operation in the vertical and / or horizontal direction on the raw image.

[0086] FIG. 5 is a diagram illustrating a process (500) for obtaining an image by an electronic device (101) according to one embodiment of the present disclosure.

[0087] In one embodiment, the electronic device (101) can capture a subject through a camera (380). For example, light reflected from the subject can pass through a lens unit (e.g., lens unit (381) of FIG. 3) and be imaged on an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3).

[0088] In one embodiment, referring to FIG. 5, a subject can be captured through a sensor pixel array (510) of an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3) of an electronic device (101). For example, as illustrated in FIG. 5, when light reflected from a subject is focused on the image sensor (383) of the electronic device (101), the image sensor (383) can output a raw image including pixel values ​​that detected the light.

[0089] In one embodiment, referring to FIG. 5, the image sensor (383) may include a plurality of unit pixels. For example, the image sensor (383) may include unit pixels having a first length in the horizontal direction and a second length in the vertical direction that is smaller than the first length. For example, referring to the sensor array (510) of the image sensor of FIG. 5, the image sensor (383) may include 10 unit pixels arranged in the horizontal direction and 10 unit pixels arranged in the vertical direction. However, the number of unit pixels of the present disclosure is not limited to the number of unit pixels illustrated in FIG. 5. For example, the unit pixel of the image sensor may have a ratio of a horizontal length to a vertical length of 1.4 to 1. However, the aspect ratio of the unit pixel is not limited thereto.

[0090] In one embodiment, the electronic device (101) may acquire a first raw image (515) through the image sensor (383) based on the first operating mode. For example, as illustrated in FIG. 5, the electronic device (101) may acquire the first raw image (515) from an image formed on the image sensor (383) through the image sensor (383). The process by which the electronic device (101) acquires the first raw image (515) will be described later.

[0091] In one embodiment, the first raw image (515) may include image pixels having an aspect ratio of 1 to 1. For example, the image pixels of the first raw image (515) illustrated in FIG. 5 may have a ratio of a horizontal length to a vertical length of 1 to 1.

[0092] In one embodiment, the first raw image (515) may be an image including one image pixel per sub-pixel included in a unit pixel of the image sensor (383). For example, the first raw image (515) illustrated in FIG. 5 may be an image including one image pixel per sub-pixel from an image sensor (383) including two sub-pixels in a unit pixel. For example, the image sensor (383) may include two sub-pixels in the horizontal direction per unit pixel, and the first raw image (515) illustrated in FIG. 5 may include one image pixel per sub-pixel. An aspect ratio of the entire first raw image (515) may be determined according to the sensor pixel array (510) and the arrangement of sub-pixels included in the unit pixel. For example, when unit pixels are arranged in an M x N array in the sensor pixel array (510) and sub-pixels are arranged in a 2 x 1 array within the unit pixel, the aspect ratio of the width and length of the first raw image (515) may be 2M:N. For example, the first raw image (515) illustrated in FIG. 5 may include 20 image pixels arranged horizontally and 10 image pixels arranged vertically. As illustrated in FIG. 5, the first raw image (515) may include 200 image pixels. However, the number of image pixels of the first raw image (515) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5.

[0093] In one embodiment, the electronic device (101) may obtain a first image (520) through a remosaic operation on the first raw image (515). For example, referring to FIG. 5 , the electronic device (101) may obtain a first image (520) through a remosaic operation on the first raw image (515) obtained based on the first operation mode. A further description of the remosaic operation by which the electronic device (101) obtains the first image (520) will be provided below.

[0094] In one embodiment, the first image (520) may include image pixels having an aspect ratio of 1 to 1 and may have a first resolution. For example, the image pixels of the first image (520) illustrated in FIG. 5 may have a ratio of a horizontal length to a vertical length of 1 to 1. For example, the first image (520) illustrated in FIG. 5 may include 20 image pixels arranged horizontally and 10 image pixels arranged vertically. As illustrated in FIG. 5, the first image (520) may have a first resolution of 20 x 10. As illustrated in FIG. 5, the first image (520) may include 200 image pixels. However, the number of image pixels of the first image (520) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5.

[0095] In one embodiment, the electronic device (101) may perform a demosaic operation and / or a scale-down operation in the horizontal direction on the first image (520) based on the first operating mode to obtain a third image (530) which is a color image. For example, as illustrated in FIG. 5, one or more processors (320) may perform a scale-down operation based on a first scale-down ratio in the horizontal direction on the first image (520) to obtain the third image (530). For example, as illustrated in FIG. 5, one or more processors (320) may perform a scale-down operation in the horizontal direction on the first image (520) or a color image obtained from the first image (520) to obtain the third image (530). A scale-down operation may not be performed in the longitudinal direction for the first image (520) or the color image acquired from the first image (520). Referring to FIG. 5, when a plurality of sub-pixels arranged horizontally are included within a unit pixel, the number of image pixels in the horizontal direction may increase as one image pixel value is acquired per sub-pixel through the re-mosaic operation. Accordingly, at least one of the first raw image (515) or the first image (520) may be acquired as an image that is expanded horizontally compared to the image formed on the sensor array (510). The electronic device may acquire a third image (530) having an aspect ratio corresponding to the aspect ratio of the image formed on the sensor array (510) by horizontally scaling down the first raw image (515) or the first image (520) that is expanded horizontally. At least one processor (320) may apply different scaling ratios in the longitudinal direction and the transverse direction. For example, at least one processor (320) may perform a scale-down operation such that the transverse scaling degree is greater than the longitudinal scaling degree.For example, as illustrated in FIG. 5, one or more processors (320) can obtain a third image (530) including 14 image pixels arranged horizontally and 10 image pixels arranged vertically through a scale-down operation. For example, as illustrated in FIG. 5, one or more processors (320) can obtain the third image (530) based on a first scale-down ratio of 0.7. However, the number of image pixels of the third image (530) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5, and the first scale-down ratio is also exemplary and not limited thereto. In one embodiment, the operation illustrated in FIG. 5 may be performed by a processor (e.g., the processor (320) of FIG. 3, the ISP (260) of FIG. 2, the AP) and / or an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3).

[0096] In one embodiment, the third image (530) acquired by the electronic device (101) may have a third resolution. The third resolution may be higher than a resolution corresponding to the number of unit pixels of the image sensor (383). For example, the third image (530) illustrated in FIG. 5 may include 14 image pixels arranged horizontally and 10 image pixels arranged vertically. As illustrated in FIG. 5, the third image (530) may have a third resolution of 14 x 10. For example, as illustrated in FIG. 5, the third resolution of the third image (530) may be higher than a resolution corresponding to the number of unit pixels of the image sensor. In FIG. 5, the image sensor (383) has an array of unit pixels of 10 x 10, and the third image (530) may have a third resolution of 14 x 10, which is higher than the resolution corresponding to the number of unit pixels of the image sensor (383).

[0097] In one embodiment, the electronic device (101) can obtain a second raw image (540) through the image sensor (383). For example, the electronic device (101) can obtain the second raw image (540) from an image formed on the image sensor (383). For example, the image sensor (383) can output a second raw image (540) having one image pixel per at least two sub-pixels through a binning operation and / or a sum operation on at least two or more of the sub-pixels of the image sensor (383). A process by which the electronic device (101) obtains the second raw image (540) will be described later.

[0098] In one embodiment, the second raw image (540) may include image pixels having an aspect ratio of 1 to 1. For example, the image pixels of the second raw image (540) illustrated in FIG. 5 may have a ratio of a horizontal length to a vertical length of 1 to 1.

[0099] In one embodiment, the second raw image (540) may be an image including one image pixel per unit pixel. For example, the second raw image (540) illustrated in FIG. 5 may be an image including one image pixel per unit pixel. For example, the second raw image (540) illustrated in FIG. 5 may include 10 image pixels arranged horizontally and 10 image pixels arranged vertically. An aspect ratio for the entire second raw image (540) may be determined according to the sensor pixel array (510) and the arrangement of the unit pixels. For example, when the unit pixels are arranged in an M x N arrangement in the sensor pixel array (510), the aspect ratio of the width and length of the second raw image (540) may be M:N. As illustrated in FIG. 5, the second raw image (540) may include 100 image pixels. However, the number of image pixels of the second raw image (540) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5.

[0100] In one embodiment, the electronic device (101) may perform a horizontal scale-up operation on the second raw image (540) or a color image acquired from the second raw image (540) based on a specified condition being satisfied when the first operation mode is set, or may perform a scale-up operation on the second image (560) to acquire a fourth image (550). Referring to FIG. 5, when the aspect ratio of a unit pixel is not 1:1 and the horizontal length of the unit pixel is longer than the vertical length, the second raw image (540) that can be acquired through the image sensor (383) may be acquired as an image that is horizontally shrunk or vertically expanded compared to an image formed on the sensor array (510). The electronic device can acquire a fourth image (550) having an aspect ratio corresponding to the aspect ratio of the image formed on the sensor array (510) by horizontally scaling up the horizontally reduced second raw image (540). For example, when an illuminance value acquired through an illuminance sensor (e.g., the sensor module (176) of FIG. 1) or an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) is below a threshold value, the electronic device (101) can acquire the fourth image (550) in the first operation mode. For example, as illustrated in FIG. 5, one or more processors (320) can perform a demosaic operation and / or a horizontally scale-up operation based on a first scale-up ratio on the second raw image (540) to acquire the fourth image (550). For example, as illustrated in FIG. 5, one or more processors (320) can obtain a fourth image (550) by performing a horizontal scale-up operation on the second raw image (540) or a color image obtained from the second raw image (540).A scale-up operation may not be performed in the longitudinal direction on the second raw image (540) or a color image acquired from the second raw image (540). At least one processor (320) may apply different scale-up ratios in the longitudinal direction and the horizontal direction. For example, at least one processor (320) may perform the scale-up operation such that the horizontal scale-up degree is greater than the vertical scale-up degree. For example, as illustrated in FIG. 5, one or more processors (320) may acquire a fourth image (550) including 14 image pixels arranged in the horizontal direction and 10 image pixels arranged in the vertical direction through the scale-up operation. For example, as illustrated in FIG. 5, one or more processors (320) may acquire the fourth image (550) based on a first scale-up ratio of 1.4. However, the number of image pixels of the fourth image (550) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5, and the first scale-up ratio is also exemplary and not limited thereto. In one embodiment, the operation illustrated in FIG. 5 may be performed by a processor (e.g., processor (320) of FIG. 3, ISP (260) of FIG. 2, AP) and / or an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3).

[0101] In one embodiment, the fourth image (550) acquired by the electronic device (101) may have a fourth resolution. The fourth resolution may be higher than a resolution corresponding to the number of unit pixels of the image sensor (383). For example, referring to FIG. 5, the fourth image (550) may include 14 image pixels arranged horizontally and 10 image pixels arranged vertically. Referring to FIG. 5, the fourth image (550) may have a third resolution of 14 x 10. For example, referring to FIG. 5, the fourth resolution of the fourth image (550) may be higher than a resolution corresponding to the number of unit pixels of the image sensor (383). Referring to FIG. 5, the resolution corresponding to the number of unit pixels of the image sensor (383) may be 10 x 10, and the fourth resolution of the fourth image (550) may be 14 x 10.

[0102] In one embodiment, the electronic device (101) may perform a demosaic operation and / or a scale-down operation based on a second scale-down ratio in the longitudinal direction on the second raw image (540) based on the second operating mode to obtain a second image (560) which is a color image. For example, referring to FIG. 5, one or more processors (320) may obtain the second image (560) by performing a scale-down operation in the longitudinal direction on the second raw image (540) or a color image obtained from the second raw image (540). A scale-down operation may not be performed in the transverse direction on the second raw image (540) or a color image obtained from the second raw image (540). Referring to FIG. 5, when the aspect ratio of a unit pixel is not 1:1 and the horizontal length of the unit pixel is longer than the vertical length, the second raw image (540) that can be acquired through the image sensor (383) can be acquired as an image that is horizontally shrunk or vertically expanded compared to the image formed on the sensor array (510). The electronic device can acquire a second image (560) having an aspect ratio corresponding to the aspect ratio of the image formed on the sensor array (510) by vertically scaling down the horizontally shrunk second raw image (540). At least one processor (320) may apply different ratios for the vertical scaling down and the horizontal scaling down. For example, at least one processor (320) may perform a scale-down operation such that the degree of vertical scaling down is greater than the degree of horizontal scaling down. For example, referring to FIG. 5, one or more processors (320) can obtain a second image (560) including 10 image pixels arranged horizontally and 7 image pixels arranged vertically through a scale-down operation. For example, as illustrated in FIG. 5, one or more processors (320) can obtain 0.A second image (560) can be acquired based on the second scale-down ratio of 7. However, the number of image pixels of the second image (560) of the present disclosure is not limited to the number of image pixels illustrated in FIG. 5, and the second scale-down ratio is also exemplary and not limited thereto. In one embodiment, the operation illustrated in FIG. 5 may be performed by a processor (e.g., the processor (320) of FIG. 3, the ISP (260) of FIG. 2, the AP) and / or an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3).

[0103] In one embodiment, the second image (560) acquired by the electronic device (101) may have a second resolution. The second resolution may be lower than a resolution corresponding to the number of unit pixels of the image sensor (383), a third resolution, and / or a fourth resolution. For example, the second image (560) illustrated in FIG. 5 may include 10 image pixels arranged horizontally and 7 image pixels arranged vertically. Referring to FIG. 5, the second image (560) may have a second resolution of 10 x 7. For example, referring to FIG. 5, the second resolution of the second image (560) may be lower than a resolution corresponding to the number of unit pixels of the image sensor (383). For example, referring to FIG. 5, the resolution corresponding to the number of unit pixels of the image sensor (383) may be 10 x 10, and the second resolution of the second image (560) may be 10 x 7. Referring to FIG. 5, the second resolution of the second image (560) may be lower than the third resolution of the third image (530) and / or the fourth resolution of the fourth image (550).

[0104] FIG. 6 is a drawing (600) illustrating an example of pixels arranged in an image sensor (383) according to one embodiment of the present disclosure.

[0105] In one embodiment, an area where an image is formed by the optical structure of the camera module (180) may be referred to as an optical format. A portion where a circular image is formed within the optical format may be referred to as an image circle (601). The electronic device (101) may include an image sensor (610) having a size corresponding to the image circle (601). For example, the image circle (601) may include a portion where a circular image is formed. For example, the size of the optical format may be expressed as the diameter of the image circle (601). For example, the size of the image sensor (610) may be determined according to the optical format. For example, the image sensor (610) may be configured to have a size that can be included within the optical format.

[0106] In one embodiment, the image sensor (610) of the electronic device (101) may include a plurality of unit pixels (620). For example, the image sensor (610) may include unit pixels (620) having a first length (621) in the longitudinal direction and a second length (622) in the transverse direction. The second length (622) may be smaller than the first length (621). For example, M (611) unit pixels (620) may be arranged in the transverse direction of the image sensor (610), and N (612) unit pixels (620) may be arranged in the longitudinal direction of the image sensor (610). For example, the transverse length of the image sensor (610) may correspond to a value obtained by multiplying the first length (621) by M (611). The longitudinal length of the image sensor (610) may correspond to the product of the second length (622) and N (612).

[0107] For example, the length ratio of the first length (621) and the second length (622) of the unit pixel (620) of the image sensor (610) may be 1.4 to 1. 4024 unit pixels (620) may be arranged in the horizontal direction of the image sensor (610). 4225 unit pixels (620) may be arranged in the vertical direction of the image sensor (610). The length ratio of the vertical and horizontal directions of the image sensor (610) may be approximately 3:4 (4225 x 1:4024 x 1.4).

[0108] FIG. 7 is a drawing illustrating an example of one unit pixel (620) arranged in an image sensor (610) according to one embodiment of the present disclosure.

[0109] In one embodiment, the sub-pixels (720) included in the image sensor may be arranged such that two sub-pixels (720) arranged in a 2 x 1 array correspond to one micro-lens (710). For example, referring to FIG. 7, two sub-pixels (720) may be arranged in a 2 x 1 array to correspond to one micro-lens (710). For example, the sub-pixels (720) may be arranged in a 2PD (2 phase detection) structure. The number and structure of the sub-pixels (720) included in the unit pixel (620) are not limited thereto. The image sensor may have a pattern in which unit pixels (620) having the structure illustrated in FIG. 7 are repeatedly arranged.

[0110] In one embodiment, a unit pixel (620) may include a plurality of sub-pixels (720). In one embodiment, a unit pixel (620) may have a first number of sub-pixels (720) arranged horizontally and a second number of sub-pixels (720) smaller than the first number arranged vertically. For example, referring to FIG. 7, two sub-pixels (720) may be arranged horizontally and one sub-pixel (720) smaller than two may be arranged vertically. For example, the structure of the unit pixel (620) of FIG. 7 may correspond to a 2PD structure. However, the structure of the plurality of sub-pixels (720) included in the unit pixel (620) is not limited to that illustrated in FIG. 7. For example, the image sensor may include a structure in which three or more sub-pixels are arranged within the unit pixel (620).

[0111] For example, referring to FIG. 7, the horizontal and vertical length ratio of a unit pixel (620) may correspond to a ratio of 1.4 to 1. Referring to FIG. 7, the horizontal and vertical length ratio of a sub-pixel (720) may correspond to a ratio of 0.7 to 1.

[0112] According to one embodiment, one light-receiving element may be arranged in one sub-pixel (720). For example, referring to FIG. 7, light-receiving elements may be arranged to correspond to the sub-pixel (720), and two light-receiving elements may be arranged in one unit pixel (620). For example, two light-receiving elements may be arranged in one micro-lens (710). The structure of the light-receiving element according to the present disclosure will be described later with reference to FIG. 7.

[0113] FIG. 8A is a diagram conceptually illustrating a process in which an electronic device (101) according to one embodiment of the present disclosure acquires a third image (840) through a demosaic operation.

[0114] In one embodiment, each of the four micro lenses (710, 821, 822, 823) arranged adjacent to each other in a 2 x 2 array included in the image sensor (383) may be arranged to correspond to a different color channel of the color filter. The light-receiving elements included in the image sensor (383) may be arranged such that two light-receiving elements arranged in a 2 x 1 array correspond to one micro lens. For example, referring to 810 of FIG. 8A, the first light-receiving element (811) and the second light-receiving element (812) may be arranged to correspond to one micro lens (710). Accordingly, two light-receiving elements arranged adjacent to each other in a 2 x 1 array may be arranged to correspond to one color channel.

[0115] In one embodiment, the image sensor (383) may be configured with a pattern repeatedly arranged as illustrated in FIG. 8A. The pattern illustrated in FIG. 8A is merely an example, and the pattern of the image sensor (383) may be replaced with another pattern that can operate by changing the resolution.

[0116] In one embodiment, while the electronic device (101) operates based on the first operating mode, the image sensor may acquire a first raw image. For example, referring to 810 of FIG. 8A, the image sensor (383) may acquire a first raw image (820) having the pattern illustrated in FIG. 8A. While the electronic device (101) operates based on the first operating mode, the first raw image (820) may be acquired when the illuminance value is greater than or equal to a threshold. For example, the number of image pixels of the first raw image (820) may be equal to the number of sub-pixels of the image sensor.

[0117] In one embodiment, the electronic device (101) may perform a remosaic operation that converts the color order on the first raw image (820) to obtain an image having a Bayer pattern. In the present disclosure, the remosaic operation may be performed on an image and / or image obtained from the image sensor (383) by one or more processors (e.g., the processor (120) of FIG. 1 , the image signal processor (260) of FIG. 2 , and the processor (320) of FIG. 3 ), but it may also be understood that the image sensor (383) may output a result of performing the remosaic operation. In FIG. 8A , the first raw image (820) may have a resolution that is twice as high in the horizontal direction compared to the number of unit pixels.

[0118] In one embodiment, the electronic device (101) may perform remosaic on the first raw image (820) to obtain the first image (830). For example, the first raw image (820) illustrated in FIG. 8A may be obtained by performing a remosaic operation. For example, a unit pixel of an image sensor may include a first number of sub-pixels in a horizontal direction. The unit pixel may include a second number of image pixels smaller than the first number in a vertical direction. The image sensor may have M unit pixels arranged in a horizontal direction and N unit pixels arranged in a vertical direction. The first image obtained through the remosaic operation may include a number of image pixels equal to M times the first number in a horizontal direction and a number of image pixels equal to N times the second number in a vertical direction. For example, in an image sensor (383) of a 2PD structure including two sub-pixels in the horizontal direction and one sub-pixel in the vertical direction per unit pixel, the first image (830) may include 2 x M image pixels in the horizontal direction and N image pixels in the vertical direction. The first image (830) may have a resolution that is twice as high in the horizontal direction compared to the number of unit pixels.

[0119] In one embodiment, the electronic device (101) may perform a horizontal scale-down operation on the first image (830) or the color image acquired from the first image (830) to acquire a third image (840). For example, the electronic device (101) may perform a horizontal scale-down operation on the first image (830) or the color image acquired from the first image (830) illustrated in FIG. 8A to acquire a third image (840) having a reduced number of horizontal image pixels. The electronic device (101) may acquire a third image (840) having a high resolution compared to the resolution corresponding to the number of unit pixels of the image sensor (383) through a scale-down operation such as the process illustrated in FIG. 8A. The electronic device (101) can obtain a third image (840) with improved resolution and clarity compared to an image obtained with one image pixel per unit pixel by utilizing a scale-down operation rather than a scale-up operation that reduces the clarity of the image.

[0120] FIG. 8b is a diagram conceptually illustrating a process in which an electronic device (101) according to one embodiment of the present disclosure acquires a second image (860).

[0121] In one embodiment, each of the four micro lenses (710, 821, 822, 823) arranged adjacent to each other in a 2 x 2 array included in the image sensor (383) may be arranged to correspond to a different color channel of the color filter. The light-receiving elements included in the image sensor (383) may be arranged such that two light-receiving elements arranged in a 2 x 1 array correspond to one micro lens. For example, referring to 810 of FIG. 8B, the first light-receiving element (811) and the second light-receiving element (812) may be arranged to correspond to one micro lens (710). Accordingly, two light-receiving elements arranged adjacent to each other in a 2 x 1 array may be arranged to correspond to one color channel.

[0122] In one embodiment, the image sensor (383) may be configured with a pattern repeatedly arranged as illustrated in FIG. 8B. The pattern illustrated in FIG. 8B illustrates one example, and the pattern of the image sensor (383) may be replaced with another pattern that can operate by changing the resolution.

[0123] According to one embodiment, the electronic device (101) may output the second raw image (850) to have one image pixel per at least two sub-pixels by performing a binning operation on at least two of the sub-pixels. For example, referring to 810 of FIG. 8B, the image sensor (383) may acquire the second raw image (850) having the pattern illustrated in FIG. 8B. For example, the electronic device (101) may output the second raw image (850) while operating based on the second operating mode. The electronic device (101) may also output the second raw image (850) when the illuminance value is less than a threshold while operating based on the first operating mode. For example, referring to FIG. 8B, the electronic device (101) may output the second raw image (850) to have one image pixel per two sub-pixels included in a unit pixel. For example, the number of image pixels of the second raw image may correspond to the number of unit pixels included in the image sensor (383). The second raw image (850) may have a Bayer pattern.

[0124] According to one embodiment, the electronic device (101) can obtain a second image (860) by performing a longitudinal scale-down operation on the second raw image (850) or the color image obtained from the second raw image (850). For example, by performing a longitudinal scale-down operation on the second raw image (850) or the color image obtained from the second raw image (850) illustrated in FIG. 8B, the electronic device (101) can obtain a second image (860) with a reduced number of image pixels in the longitudinal direction. The electronic device (101) can obtain a second image (860) with a lower resolution than the third image (840) through a scale-down operation such as the process illustrated in FIG. 8B. The electronic device (101) can obtain a second image (860) corresponding to the original shape of the subject from a raw image obtained through unit pixels having different aspect ratios through the scale-down operation.

[0125] FIG. 9 is a flowchart (900) illustrating a process for obtaining an image based on an operation mode by an electronic device (101) according to an embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of obtaining an image based on a first operation mode by an electronic device (101) according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of obtaining an image based on a second operation mode by an electronic device (101) according to an embodiment of the present disclosure. FIG. 14 is a diagram illustrating an example of obtaining an image obtained by cropping an area except for an area on the short side of a third image by an electronic device (101) according to an embodiment of the present disclosure. FIG. 9 may be described with reference to FIGS. 10, 11, and 14.

[0126] In operation 910, the electronic device (101) can check the operating mode of the camera.

[0127] In one embodiment, the electronic device (101) can determine an operating mode for operating an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) based on a user input. For example, the electronic device (101) can set the operating mode to a first operating mode based on a user input for operating a camera (e.g., the camera (380) of FIG. 3) of the electronic device (101) in a high-quality shooting mode. For example, the electronic device (101) can set the operating mode to a second operating mode based on a user input for operating a camera (e.g., the camera (380) of FIG. 3) of the electronic device (101) in a low-quality shooting mode. For example, the electronic device (101) can determine an operating mode that operates based on a user input for selecting a shooting mode of the camera.

[0128] In one embodiment, the electronic device (101) can determine an operation mode for operating an image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) based on information collected through a sensor (e.g., the sensor module (176) of FIG. 1). For example, the electronic device (101) can collect an illuminance value through an illuminance sensor (e.g., the sensor module (176) of FIG. 1) and control the image sensor based on a first operation mode when the collected illuminance value is equal to or greater than a first threshold. The electronic device (101) can collect an illuminance value through an illuminance sensor (e.g., the sensor module (176) of FIG. 1) and control the image sensor based on a second operation mode when the collected illuminance value is less than the first threshold. The electronic device (101) can determine an operation mode to be operated based on the collected illuminance value.

[0129] In one embodiment, the electronic device (101) can determine an operating mode in which it operates according to conditions set in software in the electronic device (101). For example, the electronic device (101) can operate in a second operating mode when the resource usage of the electronic device (101) is equal to or greater than a second threshold, and can operate in a first operating mode when the resource usage is less than the second threshold. The electronic device (101) can determine an operating mode in which it operates according to preset conditions.

[0130] In one embodiment, the electronic device (101) may perform operation 920 based on being in a first operation mode. The electronic device (101) may perform operation 930 based on being in a second operation mode. Operation 920 may include operations 921 to 923. Operation 930 may include operations 931 and 932.

[0131] The operating modes of the camera according to the present disclosure are not limited to the first operating mode and the second operating mode. For example, the operating mode of the camera of the electronic device (101) may include an ultra-wide-angle mode that captures images with a wider angle of view than the first angle. For example, the operating mode of the camera of the electronic device (101) may include a wide-angle mode that captures images with a narrower angle of view than the first angle and wider angle of view than the second angle.

[0132] In operation 921, the electronic device (101) can acquire a first raw image in a first operation mode.

[0133] In one embodiment, the electronic device (101) can acquire a first raw image through an image formed on the image sensor (383) in the first operating mode. For example, referring to FIG. 10, the image sensor (383) can acquire an image (1001) of an apple formed on the image sensor (383). The image formed on the image sensor (383) can correspond to the actual left-right ratio of the apple, which is the subject.

[0134] Referring to FIG. 10, the image sensor (383) can output a first raw image (1002) in a first operation mode.

[0135] In one embodiment, the aspect ratio of a unit pixel of the image sensor (383) and the aspect ratio of an image pixel of the first raw image may be different. For example, referring to FIG. 10, a unit pixel of the image sensor (383) may have a first length in the horizontal direction and a second length in the vertical direction that is smaller than the first length. Referring to FIG. 10, the horizontal length and the vertical length of the image pixel of the first raw image (1002) may be the same. For example, the vertical and horizontal length ratio of the unit pixel of the image sensor (383) may be 1.4 to 1, and the vertical and horizontal length ratio of the image pixel of the first raw image may be 1 to 1.

[0136] In one embodiment, the image sensor (383) may output a first raw image having a first aspect ratio to have one image pixel per subpixel based on the first operating mode. For example, referring to FIG. 10, in the image sensor (383), ten unit pixels may be arranged vertically and ten may be arranged horizontally. In the image sensor (383), the unit pixel may have a 2PD structure in which two subpixels are arranged horizontally and one subpixel is arranged vertically. For example, referring to FIG. 10, the image sensor (383) of the 2PD structure may output a first raw image (1002) to have one image pixel per subpixel. In the first raw image (1002), twenty image pixels may be arranged horizontally and ten may be arranged vertically. For example, in an image sensor (383) in which 4024 unit pixels are arranged in the horizontal direction and 4225 unit pixels are arranged in the vertical direction, the image sensor (383) can output a first raw image in which 8048 image pixels are arranged in the horizontal direction and 4225 image pixels are arranged in the vertical direction.

[0137] In operation 922, the electronic device (101) can obtain a first image through a remosaic operation.

[0138] In one embodiment, the electronic device (101) can obtain the first image through a re-mosaic operation on the first raw image. For example, referring to FIG. 10, the electronic device (101) can obtain the first image (1003) through a re-mosaic operation on the first raw image (1002). For example, through the re-mosaic operation, the electronic device (101) can obtain the first image (830) having the Bayer pattern illustrated in FIG. 8A from the first raw image (820) having the pattern illustrated in FIG. 8A. For example, the resolution of the first image (1003) converted through the re-mosaic operation can be twice as high as the resolution corresponding to the number of unit pixels of the image sensor (383).

[0139] In one embodiment, the resolution of the first image may be the product of the number of unit pixels arranged horizontally of the image sensor (383), the number of sub-pixels arranged horizontally per unit pixel, and the number of unit pixels arranged vertically of the image sensor (383). For example, in an image sensor (383) of a 2PD structure, when M unit pixels are arranged horizontally and N unit pixels are arranged vertically, a resolution corresponding to the number of unit pixels of the image sensor (383) may be M x N, and the resolution of the first image may be (M x 2) x N. For example, in an image sensor (383) having a resolution of 4024 x 4225, the resolution of the first image may be 8048 x 4225. For example, in an image sensor (383) of a 4PD structure, when M unit pixels are arranged in the horizontal direction and N unit pixels are arranged in the vertical direction, the resolution corresponding to the number of unit pixels of the image sensor (383) may be M x N, and the resolution of the first image may be (M x 2) x (N x 2).

[0140] In one embodiment, the re-mosaic operation may be performed by the image sensor (383) and / or the processor (320). For example, the re-mosaic operation may be performed by a computational unit (e.g., computational unit (417) of FIG. 4) of the image sensor (383). For example, the re-mosaic operation may be performed by an image signal processor (e.g., image signal processor (260) of FIG. 2). For example, the re-mosaic operation may be performed by an application processor.

[0141] In operation 923, the electronic device (101) may perform a lateral scale-down operation to obtain a third image.

[0142] In one embodiment, the electronic device (101) can obtain a third image (1004) by performing a scale-down operation based on a first scale-down ratio in the horizontal direction on the first image (1003) or a color image obtained from the first image (1003). The electronic device (101) may not perform the scale-down operation in the vertical direction. For example, referring to FIG. 10, the electronic device (101) can obtain a third image (1004) by performing a scale-down operation in the horizontal direction on the first image (1003) or a color image obtained from the first image (1003). For example, for 20 image pixels arranged horizontally in the first image (1003), the electronic device (101) can perform a scale-down operation in the horizontal direction to obtain a third image (1004) having 14 image pixels arranged horizontally.

[0143] In one embodiment, the first scale-down ratio may be determined based on a first horizontal length of the unit pixel, a second vertical length of the unit pixel, and the number of sub-pixels arranged in the unit pixel. For example, in an image sensor (383) of a 2PD structure including unit pixels having an aspect ratio of 1 to 1.4, the first scale-down ratio may be 1.4 / (1 x 2), which is obtained by dividing the first length by the second length and the first number. The first scale-down ratio may be determined so that the third image (1004) has a specified aspect ratio. For example, the first scale-down ratio may be determined so that the aspect ratio of the third image (1004) is 3:4.

[0144] In one embodiment, the scale-down operation in operation 923 may be performed in the image sensor (383) and / or the processor (320). For example, the scale-down operation may be performed in the computational unit (e.g., the computational unit (417) of FIG. 4) of the image sensor (383). For example, the scale-down operation may be performed in the image signal processor (e.g., the image signal processor (260) of FIG. 2). For example, the scale-down operation may be performed in the application processor.

[0145] For example, if the ratio of the first horizontal length of the unit pixel of the image sensor (383) of the electronic device (101) to the second vertical length is 1.4 to 1, 4024 unit pixels may be arranged horizontally and 4225 unit pixels may be arranged vertically on the image sensor (383) of the 2PD structure. According to the first scale-down ratio of 1.4 / (1 x 2) described above, the third image may include 5634 (4024 x 2 x 0.7) image pixels arranged horizontally and 4225 image pixels arranged vertically.

[0146] In one embodiment, the electronic device (101) may obtain an image obtained by cropping an area except for a short side area of ​​the third image obtained by performing a horizontal scale-down operation. For example, the electronic device (101) may obtain a cropped image using a third image obtained by using a camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, the camera (380) of FIG. 3) including an ultra-wide-angle lens (e.g., the lens unit (381) of FIG. 3). Distortion may occur in image data obtained by the electronic device (101) through the edge area of ​​the image sensor. The electronic device (101) may obtain an image obtained by cropping a portion of the third image. For example, referring to FIG. 14, the electronic device (101) can obtain an image (1420) that is a cropped image of a second area (1412) excluding a first area (1411) on the short side of a third image (1410). For example, a cropped image (1420) of a third image (1410) obtained using an ultra-wide-angle lens can be an image corresponding to an image obtained using a wide-angle lens having a field of view narrower than 120 degrees.

[0147] In one embodiment, the electronic device (101) may obtain a third image corresponding to the first angle of view image by using a lens (e.g., the lens unit (381) of FIG. 3) that supports a first angle of view (e.g., an ultra-wide angle) (or a first field of view (FoV)). For example, the electronic device (101) may obtain a first raw image based on the first angle of view mode. The electronic device (101) may obtain the first image through a re-mosaic operation based on the first raw image. The electronic device (101) may obtain the third image based on the first image or a color image obtained from the first image. The third image obtained using the lens that supports the first angle of view may include an image corresponding to the first angle of view image. The ultra-wide angle image may include an image captured based on an angle of view greater than or equal to the first angle.

[0148] In one embodiment, the electronic device (101) may acquire an image corresponding to a second angle of view image with distortion removed or reduced by using a lens supporting a first angle of view (e.g., the lens unit (381) of FIG. 3). The second angle of view (or second field of view (FoV)) may be narrower than the first angle of view. For example, the electronic device (101) may acquire a first raw image. The electronic device may acquire a third image based on the first raw image. The electronic device (101) may acquire an image cropped to a second area excluding a first area of ​​a short side of the acquired third image based on a second angle of view mode (e.g., a wide angle mode). Referring to FIG. 14, an image (1420) obtained by cropping the third image (1410) acquired using a lens supporting the first angle of view may include an image corresponding to the second angle of view image. An image (1420) obtained by cropping a portion of the third image (1410) may include an image in which a distorted area of ​​a portion of both sides of the short side (1411) of the third image is excluded.

[0149] For example, a camera (380) including a lens (e.g., a lens unit (381) of FIG. 3) supporting a first angle of view (e.g., an ultra-wide angle) (or a first field of view (FoV)) may include an image sensor (383) having an aspect ratio of 1:2. A length ratio of a first length to a second length of a unit pixel of the image sensor (383) may be 1.2:0.8. The unit pixels of the image sensor (383) may be arranged so that a ratio of the number of unit pixels arranged in a vertical direction to the number of unit pixels arranged in a horizontal direction is 9:12. The electronic device (101) may obtain a first raw image (1002) using the image sensor (383) and obtain a first image (1003) through a re-mosaic operation. Referring to FIG. 14, the electronic device (101) can obtain a third image (1410) having an aspect ratio of 9:16 using the first image (1003) or a color image obtained from the first image (1003). The third image (1410) can include an image corresponding to the first angle image. Referring to FIG. 14, the electronic device (101) can obtain an image (1420) obtained by cropping an area (1412) other than an area (1411) on the short side of the third image (1410). The area (1411) on the short side of the third image (1410) can be determined according to the aspect ratio of the cropped image (1420). Referring to FIG. 14, a short side region (1411) of a third image (1410) may be determined as an area that can match an aspect ratio of 3:4 (9:12) according to an aspect ratio of 3:4 of a cropped image (1420). An image (1420) obtained by cropping the remaining area (1412) excluding the short side region (1411) of the third image (1410) may include an image corresponding to a second angle image.

[0150] In operation 931, the electronic device (101) can acquire a second raw image based on the second operation mode.

[0151] In one embodiment, the electronic device (101) may acquire a second raw image based on the second operating mode. For example, referring to FIG. 11 , the image sensor (383) may acquire an image (1101) of an apple imaged on the image sensor (383). Referring to FIG. 11 , the image sensor (383) may output a second raw image (1102) in the second operating mode.

[0152] In one embodiment, the aspect ratio of a unit pixel of the image sensor (383) and the aspect ratio of an image pixel of the second raw image may be different. For example, referring to FIG. 11, a unit pixel of the image sensor (383) may have a first length in the horizontal direction and a second length in the vertical direction that is smaller than the first length. Referring to FIG. 11, the horizontal length and the vertical length of an image pixel of the second raw image (1102) may be the same. For example, the vertical and horizontal length ratio of a unit pixel of the image sensor (383) may be 1.4 to 1, and the vertical and horizontal length ratio of an image pixel of the second raw image (1102) may be 1 to 1.

[0153] In one embodiment, the image sensor (383) may output a second raw image having a second aspect ratio, based on the second operating mode, by performing a binning operation on at least two or more of the sub-pixels to have one image pixel per at least two sub-pixels. For example, referring to FIG. 11, in the image sensor (383), ten unit pixels may be arranged vertically and ten may be arranged horizontally. In the image sensor (383), the unit pixel may have a 2PD structure in which two sub-pixels are arranged horizontally and one sub-pixel is arranged vertically. For example, referring to FIG. 11, the image sensor (383) of the 2PD structure may output a second raw image (1102) to have one image pixel per two sub-pixels. In the second raw image (1102), ten image pixels may be arranged vertically and ten may be arranged horizontally. For example, in an image sensor (383) in which 4024 unit pixels are arranged in the horizontal direction and 4225 unit pixels are arranged in the vertical direction, the image sensor (383) can output a second raw image in which 4024 image pixels are arranged in the horizontal direction and 4225 image pixels are arranged in the vertical direction.

[0154] In operation 932, the electronic device (101) can perform a longitudinal scale-down operation to obtain a second image.

[0155] In one embodiment, the electronic device (101) can obtain the second image (1103) by performing a scale-down operation based on a second scale-down ratio on the second raw image (1102) or a color image obtained from the second raw image (1102) in the longitudinal direction. The electronic device (101) can obtain the second image by not performing the scale-down operation in the transverse direction. For example, referring to FIG. 11, the electronic device (101) can obtain the second image (1103) by performing a scale-down operation in the longitudinal direction on the second raw image (1102) or a color image obtained from the second raw image (1102) and not performing the scale-down operation in the transverse direction. For example, for 10 image pixels arranged longitudinally in the second raw image (1102), the electronic device (101) can perform a scale-down operation in the longitudinal direction to obtain a second image (1103) having 7 image pixels arranged longitudinally.

[0156] In one embodiment, the second scale-down ratio may be determined based on a first horizontal length of the unit pixel and a second vertical length of the unit pixel. For example, in an image sensor (383) of a 2PD structure including a unit pixel having an aspect ratio of 1 to 1.4, the second scale-down ratio may be 1 / 1.4, which is the second length divided by the first length. The first scale-down ratio may be determined so that the second image (1103) has a specified aspect ratio. For example, the first scale-down ratio may be determined so that the aspect ratio of the second image (1103) is 3:4.

[0157] In one embodiment, the scale-down operation in operation 932 may be performed in the image sensor (383) and / or the processor (320). For example, the scale-down operation may be performed in the computational unit (e.g., the computational unit (417) of FIG. 4) of the image sensor (383). For example, the scale-down operation may be performed in the image signal processor (e.g., the image signal processor (260) of FIG. 2). For example, the scale-down operation may be performed in the application processor.

[0158] For example, if the ratio of the first horizontal length of a unit pixel of an image sensor (383) of an electronic device (101) to the second vertical length is 1.4 to 1, 4024 unit pixels may be arranged horizontally and 4225 unit pixels may be arranged vertically on the image sensor (383) of a 2PD structure. A second image calculated by the electronic device (101) according to the second scale-down ratio of 1 / 1.4 described above may have 4024 image pixels arranged horizontally and 3018 (4225 x (1 / 1.4)) image pixels arranged vertically.

[0159] FIG. 12 is a flowchart (1200) illustrating a process for acquiring an image based on a first operation mode by an electronic device (101) according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of acquiring a fourth image by performing a horizontal scale-up operation based on the first operation mode by an electronic device (101) according to an embodiment of the present disclosure. FIG. 12 may be described with reference to FIG. 13.

[0160] In operation 1201, the electronic device (101) can determine whether the illuminance value is greater than or equal to a first threshold.

[0161] In one embodiment, the electronic device (101) can determine whether the illuminance value is greater than or equal to a threshold based on an illuminance value around the electronic device (101) collected through a sensor (e.g., the sensor module (176) of FIG. 1). For example, the electronic device (101) can obtain an illuminance value around the electronic device (101) using an illuminance sensor (e.g., the sensor module (176) of FIG. 1). For example, the electronic device (101) can determine whether the obtained illuminance value is greater than or equal to a first threshold. For example, the first threshold may be a value set by a user and / or a value set in the electronic device (101).

[0162] In one embodiment, the electronic device (101) may perform operation 920 when the illuminance value around the electronic device (101) is greater than or equal to a threshold in the first operation mode. In one embodiment, the electronic device (101) may perform operation 1210 when the illuminance value around the electronic device (101) is less than or equal to a threshold in the first operation mode.

[0163] Action 920 may include actions 921, 922, and 923. In one embodiment, action 920 of FIG. 12 may correspond to action 920 of FIG. 9.

[0164] In operation 1211, the electronic device (101) can acquire a second raw image when the illuminance value is less than the first threshold in the first operation mode.

[0165] In one embodiment, the electronic device (101) can acquire a second raw image through an image formed on the image sensor (383) when the illuminance value is less than a first threshold in the first operating mode. For example, referring to FIG. 13, the image sensor (383) can acquire an image (1301) formed on the image sensor (383).

[0166] 1301 of FIG. 13 may correspond to 1001 of FIG. 10 and 1101 of FIG. 11.

[0167] Referring to FIG. 13, the image sensor (383) can output a second raw image (1302) when the illuminance value is less than the first threshold in the first operation mode.

[0168] In one embodiment, the aspect ratio of a unit pixel of the image sensor (383) and the aspect ratio of an image pixel of the second raw image may be different. For example, referring to FIG. 13, a unit pixel of the image sensor (383) may have a first length in the horizontal direction and a second length in the vertical direction that is smaller than the first length. Referring to FIG. 13, the horizontal length and the vertical length of an image pixel of the second raw image (1302) may be the same. For example, the vertical and horizontal length ratio of a unit pixel of the image sensor (383) may be 1.4 to 1, and the vertical and horizontal length ratio of an image pixel of the second raw image (1302) may be 1 to 1.

[0169] In one embodiment, the image sensor (383) may output a second raw image having a second aspect ratio, based on the first operating mode and the illuminance value being less than a first threshold, by performing a binning operation on at least two of the sub-pixels to have one image pixel per at least two sub-pixels. For example, referring to FIG. 13, in the image sensor (383), ten unit pixels may be arranged vertically and ten may be arranged horizontally. In the image sensor (383), the unit pixel may have a 2PD structure in which two sub-pixels are arranged horizontally and one sub-pixel is arranged vertically. For example, the image sensor (383) of the 2PD structure may output a second raw image (1302) to have one image pixel per two sub-pixels. In the second raw image, ten image pixels may be arranged vertically and ten may be arranged horizontally. For example, in an image sensor (383) in which 4024 unit pixels are arranged in the horizontal direction and 4225 unit pixels are arranged in the vertical direction, the image sensor (383) can output a second raw image in which 4024 image pixels are arranged in the horizontal direction and 4225 image pixels are arranged in the vertical direction.

[0170] In operation 1212, the electronic device (101) may perform a lateral scale-up operation to obtain a fourth image.

[0171] In one embodiment, the electronic device (101) can obtain a fourth image by performing a scale-up operation based on a first scale-up ratio in the horizontal direction on the second raw image or a color image obtained from the second raw image. The electronic device (101) can obtain the fourth image by not performing the scale-up operation in the vertical direction. For example, referring to FIG. 13, the electronic device (101) can obtain the fourth image (1303) by performing a scale-up operation in the horizontal direction on the second raw image (1302) and not performing the scale-up operation in the vertical direction. For example, for 10 image pixels arranged horizontally of the second raw image (1302), the electronic device (101) can perform a scale-up operation in the horizontal direction to obtain a fourth image (1303) having 14 image pixels arranged horizontally.

[0172] For example, referring to FIG. 13, the electronic device (101) can obtain a fourth image (1303) by performing a scale-up operation in the horizontal direction on the second raw image (1302) or the color image obtained from the second raw image (1302). For example, for 10 image pixels arranged horizontally of the second raw image (1302) or the color image obtained from the second raw image (1302), the electronic device (101) can perform a scale-up operation in the horizontal direction to obtain a fourth image (1303) having 14 image pixels arranged horizontally.

[0173] In one embodiment, the first scale-up ratio may be determined based on a first horizontal length of a unit pixel and a second vertical length of the unit pixel. For example, in an image sensor (383) of a 2PD structure including a unit pixel having an aspect ratio of 1 to 1.4, the first scale-up ratio may be 1.4 / 1, which is the first length divided by the second length. The first scale-up ratio may be determined such that the fourth image (1303) has a specified aspect ratio. For example, the first scale-up ratio may be determined such that the aspect ratio of the fourth image (1303) is 3:4.

[0174] In one embodiment, the scale-up operation in operation 1212 may be performed by the image sensor (383) and / or the processor (320). For example, the scale-up operation may be performed by a computational unit (e.g., computational unit (417) of FIG. 4) of the image sensor (383). For example, the scale-up operation may be performed by an image signal processor (e.g., image signal processor (260) of FIG. 2). For example, the scale-up operation may be performed by an application processor.

[0175] For example, if the ratio of the first horizontal length of a unit pixel of an image sensor (383) of an electronic device (101) to the second vertical length is 1.4 to 1, 4024 unit pixels may be arranged horizontally and 4225 unit pixels may be arranged vertically on the image sensor (383) of a 2PD structure. A fourth image calculated by the electronic device (101) according to the first scale-up ratio of 1.4 / 1 described above may include 5634 (4024 x (1.4 / 1)) image pixels arranged horizontally and 4225 image pixels arranged vertically.

[0176] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may include a display (e.g., display module (160) of FIG. 1, display (360) of FIG. 3), a camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) including an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3) including a plurality of unit pixels), a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3) for storing instructions); and one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3) including a processing circuit. The unit pixel may have a first length in the horizontal direction and a second length in the vertical direction that is smaller than the first length, and may include a first number of sub-pixels, two or more, arranged horizontally, and a second number of sub-pixels smaller than the first number arranged vertically. The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may, when executing the instructions, acquire a first raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3)) based on the operation mode of the camera (the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being a first operation mode, acquire a first image having a first resolution and including a first number of image pixels per unit pixel in a horizontal direction through a remosaic operation on the first raw image, and acquire a third image having a third resolution by performing a horizontal scale-down operation based on the first image.The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, when executing the instructions, acquire a second raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3)) based on the operating mode of the camera (the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being a second operating mode, and perform a longitudinal scale-down operation based on the second raw image to acquire a second image having a second resolution lower than the third resolution.

[0177] In one embodiment, the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) may output the first raw image having a first aspect ratio to have one image pixel per sub-pixel based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the first operating mode. The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured, upon executing the instructions, to output the second raw image having a second aspect ratio to have one image pixel per at least two or more sub-pixels through a binning operation on at least two or more of the sub-pixels based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the second operating mode.

[0178] In one embodiment, the plurality of unit pixels may be arranged such that M unit pixels are arranged in a horizontal direction and N unit pixels are arranged in a vertical direction. The first resolution may be (the M x the first number) x the N.

[0179] In one embodiment, the one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3) may be configured not to perform a longitudinal scale-down operation based on the first image based on the operating mode of the camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) being the first operating mode when executing the instructions.

[0180] In one embodiment, the one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to perform the lateral scale-down operation based on a first scale-down ratio, based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being a first operating mode when executing the instructions. The first scale-down ratio may be a value obtained by dividing the first length by the second length and the first number, and the third resolution may be (the M x the first number x the first scale-down ratio) x the N.

[0181] In one embodiment, the one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3) may be configured not to perform a horizontal scale down operation based on the second raw image based on the operating mode of the camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) being the second operating mode when executing the instructions.

[0182] In one embodiment, the one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to perform the longitudinal scale down operation based on a second scale-down ratio, based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being a second operating mode when executing the instructions. The second scale-down ratio may be a value obtained by dividing the second length by the first length, and the second resolution may be M x (the second scale-down ratio x the N).

[0183] In one embodiment, the one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3) may be configured to operate in the first operation mode based on a user input to operate the camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) in a high-quality camera shooting mode when executing the instructions, and to operate in the second operation mode based on a user input to operate the camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) in a low-quality camera shooting mode).

[0184] In one embodiment, the one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3) may be configured to, when executing the instructions, identify an illuminance value through the at least one sensor, operate in the first operation mode when the illuminance value is equal to or greater than a first threshold, and operate in the second operation mode when the illuminance value is less than the first threshold.

[0185] In one embodiment, the one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, when executing the instructions, identify an illuminance value through the at least one sensor, and, if the illuminance value is less than a second threshold based on the operation mode of the camera (380) being the first operation mode: acquire the second raw image (540) through the image sensor (383), and perform a horizontal scale-up operation based on the second raw image (540) based on a first scale-up ratio to acquire a fourth image (550) having a fourth resolution higher than the second resolution. The first scale-up ratio may be a value obtained by dividing the first length by the second length, and the fourth resolution may be (the M x the first scale-up ratio) x the N.

[0186] In one embodiment, the one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, upon executing the instructions, acquire the first raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) based on the fact that the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) is a third operating mode for capturing a wide-angle image. The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, upon executing the instructions, acquire the first image through a re-mosaic operation on the first raw image based on the fact that the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) is a third operating mode for capturing a wide-angle image. The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, when executing the instructions, acquire a third image through the horizontal scale-down operation based on the first image, based on the fact that the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) is a third operating mode for capturing a wide-angle image. The one or more processors (e.g., the processor (120) of FIG. 1, the processor (320) of FIG. 3) may be configured to, when executing the instructions, acquire an image cropped to a second area excluding a first area of ​​a short side of the third image, based on the fact that the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) is a third operating mode for capturing a wide-angle image.

[0187] In one embodiment, an image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3) including a plurality of unit pixels having a first length in a horizontal direction and a second length smaller than the first length in a vertical direction, and including a first number of sub-pixels arranged horizontally and a second number of sub-pixels smaller than the first number arranged vertically), a camera (e.g., camera module (180) of FIGS. 1 and 2, camera (380) of FIG. 3) including the image sensor (e.g., image sensor (230) of FIG. 2, image sensor (383) of FIG. 3)) and an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) including one or more processors (e.g., processor (120) of FIG. 1, processor (320) of FIG. 3)) is provided, wherein the method of operating the camera (e.g., camera module (180) of FIGS. 1 and 2, image sensor (383) of FIG. 3) comprises: The method may include: obtaining a first raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) based on the operation mode of the camera (380) being a first operation mode; obtaining a first image having a first resolution including a first number of image pixels, which are two or more per unit pixel in the horizontal direction, through a remosaic operation on the first raw image; obtaining a third image having a third resolution by performing a horizontal scale-down operation based on the first image; and obtaining a second raw image through the image sensor (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) based on the operation mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being a second operation mode; obtaining a second raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3)).and may include an operation of performing a longitudinal scale-down operation based on the second raw image to obtain a second image having a second resolution lower than the third resolution.;

[0188] The method may include an operation in which a longitudinal scale-down operation is not performed based on the first image, based on the operation mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the first operation mode.

[0189] The method may include an operation of performing the horizontal scale-down operation based on a first scale-down ratio, based on the operation mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the first operation mode, wherein the first scale-down ratio is a value obtained by dividing the first length by the second length and the first number, and the third resolution may be (the M x the first number x the first scale-down ratio) x the N.

[0190] The method may include an operation of not performing a horizontal scale down operation based on the second raw image, based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the second operating mode.

[0191] The method may include an operation of performing the longitudinal scale down operation based on a second scale-down ratio based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the second operating mode, wherein the first scale-down ratio is a value obtained by dividing the first length by the second length and the first number, and the third resolution may be (the M x the first number x the first scale-down ratio) x the N.

[0192] The method may include an operation of not performing a horizontal scale down operation based on the second raw image, based on the operating mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the second operating mode.

[0193] The method may include an operation of performing the longitudinal scale down operation based on a second scale-down ratio, based on the operation mode of the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) being the second operation mode, wherein the second scale-down ratio is a value obtained by dividing the second length by the first length, and the second resolution may be M x (the second scale-down ratio x the N).

[0194] The method may include an operation of operating the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) in the first operation mode based on a user input of operating the camera in the high-quality camera shooting mode; and an operation of operating the camera (e.g., the camera module (180) of FIGS. 1 and 2, the camera (380) of FIG. 3) in the second operation mode based on a user input of operating the camera in the low-quality camera shooting mode.

[0195] The method may include an operation in which the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) further includes at least one sensor, and an operation in which an illuminance value is identified through the at least one sensor; an operation in which the illuminance value is greater than or equal to a first threshold value; and an operation in which the illuminance value is less than or equal to the first threshold value, and an operation in which the illuminance value is less than or equal to the first threshold value, are identified through the at least one sensor.

[0196] The method may include: an operation of acquiring the first raw image through the image sensor (e.g., the image sensor (230) of FIG. 2, the image sensor (383) of FIG. 3) based on the operation mode of the camera (e.g., the camera module (180) of FIG. 1 and FIG. 2, the camera (380) of FIG. 3) being a third operation mode for capturing a wide-angle image; an operation of acquiring the first image through a remosaic operation on the first raw image; an operation of acquiring the third image through a horizontal scale-down operation based on the first image; and an operation of acquiring an image cropped to a second area excluding a first area on a short side of the third image.

[0197] According to various embodiments, an electronic device and / or a method of operating the same may be provided that can obtain high-resolution and high-quality images by increasing the light-receiving capacity of an image sensor by using an image sensor including unit pixels with different aspect ratios to enable the use of a large area of ​​pixels in an image sensor of the same size.

[0198] According to various embodiments, when performing scale-up or / and scale-down operations on images and / or videos, an electronic device and / or an operating method thereof with high memory usage efficiency may be provided by performing operations in one of the longitudinal and transverse directions.

[0199] FIG. 15 is a diagram showing the difference between a third image acquired using an image sensor (383) of an electronic device (101) according to one embodiment of the present disclosure and an image acquired using another image sensor.

[0200] According to one embodiment, the image sensor (383) of the electronic device (101) may include an image sensor having an aspect ratio of 3 to 4. For example, referring to FIG. 15, a first horizontal length and a second vertical length of a unit pixel of the image sensor (383) may have a length ratio of 1.4 to 1. 4024 unit pixels included in the image sensor (383) may be arranged in the vertical direction and 4225 may be arranged in the horizontal direction. The optical format of the image sensor (383) may be 1 / 2.55". The third image acquired based on the electronic device (101) being in the first operation mode may be an image having an aspect ratio of 3 to 4, and the resolution of the third image may be 5634 x 4225. The resolution-priority image sensor according to FIG. 15 has the same resolution as the third image acquired from the electronic device (101) according to the present disclosure, but may have a lower light-receiving capability because the unit pixel size is smaller. The pixel size-priority image sensor according to FIG. 15 has a similar light-receiving capability when comparing the area of ​​the unit pixel with that of the electronic device (101) according to the present disclosure, but may have a lower resolution of the acquired image. In addition, the memory utilization may be reduced by performing scale-up and / or scale-down operations in the vertical and horizontal directions. Therefore, the electronic device (101) according to the present disclosure may efficiently use memory compared to other image sensors of the same size. An electronic device and / or an operating method thereof can be provided that can utilize a large pixel area and thus have excellent light-receiving capability.

[0201] According to various embodiments, an electronic device (101) having an image sensor having the same optical format and having a higher light-receiving capability and / or an operating method thereof may be provided, including a larger number of pixels. Referring to FIG. 15, when comparing an image sensor having a pixel aspect ratio of 1:1.4 according to the present disclosure and an image sensor including pixels having an aspect ratio of 1.2:1.2 that prioritizes pixel size, the image sensor (383) of the electronic device (101) according to the present disclosure may include a larger number of pixels.

[0202] According to one embodiment, in an electronic device (101) using an ultra-wide-angle lens, the electronic device (101) can obtain and use an image obtained by cropping an area except for a short side area of ​​an acquired image. By obtaining the cropped image, the electronic device (101) can achieve the effect of increasing the efficiency of memory and processor usage in the process of correcting distortion of the outer part of an image or video due to the use of an ultra-wide-angle lens.

[0203] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0204] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0205] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

[0206] In the present disclosure, the functions or operations performed by the electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The functions or operations of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include circuitry for performing calculations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operations of the electronic device described above.

[0207] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0208] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0209] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0210] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0211] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0212] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0213] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, or including a combination of two or more (including a and b, including b and c, including a and c, or including all of a, b, and c).

[0214] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In electronic devices, display; A camera comprising an image sensor comprising a plurality of unit pixels; memory for storing commands; and comprising one or more processors including processing circuitry; The above unit pixel is: It has a first length in the transverse direction and a second length in the longitudinal direction that is smaller than the first length, comprising a first number of sub-pixels, two or more, arranged horizontally, and a second number of sub-pixels smaller than the first number arranged vertically, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Based on the operation mode of the above camera being the first operation mode: Acquire a first raw image through the image sensor, Obtaining a first image having a first resolution including a first number of image pixels in the horizontal direction per unit pixel through a remosaic operation on the first raw image, A third image having a third resolution is obtained by performing a horizontal scale-down operation based on the first image, Based on the fact that the operating mode of the above camera is the second operating mode: Acquire a second raw image through the image sensor, Performing a longitudinal scale-down operation based on the second raw image to obtain a second image having a second resolution lower than the third resolution, Electronic devices.

2. In paragraph 1, The above image sensor, Based on the operation mode of the camera being the first operation mode, outputting the first raw image of the first aspect ratio so as to have one image pixel per subpixel, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Based on the operation mode of the camera being the second operation mode, the second raw image having a second aspect ratio is output so as to have one image pixel per at least two sub-pixels through a binning operation for at least two or more of the sub-pixels. Electronic devices.

3. In paragraph 2, The above multiple unit pixels are arranged such that M unit pixels are arranged in the horizontal direction and N unit pixels are arranged in the vertical direction. The above first resolution is, (the above M x the above first number) x the above N, Electronic devices.

4. In paragraph 3, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Based on the fact that the operation mode of the above camera is the first operation mode, not performing a longitudinal scale-down operation based on the first image, Electronic devices.

5. In paragraph 4, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Perform the horizontal scale-down operation based on the first scale-down ratio, The above first scale-down ratio is a value obtained by dividing the first length by the second length and the first number, The third resolution is (the M x the first number x the first scale-down ratio) x the N, Electronic devices.

6. In paragraph 3, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Based on the fact that the operation mode of the above camera is the second operation mode, a horizontal scale-down operation is not performed based on the second raw image. Electronic devices.

7. In paragraph 6, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Perform the longitudinal scale-down operation based on the second scale-down ratio, The above second scale-down ratio is a value obtained by dividing the second length by the first length, The second resolution is, M x (the second scale-down ratio x the N), Electronic devices.

8. In paragraph 1, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Operating the camera in the first operation mode based on a user input that operates the camera in a high-definition camera shooting mode, To operate the camera in the second operating mode based on a user input that operates the camera in a low-quality camera shooting mode. Electronic devices.

9. In paragraph 1, comprising at least one more sensor, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Identifying the illuminance value through at least one sensor, When the illuminance value is greater than or equal to the first threshold, the device operates in the first operation mode, and when the illuminance value is less than or equal to the first threshold, the device operates in the second operation mode. Electronic devices.

10. In paragraph 3, comprising at least one more sensor, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Identifying the illuminance value through at least one sensor, Based on the operation mode of the camera being the first operation mode, if the illuminance value is less than the second threshold: Acquiring the second raw image through the image sensor, A horizontal scale-up operation is performed based on the second raw image and the first scale-up ratio to obtain a fourth image having a fourth resolution higher than the second resolution, The above first scale-up ratio is a value obtained by dividing the first length by the second length, The fourth resolution is (the M x the first scale-up ratio) x the N. Electronic devices.

11. In paragraph 1, The above instructions are individually or collectively executed by the one or more processors, so that the electronic device: Based on the fact that the operating mode of the above camera is the third operating mode for taking wide-angle images: Acquiring the first raw image through the image sensor, Obtaining the first image through a remosaic operation on the first raw image, Obtaining the third image by performing the horizontal scale-down operation based on the first image, Obtaining an image cropped to a second area excluding the first area on the short side of the third image, Electronic devices.

12. A method for operating an electronic device including a camera including an image sensor and one or more processors, Based on the operation mode of the above camera being the first operation mode: An operation of acquiring a first raw image through the image sensor; An operation of obtaining a first image having a first resolution including a first number of image pixels, which is two or more in the horizontal direction per unit pixel, through a remosaic operation on the first raw image; An operation of performing a horizontal scale-down operation based on the first image to obtain a third image having a third resolution; Based on the fact that the operating mode of the above camera is the second operating mode: An operation of acquiring a second raw image through the image sensor; and An operation of performing a longitudinal scale-down operation based on the second raw image to obtain a second image having a second resolution lower than the third resolution, The above image sensor includes a plurality of unit pixels, The unit pixel has a first length in the horizontal direction and a second length smaller than the first length in the vertical direction, and includes a first number of sub-pixels arranged horizontally and a second number of sub-pixels smaller than the first number arranged vertically. method.

13. In paragraph 12, The above image sensor, Based on the operation mode of the camera being the first operation mode, outputting the first raw image of the first aspect ratio so as to have one image pixel per subpixel, Based on the operation mode of the camera being the second operation mode, the second raw image having a second aspect ratio is configured to be output so as to have one image pixel per at least two sub-pixels through a binning operation for at least two of the sub-pixels. method.

14. In paragraph 13, The above multiple unit pixels are arranged such that M unit pixels are arranged in the horizontal direction and N unit pixels are arranged in the vertical direction. The above first resolution is, (the above M x the above first number) x the above N, method.

15. In paragraph 14, An operation that includes not performing a longitudinal scale-down operation based on the first image, based on the operation mode of the camera being the first operation mode. method.

Images