Electronic device, method, and non-transitory computer-readable recording medium for capturing image having wide viewing angle

Abstract

This electronic device may comprise an anamorphic lens for compressing light in a second direction from among a first direction and the second direction perpendicular to the first direction, and may comprise an image sensor configured to acquire a raw image on the basis of the compressed light through the anamorphic lens. A plurality of unit pixels in the image sensor may be arranged in the first direction corresponding to a long axis direction of the image sensor and the second direction corresponding to a short axis direction of the image sensor. Each of the plurality of unit pixels may include at least two sub-pixels arranged in the short axis direction. The electronic device may comprise an image processing module configured to up-scale the raw image only in the short axis direction by using the at least two sub-pixels arranged in the short axis direction in each of the plurality of unit pixels without up-scaling the raw image in the long axis direction from among the long axis direction and the short axis direction.

Description

Electronic device, method, and non-transitory computer-readable recording medium for capturing images having a wide field of view

[0001] The following descriptions relate to an electronic device, a method, and a non-transitory computer-readable recording medium for capturing images having a wide field of view.

[0002] Image sensors included in electronic devices such as smartphones acquire images of external objects by converting light from the external object into electrical signals.

[0003] Because the field of view of the lens and the aspect ratio of the image sensor differ, electronic devices can crop the image captured by the image sensor to obtain an image with the desired aspect ratio. However, because the cropping process results in discarded areas of the image captured by the image sensor, users may find it difficult to capture images with the desired field of view.

[0004] An electronic device is disclosed. The electronic device may include an anamorphic lens that compresses light in a first direction and a second direction perpendicular to the first direction. The electronic device may include an image sensor configured to acquire a raw image based on the compressed light through the anamorphic lens. A plurality of unit pixels in the image sensor may be arranged in the first direction corresponding to a major axis direction of the image sensor and in the second direction corresponding to a minor axis direction of the image sensor. Each of the plurality of unit pixels may include at least two sub-pixels arranged in the minor axis direction. The electronic device may include an image processing module configured to up-scale the raw image only in the minor axis direction by using the at least two sub-pixels arranged in the minor axis direction in each of the plurality of unit pixels, without up-scaling in the major axis direction among the major axis direction and the minor axis direction.

[0005] A method is disclosed. The method can be performed by an electronic device. The method can include an operation of obtaining a raw image based on compressed light through an anamorphic lens, wherein the anamorphic lens compresses light in a second direction among a first direction and a second direction perpendicular to the first direction, by an image sensor. A plurality of unit pixels in the image sensor can be arranged in the first direction corresponding to a major axis direction of the image sensor and in the second direction corresponding to a minor axis direction of the image sensor. Each of the plurality of unit pixels can include at least two sub-pixels arranged in the minor axis direction. The method can include an operation of up-scaling the raw image only in the minor axis direction by using the at least two sub-pixels arranged in the minor axis direction in each of the plurality of unit pixels, without up-scaling in the major axis direction among the major axis direction and the minor axis direction.

[0006] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by at least one processor of an electronic device including an anamorphic lens and an image sensor, can cause the electronic device to acquire a raw image based on compressed light through the anamorphic lens, the compressed light being compressive in a second direction among a first direction and a second direction perpendicular to the first direction. A plurality of unit pixels within the image sensor can be arranged in the first direction corresponding to a major axis direction of the image sensor and in the second direction corresponding to a minor axis direction of the image sensor. Each of the plurality of unit pixels can include at least two sub-pixels arranged in the minor axis direction. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to upscale the raw image only in the short axis direction using the at least two sub-pixels arranged in the short axis direction in each of the plurality of unit pixels, without upscaling in the long axis direction among the long axis direction and the short axis direction.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2A is a simplified block diagram of an electronic device according to one embodiment.

[0009] FIG. 2b is a block diagram of an image sensor of an electronic device, according to one embodiment.

[0010] FIG. 3A is a drawing illustrating an imaging area in an area where an image sensor is located by light passing through a lens according to one embodiment.

[0011] FIG. 3b is a drawing comparing the sizes between the imaging area and the image sensor according to one embodiment.

[0012] FIG. 3c is a diagram illustrating a pixel sensor array of an image sensor according to one embodiment.

[0013] FIG. 3D is a drawing illustrating the structure of a unit pixel according to one embodiment.

[0014] FIG. 3e is a diagram illustrating various examples of unit pixels according to one embodiment.

[0015] FIG. 4A is a diagram illustrating images according to one embodiment.

[0016] FIG. 4b is a diagram illustrating an electronic device displaying a preview image according to one embodiment.

[0017] FIG. 5A is a drawing illustrating an imaging area in an area where an image sensor is located by light passing through a lens according to one embodiment.

[0018] FIG. 5b is a drawing comparing the sizes between the imaging area and the image sensor according to one embodiment.

[0019] FIG. 6 is a simplified block diagram of an electronic device according to one embodiment.

[0020] FIG. 7 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0021] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0022] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0023] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0024] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0025] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0029] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0030] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0031] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0035] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0037] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0040] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0042] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] FIG. 2A is a simplified block diagram of an electronic device according to an embodiment. FIG. 2B is a block diagram of an image sensor of an electronic device according to an embodiment. FIG. 3A is a diagram illustrating an imaging area in an area where an image sensor is located by light passing through a lens according to an embodiment. FIG. 3B is a diagram comparing the sizes between the imaging area and the image sensor according to an embodiment. FIG. 3C is a diagram illustrating an example of a pixel sensor array of an image sensor according to an embodiment. FIG. 3D is a diagram illustrating a structure of a unit pixel according to an embodiment. FIG. 3E is a diagram illustrating various examples of a unit pixel according to an embodiment. FIG. 4A is a diagram illustrating images according to an embodiment. FIG. 4B is a diagram illustrating an electronic device displaying a preview image according to an embodiment.

[0045] The descriptions of FIGS. 2A to 4B may refer to components or structures of the electronic device (101) described through FIG. 1.

[0046] Referring to FIG. 2A, the electronic device (200) may include a processor (120), a memory (130), a camera (210), an image signal processor (ISP) (220), and a display (230). For example, the camera (210) may be included in the camera module (180) of FIG. 1. For example, the image signal processor (220) may correspond to the auxiliary processor (123) of FIG. 1. For example, the display (230) may be included in the display module (160) of FIG. 1.

[0047] In one embodiment, the camera (210) may include a lens (211) and an image sensor (215). Referring to FIG. 2B, the image sensor (215) may include a pixel sensor array (251), a timing controller (253), a thermal decoder (255), an output circuit (257), and an image processing module (259).

[0048] In one embodiment, the lens (211) can compress light reflected by an object to be photographed and / or a landscape (e.g., light (301, 305) of FIG. 3A). In one embodiment, the lens (211) can refract light (301, 305) from a subject and / or a background located within a field of view (or angle of view). In one embodiment, the lens (211) can refract light (301, 305) in a first direction (261) and / or a second direction (265) perpendicular to the first direction (261). In one embodiment, the lens (211) can refract light (301, 305) in a second direction (265) among the first direction (261) and the second direction (265). In one embodiment, the lens (211) can refract light (301, 305) more in the second direction (265) than in the first direction (261). For example, the refractive index of the lens (211) in the first direction (261) and the refractive index of the lens (211) in the second direction (265) of light (301, 305) passing through the lens (211) may be different. For example, the refractive index of the lens (211) in the first direction (261) may be lower than the refractive index of the lens (211) in the second direction (265). For example, the lens (211) may be an anamorphic lens.

[0049] In one embodiment, the shape of the lens (211) may be circular, elliptical, square, rectangular, or rectangular with rounded corners. In one embodiment, the lens (211) may have a shape corresponding to the pixel sensor array (251) such that the imaging area (e.g., 320 in FIGS. 3A and 3B) (or field of view (FOV)) on the area (or plane) where the pixel sensor array (251) is located, which can be reached by light (301, 305) passing through the lens (211), corresponds to the size of the pixel sensor array (251). In one embodiment, the lens (211) may be elliptical, rectangular, or rectangular with rounded corners, which corresponds to the pixel sensor array (251), such that the imaging area (320) corresponds to the size of the pixel sensor array (251). In one embodiment, referring to FIG. 3a, when the shape of the lens (211) is an oval, a rectangle, or a square with rounded corners, the length of the major axis (311) of the lens (211) extending in the first direction (261) may be longer than the length of the minor axis (315) extending in the second direction (265) of the anamorphic lens. In one embodiment, referring to FIG. 3b, since the lens (211) has a shape corresponding to the pixel sensor array (251), the sizes of areas (321, 323, 325, 327) of the imaging area (320) through which light that is not received by the pixel sensor array (251) passes may be reduced.

[0050] In one embodiment, the lens (211) may be implemented as a lens assembly in which a plurality of lenses are aligned along an optical axis (e.g., 269 of FIG. 3A). In one embodiment, the lenses included in the lens assembly may have the same lens properties (e.g., angle of view, focal length, or f-number), or at least one lens may have at least one different lens property from the other lenses. In one embodiment, the lens assembly may include an anamorphic lens. In addition to the anamorphic lens, the lens assembly may further include a wide-angle lens and / or a telephoto lens.

[0051] In one embodiment, the lens (211) can be moved, for example, by an actuator (not shown). As the position of the lens (211) changes according to the movement of the lens (211), the focal length of the lens (211) can be changed.

[0052] In one embodiment, light (301, 305) passing through the lens (211) can reach an area where the pixel sensor array (251) of the image sensor (215) is located. Referring to FIGS. 3A and 3B, the pixel sensor array (251) can receive light (301, 305) within an imaging area (320) (or field of view (FOV)) on an area (or plane) where the pixel sensor array (251) is located, which can be reached by light (301, 305) passing through the lens (211).

[0053] In one embodiment, the ratio of the length of the minor axis (315) to the length of the major axis (311) of the lens (211) may be different from the ratio of the length of the minor axis (345) to the length of the major axis (341) of the imaging area (320). For example, since the lens (211) refracts light (301, 305) more in the second direction (265) than in the first direction (261), the ratio of the length of the minor axis (315) to the length of the major axis (311) of the lens (211) may be greater than the ratio of the length of the minor axis (345) to the length of the major axis (341) of the imaging area (320). Accordingly, the length of the lens (211) extending in the first direction (261) may be longer than the length of the lens (211) extending in the second direction (265).

[0054] In one embodiment, the image sensor (215) can receive compressed light (301, 305) by passing it through a lens (211) via at least two photodiodes (or photoelectric conversion elements) included in each of a plurality of unit pixels. In one embodiment, the image sensor (215) can generate electrical signals based on the light (301, 305) received through the lens (211). The image sensor (215) can generate electrical signals based on the light (301, 305) reflected from a subject.

[0055] In one embodiment, the image sensor (215) can generate image data corresponding to a raw image (e.g., 401 of FIG. 4A) by reading out electrical signals generated in response to light (301, 305) from at least two photodiodes, respectively. The image sensor (215) can output a raw image (401) including color information about an object. For example, the raw image (401) output from the image sensor (215) can represent a color value assigned to each pixel. For example, when the image sensor (215) acquires the raw image (401) through a color filter of a Bayer pattern, the image pixels included in the raw image (401) can have a Bayer pattern. Here, the electrical signals can represent the amount of charge generated in response to the light (301, 305) received by the photodiode.

[0056] In one embodiment, the image sensor (215) may include an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor. In one embodiment, the image sensor (215) may include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0057] In one embodiment, referring to FIG. 3A, the pixel sensor array (251) can receive light (301, 305) passing through a lens (211) spaced in a third direction (269). In one embodiment, the third direction (269) can be perpendicular to the first direction (261) and the second direction (265), respectively. In one embodiment, the third direction (269) can be the optical axis direction.

[0058] In one embodiment, the pixel sensor array (251) may have an aspect ratio corresponding to the aspect ratio of the lens (211). In one embodiment, the major axis (331) of the pixel sensor array (251) may have the same direction as the major axis (311) of the lens (211). In one embodiment, the minor axis (335) of the pixel sensor array (251) may have the same direction as the minor axis (315) of the lens (211).

[0059] In one embodiment, the pixel sensor array (251) may have an aspect ratio corresponding to an aspect ratio of the imaging area (320). For example, the pixel sensor array (251) may have an aspect ratio corresponding to an aspect ratio of the imaging area (320) so as to have a size that corresponds to the imaging area (320) as much as possible. In one embodiment, the major axis (331) of the pixel sensor array (251) may have the same direction as the major axis (341) of the imaging area (320). In one embodiment, the minor axis (335) of the pixel sensor array (251) may have the same direction as the minor axis (345) of the imaging area (320).

[0060] In one embodiment, referring to FIGS. 3A and 3B, the length of the pixel sensor array (251) in the major axis direction (331) may be longer than the length of the pixel sensor array (251) in the minor axis direction (335). In one embodiment, the major axis direction (331) may correspond to the first direction (261), and the minor axis direction (335) may correspond to the second direction (265).

[0061] In one embodiment, referring to FIG. 3C, the pixel sensor array (251) may include a plurality of unit pixels (350) arranged in a grid structure in a first direction (261) and a second direction (265). Each of the plurality of unit pixels (350) may include at least two photodiodes (or photoelectric conversion elements) (or photosensitive elements). The photodiodes may detect light and convert the detected light into photocharges.

[0062] In one embodiment, the optical format of the pixel sensor array (251) may be 1 / 2.86". In one embodiment, each of the plurality of unit pixels (350) included in the pixel sensor array (251) may have a size of 1.12 micrometers in the first direction (261) and 1.12 micrometers in the second direction (265). In one embodiment, the number of the plurality of unit pixels (350) arranged in the long-axis direction (331) of the pixel sensor array (251) may be greater than the number of the plurality of unit pixels (350) arranged in the short-axis direction (335) of the pixel sensor array (251). For example, the number of the plurality of unit pixels (350) arranged in the long-axis direction (331) of the pixel sensor array (251) may be 4000, and the number of the plurality of unit pixels (350) arranged in the short-axis direction (335) of the pixel sensor array (251) may be 4000. The number can be 1500.

[0063] Referring to FIG. 3D, a unit pixel (350) included in a pixel sensor array (251) may include a micro lens (361), a color filter (362), and at least two light-receiving elements (363, 365).

[0064] In one embodiment, the microlens (361) can refract and / or focus light. In one embodiment, a color filter (362) can be positioned between the microlens (361) and at least two light-receiving elements (363, 365).

[0065] In one embodiment, the color filter (362) can pass light having a specified wavelength range. In one embodiment, each of the unit pixels included in the pixel sensor array (251) can be assigned one of a plurality of reference colors (e.g., red (R), green (G), blue (B)) through the color filter (362). For example, the plurality of reference colors can be red, green, blue (RGB), red, green, blue, white (RGBW), cyan, magenta, yellow (CMY), cyan, magenta, yellow, black (CMYK), or red, yellow, blue (RYB). For example, the color filter can be formed in a Bayer pattern.

[0066] In one embodiment, at least two light-receiving elements (363, 365) may include photodiodes. In one embodiment, at least two light-receiving elements (363, 365) may output an electrical signal corresponding to incident light by the photoelectric effect. In one embodiment, each of the at least two light-receiving elements (363, 365) may be referred to as a sub-pixel.

[0067] In one embodiment, each of the plurality of unit pixels of the pixel sensor array (251) may be assigned one of the plurality of reference colors. For example, each of the plurality of unit pixels may be assigned one of the plurality of reference colors by a color filter. For example, each of the plurality of unit pixels included in the pixel sensor array (251) may be one of a red pixel for converting light in a red spectrum region into an electrical signal, a green pixel for converting light in a green spectrum region into an electrical signal, and a blue pixel for converting light in a blue spectrum region into an electrical signal. According to an embodiment, each of the plurality of unit pixels may receive all of the plurality of reference colors. For example, each of the plurality of unit pixels may generate all of the electrical signals for the reference colors.

[0068] In one embodiment, at least two sub-pixels (363, 365) included in a unit pixel (350) may be arranged in a second direction (265). In one embodiment, at least two sub-pixels (363, 365) included in a unit pixel (350) may be arranged only in the second direction (265) among the first direction (261) and the second direction (265).

[0069] In one embodiment, when the number of at least two sub-pixels (363, 365) included in a unit pixel (350) is N, the unit pixel (350) may be referred to as an NPD pixel. For example, the N sub-pixels (363, 365) may have a size that is an even division of the size of the unit pixel (350). For example, the size of each of the N sub-pixels (363, 365) in the second direction (265) may be a value obtained by dividing the size of the unit pixel (350) by N. For example, referring to FIG. 3E, the unit pixel (370) may be a 2PD pixel in which two sub-pixels (371, 373) are arranged only in the second direction (265) among the first direction (261) and the second direction (265). For example, when the size of the unit pixel (370) is 1.12 micrometers, the size of each of the two sub-pixels (371, 373) in the second direction (265) may be 0.56 micrometers. For example, the unit pixel (380) may be a 3PD pixel in which three sub-pixels (381, 383, 385) are arranged only in the second direction (265) among the first direction (261) and the second direction (265). For example, when the size of the unit pixel (380) is 1.12 micrometers, the size of each of the three sub-pixels (381, 383, 385) in the second direction (265) may have a value obtained by dividing 1.12 micrometers by 3. For example, a unit pixel (390) may be a 4PD pixel in which four sub-pixels (391, 393, 395, 397) are arranged only in the second direction (265) among the first direction (261) and the second direction (265). For example, when the size of the unit pixel (390) is 1.12 micrometers, the size of each of the four sub-pixels (391, 393, 395, 397) in the second direction (265) may be 0.28 micrometers.

[0070] In one embodiment, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may correspond to the reciprocal of the refractive index (or ratio) at which light (301, 305) is compressed by the lens (211) in the second direction (265). For example, when the lens (211) compares the refractive index of light (301, 305) in the first direction (261) to a refractive index in the second direction (265) that is N times greater, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may be N.

[0071] In one embodiment, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may correspond to a required aspect ratio of the up-scaled image (403) with respect to the aspect ratio of the pixel sensor array (251). In one embodiment, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may correspond to a required number of a plurality of unit pixels included in the second direction (265) of the up-scaled image (403) with respect to the number of a plurality of unit pixels included in the second direction (265) of the pixel sensor array (251). However, the present invention is not limited thereto. In one embodiment, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may correspond to an aspect ratio (or an inverse of the aspect ratio) of the pixel sensor array (251). For example, the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) may correspond to an inverse of a ratio of an angle of view (or a major axis angle of view) of the pixel sensor array (251) in the second direction (265) to an angle of view (or a major axis angle of view) of the pixel sensor array (251) in the first direction (261).

[0072] Referring back to FIG. 2B, in one embodiment, a plurality of unit pixels may be connected to a plurality of column lines (or control lines) arranged in a first direction (261) (or a short axis (335) of the pixel sensor array (251)). In one embodiment, the plurality of column lines may be arranged in the first direction (261) and may be connected to pixels arranged in the same column. For example, each of the plurality of column lines may transmit control signals output from the column decoder (255) to each of the pixels arranged in the same column. In one embodiment, a specific control signal (e.g., a reset signal) may be equally applied to at least two sub-pixels (363, 365) included in a unit pixel (350). In one embodiment, another specific control signal (e.g., a read-out signal) may be sequentially applied to at least two sub-pixels (363, 365) included in a unit pixel (350). Accordingly, by including at least two sub-pixels (363, 365) in a unit pixel (350), the time required to read electrical signals of a plurality of pixels included in the pixel sensor array (251) can be reduced by a specific control signal (e.g., a reset signal) that is simultaneously applied to the two sub-pixels (363, 365). Accordingly, by including at least two sub-pixels (363, 365) in a unit pixel (350), the operating power of the pixel sensor array (251) can be reduced, and / or the operating speed can be improved.

[0073] In one embodiment, a plurality of unit pixels may be connected to a plurality of row lines arranged in a second direction (265). In one embodiment, each of the plurality of row lines may be arranged in the second direction (265) and connected to pixels arranged in the same row. Each of the plurality of row lines may transmit a digital signal (or an electrical signal) of pixels to which a control signal is transmitted through a column line of the pixel sensor array (251) to an output circuit (257).

[0074] In one embodiment, the timing controller (253) may control the timing of the thermal decoder (255) and / or the output circuit (257).

[0075] In one embodiment, the column decoder (255) may generate control signals for driving the pixel sensor array (251) under the control of the timing controller (253), and provide the control signals to each of a plurality of unit pixels of the pixel sensor array (251) through a plurality of column lines. In one embodiment, the column decoder (255) may control a plurality of unit pixels of the pixel sensor array (251) to receive incident light simultaneously or in units of columns. The column decoder (255) may select unit pixels in units of columns among the plurality of unit pixels, and control the selected unit pixels (e.g., unit pixels of one column) to output digital signals (or electrical signals) through a plurality of row lines.

[0076] In one embodiment, the output circuit (257) can read out a digital signal (or electrical signal) from pixels of a column selected by a column decoder (255) among a plurality of unit pixels. In one embodiment, the output circuit (257) can convert digital signals (or electrical signals) received from the pixel sensor array (251) through a plurality of row lines into a raw image (401). In one embodiment, the output circuit (257) can temporarily store and then output a digital signal (or electrical signal) output from the pixel sensor array (251). The output circuit (257) can output the stored plurality of digital signals (or electrical signals) as a raw image (401). For example, the raw image (401) can include a plurality of pixels corresponding to the number of pixels of the pixel sensor array (251). For example, if the number of the plurality of unit pixels (350) arranged in the major axis direction (331) of the pixel sensor array (251) is 4000, and the number of the plurality of unit pixels (350) arranged in the minor axis direction (335) of the pixel sensor array (251) is 1500, the raw image (401) may include 4000 pixels in the first direction (261) and 1500 pixels in the second direction (265). For example, the raw image (401) may include a value representing a number of digital signals corresponding to the NPD for each of the 1500 pixels in the second direction (265). For example, if each of the plurality of pixels includes two sub-pixels in the second direction (265), the raw image (401) may include a value representing digital signals of 3000 sub-pixels in the second direction (265).

[0077] In one embodiment, the image processing module (259) may perform remosaic processing (or scale-up processing) (or up-scale processing) (or de-squeeze processing) on ​​the raw image (401).

[0078] In one embodiment, the image processing module (259) may perform remosaic processing (or scale-up processing) (or up-scale processing) (or decompression processing) on ​​the raw image (401) only in a specified direction (e.g., the second direction (265)).

[0079] In one embodiment, the number of at least two sub-pixels (363, 365) may correspond to a multiple of up-scaling in the second direction (265). For example, if the number of at least two sub-pixels (363, 365) arranged in the second direction (265) in the unit pixel (350) is N, the image processing module (259) may up-scale the raw image (401) by N times in the second direction (265).

[0080] In one embodiment, the image processing module (259) may not perform remosaic processing (or scale-up processing) (or up-scale processing) (or decompression processing) on ​​the raw image (401) in a direction other than the designated direction (e.g., the first direction (261)) (e.g., the second direction (265)). In one embodiment, the image processing module (259) may up-scale the raw image (401) only in the second direction (265) without up-scaling in the first direction (261). For example, the designated direction (e.g., the second direction (265)) may be a direction having a higher refractive index than the other direction (e.g., the first direction (261)). For example, the designated direction (e.g., the second direction (265)) may be a distortion direction of the lens (211). Hereinafter, a raw image (401) that has been remosaiced only in a specified direction (e.g., the second direction (265)) may be referred to as an up-scaled image (e.g., 403 in FIG. 4a).

[0081] In one embodiment, the image processing module (259) can perform remosaic processing on the raw image (401) only in a specified direction (e.g., the second direction (265)) by increasing the number of pixels in the image in the direction in which distortion occurs (e.g., the second direction (265)). For example, the image processing module (259) can upscale the raw image (401) only in the second direction (265) by determining pixel values ​​of a plurality of pixels in the second direction (265) of the up-scaled image based on electrical signals (or pixel values) represented by each of at least two sub-pixels (363, 365) in the second direction (265), and by determining pixel values ​​of a plurality of pixels in the first direction (261) of the up-scaled image based on electrical signals (or pixel values) represented by each of the plurality of unit pixels in the first direction (261). In one embodiment, the values ​​of the increased pixels in the upscaled image (403) may be interpolated values ​​based on the values ​​of the unit pixels (or two sub-pixels (363, 365)) in the raw image (401).

[0082] For example, the image processing module (259) can increase the number of pixels in the image in the direction in which distortion occurs (e.g., the second direction (265)) for the raw image (401) composed of 4000 pixels in the first direction (261) and 1500 pixels in the second direction (265) output from the image sensor (215). For example, if the raw image (401) includes 4000 pixels in the first direction (261), 1500 pixels in the second direction (265), and each of the pixels arranged in the second direction (265) includes a value representing N digital signals, the image processing module (259) can increase the number of pixels in the second direction (265) from 1500 to 1500 X N (or, 1500 X N α). In one embodiment, α may be a ratio of the number of pixels that can be increased by interpolating sub-pixels included in the NPD. For example, α may be 1.7.

[0083] For example, the image processing module (259) may perform remosaic processing on the raw image (401) such that the number of the plurality of pixels arranged along the long axis of the up-scaled image (403) corresponds to the number of the plurality of unit pixels of the pixel sensor array (251) arranged along the long axis (331), and the number of the plurality of pixels arranged along the short axis of the up-scaled image (403) corresponds to the number of at least two sub-pixels (363, 365) included in the plurality of unit pixels of the pixel sensor array (251) arranged along the short axis (335).

[0084] In one embodiment, the image signal processor (220) can perform various processing on the up-scaled image (403) output from the image sensor (215). The image signal processor (220) can perform image processing on the up-scaled image (403) acquired through the image sensor (215). In one embodiment, the up-scaled image (403) can be an image in which the number of pixels in the original image (401) is increased in the direction in which distortion occurs (e.g., the second direction (265)).

[0085] In one embodiment, the image signal processor (220) may perform image correction and image signal processing on the up-scaled image (403). For example, the image signal processor (220) may perform hot pixel correction, demosaicing, color interpolation, noise reduction, lens shading correction, defective pixel correction, green imbalance correction, crosstalk compensation, or auto dark level compensation (ADLC) to remove fixed pattern noise (FPN) for image correction.

[0086] In one embodiment, the processor (120) may obtain a final image (e.g., 405 in FIG. 4A) from the image signal processor (220).

[0087] In one embodiment, the processor (120) may generate a frame (409) that includes the final image (405) as a preview image (e.g., 407 of FIG. 4A). In one embodiment, the processor (120) may generate a frame (409) that includes the preview image (407) and / or a user interface (UI) (e.g., a UI of an application for operating the camera (210). In one embodiment, the number of the plurality of pixels arranged in the first direction (261) of the preview image (407) may correspond to the number of the plurality of pixels arranged in the first direction (261) of the up-scaled image (403). In one embodiment, the number of the plurality of pixels arranged in the second direction (265) of the preview image (407) may correspond to the number of the plurality of pixels arranged in the second direction (265) of the up-scaled image (403). In one embodiment, the aspect ratio of the preview image (407) may correspond to the aspect ratio of the up-scaled image (403).

[0088] In one embodiment, a first ratio (or image output ratio) of the number of the plurality of unit pixels of the pixel sensor array (251) arranged in the second direction (265) to the number of the plurality of unit pixels of the pixel sensor array (251) arranged in the first direction (261) may be smaller than a second ratio (or pixel number ratio) of the number of the plurality of pixels arranged in the second direction (265) to the number of the plurality of pixels arranged in the first direction (261) of the preview image (407). In one embodiment, the first ratio may be greater than a value obtained by multiplying the second ratio by a third ratio (or optical imaging ratio) of the short-axis angle of view of the pixel sensor array (251) in the first direction (261) to the long-axis angle of view of the pixel sensor array (251) in the first direction (261).

[0089] In one embodiment, referring to FIG. 4b, the processor (120) can display the generated frame (409) through the display (230).

[0090] Referring to FIG. 4B, the camera (210) of the electronic device (200) can be seen from the outside through the housing (420) forming the exterior of the electronic device (200). For example, when the camera (210) is placed on the same surface of the electronic device (200) as the display (230), the long axis (331) of the pixel sensor array (251) in the camera (210) including the lens (211) and the long axis of the display (230) may have different directions. For example, unlike the pixel sensor array (251) in which the length extending in the first direction (261) is longer than the length extending in the second direction (265), the display (230) may have a length extending in the first direction (261) that is shorter than the length extending in the second direction (265). Accordingly, the length of the camera (210) extending in the second direction (265) can be shortened, and the size of the display (230) mounted on the electronic device (200) can be increased.

[0091] As described above, the electronic device (200) can minimize wasted imaging areas (321, 323, 325, 327) by using a pixel sensor array (251) having the same shape as the lens (211). Accordingly, the electronic device (200) can minimize the size of the pixel sensor array (251) inside the electronic device (200).

[0092] In addition, as described above, the electronic device (200) can expand the angle of view of the camera (210) by compressing light using an anamorphic lens. In addition, the electronic device (200) can increase the optical sampling frequency and improve the visual resolution by decompressing the raw image acquired by the compressed light using at least two sub-pixels (363, 365) arranged in the compressed direction.

[0093] In addition, as described above, the electronic device (200) can increase the usable space within the electronic device (200) by using a relatively small-sized camera (210).

[0094] Finally, in the electronic device (200) as described above, the number of unit pixels arranged in the second direction (265) can be reduced, thereby reducing the number of column lines. Accordingly, the number of transmissions of control signals required to control the unit pixels through the column lines and the current consumption can be reduced, thereby improving the operating efficiency of the pixel sensor array (251).

[0095] FIG. 5A is a diagram illustrating an imaging area in an area where an image sensor is located by light passing through a lens, according to one embodiment. FIG. 5B is a diagram comparing the sizes between an imaging area and an image sensor, according to one embodiment.

[0096] FIG. 5a and FIG. 5b can illustrate a situation in which the shape of the lens (211) changes compared to FIG. 3a and FIG. 3b.

[0097] Referring to FIG. 5A, the shape of the lens (211) may have a shape corresponding to the pixel sensor array (251). For example, the shape of the lens (211) may be a rectangle or a square with rounded corners.

[0098] In one embodiment, referring to FIG. 5B, since the lens (211) has a shape corresponding to the pixel sensor array (251), the sizes of areas (521, 523, 525, 527) of the imaging area (320) through which light not received by the pixel sensor array (251) passes may be reduced. For example, since the lens (211) has a shape corresponding to the pixel sensor array (251), the sizes of the areas (521, 523, 525, 527) of FIG. 5B may be further reduced compared to the areas (321, 323, 325, 327) of FIG. 3B.

[0099] FIG. 6 is a simplified block diagram of an electronic device according to one embodiment.

[0100] The electronic device (600) of FIG. 6 may include an image processing module (659) in the image signal processor (220) as compared to the electronic device (200) illustrated through FIGS. 2A and 2B.

[0101] In one embodiment, as the image processing module (659) is included in the image signal processor (220), the image sensor (215) can transmit a raw image acquired through the pixel sensor array (251) to the image signal processor (220).

[0102] In one embodiment, the image signal processor (220) may perform remosaic processing (or scale-up processing) (or up-scale processing) (or decompression processing) on ​​the raw image (401) through the image processing module (659). In one embodiment, the image processing module (659) may perform remosaic processing (or scale-up processing) (or up-scale processing) (or decompression processing) on ​​the raw image (401) only in a specified direction (e.g., the second direction (265)).

[0103] In one embodiment, the image signal processor (220) can perform image correction and image signal processing on the up-scaled image (403).

[0104] In one embodiment, the image signal processor (220) may provide the final image (405) to the processor (120).

[0105] In one embodiment, the processor (120) may obtain a final image (405) from the image signal processor (220). In one embodiment, the processor (120) may generate a frame (409) including the final image (405) as a preview image (407). In one embodiment, the processor (120) may display the generated frame (409) through the display (230).

[0106] FIG. 7 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0107] Fig. 7 can be explained with reference to Figs. 2a to 6.

[0108] Referring to FIG. 7, in operation 710, the electronic device (200) may obtain a raw image (e.g., 401 of FIG. 4A) based on light compressed in a short-axis direction among a long-axis direction (e.g., a first direction (261)) and a short-axis direction (e.g., a second direction (265)). For example, the electronic device (200) may obtain a raw image (401) based on light compressed through a lens (211) (or an anamorphic lens) that compresses light on an optical path in a short-axis direction among a long-axis direction (e.g., a first direction (261)) and a short-axis direction (e.g., a second direction (265)). In one embodiment, the electronic device (200) can acquire a raw image (e.g., 401 of FIG. 4A) through an image sensor (215) in which a plurality of unit pixels are arranged in a grid, and each of the plurality of unit pixels includes at least two sub-pixels (363, 365) arranged in a short-axis direction (e.g., a second direction (265)).

[0109] In operation 720, the electronic device (200) can upscale the raw image (401) only in the short-axis direction (265) without upscaling in the long-axis direction (261). In one embodiment, the electronic device (200) can perform remosaic processing on the raw image (401) only in a specified direction (e.g., the second direction (265)) by increasing the number of pixels in the image in the direction in which distortion occurs (e.g., the second direction (265)). For example, the electronic device (200) may determine pixel values ​​of a plurality of pixels in the second direction (265) of the up-scaled image based on electrical signals (or pixel values) represented by each of at least two sub-pixels (363, 365) in the second direction (265), and may determine pixel values ​​of a plurality of pixels in the first direction (261) of the up-scaled image based on electrical signals (or pixel values) represented by each of the plurality of unit pixels in the first direction (261), thereby up-scaling the original image (401) only in the second direction (265). In one embodiment, the values ​​of the increased pixels in the up-scaled image (403) may be interpolated values ​​based on the values ​​of the unit pixels (or the two sub-pixels (363, 365)) in the original image (401).

[0110] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0111] As described above, the electronic device (200) may include an anamorphic lens (211) that compresses light in a first direction (261) and a second direction (265) perpendicular to the first direction (261). The electronic device (200) may include an image sensor (215) configured to acquire a raw image (401) based on the compressed light through the anamorphic lens (211). A plurality of unit pixels in the image sensor (215) may be arranged in the first direction (261) corresponding to the major axis direction of the image sensor (215) and in the second direction (265) corresponding to the minor axis direction of the image sensor (215). Each of the plurality of unit pixels may include at least two sub-pixels (363, 365) arranged in the minor axis direction. The electronic device (200) may include an image processing module (259) configured to upscale the raw image (401) only in the short axis direction by using at least two sub-pixels (363, 365) arranged in the short axis direction in each of the plurality of unit pixels, without upscaling in the long axis direction among the long axis direction and the short axis direction.

[0112] The length of the anamorphic lens (211) extending in the first direction (261) may be longer than the length of the anamorphic lens (211) extending in the second direction (265).

[0113] The electronic device (200) may include a housing (420), and a display (230) disposed on the front of the housing (420) where the anamorphic lens (211) is disposed, the display having a length extending in the first direction (261) shorter than a length extending in the second direction (265).

[0114] The number of the sub-pixels (363, 365) of the image sensor (215) arranged in the longitudinal direction may correspond to the number of a plurality of pixels arranged in the longitudinal direction of the up-scaled image (403). The number of the sub-pixels (363, 365) of the image sensor (215) arranged in the short axis direction may correspond to the number of the sub-pixels (363, 365) included in the plurality of pixels arranged in the short axis direction of the up-scaled image (403).

[0115] The number of sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction may correspond to the ratio at which the light is compressed in the second direction (265).

[0116] The number of sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction may correspond to a multiple of up-scaling in the short-axis direction.

[0117] The number of the sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction may correspond to the aspect ratio of the up-scaled image (403) with respect to the aspect ratio of the image sensor (215).

[0118] The image processing module (259) may be configured to upscale the raw image (401) only in the short axis direction by determining a pixel value of the up-scaled image (403) in the short axis direction based on the pixel value indicated by each of the sub-pixels (363, 365) in the short axis direction and by determining a pixel value of the up-scaled image (403) in the long axis direction based on the pixel value indicated by each of the plurality of unit pixels in the long axis direction.

[0119] The image sensor (215) may include a plurality of control lines arranged along the short axis direction.

[0120] Each of the plurality of unit pixels may include a micro lens (361) and at least two sub pixels (363, 365) that receive the compressed light passing through the micro lens (361). The at least two sub pixels (363, 365) may output different pixel values ​​depending on the optical path of the compressed light passing through the micro lens (361).

[0121] The electronic device (200) may include an image signal processor (220). The image processing module (259) may be included as a processing circuit within the image sensor (215). The image processing module (259) may be configured to transmit the up-scaled image (403) to the image signal processor (220).

[0122] The electronic device (200) may include an image signal processor (220), which includes the image processing module (259) as a processing circuit within the image signal processor (220). The electronic device (200) may be configured such that the image sensor (215) transmits the raw image (401) to the image signal processor (220). The image signal processor (220) may be configured to upscale the raw image (401) only in the short-axis direction through the image processing module (259).

[0123] The electronic device (200) may include a display (230), at least one processor (120) including a processing circuit, and a memory (130) storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (200) to display a preview image (407) based on the up-scaled image (403) through the display (230). The number of the plurality of pixels arranged in the major axis direction of the preview image (407) may correspond to the number of the plurality of pixels arranged in the major axis direction of the up-scaled image (403). The number of the plurality of pixels arranged in the minor axis direction of the preview image (407) may correspond to the number of the plurality of pixels arranged in the minor axis direction of the up-scaled image (403).

[0124] A first ratio of the number of the plurality of unit pixels of the image sensor (215) arranged in the short axis direction to the number of the plurality of unit pixels of the image sensor (215) arranged in the long axis direction may be smaller than a second ratio of the number of the plurality of pixels arranged in the short axis direction to the number of the plurality of pixels arranged in the long axis direction of the preview image (407), and may be larger than a value obtained by multiplying the second ratio by a third ratio of the short axis angle of view of the image sensor (215) in the short axis direction to the long axis angle of view of the image sensor (215) in the long axis direction.

[0125] As described above, the method can be performed by the electronic device (200). The method can include an operation of obtaining a raw image (401) based on the compressed light through an anamorphic lens (211) that compresses light in a first direction (261) and a second direction (265) perpendicular to the first direction (261) by the image sensor (215). A plurality of unit pixels in the image sensor (215) can be arranged in the first direction (261) corresponding to the major axis direction of the image sensor (215) and in the second direction (265) corresponding to the minor axis direction of the image sensor (215). Each of the plurality of unit pixels can include at least two sub-pixels (363, 365) arranged in the minor axis direction. The method may include an operation of upscaling the raw image (401) only in the short axis direction using at least two sub-pixels (363, 365) arranged in the short axis direction in each of the plurality of unit pixels, without upscaling in the long axis direction among the long axis direction and the short axis direction.

[0126] The method may include an operation of displaying a preview image (407) based on the up-scaled image (403) through a display (230). The number of a plurality of pixels arranged in the major axis direction of the preview image (407) may correspond to the number of a plurality of pixels arranged in the major axis direction of the up-scaled image (403). The number of a plurality of pixels arranged in the minor axis direction of the preview image (407) may correspond to the number of a plurality of pixels arranged in the minor axis direction of the up-scaled image (403).

[0127] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by at least one processor (120) of an electronic device (200) including an anamorphic lens (211) and an image sensor (215), can cause the electronic device (200) to obtain a raw image (401) based on compressed light through the anamorphic lens (211) that compresses light in a first direction (261) and a second direction (265) perpendicular to the first direction (261) by the image sensor (215). A plurality of unit pixels within the image sensor (215) may be arranged in the first direction (261) corresponding to the major axis direction of the image sensor (215) and the second direction (265) corresponding to the minor axis direction of the image sensor (215). Each of the plurality of unit pixels may include at least two sub-pixels (363, 365) arranged in the minor axis direction. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device (200) to up-scale the raw image (401) only in the minor axis direction by using the at least two sub-pixels (363, 365) arranged in the minor axis direction in each of the plurality of unit pixels, without up-scaling in the major axis direction among the major axis direction and the minor axis direction.

[0128] The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device (200) to display a preview image (407) based on the up-scaled image (403) through the display (230) of the electronic device (200). The number of the plurality of pixels arranged in the major axis direction of the preview image (407) may correspond to the number of the plurality of pixels arranged in the major axis direction of the up-scaled image (403). The number of the plurality of pixels arranged in the minor axis direction of the preview image (407) may correspond to the number of the plurality of pixels arranged in the minor axis direction of the up-scaled image (403).

[0129] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0130] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0131] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0132] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0133] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0134] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0135] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (200), An anamorphic lens (211) that compresses light in a first direction (261) and a second direction (265) perpendicular to the first direction (261), An image sensor (215) configured to acquire a raw image (401) based on the compressed light through the anamorphic lens (211), a plurality of unit pixels within the image sensor (215) are arranged in the first direction (261) corresponding to the major axis direction of the image sensor (215) and the second direction (265) corresponding to the minor axis direction of the image sensor (215), and each of the plurality of unit pixels includes at least two sub-pixels (363, 365) arranged in the minor axis direction, and An image processing module (259) configured to upscale the raw image (401) only in the short axis direction using at least two sub-pixels (363, 365) arranged in the short axis direction in each of the plurality of unit pixels without upscaling in the long axis direction among the long axis direction and the short axis direction, Electronic devices.

2. In claim 1, The length of the anamorphic lens (211) extending in the first direction (261) is longer than the length of the anamorphic lens (211) extending in the second direction (265). Electronic devices.

3. In claim 2, Housing (420), and A display (230) is disposed on the front of the housing (420) where the anamorphic lens (211) is disposed, and the length extending in the first direction (261) is shorter than the length extending in the second direction (265). Electronic devices.

4. In any one of claims 1 to 3, The number of the sub-pixels (363, 365) of the image sensor (215) arranged in the longitudinal direction corresponds to the number of a plurality of pixels arranged in the longitudinal direction of the up-scaled image (403), The number of the sub-pixels (363, 365) of the image sensor (215) arranged in the short-axis direction corresponds to the number of sub-pixels (363, 365) included in the plurality of pixels arranged in the short-axis direction of the up-scaled image (403). Electronic devices.

5. In any one of claims 1 to 4, The number of sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction corresponds to the ratio at which the light is compressed in the second direction (265). Electronic devices.

6. In any one of claims 1 to 5, The number of sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction corresponds to a multiple of up-scaling in the short-axis direction. Electronic devices.

7. In any one of claims 1 to 6, The number of the sub-pixels (363, 365) included in each of the plurality of unit pixels arranged in the short-axis direction corresponds to the aspect ratio of the up-scaled image (403) with respect to the aspect ratio of the image sensor (215). Electronic devices.

8. In any one of claims 1 to 7, The above image processing module (259) Based on the pixel value indicated by each of the sub-pixels (363, 365) in the above-mentioned short-axis direction, the value of the pixel in the above-mentioned short-axis direction of the up-scaled image (403) is determined, By determining the pixel value of the up-scaled image (403) in the longitudinal direction based on the pixel value represented by each of the plurality of unit pixels in the longitudinal direction, The above raw image (401) is configured to be upscaled only in the short-axis direction. Electronic devices.

9. In any one of claims 1 to 8, The image sensor (215) includes a plurality of control lines arranged along the short axis direction. Electronic devices.

10. In any one of claims 1 to 9, Each of the above plurality of unit pixels, Micro lens (361), and comprising at least two sub-pixels (363, 365) that receive the compressed light passing through the micro lens (361); The above at least two sub-pixels (363, 365) output different pixel values ​​according to the optical path of the compressed light passing through the micro lens (361). Electronic devices.

11. In any one of claims 1 to 10, Includes an image signal processor (220), The image processing module (259) is included as a processing circuit within the image sensor (215), The image processing module (259) is configured to transmit the up-scaled image (403) to the image signal processor (220). Electronic devices.

12. In any one of claims 1 to 11, An image signal processor (220) including an image processing module (259) as a processing circuit within the image signal processor (220), The image sensor (215) is configured to transmit the raw image (401) to the image signal processor, The image signal processor (220) is configured to upscale the raw image (401) only in the short-axis direction through the image processing module (259). Electronic devices.

13. In any one of claims 1 to 12, Display (230), At least one processor (120) including a processing circuit, and A memory (130) storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (200) to: Causing a preview image (407) based on the above up-scaled image (403) to be displayed through the display (230), The number of the plurality of pixels arranged in the longitudinal direction of the above preview image (407) corresponds to the number of the plurality of pixels arranged in the longitudinal direction of the above up-scaled image (403), The number of the plurality of pixels arranged in the short-axis direction of the above preview image (407) corresponds to the number of the plurality of pixels arranged in the short-axis direction of the above up-scaled image (403). Electronic devices.

14. In claim 13, A first ratio of the number of the plurality of unit pixels of the image sensor (215) arranged in the short-axis direction to the number of the plurality of unit pixels of the image sensor (215) arranged in the long-axis direction is is smaller than a second ratio of the number of the plurality of pixels arranged in the short-axis direction to the number of the plurality of pixels arranged in the long-axis direction of the preview image (407), and A value greater than the product of the third ratio of the short-axis angle of view of the image sensor (215) in the short-axis direction to the long-axis angle of view of the image sensor (215) in the long-axis direction and the second ratio Electronic devices.

15. In the method of electronic device (200), An operation of obtaining a raw image (401) based on the compressed light through an anamorphic lens (211) by compressing light in a first direction (261) and a second direction (265) perpendicular to the first direction (261) by an image sensor (215), wherein a plurality of unit pixels in the image sensor (215) are arranged in the first direction (261) corresponding to the long-axis direction of the image sensor (215) and in the second direction (265) corresponding to the short-axis direction of the image sensor (215), and each of the plurality of unit pixels includes at least two sub-pixels (363, 365) arranged in the short-axis direction, and An operation of upscaling the raw image (401) only in the short axis direction using at least two sub-pixels (363, 365) arranged in the short axis direction in each of the plurality of unit pixels without upscaling in the long axis direction among the long axis direction and the short axis direction, method.

Images