Electronic device comprising camera, method by which electronic device captures image, and non-transitory storage medium

Abstract

The present document relates to an electronic device comprising a camera, a method by which the electronic device captures an image, and a non-transitory storage medium. According to one embodiment, the electronic device may perform the operations of: using an image sensor included in a camera assembly of the electronic device, so as to acquire an image corresponding to a subject; acquiring, from a memory of the electronic device, pre-designated shading correction data for shading correction of a first region of interest (ROI) designated in the image; identifying a movement position of the image sensor on the basis of the shading correction data so as to correct a shading deviation between first sub-photodiodes corresponding to respective first microlenses disposed in a first region corresponding to the first ROI at an image height of the image sensor; on the basis of the movement position, controlling an actuator included in an image stabilizer of the camera assembly, so as to move the position of the image sensor; and using the image sensor so as to acquire first image data of the first ROI in which the shading deviation between the first sub-photodiodes is corrected. Various embodiments are also possible.

Description

Electronic device including a camera, method for capturing images in the electronic device, and non-transitory storage medium

[0001] The present disclosure relates to an electronic device including a camera, a method for capturing an image in the electronic device, and a non-transitory storage medium.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] Recently, technologies have been developed to enable electronic devices to capture high-quality images in various ways, including high-performance cameras. Cameras may include image sensors that detect objects. Electronic devices can detect objects or images through image sensors. Image sensors may be composed of multiple pixel units, and each pixel unit may be composed of multiple subpixels. Image sensors may be composed of arrays of small photodiodes, called pixels or photosites.

[0004] As the technology of cameras incorporated into electronic devices has advanced, they have been able to provide focus control or image stabilization functions to improve focus detection performance and image quality through image stabilization when shooting video.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0006] Due to limitations in the design and manufacturing process of the image sensor of the camera included in the electronic device, the chief ray angle (CRA) of the light that does not cause shading on the pixel (e.g., unit pixel), which is the gaze direction of the micro lens placed directly on the photodiode, is fixed. Therefore, CRA error may occur if the lens and micro lens are not aligned with the CRA in various shooting environments. Electronic devices apply a pixel-by-pixel correction coefficient to correct the error, but this causes a larger photodiode shading (PD shading) error to occur in some areas of the image sensor's image height (e.g., the edge). In this case, the gain for shading compensation also increases, which may cause a deterioration in the remosaic quality or focus (AF (auto focus)) performance due to shading, resulting in a deterioration in image quality.

[0007] According to one embodiment of the present disclosure, an electronic device includes a display, a camera assembly including a camera circuit, at least one processor including a processing circuit, and a memory storing instructions.

[0008] In one embodiment, the camera assembly may include an image stabilizer including a lens, an image sensor configured to provide an electrical signal corresponding to light received through the lens, and an actuator configured to move the lens or the image sensor to stabilize the image.

[0009] According to one embodiment, the image sensor includes a plurality of unit photodiodes including a plurality of sub-photodiodes and a plurality of micro lenses corresponding to each of the plurality of unit photodiodes, wherein the plurality of sub-photodiodes included in one unit photodiode correspond to one micro lens.

[0010] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to obtain an image corresponding to a subject using the image sensor.

[0011] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the memory, pre-designated shading correction data for shading correction of a designated first region of interest (ROI) within the image. According to one embodiment, the shading correction data includes position information in which a chief ray angle of the image sensor or a chief ray angle of the lens is set to match a designated angular range.

[0012] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to determine a movement position of the image sensor based on the shading correction data, so as to correct a shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first region corresponding to the first ROI in the image sensor.

[0013] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to control the actuator to move the position of the image sensor based on the movement position.

[0014] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to obtain, using the image sensor, first image data of the first ROI in which the shading deviation between the first sub-photodiodes is corrected.

[0015] According to one embodiment of the present disclosure, an electronic device includes a display, a camera assembly including a camera circuit, at least one processor, and a memory storing instructions.

[0016] In one embodiment, the camera assembly includes an image stabilizer including a lens, an image sensor configured to provide an electrical signal corresponding to light received through the lens, and an actuator configured to move the lens or the image sensor to stabilize the image.

[0017] According to one embodiment, the image sensor includes a plurality of unit photodiodes including a plurality of sub-photodiodes and a plurality of micro lenses corresponding to each of the plurality of unit photodiodes.

[0018] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to acquire an image corresponding to a subject using the image sensor.

[0019] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, from the memory, pre-specified shading correction data for shading correction of a first region of interest (ROI) specified within the image.

[0020] According to one embodiment, the shading correction data includes position information in which the chief ray angle of the image sensor or the chief ray angle of the lens is set to match a specified angular range.

[0021] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to determine a target movement position of the lens based on the shading correction data, so as to correct shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first region corresponding to the first ROI in the image sensor.

[0022] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to control the actuator to move the position of the lens based on the target movement position of the lens.

[0023] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to acquire second image data of the first ROI, in which shading deviation is corrected by moving the position of the lens, using the image sensor.

[0024] According to one embodiment, a method of operating in an electronic device may include an operation of acquiring an image corresponding to a subject using an image sensor included in a camera assembly of the electronic device.

[0025] According to one embodiment, the operating method may include an operation of acquiring pre-specified shading correction data from a memory of the electronic device for shading correction of a first region of interest (ROI) specified within the image. According to one embodiment, the shading correction data includes position information of the image sensor or position information of the lens, wherein the chief ray angle of the image sensor or the chief ray angle of the lens is set to match a specified angular range.

[0026] According to one embodiment, the operating method may include an operation of determining a movement position of the image sensor based on the shading correction data to correct a shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first region corresponding to the first ROI in the image sensor.

[0027] According to one embodiment, the operating method may include controlling an actuator included in an image stabilizer of the camera assembly to move a position of the image sensor based on the movement position.

[0028] According to one embodiment, the operating method may include an operation of obtaining first image data of the first ROI in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

[0029] According to one embodiment, in a non-transitory storage medium storing one or more programs, the one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform an operation of acquiring an image corresponding to a subject using an image sensor included in a camera assembly of the electronic device.

[0030] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of acquiring, from a memory of the electronic device, pre-specified shading correction data for shading correction of a first region of interest (ROI) specified within the image. The shading correction data may include position information in which an angular range (chief ray angle) of the image sensor or an angular range (chief ray angle) of the lens is set to match a specified angular range.

[0031] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of determining a movement position of the image sensor based on the shading correction data to correct a shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first region corresponding to the first ROI in the image sensor.

[0032] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to execute an operation of controlling an actuator included in an image stabilizer of the camera assembly to move a position of the image sensor based on the movement position.

[0033] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of acquiring first image data of the first ROI in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

[0034] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0035] FIG. 2 is a diagram illustrating an example of a camera circuit of an electronic device according to one embodiment.

[0036] FIG. 3 is a diagram showing the configuration of an electronic device that outputs image data according to one embodiment.

[0037] FIG. 4A is a diagram showing an example of the structure of an image sensor and lens having two photodiodes according to one embodiment.

[0038] FIG. 4b is a drawing showing an example of the structure of a unit pixel having two photodiodes according to one embodiment.

[0039] FIG. 4c is a drawing showing an example of the structure of a unit pixel having four photodiodes according to one embodiment.

[0040] FIG. 5 is a diagram illustrating an example for shading compensation in an electronic device according to one embodiment.

[0041] FIG. 6A is a diagram showing an example of an image sensor of a camera circuit according to one embodiment.

[0042] FIG. 6b is a diagram illustrating an example of a lookup table including shading correction data according to one embodiment.

[0043] FIG. 7 is a diagram illustrating an example of CRA alignment in a camera circuit of an electronic device according to one embodiment.

[0044] FIG. 8 is a diagram illustrating an example of a sensor shift operation in a camera circuit of an electronic device according to one embodiment.

[0045] FIG. 9 is a diagram illustrating an example of a lens shift operation in a camera circuit of an electronic device according to one embodiment.

[0046] FIG. 10 is a drawing showing an example of an operating method in an electronic device according to one embodiment.

[0047] FIG. 11 is a drawing showing an example of shading compensation according to an operating method in an electronic device according to one embodiment.

[0048] Fig. 12 is a diagram illustrating an example of shading correction according to one embodiment.

[0049] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0050] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness. The term "user" used in the embodiments of the present disclosure may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0051] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0052] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0053] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0054] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0055] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0056] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0057] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0058] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0059] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0060] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0061] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0062] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0063] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0064] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0065] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0066] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0067] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0068] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0069] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing 1eMBB, a loss coverage (e.g., 164 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for realizing URLLC.

[0070] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0071] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0072] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0073] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0074] FIG. 2 is a diagram illustrating an example of a camera circuit of an electronic device according to one embodiment.

[0075] Referring to FIGS. 1 and 2, an electronic device (101) according to one embodiment may include a processor (120), a memory (130), a display (160), and a camera circuit (200) (e.g., a camera module (180) or a camera assembly of FIG. 1). In addition, the electronic device (101) may further include other components illustrated in FIG. 1.

[0076] According to one embodiment, the operation of the electronic device (101) may be controlled by the processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 and / or the image signal processor (260) of FIG. 2). The electronic device (101) performing a specific operation may be that the electronic device (101) or a component included in the electronic device (101) is controlled by the processor (120) of the electronic device (101). The electronic device (101) may include one or more processors (120), and for the convenience of explanation, in the following, even when a plurality of processors (120) are implemented, the term “operation of the electronic device (101)” or “operation of the processor (120)” will be used.

[0077] According to one embodiment, the processor (120) of the electronic device (101) can control to process an image captured by the camera circuit (200) and output the image to the display (160). The processor (120) of the electronic device (101) can include the image signal processor (260) illustrated in FIG. 2 or control the processing operation of the image signal processor (260).

[0078] According to one embodiment, a camera assembly of an electronic device (101) (e.g., the electronic device (101) of FIG. 1) may include a camera circuit (200) (e.g., the camera module (180) of FIG. 1). The camera assembly may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera circuit (200) may include a plurality of lens assemblies (210). In this case, the camera circuit (200) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0079] In one embodiment, the flash (220) may emit light that is used to enhance light emitted or reflected from a subject. In one embodiment, the flash (220) may include one or more light emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or a xenon lamp.

[0080] According to one embodiment, the image sensor (230) can obtain an image (e.g., a Bayer image or RAW image data) corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0081] In one embodiment, the image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting exposure timing) in response to movement of the camera circuit (200) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) may detect such movement of the camera circuit (200) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera circuit (200). In one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer.

[0082] According to one embodiment, the memory (250) can temporarily store at least a portion of an image acquired through the image sensor (230) for subsequent image processing operations. According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) of FIG. 1, or as a separate memory that operates independently therefrom.

[0083] According to one embodiment, the image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, or read-out timing control) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120) of FIG. 1. It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0084] According to one embodiment, the electronic device (101) may include a plurality of camera circuits, each having different properties or functions. In this case, for example, at least one of the plurality of camera circuits may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera circuits may be a front-facing camera, and at least another may be a rear-facing camera.

[0085] FIG. 3 is a diagram showing the configuration of an electronic device that outputs image data according to one embodiment.

[0086] Referring to FIG. 3, according to one embodiment, the lens (211) may be an imaging lens. The lens (211) may be implemented with a size corresponding to, for example, the pixel array (331) and may be configured to form an image of an object (201).

[0087] According to one embodiment, the image sensor (230) may be functionally connected to an image signal processor (260) (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2), and, under the control of the image signal processor (260), may convert an optical signal entering through the lens (211) into an electrical signal to obtain image data (e.g., a Bayer image or raw data) corresponding to the object (201). Here, the “raw data” may be an image corresponding to data on the object (201) (e.g., a subject) obtained through the image sensor (230).

[0088] According to one embodiment, the image signal processor (260) can output image data sensed and generated by the image sensor (230) to the display (160). For example, the display (160) can be implemented as a dedicated display such as a monitor or a display formed in an electronic device such as a computer, a mobile phone, a TV, or a camera. The image sensor (230) is described as a component of an image sensor and a camera for convenience of explanation, but is not limited thereto and various modifications are possible.

[0089] According to one embodiment, the image sensor (230) may include a pixel array (331) including a plurality of unit pixels, a row driver (332) that controls the pixel array (331) in units of rows, a readout (333) that outputs a signal from the pixel array (331), a timing generator (334) that provides a clock signal to the row driver (332) and the readout (333), and / or a control register (335) that stores various commands necessary for the operation of the image sensor (230). The image sensor (230) may output color information including color information of at least one of R (red), G (green), and B (blue). The R may include, for example, red, and the G may include, for example, green. In addition, the B may include, for example, blue.

[0090] According to one embodiment, the pixel array (331) may include a plurality of unit pixels (e.g., unit photodiodes). Each unit pixel may include, for example, a plurality of sub-pixels (e.g., sub-photodiodes). Each unit pixel may include two sub-photodiodes or four sub-photodiodes. Each sub-pixel may sense light incident through the lens (211) under the control of the row driver (332) and output at least one sub-pixel signal.

[0091] According to one embodiment, the pixel array (331) can output a sub-pixel signal from a row selected by each control signal provided from the row driver (332) to the readout (333). According to one embodiment, the pixel array (331) can output a signal of each sub-pixel level along a column line in units of rows under the control of the timing generator (334).

[0092] According to one embodiment, the pixel array (331) can output a signal at the sub-pixel level as many times as the product of the entire unit pixels and the sub-pixels of each unit pixel, or can output a signal as many times as the number of the entire unit pixels by adding up the signals at the sub-pixel level of one unit pixel.

[0093] According to one embodiment, a filter array including each color filter for transmitting or blocking light of a specific spectral region may be arranged on top of each unit pixel constituting the pixel array (331). In addition, micro lenses for increasing the light gathering power of each unit pixel may be arranged on top of each unit pixel constituting the pixel array (331).

[0094] According to one embodiment, the row driver (332) may drive control signals to the pixel array (331) for controlling the operation of each of the plurality of sub-pixels under the control of the timing generator (334). For example, the plurality of control signals may include a signal for controlling the transmission of photocharges generated by each of the plurality of sub-pixels, a signal for selecting each of the plurality of sub-pixels, or a signal for resetting each of the plurality of sub-pixels.

[0095] According to one embodiment, the readout (333) may include various components (e.g., a counter, a memory (e.g., a plurality of column memories), a readout circuit, or a sense amplifier (SA) circuit) for processing a sub-pixel level signal output from the pixel array (331). According to one embodiment, the readout (333) may temporarily store a sub-pixel signal output from the pixel array (331), and then sense, amplify, and output the same.

[0096] According to one embodiment, the readout (333) can output a sub-pixel level signal corresponding to each sub-pixel.

[0097] According to one embodiment, the timing generator (334) may output a control signal or a clock signal to each of the row driver (332) and / or the readout (333) to control the timing of the row driver (332) and / or the readout (333). At this time, the control register (335) operates under the control of the processor (120) and may store commands necessary for the operation of the image sensor (230).

[0098] According to one embodiment, the image signal processor (260) can process image data output from the image sensor (230). The image signal processor (260) can be one of several processors included in the electronic device (101). The image signal processor (260) described in the present disclosure can be replaced with another processor that can process image data included in the electronic device (101).

[0099] According to one embodiment, the image signal processor (260) can process image data output from an R pixel, image data output from a B pixel, and image data output from a G pixel. The image signal processor (260) can process sub-pixel level signals or unit pixel level image data output from the readout (333) in sub-pixel units or unit pixel units.

[0100] According to one embodiment, the image sensor (230) can output the signals of each sub-pixel level of a unit pixel as one image data. The image sensor (230) can output, for example, information for calculating a phase difference between sub-pixels (e.g., sub-photodiodes) included in each unit pixel (e.g., unit photodiode). For example, the unit pixel can output information for calculating a phase difference of light entering each photodiode together, and output color information by adding signals of two sub-pixel levels (e.g., sub-photodiodes).

[0101] According to one embodiment, the image processing operation in the image sensor (230) may be implemented by a combination of at least one of software, firmware, or hardware. At least a portion of the processor (120) may include, for example, modules, programs, routines, sets of instructions, or processes for performing one or more functions.

[0102] According to one embodiment, the electronic device (101) may be implemented by integrating, for example, a camera control circuit (261), an image signal processor (260) with a processor (e.g., the processor (120) of FIG. 1), and may be implemented in a form that is stored in software form in a (dedicated) memory area accessible to the processor and executed by the processor.

[0103] FIG. 4a is a diagram showing an example of the structure of an image sensor and a lens having two photodiodes corresponding to one micro lens according to one embodiment, FIG. 4b is a diagram showing an example of the structure of a unit pixel having two photodiodes according to one embodiment, and FIG. 4c is a diagram showing an example of the structure of a unit pixel having four photodiodes according to one embodiment.

[0104] Referring to FIGS. 3 and 4A to 4C, an image sensor (230) according to one embodiment can output a voltage signal of a level that can be processed by varying electron-hole generated in the photoconductor according to the wavelength and intensity of the light when light enters the photoconductor through a color filter (420). This image sensor (230) can be classified into, for example, a charge coupled device (CCD) type image sensor and a complementary metal oxide semiconductor (CMOS) type image sensor according to a method. The image sensor (230) can be configured with a plurality of unit pixels (e.g., unit photodiodes) (410), and an image sensor array in which a plurality of unit pixels are arranged in predetermined columns and rows can be used to obtain image data of a predetermined standard.

[0105] In one embodiment, the lens (211) may be functionally connected to an image stabilizer (240) that includes an actuator for image stabilization (e.g., optical image stabilization (OIS) or auto focus (AF)).

[0106] According to one embodiment, each of the unit pixels (410) of the image sensor (230) may include two sub-photodiodes (411, 412), a color filter (420), and / or a micro-lens (430). In addition, the unit pixel of the image sensor (230) may include, for example, four sub-photodiodes, a color filter (420), and / or a micro-lens (430). The number of sub-pixels per unit pixel according to one embodiment may be any number, and FIG. 4B illustrates a case where there are two sub-pixels, and FIG. 4C illustrates a case where there are four sub-pixels. For each unit pixel, only light of the corresponding color may pass through a color filter (420) of red (R), green (G), or blue (B) to detect the corresponding color. As illustrated in FIGS. 4B and 4C, green may be arranged one by one for every two unit pixels, and blue and red may be arranged one by one for every four unit pixels. The unit pixel of the image sensor (230) can be configured in various ways, such as Octa and Nona configurations, in addition to the configurations shown in FIGS. 4b and 4c.

[0107] According to one embodiment, the image sensor (230) may include at least one color filter (420) among an R (e.g., red) filter, a G (e.g., green) filter, a B (e.g., blue) filter, a yellow filter, a magenta filter, a cyan filter, and a white filter. According to one embodiment, the color filter (420) may be formed on a unit photodiode (410) (e.g., a unit pixel) including sub-photodiodes (PD1, PD2) (411, 412) based on an incident angle of incident light, and may have a Bayer pattern. The Bayer pattern may arrange filters that receive the brightness of each of red, green, and blue on a two-dimensional plane to collect the brightness and color of an object and create image data composed of points. Each unit pixel forming a grid under the color filter of the Bayer pattern may recognize only the assigned color among red, green, and / or blue and interpolate it.

[0108] According to one embodiment, the micro lens (430) may be formed to correspond to a unit photo diode (410) including sub-photo diodes (411, 412) on a color filter (420). A barrier (440) may be positioned between each of the photo diodes (410). For example, at least one color filter (420) may be positioned on the plurality of photo diodes (410). In addition, for example, at least one micro lens (430) may be positioned on the plurality of unit photo diodes (410). The micro lens (430) may be positioned, for example, on the color filter (420).

[0109] According to one embodiment, the sub-photo diodes (PD1, PD2) (411, 412) can receive light that has passed through the same micro lens (430). For example, each of the two sub-photo diodes (411, 412) can receive light that has passed through a color filter region and generate a charge corresponding to the received light energy.

[0110] According to one embodiment, the image sensor (230) of the camera circuit (200) of the electronic device (101) may be designed to match the angular range (CRA: chief ray angle) of the image sensor (230) and the CRA of the lens (211) to minimize shading deviation between the plurality of sub-photodiodes (411, 412), and the gaze position (e.g., CRA) of the micro lens (430) of the image sensor (230) may be modified (e.g., micro lens shrink) according to each field distinguished from the above to match the CRA of the lens (211). Here, the angular range (CRA) may mean the upper limit of the incident angle of light at which shading does not occur in a pixel (e.g., unit pixel). The CRA state of the lens (211) may vary depending on the focus control (AF) operation. According to one embodiment, the electronic device (101) can perform an operation to correct the shading deviation between the sub-photodiodes (411, 412) of such a region of interest (ROI).

[0111] According to one embodiment, since the CRA of the micro lens (430) of the aligned image sensor (230) is fixed, when the CRA of the lens (211) changes according to a focus adjustment operation, a zoom operation, or hand shaking of the lens (211), shading due to a CRA alignment error of the plurality of photodiodes (410) due to the change in the CRA of the lens (211) at the image height may increase. According to one embodiment, the electronic device (101) may perform a shading compensation operation of moving the image sensor (230) or the lens (211) by using the image stabilizer (240) included in the camera circuit (200) to reduce shading deviation between pixels occurring in ROIs (e.g., edge ROIs of the image) located at various image heights, thereby improving CRA alignment accuracy, thereby obtaining high focus detection performance and image quality.

[0112] FIG. 5 is a diagram showing an example for shading correction in an electronic device according to one embodiment, FIG. 6a is a diagram showing an example of an image sensor of a camera circuit according to one embodiment, and FIG. 6b is a diagram showing an example of a lookup table including shading correction data according to one embodiment.

[0113] Referring to FIGS. 5, 6A, and 6B, according to one embodiment, the electronic device (101) may acquire an image (501) (e.g., image data) corresponding to an object (e.g., a subject) through a lens (211) of a camera circuit (200), and display an image (e.g., a preview image) obtained by performing an image processing process on the acquired image (501) on a display (160).

[0114] According to one embodiment, the image signal processor (260) of the camera circuit (200) (e.g., the processor (120) of FIG. 1) can automatically designate a region of interest (ROI) (503) in an image (501) based on user input or specified conditions while performing a focus operation, a zoom operation, or an image stabilization operation. According to one embodiment, the image signal processor (260) can determine the location of the region of interest (ROI) (503) based on the locations of regions distinguished from an upper portion (e.g., an upper surface) of the image sensor (230), and can determine the location of the image sensor (230) and / or the lens (211) using at least one sensor (e.g., a gyro sensor or a Hall sensor).

[0115] According to one embodiment, the image signal processor (260) may divide an area (e.g., the upper part) corresponding to the pixel array (331) of the image sensor (230) into a specified size before designating the region of interest. For example, the size of each divided area may correspond to the region of interest (503) (e.g., the size of the CRA). For example, as shown in FIG. 6A, the central area (611) in the image (501) may obtain a uniform shake correction effect in all directions, so that no shading occurs or the shading occurs to a minimum, and the edge areas (612, 613, 614, 615) (e.g., some areas of the upper part) may have increased shading due to a focus adjustment operation, a zoom operation, or hand shaking.

[0116] According to one embodiment, the image signal processor (260) may store in advance in the form of a lookup table, shading correction data for all regions distinguished from the image sensor (230), in the memory (250) (or the memory (130) of FIG. 1), so that the shading deviation occurring in the pixels of the ROI (e.g., between sub-photodiodes) can be corrected, as illustrated in FIG. 6B. According to one embodiment, the image signal processor (260) may control the image stabilizer (240) based on the shading correction data (e.g., coordinate values ​​or movement values ​​for controlling the position of the lens or image sensor using OIS) so that the image sensor (230) or the lens (211) is accurately positioned at a designated position. According to one embodiment, the image signal processor (260) may set the actuator setting value (actuator setting value) of the optimal OIS of all regions (e.g., ROIs divided into a certain size in an image (e.g., 400 x 300 ROIs divided into a 4000 x 3000 image)) of the image stabilizer (240) by using at least one sensor (e.g., a gyro sensor or a hall sensor) to the actuator setting value for shading correction, and may include the set actuator setting value in the shading correction data. For example, the image signal processor (260) can check the position information (e.g., coordinate values) of the image sensor (230) moved by the actuator or the position information (e.g., coordinate values) of the lens (211) where the shading difference or the dispersion of pixel gain is minimized for each region.The image signal processor (260) may store in advance in the memory (250) in the form of a lookup table (e.g., a 10 x 10 LUT) the shading correction data including the position information (coordinate values) of the image sensor (230) or the lens (211) for CRA alignment of each area, i.e., the position information (e.g., actuator setting values) where the shading difference or the dispersion of pixel gain is minimized as shown in FIG. 6B. The shading correction data may further include other information related to shading correction in addition to the position information where the shading difference or the dispersion of pixel gain is minimized. For example, since the lens (211) may have a different viewing angle (e.g., CRA) depending on the focusing operation (AF operation), the image signal processor (260) may preset two lookup tables (LUTs) of a subject of a flat light source, two positions of AF operation (e.g., designated points for autofocusing (farthest (FAR), nearest (NEAR)), and perform a fitting operation to correspond to the CRA change at all AF positions. For example, since the micro lens (430) is configured (e.g., designed) to set (fix) the CRA so that the shading of pixels can be minimized based on the FAR, if the CRA of the lens (211) changes due to the focusing operation, zoom operation, or optical image stabilization (OIS) operation in the imaging environment, the shading deviation may increase. When focusing on the subject of the FAR (farthest) by adjusting the vertical position of the lens (211) (e.g., when the sub photodiode (412) When performing shading correction, the electronic device (101) may preset a look-up table (LUT) of the FAR and store it in the memory (250) so that the shading deviation increases due to misalignment of the CRA at a position outside the set point (e.g., optimized point) of the micro lens (430) (receiving the lens CRA light).When focusing on a subject at NEAR (nearest) (e.g., the sub-photodiode (411) receives lens CRA light), the electronic device (101) may preset a lookup table (LUT) at NEAR and store it in the memory (250) to perform shading correction because the shading deviation increases due to misalignment of the CRA.

[0117] According to one embodiment, the image signal processor (260) can determine whether compensation for increased shading due to a focus adjustment operation, a zoom operation, or hand shaking is required. If the image signal processor (260) determines that shading compensation is required, the image signal processor (260) can designate a ROI (503) on the image (501) and obtain shading compensation data stored in the memory (250) corresponding to the position of the designated ROI (503). The image signal processor (260) can determine position information where the shading difference or the variance of pixel gain corresponding to the position of the designated ROI (503) is minimized in the shading compensation data.

[0118] According to one embodiment, the image signal processor (260) can identify a position (e.g., a target position) to move the image sensor (230) or the lens (211) based on the position information identified in the shading correction data, and control the actuator of the image stabilizer (240) to move the image sensor (230) or the lens (211) to the position to move. According to one embodiment, the electronic device (101) can correct the changed second position (505) to a third position (507) corresponding to the first position (503) since the first position (e.g., coordinates) (503) of the ROI (e.g., an area on the image corresponding to the object of interest) specified on the image (501) (e.g., coordinates of the image) is changed to a second position (505) by moving the image sensor (230) or the lens (211). According to one embodiment, the image signal processor (260) can obtain image data of an ROI in which shading deviation is corrected as the image sensor (230) or the lens (211) moves. The electronic device (101) according to one embodiment can display a captured image (e.g., a final captured image or a corrected image) on a display (160) or store it in a memory (130).

[0119] FIG. 7 is a diagram illustrating an example of CRA alignment in a camera circuit of an electronic device according to one embodiment. FIG. 8 is a diagram illustrating an example of a sensor shift operation in a camera circuit of an electronic device according to one embodiment. FIG. 9 is a diagram illustrating an example of a lens shift operation in a camera circuit of an electronic device according to one embodiment.

[0120] Referring to FIGS. 7 and 8, according to one embodiment, the camera circuit (200) of the electronic device (101) can designate a ROI (701) at an image height of an image (501) and identify unit pixels (e.g., unit photodiodes (410a, 410b, 410c)) of the image sensor (230) at the location of the designated ROI (701). Here, the micro lenses (430) of the image sensor (230) are optimized to minimize shading based on a specific point (e.g., a focus of the FAR subject (711)). Each of the unit photodiodes (410a, 410b, 410c) may include two or four sub-photodiodes. For example, the first unit photodiode (410a) may include two sub-photodiodes (L1, R1). For example, the second unit photodiode (410b) may include two sub-photodiodes (L2, R2). For example, the third unit photodiode (410c) may include two sub-photodiodes (L3, R3).

[0121] According to one embodiment, the image signal processor (260) of the camera circuit (200) can determine that shading correction is not necessary if the CRA alignment of the unit pixels (e.g., the unit photodiodes (410a, 410b, 410c)) at the location of the designated ROI (701) is correct. The correct CRA alignment of the unit pixels may mean that the CRA (721) of the lens (211) and the CRA (723) of the image sensor (230) are not misaligned and are identical or approximately identical. When the CRA is out of focus, it may be more misaligned than when it is in focus. The image signal processor (260) can compensate for the shading between the sub-photodiodes by moving the position of the image sensor when the CRA is out of focus. The misalignment of the CRA in the focused state can be compensated by a method of correcting the photodiode value. The image signal processor (260) can correct an error caused by a method of correcting a photodiode value by moving the position of the image sensor. According to one embodiment, the image signal processor (260) of the camera circuit (200) can determine that the position of the image sensor (230) or the lens (211) is changed by an actuator included in the image stabilizer (240) due to hand shake or camera shake when performing a focus adjustment operation, a zoom operation, or a shake correction operation. The image signal processor (260) can identify that the CRA alignment of the unit pixels (e.g., the unit photodiodes (410a, 410b, 410c)) of the position of the specified ROI (701) is not correct as the position of the image sensor (230) or the lens (211) is changed by the actuator.For example, as illustrated in FIG. 7, the image signal processor (260) can check if the CRA alignment of the unit pixels (e.g., the unit photodiodes (410a, 410b, 410c)) at the position of the ROI (701) specified by the movement of the lens (211) (e.g., adjusting to the focus of the Far subject (711) or adjusting to the focus of the Near subject (713)) is misaligned (e.g., L1-R1>0, L2-R2>0, L3-R3>0 or L1-R1<0, L2-R2<0, L3-R3<0). The image signal processor (260) can determine that the shading deviation between sub-photodiodes (e.g., sub-pixels) of the ROI (701) increases (e.g., the deviation value between L1 and R1 in the ROI (701) is greater than a threshold value and / or the deviation value between L2 and R2 is greater than a threshold value) as the CRA alignment of the unit pixels (e.g., the unit photodiodes (410a, 410b, 410c)) at the location of the designated ROI (701) is misaligned.

[0122] According to one embodiment, the image signal processor (260) can determine that the lens CRA (721) changes as the lens (211) moves by a focus adjustment operation, a zoom operation, or an image shake correction operation during imaging. When the image signal processor (260) determines that the lens CRA (721) changes by the focus adjustment operation, the zoom operation, or the image shake correction operation, the image signal processor (260) can perform a sensor shift operation that changes the optical system, i.e., changes the position of the image sensor (230), using a sensor shift method as illustrated in FIG. 8, thereby minimizing the shading deviation caused by the misalignment of the sensor CRA (723) and the lens CRA (721) in the ROI (701) (e.g., so that the shading deviation becomes less than a specified threshold value or minimum value). For example, when the alignment of the lens CRA (721) in the designated ROI (701) is misaligned due to the position of the lens (211) being focused on a far subject, the image signal processor (260) can correct the shading deviation (e.g., shading deviation between sub-pixels) that occurred in the designated ROI (701) by performing a first sensor shift (801) (e.g., moving the image sensor (230)) operation in a direction (e.g., a first direction) that aligns the sensor CRA (723) of the unit photodiodes (410a, 410b, and 410c) of the image sensor (230) with the lens CRA (721) of the lens (211). The image signal processor (260) can obtain image data of the ROI (811) in which the shading deviation (e.g., shading deviation between sub-pixels) is corrected through the first sensor shift (801).According to one embodiment, when the alignment of the lens CRA (721) in the designated ROI (701) is misaligned due to focusing on a near subject, the image signal processor (260) can compensate for the shading deviation that occurred in the designated ROI (701) by performing a second sensor shift (803) (e.g., moving the image sensor (230)) in a second direction to align the sensor CRA (723) of the unit photodiodes (410a, 410b, and 410c) of the image sensor (230) with the lens CRA (721) of the lens (211) through the sensor shift (803). The image signal processor (260) can obtain image data of the ROI (813) in which the shading deviation is compensated through the second sensor shift (803). Through this shading correction, the image signal processor (260) can reduce the loss of dynamic range in the image and improve noise.

[0123] According to one embodiment, when the image signal processor (260) determines that the lens CRA (721) is changed by a focus adjustment operation, a zoom operation, or a hand shake correction operation, as illustrated in FIG. 9, the image signal processor (260) changes the optical system by using a lens shift method, that is, changes (moves) the position of the lens (211) in a direction to match the sensor CRA (723) of the unit photodiodes (410a, 410b, and 410c) and the lens CRA (721) of the lens (211), thereby correcting the position (e.g., coordinates) of the ROI (701), thereby minimizing the shading deviation that occurred in the ROI (701) (e.g., so that the shading deviation becomes less than a specified threshold value or minimum value). For example, the image signal processor (260) can correct the shading deviation that occurred in the designated ROI (701) by changing the position of the lens (211) in a first direction that matches the sensor CRA (723) and the lens CRA (721) using OIS even if the position of the lens (211) is adjusted to the focus of the Far subject and the CRA alignment is misaligned in the designated ROI (701) through the first lens shift (901). The image signal processor (260) can obtain image data of the ROI (911) in which the shading deviation is corrected through the first lens shift (901). According to one embodiment, when focusing on a near subject, the image signal processor (260) can correct the shading deviation for a designated ROI (701) by changing the position of the lens (211) through a second lens shift (903) that changes the position of the lens (211) in a second direction to align the sensor CRA (723) and the lens CRA (721) using OIS. The image signal processor (260) can obtain image data of the ROI (913) with the shading deviation corrected through the first lens shift (901). Through this shading correction, the image signal processor (260) can reduce the loss of dynamic range and improve noise in the image.

[0124] As such, in one embodiment, the main components of the electronic device have been described through the electronic device (101) of FIGS. 1 and 2. However, in various embodiments, not all of the components illustrated through FIGS. 1 and 2 are essential components, and the electronic device (101) may be implemented with more components than the illustrated components, or with fewer components. In addition, the positions of the main components of the electronic device (101) described above through FIGS. 1 and 2 may be changed according to various embodiments.

[0125] The present disclosure provides an electronic device, method, and non-transitory storage medium for capturing images, which correct shading caused by alignment errors of an image sensor and a lens due to changes in the CRA of the lens when capturing images.

[0126] According to one embodiment, an electronic device (e.g., electronic device (101) of FIGS. 1 and 2) may include a display (e.g., display module (160) of FIG. 1, display (160) of FIG. 3), a camera assembly including a camera circuit (e.g., camera module (180) of FIG. 1, camera circuits (180, 200) of FIG. 2), at least one processor including a processing circuit (e.g., processor (120) of FIG. 1, image signal processor (260) of FIGS. 2 and 3), and a memory storing instructions (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2).

[0127] According to one embodiment, the camera assembly may include a lens (e.g., lens assembly (210) of FIG. 2, lens (211) of FIG. 3), an image sensor (e.g., image sensor (230) of FIGS. 2 and 3) configured to provide an electrical signal corresponding to light received through the lens, and an image stabilizer (e.g., image stabilizer (240) of FIGS. 2 and 3) including an actuator configured to move at least one of the lens or the image sensor to stabilize the image.

[0128] According to one embodiment, the image sensor includes a plurality of unit photodiodes (e.g., unit photodiodes (410) of FIG. 4A) including a plurality of sub-photodiodes (e.g., sub-photodiodes (411, 412) of FIG. 4A) and a plurality of micro lenses corresponding to each of the plurality of unit photodiodes, and the plurality of sub-photodiodes included in one unit photodiode may correspond to one micro lens.

[0129] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to obtain an image corresponding to a subject using the image sensor, obtain pre-specified shading correction data for shading correction of a designated first region of interest (ROI) within the image from the memory, the shading correction data including position information in which an angular range (chief ray angle) of the image sensor or an angular range (chief ray angle) of the lens is set to match a designated angular range, and determine a movement position of the image sensor based on the shading correction data to correct a shading deviation between first sub-photodiodes corresponding to each of a plurality of first micro lenses arranged in a first region corresponding to the first ROI in an image height of the image sensor, and control the actuator to move the position of the image sensor based on the movement position, and obtain first image data of the first ROI in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

[0130] According to one embodiment, the position information included in the shade correction data may include at least one of position information of the image sensor controlled by the actuator or position information of the lens, wherein the shade deviation between the plurality of sub-photodiodes is identified as being less than a specified minimum value.

[0131] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, before acquiring the image, divide regions of a specified size from the image sensor, identify at least one of position information of the image sensor or position information of the lens in each of the divided regions, set the shading correction data including at least one of position information of the image sensor or position information of the lens, and store the shading correction data in the memory.

[0132] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: set the specified angular range such that the shading deviation is less than the minimum value; and may be an incident angle of light such that the angular range of the lens and the angular range of the image sensor are aligned so as not to misalign.

[0133] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a target movement position of the lens based on the shading correction data, and to control the actuator to move the lens about a designated axis based on the target movement position of the lens.

[0134] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to: identify, when receiving the image while performing a hand shake correction, focus adjustment or zoom operation, that an incoming angle of light received through the lens and the plurality of micro lenses is outside the specified angular range; and control the actuator to move the image sensor in a direction such that the angular range of the lens and the angular range of the image sensor are aligned without being misaligned based on the movement position so as to correct the shading deviation that occurs in the first ROI as the incoming angle of light is outside the specified angular range.

[0135] According to one embodiment, an electronic device (e.g., electronic device (101) of FIGS. 1 and 2) may include a display (e.g., display module (160) of FIG. 1, display (160) of FIG. 3), a camera assembly including a camera circuit (e.g., camera module (180) of FIG. 1, camera circuit (200) of FIG. 2), at least one processor including a processing circuit (e.g., processor (120) of FIG. 1, image signal processor (260) of FIGS. 2 and 3), and a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2)) that stores instructions.

[0136] According to one embodiment, the camera assembly may include a lens (e.g., lens assembly (210) of FIG. 2, lens (211) of FIG. 3), an image sensor (e.g., image sensor (230) of FIGS. 2 and 3) configured to provide an electrical signal corresponding to light received through the lens, and an image stabilizer (e.g., image stabilizer (240) of FIGS. 2 and 3) including an actuator configured to move at least one of the lens or the image sensor to stabilize the image.

[0137] According to one embodiment, the image sensor includes a plurality of unit photodiodes (e.g., unit photodiodes (410) of FIG. 4A) including a plurality of sub-photodiodes (e.g., sub-photodiodes (411, 412) of FIG. 4A) and a plurality of micro lenses corresponding to each of the plurality of unit photodiodes, and the plurality of sub-photodiodes included in one unit photodiode may correspond to one micro lens.

[0138] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to obtain an image corresponding to a subject using the image sensor, obtain pre-specified shading correction data for shading correction of a designated first region of interest (ROI) within the image from the memory, and the shading correction data includes position information in which an angular range (chief ray angle) of the image sensor or an angular range (chief ray angle) of the lens is set to match a designated angular range, and determine a target movement position of the lens based on the shading correction data to correct shading deviation between first sub-photodiodes corresponding to each of a plurality of first micro lenses arranged in a first area corresponding to the first ROI in the image sensor, and control the actuator to move the position of the lens based on the target movement position of the lens, and obtain second image data of the first ROI in which the shading deviation is corrected by moving the position of the lens using the image sensor.

[0139] According to one embodiment, the position information included in the shade correction data may include at least one of position information of the image sensor controlled by the actuator or position information of the lens, wherein the shade deviation between the plurality of sub-photodiodes is identified as being less than the specified minimum value.

[0140] According to one embodiment, the specified angular range may be an incident angle of light that is set so that the shading deviation is less than a specified minimum value and is aligned so that the angular range of the lens and the angular range of the image sensor do not misalign.

[0141] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to: identify, when receiving the image while performing a hand shake operation, a focus adjustment or a zoom operation, that an incoming angle of light received through the lens and the plurality of micro lenses is outside the specified angular range; and control the actuator to move the lens in a direction such that the angular range of the lens and the angular range of the image sensor are aligned without being misaligned, based on the target movement position of the lens, so as to correct the shading deviation that occurs in the first ROI as the incoming angle of light is outside the specified angular range.

[0142] FIG. 10 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment, and FIG. 11 is a diagram illustrating an example of shading correction according to an operating method in an electronic device according to an embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0143] Referring to FIG. 10, in operation 1001, an electronic device according to an embodiment (e.g., the electronic device (101) of FIGS. 1 and 2) may acquire image data corresponding to an object (e.g., a subject) through a lens (211) of a camera circuit (e.g., the camera module (180) of FIG. 1 and the camera circuit (200) of FIG. 2), and display an image (e.g., a preview image) for the acquired image data on a display (e.g., the display module (160) of FIG. 1 and the display (160) of FIG. 3).

[0144] In operation 1003, the electronic device can automatically designate a region of interest (ROI) in an image by a user input or specified conditions by an image signal processor of a camera circuit (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2).

[0145] In operation 1005, the electronic device can obtain shading compensation data in a specified ROI from a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2). The electronic device can identify shading compensation data corresponding to a height corresponding to the specified ROI. Here, the shading compensation data may be pre-specified before performing the operation and stored in the memory in the form of a lookup table. The shading compensation data may be set to compensate for shading (e.g., shading deviation between sub-photodiodes) that occurs in the specified ROI due to an incident angle of light entering a plurality of micro lenses included in the image sensor deviating from a specified angular range (CRA). The shading correction data may include information indicating a position where the shading deviation between sub-pixels (e.g., sub-photodiodes) included in each of a plurality of unit pixels (e.g., unit photodiodes) is minimum (e.g., the shading deviation is less than a specified threshold value or minimum value) (e.g., position information (coordinate values) where the variance of pixel gain measured to correct shading is minimum). The shading correction data may include position information (e.g., position information of the image sensor and / or position information of the lens) set so that the angular range of the image sensor (e.g., sensor CRA) or the angular range of the lens (e.g., lens CRA) matches the specified angular range. In operation 1007, the electronic device may determine a movement position (e.g., a position to move, a shift amount) for moving the image sensor (230) and / or the lens (211) based on the shading correction data. The electronic device may determine a position of a region of interest based on positions of regions distinguished from an image sensor (e.g., the image sensor (230) of FIG. 2) by an image signal processor, and may include at least one sensor (e.g., The position of the image sensor (230) and / or the lens (211) (e.g., the position of the actuator for image stabilization) can be determined using a gyro sensor or a Hall sensor.The electronic device can identify, as a target movement position, position information (e.g., position information where the variance of pixel gains measured to correct shading is minimized) where the shading deviation included in the shading correction data is minimal (e.g., the shading deviation is less than a specified threshold or minimum value). In operation 1007, the electronic device can identify a movement position for moving the image sensor (230) and / or the lens (211) based on the pre-specified shading correction data.

[0146] In operation 1009, the electronic device may control an image stabilizer (e.g., an actuator) included in the camera circuit to move the position of the image sensor and / or the position of the lens based on the identified movement position. In operation 1009, when the electronic device moves the position of the image sensor (230), the electronic device may control the image stabilizer (e.g., an actuator) to move the position of the image sensor using a sensor shift method. In operation 1009, when the electronic device moves the position of the lens (211), the electronic device may control the image stabilizer (e.g., an actuator) to move the lens (211) to the identified target movement position of the lens (211) using a lens shift method of the OIS. According to one embodiment, the electronic device may correct the position of the specified ROI as the image sensor and / or the lens is moved.

[0147] In operation 1011, the electronic device can acquire image data of an ROI in which the shading deviation is corrected (e.g., the shading is reduced) as the CRA alignment is corrected as the image sensor and / or the lens is moved. For example, if shading correction is not performed in a designated ROI, the ROI may have a large shading deviation (e.g., green standard deviation: 0.0588) between sub-pixels (e.g., sub-photodiodes) included in unit pixels, as in the pre-correction ROI image (1101) of FIG. 11. If shading correction is performed in the designated ROI through the operation method of FIG. 10 described above, the ROI may have a reduced shading deviation (e.g., green standard deviation) (e.g., green standard deviation: 0.0233) between sub-pixels (e.g., sub-photodiodes) included in unit pixels, as in the post-correction ROI image (1103) of FIG. 11.

[0148] An electronic device according to one embodiment can move the position of an image sensor (e.g., the image sensor (230) of FIGS. 2 and 3) and / or a lens (e.g., the lens (211) of FIG. 3) based on pre-specified shading correction data in the same manner as the operating method of FIG. 10 described above, and can obtain shading-corrected ROI image data according to the movement of the position of the image sensor and / or the lens.

[0149] According to one embodiment, when shading correction is completed in an image through the operation method of FIG. 10 as described above, the electronic device performs an image preprocessing operation and then captures an image of a subject with improved quality according to a shooting request (e.g., pressing a shooting button). The electronic device (101) can display the captured image (e.g., a final captured image or a corrected image) on the display (160) or store it in the memory (130).

[0150] According to one embodiment, the electronic device may preset (e.g., designate, measure) shading correction data for all areas distinguished from the image sensor for shading correction before performing the operation method of FIG. 10 as described above, and store the preset shading correction data in the memory (250) (or the memory (130) of FIG. 1) in the form of a lookup table. According to one embodiment, in order to preset the shading correction data, the electronic device may control the image stabilizer so that the image sensor and / or the lens are accurately positioned at a designated position (e.g., a position designated during design). According to one embodiment, the electronic device may identify an image of an area in which the shading deviation or the dispersion of pixel gain is minimized for each area, and confirm the position of an actuator corresponding to the image of the identified area. The electronic device may store shading correction data including the position of the actuator for CRA alignment of each area (e.g., position information for which the dispersion of pixel gain measured for shading correction is minimized) in the memory in the form of a lookup table. Here, the shading correction data may further include other information related to shading correction in addition to the position information where the shading deviation or the dispersion of pixel gain is minimized. For example, since the lens (211) may have a different viewing angle (e.g., CRA) depending on the focusing operation (AF operation), the electronic device may set two lookup tables (LUTs) for the subject of a flat light source and two positions (Farthest, Nearest) of the AF operation, and perform an appropriate fitting operation to correspond to the CRA change at all focus (AF) positions.

[0151] An electronic device according to one embodiment, when remosaicing an image (e.g., raw image data), can perform shading correction by using a remosaic zoom operation to designate a cropped image as an ROI and using the operation method of FIG. 10 described above.

[0152] Fig. 12 is a diagram illustrating an example of shading correction according to one embodiment.

[0153] An electronic device according to one embodiment may perform a shading correction operation while performing optical image stabilization (OIS). When the electronic device performs the shading correction operation while performing OIS, as illustrated in FIG. 12, the range of lens movement for the OIS of the image stabilizer may be limited as the lens moves for shading correction. Accordingly, the electronic device may perform shading correction while reducing the range of motion (e.g., margin) of the OIS. As illustrated in FIG. 12, the lens may be moved in all directions by a specified range in a center ROI or in a state where shading correction is not performed. Accordingly, a uniform shake correction effect may be obtained in all directions in a state where the center ROI or shading correction is not performed.

[0154] An electronic device according to one embodiment can adjust the lens position using OIS when light is received at an angle outside the specified range (CRA) of the lens, if flare occurs throughout the image.

[0155] An electronic device according to one embodiment may only perform shade compensation operations when OIS compensation is not performed.

[0156] An electronic device according to one embodiment may perform only a shade correction operation while performing a focus operation or a zoom operation (e.g., digital zoom) without performing OIS compensation.

[0157] According to one embodiment, an operating method in an electronic device (e.g., an electronic device (101) of FIGS. 1 and 2) may include an operation of acquiring an image corresponding to a subject by using an image sensor (e.g., an image sensor (230) of FIGS. 2 and 3) included in a camera assembly of the electronic device (e.g., a camera assembly including a camera module (180) of FIG. 1 and a camera circuit (200) of FIG. 2).

[0158] According to one embodiment, the operating method may include an operation of acquiring pre-specified shading correction data for shading correction of a specified first region of interest (ROI) within the image from a memory of the electronic device (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2). According to one embodiment, the shading correction data may include position information in which an angular range (CRA: chief ray angle) of the image sensor or an angular range of the lens is set to match a specified angular range.

[0159] According to one embodiment, the operating method may include an operation of checking a movement position of the image sensor based on the shading correction data to correct a shading deviation between first sub-photodiodes (e.g., sub-photodiodes (411, 412) of FIG. 4A) corresponding to each of a plurality of first micro lenses (e.g., micro lenses (430) of FIG. 4A) arranged in a first region corresponding to the first ROI in the image sensor.

[0160] According to one embodiment, the operating method may include controlling an actuator included in an image stabilizer of the camera assembly (e.g., the image stabilizer (240) of FIGS. 2 and 3) to move a position of the image sensor based on the movement position.

[0161] According to one embodiment, the operating method may include an operation of obtaining first image data of a first ROI in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

[0162] According to one embodiment, the shading correction data may include at least one of position information of the image sensor controlled by the actuator or position information of the lens, wherein the shading deviation between the plurality of sub-photodiodes is identified as being less than a specified minimum value.

[0163] According to one embodiment, the method may further include, before acquiring the image, an operation of dividing the image sensor into regions of a specified size, an operation of identifying at least one of position information of the image sensor or position information of the lens in each of the divided regions, an operation of setting the shading correction data including at least one of position information of the image sensor or position information of the lens, and an operation of storing the shading correction data in the memory.

[0164] According to one embodiment, the specified angular range may be an incident angle of light that is set so that the shading deviation is less than the specified minimum value and is aligned so that the angular range of the lens and the angular range of the image sensor do not misalign.

[0165] According to one embodiment, the method may further include an operation of determining a movement position of the lens based on the shading correction data, and an operation of controlling the actuator to move the lens along a specified axis based on the target movement position of the lens.

[0166] According to one embodiment, the method may include an operation of controlling the actuator to move the image sensor in a direction in which the angular range of the lens and the angular range of the image sensor are aligned without being misaligned, based on the movement position, so as to correct the shading deviation that occurs in the first ROI as the image is received while performing a hand shake correction, focus adjustment or zoom operation, and to identify that an incoming angle of light received through the lens and the plurality of micro lenses is outside the specified angular range.

[0167] According to one embodiment, in a non-transitory storage medium storing one or more programs, the one or more programs, when executed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2 and FIG. 3) of an electronic device (e.g., the electronic device (101) of FIG. 1 and FIG. 2), cause the electronic device to: obtain an image corresponding to a subject using an image sensor (e.g., the image sensor (230) of FIG. 2 and FIG. 3) included in a camera assembly of the electronic device; obtain pre-specified shading correction data for shading correction of a first region of interest (ROI) specified in the image from a memory of the electronic device (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2); the shading correction data includes position information that is set such that an angular range (CRA: chief ray angle) of the image sensor or an angular range of the lens is matched to a specified angular range; and a first plurality of first regions corresponding to the first ROI are arranged in an image of the image sensor. The method may include commands for performing an operation of checking a movement position of the image sensor based on the shading correction data to correct a shading deviation between the first sub-photodiodes corresponding to each of the micro lenses, an operation of controlling an actuator included in an image stabilizer of the camera assembly (e.g., the image stabilizer (240) of FIGS. 2 and 3) to move the position of the image sensor based on the movement position, and an operation of acquiring first image data of a first ROI in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

[0168] According to one embodiment, the specified angular range may be an incident angle of light that is set so that the shading deviation is less than a specified minimum value and is aligned so that the angular range of the lens and the angular range of the image sensor do not misalign.

[0169] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor (120, 260) of the electronic device (101), cause the electronic device to perform an operation of confirming a target movement position of the lens based on the shading correction data and an operation of controlling the actuator to move the lens relative to a specified axis based on the target movement position of the lens.

[0170] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor (120, 260) of the electronic device (101), cause the electronic device to: identify, when receiving the image while performing a hand shake operation, a focus adjustment operation, or a zoom operation, that an incoming angle of light received through the lens and the plurality of micro lenses is outside the specified angular range; and control the actuator to move the lens in a direction in which the angular range of the lens and the angular range of the image sensor are aligned without being misaligned, based on the movement position, so as to correct the shading deviation that occurs in the first ROI as the incoming angle of light is outside the specified angular range.

[0171] According to one embodiment of the present document, an electronic device can minimize (e.g., so that the shading deviation is less than a specified minimum value) the shading deviation between sub-photodiodes (e.g., the shading deviation between pixels) by changing the optical system itself with a sensor shift or OIS when the CRA of the lens changes due to a focus adjustment operation, a zoom operation, or an optical image stabilization (OIS) operation, thereby improving the focus detection capability, and improving the optical / electrical image stabilization and subject tracking performance of a camera circuit. According to one embodiment of the present document, when the electronic device utilizes a phase detection pixel itself as an image pixel through remosaic, the quality degradation of the image due to the shading can be improved. In addition, various effects that can be directly or indirectly understood through the present document can be provided. The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

[0172] The embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content, and do not limit the scope of the technology described in this document. Therefore, the scope of this document should be interpreted to include all modifications or various embodiments based on the technical concepts of this document.

[0173] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0174] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0175] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0176] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0177] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0178] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display (160); A camera assembly including a camera circuit (180, 200); At least one processor (120, 260) comprising a processing circuit; and It includes memory (130, 250) for storing instructions, The above camera assembly: Lens (210, 211); An image sensor (230) configured to provide an electrical signal corresponding to light received through the lens; and An image stabilizer (240) comprising an actuator configured to move at least one of the lens or the image sensor for image stabilization, The above image sensor: a plurality of unit photodiodes including a plurality of sub-photodiodes; and It includes a plurality of micro lenses corresponding to each of the plurality of unit photodiodes, and the plurality of sub-photodiodes included in one unit photodiode correspond to one micro lens, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Using the above image sensor, an image corresponding to the subject is acquired, Pre-specified shading correction data is acquired from the memory for shading correction of a first region of interest (ROI) specified within the image, and the shading correction data includes position information set to match the chief ray angle of the image sensor or the chief ray angle of the lens to the specified angular range. In order to correct the shading deviation between the first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in the first area corresponding to the first area of ​​interest in the image sensor, the movement position of the image sensor is confirmed based on the shading correction data, Based on the above movement position, controlling the actuator to move the position of the image sensor, An electronic device that uses the image sensor to obtain first image data of a first region of interest in which the shading deviation between the first sub-photodiodes is corrected.

2. In paragraph 1, The position information included in the shade correction data is position information in which the shade deviation between the plurality of sub-photodiodes is specified to be less than the minimum value, An electronic device in which the position information included in the shade correction data includes at least one of position information of the image sensor controlled by the actuator or position information of the lens.

3. In the first or second paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device: Before acquiring the above image, areas of a specified size are distinguished from the image sensor, In each of the above-described areas, at least one of the position information of the image sensor or the position information of the lens is identified, Setting the shading correction data including at least one of the position information of the image sensor or the position information of the lens, An electronic device that stores the above shade correction data in the memory.

4. In any one of paragraphs 1 to 3, The above specified angular range is set so that the shade deviation is less than the minimum value, An electronic device wherein the above-mentioned specified angular range is an angle of light inflow at which the angular range of the lens and the angular range of the image sensor are aligned without misalignment.

5. In any one of paragraphs 1 to 4, when the instructions are individually or collectively executed by the at least one processor, the electronic device: Based on the above shading correction data, the movement position of the lens is confirmed, An electronic device that controls the actuator to move the lens about a specified axis based on the movement position of the lens.

6. In any one of paragraphs 1 to 5, when the instructions are individually or collectively executed by the at least one processor, the electronic device: When receiving the image while performing a hand shake correction operation, focus adjustment or zoom operation, it is determined that the angle of incidence of light received through the lens and the plurality of micro lenses is outside the specified angle range, An electronic device that controls the actuator to move the image sensor in a direction in which the angular range of the lens and the angular range of the image sensor are aligned without misalignment, based on the movement position of the image sensor, so as to correct the shading deviation that occurs in the first area of ​​interest as the incoming angle of the light deviates from the specified angular range.

7. In the operating method in the electronic device (101), An operation of acquiring an image corresponding to a subject using an image sensor (230) included in a camera assembly of the electronic device; An operation of acquiring pre-specified shading correction data from a memory (130, 250) of the electronic device for shading correction of a first region of interest (ROI) specified within the image, the shading correction data including position information set such that the angular range (CRA: chief ray angle) of the image sensor or the angular range of the lens matches the specified angular range; An operation of confirming the movement position of the image sensor based on the shading correction data to correct shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first area corresponding to the first area of ​​interest in the image sensor; An operation of controlling an actuator included in an image stabilizer (240) of the camera assembly to move the position of the image sensor based on the movement position; and A method comprising an operation of obtaining first image data of the first region of interest in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

8. In paragraph 7, The position information included in the shade correction data is position information in which the shade deviation between the plurality of sub-photodiodes is specified to be less than the minimum value, A method wherein the position information included in the shade correction data includes at least one of position information of the image sensor controlled by the actuator or position information of the lens.

9. In the 7th or 8th paragraph, the method comprises: an operation of dividing the image sensor into areas of a specified size before acquiring the image; An operation of identifying at least one of the position information of the image sensor or the position information of the lens in each of the above-described areas; An operation of setting the shading correction data including at least one of position information of the image sensor or position information of the lens; and A method further comprising an operation of storing the shade correction data in the memory.

10. In any one of paragraphs 7 to 9, The above specified angular range is set so that the shade deviation is less than the minimum value, A method wherein the above-mentioned specified angular range is an angle of light inflow at which the angular range of the lens and the angular range of the image sensor are aligned without misalignment.

11. In any one of the 7th to 10th clauses, the method, An operation of confirming the movement position of the lens based on the above shading correction data; and A method further comprising an action of controlling the actuator to move the lens about a specified axis based on the movement position of the lens.

12. In any one of paragraphs 7 to 11, the method, An operation of identifying that an angle of incidence of light received through the lens and the plurality of micro lenses is outside the specified angle range when receiving the image while performing hand shake correction, focus adjustment or zoom operation; and A method comprising: controlling the actuator to move the image sensor in a direction in which the angular range of the lens and the angular range of the image sensor are aligned without misalignment, based on the movement position of the image sensor, so as to correct the shading deviation that occurs in the first region of interest as the incoming angle of the light deviates from the specified angular range.

13. In a non-transitory storage medium storing one or more programs, the one or more programs, when executed by at least one processor (120, 260) of an electronic device (101), cause the electronic device to: An operation of acquiring an image corresponding to a subject using an image sensor (230) included in a camera assembly of the electronic device; An operation of acquiring pre-specified shading correction data from a memory (130, 250) of the electronic device for shading correction of a first region of interest (ROI) specified within the image, the shading correction data including position information set such that the angular range (CRA: chief ray angle) of the image sensor or the angular range of the lens matches the specified angular range; An operation of confirming the movement position of the image sensor based on the shading correction data to correct shading deviation between first sub-photodiodes corresponding to each of the first plurality of micro lenses arranged in a first area corresponding to the first area of ​​interest in the image sensor; An operation of controlling an actuator included in an image stabilizer (240) of the camera assembly to move the position of the image sensor based on the movement position; and A non-transitory storage medium comprising commands for executing an operation of obtaining image data of the first region of interest in which the shading deviation between the first sub-photodiodes is corrected using the image sensor.

14. In paragraph 13, The above specified angular range is set so that the shade deviation is less than the specified minimum value, The above-mentioned specified angular range is an angle of light in which the angular range of the lens and the angular range of the image sensor are aligned without misalignment, and the one or more programs, when executed by at least one processor (120, 260) of the electronic device (101), cause the electronic device to: determine a target movement position of the lens based on the shading correction data; and A non-transitory storage medium comprising commands for executing an operation of controlling the actuator to move the lens about a specified axis based on the target movement position of the lens.

15. In any one of paragraphs 13 and 14, the one or more programs, when executed by at least one processor (120, 260) of the electronic device (101), cause the electronic device to: An operation of identifying that the angle of incidence of light received through the lens and the plurality of micro lenses is outside the specified angle range when receiving the image while performing a hand shake operation, focus adjustment or zoom operation; and A non-transitory storage medium comprising commands for controlling the actuator to move the lens in a direction in which the angular range of the lens and the angular range of the image sensor are aligned without misalignment, based on the movement position of the image sensor, so as to correct the shading deviation that occurs in the first region of interest as the incoming angle of the light deviates from the specified angular range.

Images