Electronic device performing image correction and operation method therefof

Abstract

An electronic device according to an embodiment may comprise: a motion sensor for outputting a signal for acquiring motion data corresponding to a movement of the electronic device; a camera module including an image sensor, and including an OIS module configured to perform optical image stabilization (OIS) on the basis of the motion data; a processor; and a memory for storing at least one instruction executable by the processor. Various other embodiments are possible.

Description

Electronic device for performing image correction and method of operation thereof

[0001] Various embodiments of the present disclosure relate to an electronic device for performing image correction and a method of operating the same.

[0002] Electronic devices, including cameras, can process digital data acquired through image sensors to provide improved images. A representative example of image enhancement is lens shading correction (LSC).

[0003] Lens shading compensation compensates for the optical characteristics of lenses, which cause the edges of an image or video to darken. Specifically, lens shading compensation applies different gain values ​​between the center and periphery of the image frame, thereby achieving overall balance within the image frame.

[0004] Additionally, the electronic device can perform optical image stabilization (OIS) to prevent image blur caused by hand shake. OIS is a method of reducing shaking by moving the lens assembly or image sensor included in the camera module.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0006] An electronic device according to one embodiment may include a motion sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device, a camera module including an optical image stabilization (OIS) module that includes an image sensor and is configured to perform optical image stabilization (OIS) based on the motion data, a processor, and a memory that stores at least one instruction executable by the processor. At least one instruction may cause the electronic device to obtain a raw image from the camera module during a movement of the electronic device. At least one instruction may obtain movement information of the OIS module based on the motion data and an LSC (lens shading correction) table for compensating for brightness of the raw image. The LSC table may include gain values ​​for each channel for compensating for brightness of each block of the raw image. At least one instruction may be configured to shift the LSC table based on movement information with respect to a center of the image sensor, and process brightness of each block of the raw image for each channel of the shifted LSC table to generate a corrected image.

[0007] An electronic device according to one embodiment may include a motion sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device, a camera module including a lens assembly, an image sensor, and a circuit board on which the image sensor is arranged, and a carrier that moves the circuit board in a direction crossing an optical axis to perform optical image stabilization (OIS) based on the motion data, a processor, and a memory that stores at least one instruction executable by the processor. At least one instruction may cause the electronic device to obtain a raw image from the camera module during the movement of the electronic device. At least one instruction may obtain movement information of the circuit board based on the motion data and an LSC table for compensating brightness of the raw image. The LSC table may include gain values ​​for each channel for compensating brightness of each block of the raw image. At least one instruction may be configured to shift the LSC table based on movement information with respect to the optical axis, and process brightness of each block of the raw image for each channel of the shifted LSC table to generate a corrected image.

[0008] A computer-readable non-transitory recording medium according to one embodiment of the present disclosure may store at least one command and / or instructions that, when executed, cause an electronic device to perform the method or operation of the electronic device described above.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram of a camera module according to various embodiments.

[0011] FIG. 3 is a drawing for explaining lens shading correction according to one embodiment.

[0012] FIG. 4 is a drawing for explaining the relative movement between a lens and an image sensor in a camera module according to one embodiment.

[0013] Figure 5 illustrates shading caused by relative movement between the lens and the image sensor.

[0014] FIG. 6 is a flowchart of an operating method when the OIS of an electronic device according to one embodiment is a lens shift type.

[0015] FIG. 7 is a diagram for explaining an LSC table according to one embodiment.

[0016] Figure 8 is a flowchart of an operating method according to one embodiment.

[0017] FIG. 9 and FIG. 10 are drawings for explaining operations 803 to 805 of FIG. 8 in more detail.

[0018] FIG. 11 and FIG. 12 are drawings for explaining operations 807 to 811 of FIG. 8 in more detail.

[0019] Fig. 13 is a drawing for explaining an extended LSC table, etc. according to one embodiment.

[0020] FIG. 14 is a flowchart of an operating method when the OIS of an electronic device according to one embodiment is a sensor shift method.

[0021] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terminology used herein is for the purpose of describing embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms, unless specifically stated otherwise.

[0023] The terms "comprises" and / or "comprising" as used in the specification do not exclude the presence or addition of one or more other components, operations, operations and / or elements.

[0024] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0047] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0048] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0049] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0050] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0051] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0052] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0053] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0054] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0055] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be a processor (120). It may be configured as a separate processor that is at least partially composed of, or operates independently of, the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) by the processor (120) as is or after undergoing additional image processing.

[0056] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0057] Meanwhile, according to one embodiment, the electronic device (101) can generate raw data for generating a raw image through the camera module (180). The camera module (180) according to one embodiment can include an image signal processor (260). The image signal processor (260) according to one embodiment can perform at least one image processing process on the raw image, which is raw data acquired through the image sensor (230). For example, the image signal processor (260) can perform lens shading correction (LSC) and image stabilization (e.g., video digital image stabilization (VDIS), electrical image stabilization (EIS), optical image stabilization (OIS)). The electronic device (101) can provide an image with balanced brightness by compensating for an insufficient amount of light in the periphery of the captured raw image through lens shading correction. In addition, the electronic device (101) can provide an image with less shaking by extracting a specific area from input frames with shaking among raw images as output frames by considering the movement of the electronic device (101) through VDIS correction. For example, the camera module (180) can include a method of correcting image shaking by performing software processing on the data output value of the image sensor (230). The camera module (180) can extract a motion vector based on the difference (different image) between frames of the image through VDIS, which is a digital shake correction, and increase the clarity through image processing. In addition, the camera module (180) can extract a motion vector based on the image through VDIS, and recognize the movement of the subject itself as shaking in addition to the shaking of the electronic device (100).Below, lens shading correction and VDIS correction are described in detail with reference to FIGS. 3 and 4, respectively.

[0058] FIG. 3 is a drawing for explaining lens shading correction according to one embodiment.

[0059] Lens shading is a phenomenon in which the periphery of an image / video becomes darker as it approaches the periphery due to the optical characteristics of the lens. Referring to Fig. 3, the periphery of the first image (A) is darker than the center due to a relatively small amount of light. This phenomenon may occur due to a decrease in the amount of light in the periphery because the diameter of the image circle of the lens is shorter than the diagonal length of the shooting screen. LSC (lens shading correction) can alleviate the brightness imbalance between the center and the periphery by dividing the image into blocks (pixels of the image sensor) and applying predetermined gain values ​​to compensate for the phenomenon of the periphery becoming darker as it approaches the periphery. LSC can be performed by an image signal processor (260), and can balance the brightness within the image by applying a higher gain value as the area gets farther from the center of the image (or the center of the lens). Referring to Fig. 3, the periphery of the second image (B) processed with LSC is confirmed to be brighter than the periphery of the first image (A).

[0060] An image signal processor (260) according to one embodiment divides a raw image into multiple blocks and stores gain values ​​for each block in a memory (250), so that the stored gain values ​​can be used when processing LSC. The gain values ​​generally increase from the center of the lens to the periphery. The gain values ​​can be assigned to each channel in the LSC table.

[0061] An image signal processor (260) according to one embodiment can perform LSC after converting a raw image into a format having a raw Bayer pattern. The Bayer pattern corresponds to a format of image data provided in a form in which G is cross-placed at 50% and R and B are each 25% according to human visual characteristics. For example, the image signal processor (260) can perform LSC after a remosaic operation in which the image signal processor (260) converts the raw image into a pattern (e.g., a Bayer pattern) on which a demosaic operation can be performed. The image signal processor (260) according to one embodiment can compensate for the brightness of the raw image by multiplying the pre-stored gain values ​​by the R, G, and B channels as they are.

[0062] An image signal processor (260) according to one embodiment may perform demosaicing after performing LSC. Demosaicing is a process of inferring monochrome pixels back into a three-color (RGB) image using information from surrounding pixels. For example, the image signal processor (260) may perform an operation of converting an image frame output from an image sensor (230) into an image (e.g., a three-color image) having color values ​​expressed similarly to those perceived by the human eye.

[0063] In one embodiment, the re-mosaic, LSC, and demosaic operations are not limited to being performed in the image signal processor (260). For example, the re-mosaic, LSC, and demosaic operations may be performed in the image sensor (230) or the processor (120) (e.g., an application processor (AP)).

[0064] FIG. 4 is a drawing for explaining relative movement between a lens and an image sensor in a camera module according to one embodiment, and FIG. 5 illustrates shading caused by relative movement between a lens and an image sensor.

[0065] OIS (optical image stabilizer), an optical image stabilization technology, is intended to prevent blur caused by hand shaking. By moving the lens in the X-axis, Y-axis, and Z-axis through the OIS module in the camera module, light reflected from a target (not shown) can be received at the center of the image sensor (230). As the lens moves, the path along which light gathers on the image sensor (230) and the amount of light received at each position of the image sensor (230) change, but a general LSC compensates for brightness only at a fixed position based on the image sensor (230).

[0066] Referring to FIGS. 4 and 5, the lens (or lens assembly, 210) can move relative to the image sensor (230) on the ZX plane, and the center of the lens and the center of the image sensor (230) can be located at different points on the ZX plane due to the OIS operation.

[0067] As shown in 5A of FIG. 5, when the OIS operation is off, the center of the lens and the center of the image sensor (230) coincide, so the shading distribution by the lens between the center and the periphery is symmetrical with respect to the Z-axis and the X-axis.

[0068] On the other hand, as shown in 5B or 5C of FIG. 5, when the OIS operation is on, the center of the lens and the center of the image sensor (230) do not coincide, so the shading distribution by the lens between the center and the periphery is not symmetrical but has a bias.

[0069] The OIS method may include a lens shifting method that fixes the image sensor (230) and moves the lens, and a sensor shifting method that fixes the lens and moves the image sensor (230). As described above, if there is relative movement between the lenses or the image sensors (230), the position where light is gathered on the image sensor (230) changes. In principle, LSC can compensate for the brightness of the image based on the center of the image sensor (230) regardless of the position of the lens. Therefore, if the LSC table is processed into the image without considering the relative movement between the lenses or the image sensors (230), a brightness difference between the center and the periphery may occur even if LSC is performed.

[0070] Meanwhile, the LSC table may be data that assigns gain values ​​to each location of the image based on shading profile data. The LSC table may be data that assigns gain values ​​for each channel corresponding to the location. The LSC table may have multiple channels arranged in rows and columns, and the location of each channel may correspond to the location of a block of the image. For example, a gain value assigned to one channel in the LSC table may be applied to a block of the image corresponding to the channel. The shading profile data may include a brightness distribution for each area within the image and a brightness value according to the block location. Since the brightness for each location in the image becomes darker from the center to the periphery, the gain value may have a relatively larger value from the center to the periphery. Here, the gain value may correspond to a correction value multiplied by the RGB data of the image and / or the Y data of YUV. In YUV, Y represents a luma component or brightness, and UV represents chrominance components or color information.

[0071] FIG. 6 is a flowchart (600) of an operating method when the OIS of an electronic device according to one embodiment is a lens shift method, and FIG. 7 is a drawing for explaining an LSC table according to one embodiment.

[0072] According to one embodiment, a processor (120, 260) can acquire a raw image (601) through an image sensor (230). The raw image has a characteristic in which the periphery of the image is darker than the center due to lens shading.

[0073] According to one embodiment, a processor (120, 260) divides a raw image into a plurality of blocks and stores gain values ​​corresponding to each block in a memory (130, 250), so that the stored gain values ​​can be used when processing LSC. The gain values ​​generally become larger as they move from the center of the lens to the periphery.

[0074] According to one embodiment, the processor (120, 260) may perform LSC after converting the raw image into a format having a raw Bayer pattern. The Bayer pattern corresponds to a format of image data provided in a form in which G is cross-placed to 50% and R and B are each 25% according to human visual characteristics. According to one embodiment, the processor (120, 260) may compensate for the brightness of the raw image by multiplying the R, G, and B data by pre-stored gain values. In addition, the processor (120, 260) according to one embodiment may compensate for the brightness of the raw image by multiplying the R, G, and B average values ​​by pre-stored gain values. In addition, the processor (120, 260) according to one embodiment may compensate for the brightness of the raw image by applying a gain value to Y data of YUV.

[0075] According to one embodiment, a processor (120, 260) can obtain movement information and an LSC table of an OIS module (603).

[0076] For example, movement information corresponds to data required for shifting (coordinate movement) of the LSC table. When the electronic device (101) takes a picture or a video using a camera, data on the amount of shaking can be transmitted from a motion sensor (not shown) within the electronic device (101) to the processor (120, 260). The processor (120, 260) can provide a target signal to the OIS module (actuator) to move the lens in a direction opposite to the shaking. The processor (120, 260) can perform OIS correction by controlling the OIS module to move the lens (lens assembly, 210) relative to the image sensor (230) based on the target signal.

[0077] According to one embodiment, the motion sensor provides motion data to the processor (120, 260) so that the processor (120, 260) processes the motion data to obtain a motion level. The motion level is a measure of the movement of the electronic device (101). The motion data is data on the amount of physical movement of the electronic device (101) and can be obtained through a signal output from the motion sensor. Specifically, the motion data can include at least one of data processed from a signal generated by an acceleration sensor, a gyro sensor (gyroscope), a magnetic sensor, or a Hall sensor.

[0078] According to one embodiment, the motion sensor may include at least one of an acceleration sensor, a gyro sensor (gyroscope), a magnetic sensor, or a Hall sensor. For example, the acceleration sensor may measure acceleration acting along three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101). As another example, the gyro sensor may measure a rotation angle or inclination about three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101). However, the above sensors are exemplary, and the motion sensor may further include at least one other type of sensor.

[0079] According to one embodiment, the motion sensor may include at least one of an acceleration sensor that outputs a signal for acquiring acceleration data or a gyro sensor that outputs a signal for acquiring angular velocity data. The processor may acquire motion data corresponding to the movement of the electronic device (101) based on at least one of the acceleration data or the angular velocity data.

[0080] The LSC table is data for reducing the brightness difference between the center and periphery of an image. The LSC table consists of multiple channels, each of which can be assigned a gain value. These gain values ​​are reflected in an MxN matrix, and the row-column location of each component can correspond to the location of a block within the image.

[0081] A gain value belonging to an LSC table according to one embodiment can be determined based on auto exposure (AE) and auto white balance (AWB) information.

[0082] According to one embodiment, a processor (120, 260) can shift an LSC table based on the center of an image sensor (230) (605).

[0083] When an electronic device (101) captures a scene for a specific field of view, the path along which light gathers may change depending on the movement of the lens position. Referring to FIG. 7, as the central axis of the lens moves (O -> O”) due to the OIS operation, the shading distribution for the region of interest captured by the image sensor (e.g., the region corresponding to 12 in FIG. 7) changes. If the LSC table applied to the region of interest is always fixed to the image sensor (230), the brightness deviation between each block within the region of interest may not be resolved. Therefore, the present disclosure shifts the LSC table in consideration of the lens movement due to the OIS operation (e.g., the region corresponding to 11 in FIG. 7), thereby allowing an appropriate gain value for each block to be reflected. Therefore, the electronic device (101) according to the disclosure can minimize the brightness deviation of the acquired image even if the path along which light gathers is applied differently due to the movement of the lens.

[0084] According to one embodiment, the OIS module may include a lens assembly (210) and an actuator (e.g., an image stabilizer (240) of FIG. 2) that moves the lens assembly (210) in a direction intersecting the optical axis. The processor (120, 260) may shift the LSC table based on movement information of the OIS module. The movement information includes a first displacement value and a second displacement value, which are center coordinate values ​​of the lens assembly (210) with the center of the image sensor (230) as an origin. The processor may shift the LSC table based on the first displacement value and the second displacement value.

[0085] According to one embodiment, a processor (120, 260) can generate a corrected image by applying a shifted LSC table to a raw image (607). As the shifted LSC table is applied to the raw image, brightness deviation of the image can be minimized, thereby improving image quality stability.

[0086] In one embodiment, blocks of a raw image may be assigned R (red) values, G (green) values, and B (blue) values, respectively. The processor may compensate for the brightness of the raw image by applying gain values ​​included in the LSC table shifted to the R (red) values, G (green) values, and B (blue).

[0087] Meanwhile, the LSC table applied to the raw image can have a series of additional processing steps based on the number of channels in the LSC table relative to the size of the image sensor. For example, the size ratio of the LSC table can match the size (number of pixels) ratio of the image sensor. Therefore, by matching the number of channels in the LSC table to the number of blocks in the image, a gain value can be applied to all data for each block in the image. For example, when the number of channels in the LSC table is MxN, the number of blocks in the image can also be MxN.

[0088] Figure 8 is a flowchart (800) of an operating method according to one embodiment.

[0089] Referring to FIG. 8, a processor according to one embodiment may extend an LSC table based on a comparison (801) of the number of channels in the LSC table and the number of blocks in the raw image.

[0090] If the number of channels in the LSC table is greater than the number of blocks in the raw image, the processor according to one embodiment may determine some of the plurality of channels included in the LSC table based on the movement information of the OIS module (803), and may generate a corrected image using some of the plurality of channels (805).

[0091] For example, referring to FIG. 9, the image sensor (230) has 4000x2250 cells, and when the size of the Bayer pattern block is set to 200x150, the number of blocks in the image becomes 20x15. At this time, since the number of channels of the LSC table is 25x20, which is greater than the number of blocks in the image (9A of FIG. 9), the LSC table is sufficient to apply gain values ​​to all RGB values ​​or Y values ​​corresponding to all blocks in the image. Meanwhile, the LSC table may be composed of multiple LSC tables having different gain values ​​depending on the brightness determined by the AE (auto exposure) algorithm and the color temperature determined by the AWB (auto white balance) algorithm, as illustrated in 9B of FIG. 9. This will be described later with reference to FIG. 13.

[0092] Referring to FIG. 10, when the OIS operation is not performed, i.e., when the center axes of the lens assembly (210) and the image sensor (230) are aligned, a corrected image can be generated by applying only a part (L2) of the LSC table (L1) to the raw image (10A-1, 10B-1, 10C-1).

[0093] When the OIS operation is being performed, i.e., when the electronic device (101) detects hand shake and the center axes between the lens assembly (210) and the image sensor (230) do not match, only a part (L2) of the shifted LSC table (L1) can be applied to the raw image to generate a corrected image (10A-2, 10B-2, 10C-2).

[0094] According to one embodiment, a processor (120, 260) may generate a corrected image using some of the plurality of channels included in the LSC table when the number of channels in the LSC table is greater than the number of blocks in the raw image. In this case, the range of some of the plurality of channels may be determined based on movement information of the OIS module.

[0095] Meanwhile, when the number of channels in the LSC table is equal to the number of blocks in the raw image (or when the number of channels in the LSC table is less than the number of blocks in the raw image), the processor (120, 260) according to one embodiment may add channels from the outermost part of the LSC table and extend the LSC table (807), and generate a corrected image using a part of the extended LSC table (811). For example, when the number of channels in the LSC table is 20x15, if one channel is added from each of the outermost channels, the number of channels in the extended LSC table becomes 22x17. Additionally, when the processor according to one embodiment extends the LSC table by adding channels to the outermost part of the LSC table, the processor may also perform an operation (809) of storing the extended LSC table for use in the LSC process in the future.

[0096] For example, referring to FIG. 11, when the image sensor (230) has 4000x3000 cells and the Bayer pattern block size is set to 200x200, the number of blocks in the image becomes 20x15. At this time, since the number of channels (e.g., 20x15) of the LSC table and the number of blocks of the image are the same, if the LSC table moves out of the region of interest (S) due to the OIS operation, the gain value cannot be applied to all RGB values ​​or Y values ​​of the blocks corresponding to the entire region of interest (S) (11A of FIG. 11). Therefore, the processor according to one embodiment can expand the LSC table so that the brightness of all blocks within the region of interest (S) can be adjusted by adding the number of channels for the LSC table when the number of channels of the LSC table is the same as the number of blocks of the raw image. Meanwhile, the LSC table may be composed of multiple LSC tables with different gain values ​​depending on the brightness determined by the AE (auto exposure) algorithm and the color temperature determined by the AWB (auto white balance) algorithm, as illustrated in 11B of Fig. 11. This will be described later with reference to Fig. 13.

[0097] Referring to FIG. 12, when no operation is performed, i.e., when the center axes of the lens assembly (210) and the image sensor (230) are aligned, the entire LSC table (L1) can be applied to the raw image to generate a corrected image (12A-1, 12B-1, 12C-1, 12D-1).

[0098] If an OIS operation is performed, a part of the LSC table (L1) cannot provide gain values ​​for all blocks in the region of interest (S), so the processor (120, 260) according to one embodiment can generate an extended LSC table (L1+) by adding channels from the outermost part to the LSC table. Then, the processor according to one embodiment can generate a corrected image by applying only a part (L2) of the extended LSC table (L1+) to the raw image (12A-2, 12B-2, 12C-2, 12D-2).

[0099] According to one embodiment, the processor (120, 260) may extend the LSC table by adding channels from the outermost part of the LSC table when the number of channels of the LSC table is equal to the number of blocks of the raw image. At this time, the gain value belonging to the channel added adjacent to the outermost part may have a value equal to or greater than the gain value of the adjacent channel (considering the characteristic that the amount of light decreases as it goes toward the outer edge of the lens). According to one embodiment, the processor (120, 260) may generate a corrected image using some of the plurality of channels included in the extended LSC table.

[0100] According to one embodiment, a processor (120, 260) can generate a FULL LSC table based on the gain value of an added channel after shifting the LSC table based on the movement information of the OIS module, and store and reuse the generated FULL LSC table.

[0101] Fig. 13 is a drawing for explaining an extended LSC table, etc. according to one embodiment.

[0102] Basically, the LSC table can have relatively low gain values ​​toward the channel corresponding to the center and relatively high gain values ​​toward the channel corresponding to the periphery, reflecting the geometric and optical characteristics of the lens.

[0103] According to one embodiment, the gain value in the LSC table may be determined based on the ambient illumination of the electronic device (101). As the amount of light received by the image sensor (230) increases, the brightness difference between the center and the periphery of the image decreases, and thus, the standard deviation for the gain value may decrease as one moves to the right in the table according to FIG. 13.

[0104] According to one embodiment, the gain value in the LSC table may be determined based on the brightness determined by the AE (auto exposure) algorithm and the color temperature determined by the AWB (auto white balance) algorithm. For example, based on the AE algorithm, an LSC table including gain values ​​with smaller standard deviations as the amount of IR (infrared) decreases may be applied to image correction. Additionally, based on the AWB algorithm, an LSC table including gain values ​​with smaller standard deviations as the color temperature increases may be applied to image correction.

[0105] According to one embodiment, a processor may distinguish between outdoors and indoors based on illuminance, and if it is determined to be outdoors, an outdoor LSC table (LSC table 4) may be applied, and if it is determined to be indoors, a gain value of the periphery may be adjusted based on color temperature and / or IR amount.

[0106] Each of the FULL LSC tables (Full LSC tables 1 to 4) according to one embodiment may further include per-channel gain values ​​estimated from gain values ​​in the periphery of the LSC table.

[0107] An electronic device (101) according to one embodiment may further include an illuminance sensor that outputs a signal for measuring illuminance around the electronic device. A processor according to one embodiment may apply weights to gain values ​​for each channel of an LSC table based on illuminance.

[0108] According to one embodiment, a processor may select one LSC table from among a plurality of pre-stored LSC tables, and if the illuminance exceeds a predetermined illuminance, select one LSC table from among the plurality of LSC tables and apply it to the raw image. Here, the selected LSC table may correspond to an outdoor LSC table (e.g., LSC table 4, Full LSC table 4) in FIG. 13.

[0109] According to one embodiment, a processor may select one LSC table from among a plurality of pre-stored LSC tables, and if the illuminance is below a predetermined illuminance, select one LSC table from among the plurality of LSC tables and apply the selected LSC table to the raw image. Here, the selected LSC table may correspond to an indoor LSC table (e.g., LSC tables 1 to 3, Full LSC tables 1 to 3) in FIG. 13.

[0110] According to one embodiment, the channel-specific gain values ​​of the LSC table can be determined based on the color temperature and the amount of infrared light.

[0111] FIG. 14 is a flowchart (1400) of an operating method when the OIS of an electronic device according to one embodiment is a sensor shift method.

[0112] According to one embodiment, a processor (120, 260) can acquire a raw image (1401) through an image sensor (230). The raw image has a characteristic in which the periphery of the image is darker than the center due to lens shading.

[0113] According to one embodiment, a processor (120, 260) divides a raw image into a plurality of blocks and stores gain values ​​corresponding to each block in a memory (130, 250), so that the stored gain values ​​can be used when processing LSC. The gain values ​​generally become larger as they move from the center of the lens to the periphery.

[0114] According to one embodiment, the processor (120, 260) may perform LSC after converting the raw image into a format having a raw Bayer pattern. The Bayer pattern corresponds to a format of image data provided in a form in which G is cross-placed to 50% and R and B are each 25% according to human visual characteristics. According to one embodiment, the processor (120, 260) may compensate for the brightness of the raw image by multiplying the R, G, and B data by pre-stored gain values. In addition, the processor (120, 260) according to one embodiment may compensate for the brightness of the raw image by multiplying the R, G, and B average values ​​by pre-stored gain values. In addition, the processor (120, 260) according to one embodiment may compensate for the brightness of the raw image by applying a gain value to Y data of YUV.

[0115] A processor (120, 260) according to one embodiment can obtain movement information and an LSC table of a circuit board (e.g., a board including / moving an image sensor (230)) (1403).

[0116] The present embodiment may be a sensor shifting method in which OIS is performed by moving the image sensor (230) in a direction substantially intersecting the optical axis, rather than a lens shift method in which the lens assembly (210) moves relative to the image sensor (230).

[0117] For example, movement information corresponds to data required for shifting (coordinate movement) of the LSC table. When the electronic device (101) takes a picture or a video using a camera, data on the amount of shaking can be transmitted from a motion sensor (not shown) within the electronic device (101) to the processor (120, 260). The processor (120, 260) can provide a target signal to the OIS module (actuator) to move a circuit board that moves the image sensor in a direction opposite to the shaking. The processor (120, 260) can control the circuit board to move the image sensor (230) based on the target signal to perform OIS correction.

[0118] According to one embodiment, the motion sensor provides motion data to the processor (120, 260) so that the processor (120, 260) processes the motion data to obtain a motion level. The motion level is a measure of the movement of the electronic device (101). The motion data is data on the amount of physical movement of the electronic device (101) and can be obtained through a signal output from the motion sensor. Specifically, the motion data can include at least one of data processed from a signal generated by an acceleration sensor, a gyro sensor (gyroscope), a magnetic sensor, or a Hall sensor.

[0119] According to one embodiment, the motion sensor may include at least one of an acceleration sensor, a gyro sensor (gyroscope), a magnetic sensor, or a Hall sensor. For example, the acceleration sensor may measure acceleration acting along three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101). As another example, the gyro sensor may measure a rotation angle or inclination about three axes (e.g., X-axis, Y-axis, or Z-axis) of the electronic device (101). However, the above sensors are exemplary, and the motion sensor may further include at least one other type of sensor.

[0120] According to one embodiment, the motion sensor may include at least one of an acceleration sensor that outputs a signal for acquiring acceleration data or a gyro sensor that outputs a signal for acquiring angular velocity data. The processor may acquire motion data corresponding to the movement of the electronic device (101) based on at least one of the acceleration data or the angular velocity data.

[0121] The LSC table is data for reducing the brightness difference between the center and periphery of an image. The LSC table consists of multiple channels, each of which can be assigned a gain value. These gain values ​​are reflected in an MxN matrix, and the row-column location of each component can correspond to the location of a block within the image.

[0122] A gain value belonging to an LSC table according to one embodiment can be determined based on auto exposure (AE) and auto white balance (AWB) information.

[0123] According to one embodiment, a processor (120, 260) can shift an LSC table based on the optical axis (1405).

[0124] According to one embodiment, a processor (120, 260) can generate a corrected image by applying a shifted LSC table to a raw image (1407). As the shifted LSC table is applied to the raw image, brightness deviation of the image can be minimized, thereby improving image quality stability.

[0125] In one embodiment, blocks of a raw image may be assigned R (red) values, G (green) values, and B (blue) values, respectively. The processor may compensate for the brightness of the raw image by applying gain values ​​included in the LSC table shifted to the R (red) values, G (green) values, and B (blue).

[0126] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0127] An electronic device (101) according to one embodiment may include a motion sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device (101), a camera module (180) that includes an image sensor (230) and an OIS module configured to perform optical image stabilization (OIS) based on the motion data, a processor (120), and a memory (130) that stores at least one instruction executable by the processor (120). At least one instruction may be configured to cause the electronic device (101) to obtain a raw image from the camera module (180) during movement of the electronic device (101), obtain movement information of an OIS module based on motion data and an LSC (lens shading correction) table for compensating brightness of the raw image, wherein the LSC table includes gain values ​​for each channel for compensating brightness of each block of the raw image, shift the LSC table based on the movement information with respect to the center of the image sensor (230), and process the brightness of each block of the raw image for each channel of the shifted LSC table to generate a corrected image.

[0128] An OIS module according to one embodiment may include a lens assembly and an actuator that moves the lens assembly in a direction intersecting an optical axis. The movement information according to one embodiment may include a first displacement value and a second displacement value, which are coordinate values ​​of a center of the lens assembly with the center of the image sensor as an origin. At least one instruction according to one embodiment may shift an LSC table based on the first displacement value and the second displacement value.

[0129] An electronic device according to one embodiment can generate a corrected image using some of the plurality of channels included in the LSC table when the number of channels in the LSC table is greater than the number of blocks of the raw image.

[0130] An electronic device according to one embodiment can extend an LSC table by adding channels from the outermost part of the LSC table when the number of channels in the LSC table is equal to the number of blocks of the raw image, and can generate a corrected image using some of the plurality of channels included in the extended LSC table.

[0131] An electronic device according to one embodiment may further include an illuminance sensor that outputs a signal for measuring illuminance around the electronic device. The electronic device according to one embodiment may apply weights to gain values ​​for each channel of an LSC table based on illuminance.

[0132] An electronic device according to one embodiment may select one LSC table from among a plurality of pre-stored LSC tables, and may select a first LSC table from among the plurality of LSC tables when the illuminance exceeds a predetermined illuminance.

[0133] An electronic device according to one embodiment may select one LSC table from among a plurality of pre-stored LSC tables, and if the illuminance is below a predetermined illuminance, select a second LSC table from among the plurality of LSC tables.

[0134] The channel-specific gain values ​​of the LSC table according to one embodiment can be determined based on the color temperature and the amount of infrared light.

[0135] A motion sensor according to one embodiment may include at least one of an acceleration sensor that outputs a signal for acquiring acceleration data and a gyro sensor that outputs a signal for acquiring angular velocity data. An electronic device according to one embodiment may acquire motion data corresponding to a movement of the electronic device based on at least one of the acceleration data and the angular velocity data.

[0136] According to one embodiment, blocks of a raw image may be assigned R (red) values, G (green) values, and B (blue) values, respectively. An electronic device according to one embodiment may compensate for the brightness of the raw image by applying a gain value to the R (red) values, G (green) values, and B (blue) values.

[0137] An electronic device (101) according to one embodiment may include a motion sensor that outputs a signal for obtaining motion data corresponding to a movement of the electronic device (101), a camera module (180) including a lens assembly (210), an image sensor (230), and a circuit board on which the image sensor (230) is arranged, and a carrier that moves the circuit board in a direction intersecting an optical axis to perform optical image stabilization (OIS) based on the motion data, a processor (120), and a memory (130) that stores at least one instruction executable by the processor (120). At least one instruction may be configured to cause the electronic device (101) to obtain a raw image from the camera module (180) during movement of the electronic device (101), obtain movement information of a circuit board based on motion data and an LSC table for compensating brightness of the raw image, wherein the LSC table includes gain values ​​for each channel for compensating brightness of each block of the raw image, shift the LSC table based on the movement information with respect to an optical axis, and process the brightness of each block of the raw image for each channel of the shifted LSC table to generate a corrected image.

[0138] The movement information according to one embodiment may include a first displacement value and a second displacement value, which are coordinate values ​​of the center of the image sensor with the center of the lens assembly as the origin. The electronic device according to one embodiment may shift the LSC table based on the first displacement value and the second displacement value.

[0139] An electronic device according to one embodiment can generate a corrected image using some of the plurality of channels included in the LSC table when the number of channels in the LSC table is greater than the number of blocks in the raw image.

[0140] An electronic device according to one embodiment can extend an LSC table by adding channels from the outermost part of the LSC table when the number of channels in the LSC table is equal to the number of blocks of the raw image, and can generate a corrected image using some of the plurality of channels included in the extended LSC table.

[0141] An electronic device according to one embodiment may further include an illuminance sensor that outputs a signal for measuring illuminance around the electronic device. The electronic device according to one embodiment may apply weights to gain values ​​for each channel of an LSC table based on illuminance.

[0142] An electronic device according to one embodiment may select one LSC table from among a plurality of pre-stored LSC tables, and may select a first LSC table from among the plurality of LSC tables when the illuminance exceeds a predetermined illuminance.

[0143] An electronic device according to one embodiment may select one LSC table from among a plurality of pre-stored LSC tables, and if the illuminance is below a predetermined illuminance, select a second LSC table from among the plurality of LSC tables.

[0144] The channel-specific gain values ​​of the LSC table according to one embodiment can be determined based on the color temperature and the amount of infrared light.

[0145] A motion sensor according to one embodiment may include at least one of an acceleration sensor that outputs a signal for acquiring acceleration data and a gyro sensor that outputs a signal for acquiring angular velocity data. An electronic device according to one embodiment may acquire motion data corresponding to a movement of the electronic device based on at least one of the acceleration data and the angular velocity data.

[0146] A motion sensor according to one embodiment may include at least one of an acceleration sensor that outputs a signal for acquiring acceleration data and a gyro sensor that outputs a signal for acquiring angular velocity data. An electronic device according to one embodiment may acquire motion data corresponding to a movement of the electronic device based on at least one of the acceleration data and the angular velocity data.

[0147] According to one embodiment, blocks of a raw image may be assigned R (red) values, G (green) values, and B (blue) values, respectively. An electronic device according to one embodiment may compensate for the brightness of the raw image by applying a gain value to the R (red) values, G (green) values, and B (blue) values.

[0148] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0149] As used herein, the term “if” will be understood to mean “when, upon,” “in response to deciding,” or “in response to detecting,” depending on the context. Similarly, “if it is decided to do,” or “if [the stated condition or event] is detected,” will optionally be understood to mean “upon deciding,” or “in response to deciding,” “upon detecting [the stated condition or event],” or “in response to detecting [the stated condition or event].”

[0150] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. A processing device (or processing circuit) may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0151] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0152] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0153] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0154] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, A motion sensor that outputs a signal for obtaining motion data corresponding to the movement of the electronic device; A camera module including an image sensor and an optical image stabilization (OIS) module configured to perform OIS based on the motion data; processor; a memory storing at least one instruction executable by the processor; At least one of the instructions above causes the electronic device to: Acquire raw images from the camera module during movement of the electronic device; Obtaining movement information of the OIS module based on the motion data and an LSC (lens shading correction) table for compensating for the brightness of the raw image, wherein the LSC table includes gain values ​​for each channel for compensating for the brightness of each block of the raw image, Shifting the LSC table based on the movement information with respect to the center of the image sensor, and An electronic device configured to process block-wise brightness of the raw image for each channel of the shifted LSC table to generate a corrected image.

2. In paragraph 1, The above OIS module, A lens assembly comprising an actuator for moving the lens assembly in a direction intersecting the optical axis; The above movement information is, Including a first displacement value and a second displacement value, which are coordinate values ​​of the center of the lens assembly with the center of the image sensor as the origin, At least one of the above instructions, An electronic device shifting the LSC table based on the first displacement value and the second displacement value.

3. In paragraph 2, At least one of the above instructions, If the number of channels in the above LSC table is greater than the number of blocks in the above raw image: An electronic device that generates the compensation image by using some of the plurality of channels included in the above LSC table.

4. In paragraph 2, At least one of the above instructions, If the number of channels in the above LSC table is equal to the number of blocks in the above raw image: An electronic device for extending the LSC table by adding channels from the outermost part of the LSC table, and generating the correction image by using some of the plurality of channels included in the extended LSC table.

5. In paragraph 1, Further comprising a light sensor for outputting a signal for measuring the light intensity around the electronic device; At least one of the above instructions, An electronic device that applies weights to channel-specific gain values ​​of the LSC table based on the above illuminance.

6. In paragraph 5, At least one of the above instructions, Select one LSC table from among multiple pre-stored LSC tables. An electronic device that selects a first LSC table among the plurality of LSC tables when the above illuminance exceeds a predetermined illuminance.

7. In paragraph 5, At least one of the above instructions, Select one LSC table from among multiple pre-stored LSC tables. An electronic device that selects a second LSC table among the plurality of LSC tables when the illuminance is less than or equal to a predetermined illuminance.

8. In paragraph 1, The gain values ​​for each channel in the above LSC table are: An electronic device that is determined based on color temperature and the amount of infrared light.

9. In paragraph 1, The above motion sensor, At least one of an acceleration sensor outputting a signal for acquiring acceleration data or a gyro sensor outputting a signal for acquiring angular velocity data is included, At least one of the above instructions, An electronic device that obtains motion data corresponding to movement of the electronic device based on at least one of the acceleration data and the angular velocity data.

10. In paragraph 1, The blocks of the above raw image are each assigned an R (red) value, a G (green) value, and a B (blue) value. At least one of the above instructions, An electronic device that compensates for the brightness of a raw image by applying a gain value to the R (red) value, the G (green) value, and the B (blue) value.

11. In electronic devices, A motion sensor that outputs a signal for obtaining motion data corresponding to the movement of the electronic device; A camera module comprising a lens assembly, an image sensor, and a circuit board on which the image sensor is arranged, and a carrier for moving the circuit board in a direction intersecting an optical axis to perform optical image stabilization (OIS) based on the motion data; processor; a memory storing at least one instruction executable by the processor; At least one of the instructions above causes the electronic device to: Acquire raw images from the camera module during movement of the electronic device; Obtaining movement information of the circuit board based on the motion data and an LSC table for compensating brightness of the raw image, wherein the LSC table includes gain values ​​for each channel for compensating brightness for each block of the raw image, Shifting the LSC table based on the movement information based on the optical axis, An electronic device configured to process block-wise brightness of the raw image for each channel of the shifted LSC table to generate a corrected image.

12. In paragraph 11, The above movement information is, Including a first displacement value and a second displacement value, which are coordinate values ​​of the center of the image sensor with the center of the lens assembly as the origin, At least one of the above instructions, An electronic device shifting the LSC table based on the first displacement value and the second displacement value.

13. In paragraph 12, At least one of the above instructions, If the number of channels in the above LSC table is greater than the number of blocks in the above raw image: An electronic device that generates the compensation image by using some of the plurality of channels included in the above LSC table.

14. In paragraph 12, At least one of the above instructions, If the number of channels in the above LSC table is equal to the number of blocks in the above raw image: An electronic device for extending the LSC table by adding channels from the outermost part of the LSC table, and generating the correction image by using some of the plurality of channels included in the extended LSC table.

15. In paragraph 11, Further comprising a light sensor for outputting a signal for measuring the light intensity around the electronic device; At least one of the above instructions, An electronic device that applies weights to channel-specific gain values ​​of the LSC table based on the above illuminance.

Images