Electronic device comprising camera and operating method thereof

Abstract

An operating method of an electronic device, according to a disclosed embodiment, may comprise an operation of acquiring a first image by using a camera of the electronic device. The operating method of the electronic device may comprise an operation of dividing the first image into a plurality of regions. The operating method of the electronic device may comprise an operation of acquiring a first histogram of a first region among the plurality of regions. The operating method of the electronic device may comprise an operation of acquiring a second histogram of the first region by redistributing frequencies of the first histogram. The operating method of the electronic device may comprise an operation of acquiring a first gain map of the first region on the basis of the second histogram of the first region. The operating method of the electronic device may comprise an operation of acquiring a second gain map of the first region by calculating the weight sum of the first gain map of the first region and a first gain map of a second region among regions adjacent to the first region. The operating method of the electronic device may comprise an operation of acquiring, by using the second gain map of the first region, a gain value of a first pixel included in the first region. The operating method of the electronic device may comprise an operation of acquiring a second image by applying the gain value of the first pixel to a pixel value of the first pixel. The operating method of the electronic device may comprise an operation of calculating the weighted sum of the first histogram of the first region and a first histogram of the second region among the regions adjacent to the first region. The operating method of the electronic device may comprise an operation of calculating the weighted sum of the second histogram of the first region and a second histogram of the second region among the regions adjacent to the first region.

Description

Electronic device including a camera, method of operation thereof

[0001] The present disclosure relates to an electronic device including a camera and a method of operating the same.

[0002] With the development of digital technology, various types of electronic devices such as mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smartphones, tablet PCs (personal computers), or wearable devices are widely used. Electronic devices can provide various functions. For example, electronic devices can provide at least one function by running at least one application in the foreground and / or background.

[0003] The electronic device has a specified operating system (e.g., the Android operating system (Android) TM Various functions can be provided using the operating system. For example, an electronic device can support multiple functions provided through a camera.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] A method of operation of an electronic device according to a disclosed embodiment may include an operation of acquiring a first image using a camera of the electronic device. A method of operation of the electronic device may include an operation of dividing the first image into a plurality of regions. A method of operation of the electronic device may include an operation of acquiring a first histogram of a first region among the plurality of regions. A method of operation of the electronic device may include an operation of acquiring a second histogram of a first region by redistributing the frequencies of the first histogram. A method of operation of the electronic device may include an operation of acquiring a first gain map of a first region based on the second histogram of the first region. A method of operation of the electronic device may include an operation of acquiring a second gain map of a first region by weighting the first gain map of the first region and the first gain map of a second region among regions adjacent to the first region. A method of operation of the electronic device may include an operation of acquiring a gain value of a first pixel included in the first region using the second gain map of the first region. A method of operation of the electronic device may include an operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0006] An electronic device according to a disclosed embodiment may include a camera. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory including instructions. The instructions may be executed by at least one processor. The electronic device may acquire a first image using the camera. The electronic device may divide the first image into a plurality of regions. The electronic device may acquire a first histogram of a first region among the plurality of regions. The electronic device may acquire a second histogram of a first region by redistributing the frequencies of the first histogram. The electronic device may acquire a first gain map of a first region based on the second histogram of the first region. The electronic device may acquire a second gain map of a first region by performing a weighted sum of the first gain map of the first region and the first gain map of a second region among regions adjacent to the first region. The electronic device may acquire a gain value of a first pixel included in the first region using the second gain map of the first region. The electronic device can acquire a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0007] According to a disclosed embodiment, a method of operation of an electronic device may include an operation of acquiring a first image using a camera of the electronic device. The method of operation of the electronic device may include an operation of dividing the first image into a plurality of regions. The method of operation of the electronic device may include an operation of acquiring a first histogram of a first region among the plurality of regions. The method of operation of the electronic device may include an operation of acquiring a second histogram of a first region by weighting the first histogram of the first region and the first histogram of a second region among regions adjacent to the first region. The method of operation of the electronic device may include an operation of acquiring a third histogram of a first region by redistributing the frequencies of the second histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a first gain map of a first region based on the third histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a gain value of a first pixel included in the first region using the first gain map of the first region. The method of operation of the electronic device may include an operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0008] According to a disclosed embodiment, a method of operation of an electronic device may include an operation of acquiring a first image using a camera of the electronic device. The method of operation of the electronic device may include an operation of dividing the first image into a plurality of regions. The method of operation of the electronic device may include an operation of acquiring a first histogram of a first region among the plurality of regions. The method of operation of the electronic device may include an operation of acquiring a second histogram of the first region by redistributing the frequencies of the first histogram. The method of operation of the electronic device may include an operation of acquiring a third histogram of the first region by weighting the second histogram of the first region and the second histogram of the second region among regions adjacent to the first region. The method of operation of the electronic device may include an operation of acquiring a first gain map of the first region based on the third histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a gain value of a first pixel included in the first region using the first gain map of the first region. The method of operation of the electronic device may include an operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0009] FIG. 1 is a block diagram illustrating an electronic device in a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0012] FIG. 4 is a conceptual diagram illustrating the configuration of an image sensor according to one embodiment.

[0013] FIG. 5 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0014] FIG. 6 is a block diagram illustrating the configuration of an image statistical processing operation block of an electronic device according to one embodiment.

[0015] Figure 7 is a flowchart regarding how an electronic device statistically processes images.

[0016] FIG. 8 is a diagram illustrating the operation of an electronic device according to one embodiment generating a black and white image.

[0017] FIG. 9 is a drawing for explaining a histogram generated by an electronic device according to one embodiment.

[0018] FIG. 10 is a diagram illustrating the operation of an electronic device redistributing the degrees of a histogram according to one embodiment.

[0019] FIG. 11 is a drawing for illustrating a histogram in which the frequencies are redistributed by an electronic device according to one embodiment.

[0020] FIG. 12 is a diagram illustrating the operation of an electronic device according to one embodiment generating a cumulative distribution function.

[0021] FIG. 13 is a diagram illustrating the operation of an electronic device generating a gain map according to one embodiment.

[0022] FIG. 14 is a diagram illustrating a tone curve generated by an electronic device according to one embodiment.

[0023] FIG. 15 is a diagram illustrating the operation of an electronic device according to one embodiment performing a two-dimensional blending operation.

[0024] FIG. 16 is a diagram illustrating the operation of an electronic device according to one embodiment performing a three-dimensional blending operation.

[0025] FIG. 17 is a drawing for explaining the comparison between an image before the gain map of the present disclosure is blended and an image after the gain map of the present disclosure is blended.

[0026] FIG. 18 is a flowchart relating to a method of statistically processing an image by an electronic device according to one embodiment.

[0027] FIG. 19 is a flowchart relating to a method of statistically processing an image by an electronic device according to one embodiment.

[0028] FIG. 20 is a drawing for explaining the comparison between an image before the histogram of the present disclosure is blended and an image after the histogram of the present disclosure is blended.

[0029] FIG. 21 is a block diagram illustrating the configuration of a pixel processing block according to one embodiment.

[0030] Figure 22 is a flowchart regarding how an electronic device processes pixels of an image.

[0031] FIG. 23 is a diagram illustrating the operation of an electronic device according to one embodiment performing spatial bilinear interpolation operations.

[0032] FIG. 24 is a diagram illustrating the operation of an electronic device according to one embodiment performing intensity linear interpolation.

[0033] FIG. 25 is a drawing for explaining the comparison between an image before the local tone mapping of the present disclosure is applied and an image after the local tone mapping of the present disclosure is applied.

[0034] An electronic device can acquire images through a camera module. The electronic device can process images using an image signal processor. For example, the image signal processor can perform tone mapping operations to represent the dynamic range of the image. By performing tone mapping operations, the electronic device can improve image contrast to provide the user with an image that has a sense of depth. However, halo or ringing artifacts may occur due to tone mapping operations.

[0035] The technical problems intended to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description in this disclosure.

[0036] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0037] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0038] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0039] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0040] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0041] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0042] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0043] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0044] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0045] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0046] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0047] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0048] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0050] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0052] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0054] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0055] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0056] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0057] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0058] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0059] FIG. 2 is a block diagram relating to an electronic device (101) including a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260).

[0060] The lens assembly (210) can collect light emitted from a subject that is the subject of image capture. The lens assembly (210) may include one or more lenses.

[0061] According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0062] The flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp.

[0063] The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include, for example, one image sensor selected from image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0064] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction in response to the movement of the camera module (180) or the electronic device (101) including it, or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing). This allows for compensating for at least some of the negative effects caused by the movement on the image being captured.

[0065] According to one embodiment, the image stabilizer (240) can detect the movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). For example, the image stabilizer (240) may be implemented as an optical image stabilizer.

[0066] The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed, for example, by an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0067] According to one embodiment, the memory (250) can store images acquired and output as preview images at least temporarily. The preview image may include an image provided by an electronic device (101) that allows the user to check the location, lighting, or composition of the object to be photographed so that the user can acquire the desired image.

[0068] The image signal processor (260) can perform one or more image processing operations on an image obtained through the image sensor (230) or an image stored in memory (250). The above one or more image processing methods may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) may perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). The image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be composed of at least a part of the processor (120), or It may be configured as a separate processor that operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) either as is or after additional image processing by the processor (120).

[0069] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. For example, among the plurality of camera modules (180), at least one may be a wide-angle camera and at least another may be a telephoto camera. Similarly, among the plurality of camera modules (180), at least one may be a front camera and at least another may be a rear camera.

[0070] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0071] Referring to FIG. 3, an electronic device (101) according to one embodiment may include a camera module (380), a memory (330), and a processor (320). An electronic device (101) according to one embodiment may further include a display (360). The display (360) may be replaced by an external display connected to the electronic device (101). The electronic device (101), processor (320), memory (330), camera module (380), and display (360) may correspond to the electronic device (101), processor (120), memory (130), camera module (180), and display module (160) described above with reference to FIG. 1 and FIG. 2, respectively. However, the components of the electronic device (101) shown in FIG. 3 are for the purpose of explaining one embodiment, and the electronic device (101) may include more components than those shown in FIG. 3, or may include other components that can replace at least some of the components. For example, the memory (330) is not limited to a storage medium included in the electronic device (101) and may include a cloud storage outside the electronic device (101).

[0072] According to one embodiment, the camera module (380) may include a lens unit (381) (e.g., lens assembly (210) of FIG. 2) comprising at least one lens for collecting light, and an image sensor (383) (e.g., image sensor (230) of FIG. 2) for converting a light signal passing through the lens unit (381) into a digital signal.

[0073] The image sensor (383) may include a color filter array comprising a plurality of light-receiving elements, a plurality of microlenses, and a plurality of color channels. The plurality of light-receiving elements may include photodiodes arranged to correspond to one microlens in an array having M rows and N columns. Here, M and N may each be natural numbers greater than or equal to 1.

[0074] The color filter array included in the image sensor (383) may be composed of a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern). The colors of the element groups of the color filter array composed of the non-Bayer pattern may be matched to correspond to the Bayer pattern.

[0075] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a non-Bayer pattern. For example, the image sensor (383), which includes a color filter array composed of a non-Bayer pattern, can output an image signal composed of a non-Bayer pattern by determining a pixel value using the output value of a light-receiving element corresponding to an element. The image signal output from the image sensor (383) may be data in which the color pattern is maintained, as the color order of the color pattern of the image sensor (383) is not changed.

[0076] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a Bayer pattern. For example, the image sensor (383) including a color filter array composed of a non-Bayer pattern can output an image signal composed of a Bayer pattern by binning the output values ​​of light-receiving elements corresponding to the element groups constituting the color filter array.

[0077] According to one embodiment, the image sensor (383) may operate in a high-resolution mode or a crop mode. The high-resolution mode may include a mode in which each of the output values ​​of the light-receiving elements included in the image sensor (383) is used as the pixel value of a pixel corresponding to each of the light-receiving elements. In the present disclosure, the term 'pixel' may mean the smallest unit constituting a digital image. The resolution of an image may be expressed as the number of pixels included in the image. For example, if an image consists of axb pixels arranged in a rows and b columns, the resolution of the image may be indicated as axb.

[0078] For example, an electronic device (101) can acquire high-resolution image data by using an image sensor (383) composed of 50 Mp light-receiving elements, and using each of the output values ​​of the 50 Mp light-receiving elements as the pixel value of the pixel corresponding to the light-receiving elements. The high-resolution mode can be understood as a full pixel mode.

[0079] The crop mode may include a mode in which the output values ​​of a predetermined number of light-receiving elements (for example, a predetermined number of light-receiving elements located in the center of the image sensor) among the light-receiving elements are used as pixel values. The electronic device (101) can acquire an image with a narrowed field of view (FOV) through the crop mode and can provide the user with an experience similar to that of a zoom-in function. For example, when the electronic device (101) operates in a crop mode in which the output values ​​of 12.5 Mp light-receiving elements located in the center of an image sensor (383) composed of 50 Mp light-receiving elements are used as pixel values, the user can be provided with an experience similar to that of a 2x zoom-in function.

[0080] According to one embodiment, the image sensor (383) may operate in a low-light mode or a multi-frame composite mode. The low-light mode may include a mode in which the pixel values ​​of a first pixel corresponding to the first light-receiving elements are used based on values ​​output from first light-receiving elements corresponding to elements of a color filter array matched to the same color of the light-receiving elements included in the image sensor (383). The multi-frame composite mode may include a mode in which the image sensor (383) acquires a plurality of frames in which there is a difference in exposure values.

[0081] According to one embodiment, the memory (330) can store instructions that can be executed by the processor (320). By executing the instructions stored in the memory (330), the processor (320) can perform operations or control components of the electronic device (101).

[0082] In the present disclosure, the operation of the electronic device (101) may be understood as being performed by at least one processor (320) executing instructions. According to one embodiment, the processor (320) may include at least one of an application processor (AP), a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, at least one processor (320) may include an application processor. At least one processor (320) may acquire image data based on an image frame containing information read out from an image sensor (383).

[0083] According to one embodiment, at least one processor (320) can control a camera module (380) by executing a camera application. For example, the processor (320) can initiate the operation of the camera module (380) through the camera application. For example, the processor (320) can provide a request to the camera module (380) through the camera application for the acquisition of at least one frame (e.g., a frame for preview, a frame for capture). For example, the processor (320) can acquire one or more frames through the camera application. For example, the processor (320) can identify user input received through the camera application (e.g., user input regarding the acquisition of a capture image). For example, the capture image may include an image in which one or more preset operations are applied to an image signal acquired by the camera module (380). For example, the captured image may include an image to which at least one operation has been applied, such as demosaicing, white balance, contrast, saturation value adjustment, gamma correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement. For example, the processor (320) may acquire image data (e.g., preview image data, draft image data, captured image data, video data) through a camera application. For example, the processor (320) may display the acquired image data using a display (360).

[0084] According to one embodiment, at least one processor (320) can transmit a control signal to an image sensor (383) to perform a read operation. For example, at least one processor (320) can transmit the control signal based on a specified communication method (e.g., I2C (inter-integrated circuit), I3C (improved inter-integrated circuit)). At least one processor (320) can acquire image data output from the image sensor (383). The image data may include image frames. For example, at least one processor (320) can receive image data through an interface connected to the image sensor (383) (e.g., MIPI (mobile industry processor interface)).

[0085] According to one embodiment, at least one processor (320) can control an image sensor (383) to acquire an image corresponding to a user input. The processor (320) can control the image sensor (383) to operate based on a selected shooting mode. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out each output of a light receiving element constituting the image sensor (383) based on a first shooting mode (e.g., high-resolution shooting mode) so that each output corresponds to a pixel corresponding to each of the light receiving elements. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out each output of a predetermined number of first light receiving elements among a plurality of light receiving elements based on a second shooting mode (e.g., crop shooting mode) so that each output corresponds to a pixel corresponding to each of the first light receiving elements. For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out a first group of light-receiving elements based on a third shooting mode (e.g., low-light shooting mode) such that the output of the first group corresponds to a first pixel corresponding to the first group. For example, the image sensor (383) can output a short exposure image signal and a long exposure image signal by controlling the exposure time of the image sensor (383) based on a fourth shooting mode (e.g., HDR shooting mode) so that there is a preset difference in exposure.

[0086] According to one embodiment, at least one processor (320) can acquire image data by performing operations on an image signal (e.g., raw image data) output from an image sensor (383). For example, the processor (320) can perform at least one operation on the image signal (e.g., demosaicing (or Bayer interpolation), an operation to adjust white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement).

[0087] For example, the processor (320) can generate a second image signal (for example, raw image data composed of a Bayer pattern) by performing remosaicking on a first image signal output from an image sensor (383) (for example, raw image data composed of a non-Bayer pattern). For example, the processor (320) can perform at least one operation (for example, demosaicking, operation to adjust white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) by inputting the image signal output from the image sensor (383) to an image signal processor (e.g., image signal processor (260) of FIG. 2). The processor (320) can store the generated image data in memory (330).

[0088] For example, the processor (320) can identify objects included in an image using an image signal. For example, the processor (320) can identify objects included in an image by performing an object recognition operation on the image signal. For example, the processor (320) can identify a main object and a sub-object among the objects included in the image. For example, the processor (320) can identify a face included in an image by performing face recognition.

[0089] According to one embodiment, at least one processor (320) can initiate the operation of a camera module (380). For example, the processor (320) can initiate the operation of the camera module (380) by providing a control signal to a hardware abstraction layer (HAL) that includes the identification of the camera and a command to initiate the operation of the camera.

[0090] According to one embodiment, at least one processor (320) can acquire one or more frames. For example, the processor (320) can acquire one or more frames by controlling the camera module (380) through a request to acquire frames. For example, the processor (320) can acquire a preview frame composed of a Bayer pattern. For example, the processor (320) can acquire a capture frame composed of a non-Bayer pattern.

[0091] According to one embodiment, at least one processor (320) can convert a first image signal composed of a first color pattern (e.g., a non-Bayer pattern) into a second image signal composed of a second color pattern (e.g., a Bayer pattern). For example, the processor (320) can convert the first image signal into a second image signal by performing remosaicing on the first image signal. For example, the processor (320) can convert the first image signal into a second image signal by performing binning on the first image signal. For example, the processor (320) can control the operation unit (e.g., the operation unit (417) of FIG. 4) of the image sensor (383) to convert the first image signal into a second image signal.

[0092] According to one embodiment, at least one processor (320) can identify received user input. For example, the processor (320) can identify user input changing a shooting mode (e.g., high resolution mode, crop mode, low light mode, multi-frame composite mode, still image shooting mode, video shooting mode). For example, the processor (320) can identify user input regarding the acquisition of still images and / or videos. For example, the processor (320) can identify user input requesting the acquisition of a multi-frame composite image.

[0093] According to one embodiment, at least one processor (320) can acquire preview image data. For example, the processor (320) can generate preview image data by applying an image signal composed of a Bayer pattern output from an image sensor (383) including a Bayer pattern color filter to an image signal processor (260). For example, the processor (320) can control a processing unit (e.g., the processing unit (417) of FIG. 4) of the image sensor (383) so that an image signal composed of a Bayer pattern is output by binning the output values ​​of the light receiving elements of the image sensor (383) including a color filter of a non-Bayer pattern (e.g., tetra, nonar, hexadeca). For example, the tetra pattern may be a non-Bayer pattern in which a 2x2 array of light receiving elements corresponds to a color filter of the same color. For example, the nonar pattern may be a non-Bayer pattern in which a 3x3 array of light receiving elements corresponds to a color filter of the same color. For example, a hexadecca pattern may be a non-Bayer pattern in which a 4x4 array of light-receiving elements corresponds to a color filter of the same color.

[0094] According to one embodiment, at least one processor (320) can acquire a capture image. For example, the processor (320) can generate a capture image by performing at least one operation (e.g., demosaicking, white balance, contrast, saturation value adjustment operation, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on a first image signal composed of a non-Bayer pattern output from an image sensor (383). For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a high-resolution mode. For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a crop mode.

[0095] According to one embodiment, at least one processor (320) can display acquired image data using a display (360). For example, the processor (320) can display at least one of preview image data and captured image data in at least a part area of ​​a camera application.

[0096] According to one embodiment, the display (360) can display one or more pieces of information through the control of at least one processor (320). For example, the display (360) can display a user interface (UI) of an electronic device. For example, the display (360) can display an execution screen of an application running on an electronic device. For example, the display (360) can display preview image data and captured image data.

[0097] FIG. 4 is a conceptual diagram illustrating the configuration of an image sensor according to one embodiment. The image sensor of FIG. 4 may correspond to the image sensor (230, 383) described above with reference to FIG. 2 and FIG. 3.

[0098] Referring to FIG. 4, an image sensor according to one embodiment may include a micro lens array (MLA) (411), a color filter array (CFA) (413), a light receiving unit (415), and a processing unit (417).

[0099] In one embodiment, the micro-lens array (411) may be arranged so that a light beam (421) passing through a lens assembly (e.g., lens assembly (210) of FIG. 2) is collected by a light receiving element of a light receiving unit (415). As the light beam (423) passing through the micro-lens array (411) passes through a color filter array (413), at least some wavelengths other than the band corresponding to a specific color may be blocked.

[0100] According to one embodiment, the color filter array (413) may be positioned at a location corresponding to a light receiving element of the image sensor (230). Light beams (425) passing through the color filter array (413) may be detected by a light receiving element (e.g., a photodiode) of the light receiving unit (415).

[0101] In one embodiment, the light receiving elements of the light receiving unit (415) may include photodiodes arranged in an array having M rows and N columns to correspond to a single microlens. Here, M and N may each be natural numbers greater than or equal to 1. The light receiving unit (415) may include a light receiving element (e.g., including a light receiving circuit) that generates a charge and converts it into an electrical signal upon receiving light, and a circuit that selectively reads out the charge of the light receiving element. Between the light receiving unit (415) and the processing unit (417), a circuit for digitizing the signal read out from the light receiving unit (415) or for reducing noise may be further arranged.

[0102] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light beam (421) is incident, the area where a single microlens included in the microlens array (411) is arranged may overlap at least partially with the area where a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter array (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.

[0103] In one embodiment, the color pattern of the color filter array (413) is illustrated based on a Bayer pattern, but the color pattern of the color filter array (413) is not limited to that illustrated in FIG. 4. Regions within the color filter array (413) corresponding to a plurality of adjacent microlenses may be configured to include the same color channel.

[0104] In one embodiment, the calculation unit (417) can perform a calculation to process electrical data (or signal) (427) output from the light receiving unit (415). The calculation unit (417) can output the acquired data based on the calculation result. The output of the calculation unit (417) can be the sensor output (429) of the image sensor (230).

[0105] FIG. 5 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0106] Referring to FIG. 5, the electronic device (500) may include a processor (510) and a memory (520). The electronic device (500) may correspond to at least one of the electronic device (101) described with reference to FIG. 1, the electronic device (101) described with reference to FIG. 2, or the electronic device (101) described with reference to FIG. 3. The processor (510) may correspond to at least one of the processor (120) described with reference to FIG. 1, the image signal processor (260) described with reference to FIG. 2, or the processor (320) described with reference to FIG. 3. The memory (520) may correspond to at least one of the memory (130) described with reference to FIG. 1, the memory (250) described with reference to FIG. 2, or the memory (330) described with reference to FIG. 3. Redundant content is omitted.

[0107] According to one embodiment, the processor (510) may include at least one operation block for processing an image. For example, the operation block may be implemented as an operation circuit to perform a defined logical operation. For example, the operation block may be implemented as an algorithm to perform a defined logical operation. For example, the processor (510) may include a local tone mapping operation block (511) for performing local tone mapping of an image. For example, the local tone mapping operation block (511) may include an image statistics processing operation block (512) and a pixel processing block (513).

[0108] According to one embodiment, an electronic device (500) can acquire a first image (501). For example, the electronic device (500) can acquire the first image (501) by taking a picture using a camera (e.g., the camera module (180) of FIG. 1). For example, the first image (501) may be image data output from a camera (e.g., the camera module (180) of FIG. 1). For example, the first image (501) may be image data processed from image data output from a camera (e.g., the camera module (180) of FIG. 1) (e.g., downsampled image data). For example, the first image (501) may be image data containing a defined color space. For example, the first image (501) may be image data formed in an RGB, HSV, or YUV color space. The following description is based on the assumption that the first image (501) is image data formed in an RGB color space, but is not limited thereto, and can be applied by analogy even if the first image (501) is image data formed in a different color space.

[0109] According to one embodiment, the electronic device (500) may store a first image (501) in a memory (520). For example, the electronic device (500) may temporarily store image data output from a camera (e.g., camera module (180) of FIG. 1) in the memory (520). For example, the memory (520) may store data (e.g., metadata) related to processing the first image (501). For example, the memory (520) may store setting values ​​used for the operation of the pixel processing block (513). For example, the memory (520) may store setting values ​​used for adjusting the quality of the first image (501). For example, the memory (520) may store data output from the pixel processing block (513) (e.g., a second image (502)).

[0110] According to one embodiment, the image statistical processing operation block (512) can perform an operation to statistically process the first image (501). For example, the image statistical processing operation block (512) can divide the first image (501) into a plurality of regions.

[0111] For example, the image statistical processing operation block (512) can generate a histogram from the first image (501). For example, the image statistical processing operation block (512) can generate a histogram from each of the multiple regions into which the first image (501) is divided.

[0112] For example, the image statistical processing operation block (512) can generate at least one of a gain map, a tone curve, or a tone map from the first image (501) stored in memory (520). For example, the image statistical processing operation block (512) can generate at least one of a gain map, a tone curve, or a tone map for each of the multiple regions into which the first image (501) is divided. For example, the image statistical processing operation block (512) can generate at least one of a gain map, a tone curve, or a tone map using a histogram.

[0113] According to one embodiment, the image statistical processing operation block (512) can store the result of statistically processing the first image (501) in memory (520). For example, the image statistical processing operation block (512) can store a plurality of regions into which the first image (501) is divided in memory (520). For example, the image statistical processing operation block (512) can store a histogram in memory (520). For example, the image statistical processing operation block (512) can store a gain map in memory (520). For example, the image statistical processing operation block (512) can store a tone curve or a tone map in memory (520).

[0114] According to one embodiment, the pixel processing block (513) can perform operations on the pixel values ​​of the pixels of an image based on data output from the image statistical processing operation block (512). For example, the pixel processing block (513) can perform operations on data regarding the result of statistically processing a first image (501) stored in memory (520). For example, the pixel processing block (513) can obtain gain values ​​to be applied to each of the pixels of the first image (501) from a gain map stored in memory (520) and generate a second image (502) to which the gain values ​​are applied. For example, the pixel processing block (513) can perform an operation to obtain the second image (502) by applying a tone curve or tone map stored in memory (520) to each of the pixels of the first image (501). For example, the pixel processing block (513) can obtain gain values ​​to be applied to each of the pixels of the first image (501) using a tone curve or tone map stored in memory (520), and obtain a second image (502) to which the gain values ​​are applied.

[0115] According to one embodiment, the pixel processing block (513) can store the result of applying operations to the first image (501) in memory (520). For example, the pixel processing block (513) can store a second image (502) in memory (520) in which gain values ​​are applied to the first image (501). For example, the pixel processing block (513) can store the second image (502) obtained by applying a tone curve to each of the pixels of the first image (501) in memory (520). For example, the pixel processing block (513) can store the second image (502) obtained by applying gain values ​​obtained using a tone map to each of the pixels of the first image (501) in memory (520).

[0116] According to one embodiment, the electronic device (500) can output a second image (502). For example, the electronic device (500) can output the second image (502) stored in memory (520) using a display (e.g., the display module (160) of FIG. 1). For example, the electronic device (500) can output the second image (502) to an external electronic device (e.g., the electronic device (102), electronic device (104), server (108) of FIG. 1) using a communication module (e.g., the communication module (190) of FIG. 1).

[0117] FIG. 6 is a block diagram illustrating the configuration of an image statistical processing operation block of an electronic device according to one embodiment. The electronic device (600) of FIG. 6 may correspond to the electronic device (500) described with reference to FIG. 5. The image statistical processing operation block (610) of FIG. 6 may correspond to the image statistical processing operation block (512) described with reference to FIG. 5.

[0118] According to one embodiment, the image statistical processing operation block (610) may receive the first data (601).

[0119] According to one embodiment, the first data (601) may include a first image (e.g., the first image (501) of FIG. 5). For example, the first image (e.g., the first image (501) of FIG. 5) may be image data output from a camera (e.g., the camera module (180) of FIG. 1). For example, the first image (e.g., the first image (501) of FIG. 5) may be image data processed from image data output from a camera (e.g., the camera module (180) of FIG. 1) (e.g., downsampled image data). For example, the first image (e.g., the first image (501) of FIG. 5) may be image data including a defined color space. For example, the first image (e.g., the first image (501) of FIG. 5) may be image data formed in an RGB, HSV, or YUV color space.

[0120] According to one embodiment, the first data (601) may include information regarding a grid that divides the first image (e.g., the first image (501) of FIG. 5) into a plurality of regions. For example, the first data (601) may include information regarding the number of grids. For example, the first data (601) may include information regarding the size of the grids. For example, the first data (601) may include information regarding the number of pixels of each of the plurality of regions of the first image (e.g., the first image (501) of FIG. 5) divided by the grid. For example, the first data (601) may include the pixel values ​​of each of the plurality of regions of the first image (e.g., the first image (501) of FIG. 5) divided by the grid.

[0121] According to one embodiment, the first data (601) may include information regarding the histogram of the first image (e.g., the first image (501) of FIG. 5). For example, the first data (601) may include information regarding the bins of the histogram. For example, the first data (601) may include information regarding the size of the bins of the histogram. For example, the first data (601) may include the number of bins of the histogram. For example, the first data (601) may include information regarding the histograms of each of the plurality of regions into which the first image (e.g., the first image (501) of FIG. 5) is divided. For example, the first data (601) may store setting values ​​used to adjust the quality of the first image (501).

[0122] According to one embodiment, the image statistical processing operation block (610) may include, but is not limited to, a grey value generating block (620), a histogram generating block (630), a histogram analysis block (640), a gain map generating block (650), and a blending operation block (660). To correspond to the use and / or function of the operation block, there may be more blocks than the operation block shown in FIG. 6, or two or more operation blocks may be implemented as a single operation block. At least one of the operation blocks disclosed in FIG. 6 may be implemented as an operation circuit to perform a defined logical operation. At least one of the operation blocks disclosed in FIG. 6 may be implemented as an algorithm for performing a defined logical operation.

[0123] According to one embodiment, a grey value generating block (620) can obtain grey values ​​from a first image (e.g., the first image (501) of FIG. 5) included in the input first data (601). For example, the grey value generating block (620) can convert the first image (e.g., the first image (501) of FIG. 5) included in the first data (601) into a grey image. For example, the grey value generating block (620) can convert the luminance value of each pixel of an image having an RGB or YUV color space into a grey value. For example, the grey value generating block (620) can be bypassed if the first image (e.g., the first image (501) of FIG. 5) included in the first data (601) is a grey image.

[0124] According to one embodiment, a histogram generating block (630) can generate a first histogram from a black and white image. For example, the histogram generating block (630) can generate a first histogram from a black and white value output from a black and white value generating block (620). For example, the histogram generating block (630) can generate a first histogram from a black and white image included in the first data (601).

[0125] For example, the histogram generation block (630) can generate a first histogram based on information regarding the intervals (bins) of the histogram included in the first data (601). For example, the histogram generation block (630) can generate a first histogram containing 64 intervals (bins) based on the number of intervals set to 64 (6 bits).

[0126] For example, the histogram generation block (630) can generate a first histogram based on the count of black and white values ​​corresponding to each of the bins of the histogram. For example, the histogram generation block (630) can generate a first histogram by plotting the count of black and white values ​​corresponding to each of the bins of the histogram.

[0127] For example, the histogram generation block (630) can generate a first histogram for each of multiple regions in which a black and white image is divided by a grid. For example, the histogram generation block (630) can generate a first histogram for each of multiple regions in which a black and white image is divided by mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0128] According to one embodiment, a histogram analysis block (640) can obtain a second histogram from a first histogram. For example, the histogram analysis block (640) can generate a second histogram by performing a histogram redistribution operation on the frequencies of the first histogram. For example, the histogram redistribution operation may include an operation (or method) of adjusting the histogram using a contrast limited adaptive histogram equalization (CLAHE) algorithm or a rescale algorithm. For example, the operation of adjusting the histogram using a CLAHE algorithm or a rescale algorithm may include an operation of adjusting (e.g., redistributing) the values ​​(or distribution) of pixels exceeding a threshold value (or clip limit value) or a specified value in the histogram using the corresponding algorithm (e.g., CLAHE algorithm or rescale algorithm).

[0129] For example, the histogram analysis block (640) can generate a second histogram using a clip limit value identified from the first histogram. For example, the histogram analysis block (640) can identify the clip limit value based on the operation described below with reference to FIG. 10. For example, the histogram analysis block (640) can generate a second histogram by redistributing the frequencies of the first histogram that exceed the clip limit value to the intervals of the first histogram.

[0130] According to one embodiment, a gain map generating block can generate at least one of a gain map, a tone curve, or a tone map from a histogram. For example, the gain map generating block (650) can generate at least one of a gain map, a tone curve, or a tone map of a plurality of regions divided by a grid. For example, the gain map generating block (650) can generate at least one of a gain map, a tone curve, or a tone map from a first histogram output from a histogram generating block (630). For example, the gain map generating block (650) can generate at least one of a gain map, a tone curve, or a tone map from a second histogram output from a histogram analysis block (640). For example, the gain map generating block (650) can obtain a cumulative distribution function (CDF) from a histogram. The cumulative distribution function may be referred to as a tone curve or a tone map. According to one embodiment, the gain map generation block (650) can obtain a gain map using a cumulative distribution function.

[0131] According to one embodiment, the blending operation block (660) can blend histograms. For example, the blending operation block (660) can perform a weighted sum of histograms of multiple regions divided by a grid. For example, the blending operation block (660) can blend first histograms of multiple regions output from the histogram generation block (630). For example, the blending operation block (660) can blend second histograms of multiple regions output from the histogram analysis block (640).

[0132] According to one embodiment, the blending operation block (660) can perform a weighted sum of at least one of a gain map, a tone curve, or a tone map. For example, the blending operation block (660) can blend gain maps of a plurality of regions divided by a grid. For example, the blending operation block (660) can blend gain maps of a plurality of regions output from the gain map generation block (650).

[0133] According to one embodiment, the blending operation block (660) can blend a first region among a plurality of regions and at least one second region among regions adjacent to the first region. For example, the blending operation block (660) can blend a first histogram of the first region and a first histogram of the second region. For example, the blending operation block (660) can blend a second histogram of the first region and a second histogram of the second region. For example, the blending operation block (660) can blend a gain map of the first region and a gain map of the second region. The following description is based on an embodiment that blends gain maps, but this can be applied by analogy to an embodiment that blends histograms, tone curves, or tone maps.

[0134] According to one embodiment, the blending operation block (660) can blend the gain map using a mask. For example, the blending operation block (660) can perform a weighted sum of the gain map using a mask in which weights corresponding to each of the first region and the regions adjacent to the first region are set. For example, the blending operation block (660) can blend the gain map using weights that are set as fixed values. For example, the blending operation block (660) can blend the gain map using dynamically set weights.

[0135] For example, the blending operation block (660) can blend the gain map using a mask of an mxn array. For example, m and n can be odd numbers greater than or equal to 3. For example, the blending operation block (660) can set at least one of m or n larger as the clip limit value identified in the histogram analysis block (640) increases.

[0136] For example, the blending operation block (660) can blend the gain map by performing a spatial 2D convolution operation on the gain map and the mask. For example, the blending operation block (660) can blend the gain map by performing a 2D convolution operation on the x-axis and y-axis of the grid.

[0137] For example, the blending operation block (660) can blend the gain map by performing a 3D convolution operation on the gain map and the mask. For example, the blending operation block (660) can blend the gain map by performing a 3D convolution operation on the x-axis and y-axis of the grid and on the bins of the gain map. For example, the blending operation block (660) can blend the gain map by using a fixed setting value for the bins of the gain map.

[0138] According to one embodiment, the image statistical processing operation block (610) may output second data (602). For example, the second data (602) may include at least one data among the data generated by the image statistical processing operation block (610). For example, the second data (602) may include a black and white value output from the black and white value generation block (620). For example, the second data (602) may include a first histogram output from the histogram generation block (630). For example, the second data (602) may include a second histogram output from the histogram analysis block (640). For example, the second data (602) may include at least one of a gain map, a tone curve, or a tone map output from the gain map generation block (650). For example, the second data (602) may include at least one of a blended gain map, a blended tone curve, a blended tone map, a blended first histogram, or a blended second histogram output from a blending operation block (660). For example, the second data (602) may be stored in memory (e.g., memory (130) of FIG. 1). For example, the second data (602) may be input to a pixel processing block (e.g., pixel processing block (513) of FIG. 5).

[0139] FIG. 7 is a flowchart relating to a method for an electronic device to statistically process an image. FIG. 7 may correspond to an embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The operation of the electronic device illustrated in FIG. 7 (e.g., the electronic device (101) of FIG. 1) may be performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3) performing operations or by controlling components of the electronic device (e.g., the electronic device (101) of FIG. 1). For example, the operation of the electronic device illustrated in FIG. 7 (e.g., the electronic device (101) of FIG. 1) may be performed by an operation block of a processor implemented as an operation circuit. For example, the operation of the electronic device illustrated in FIG. 7 (e.g., the electronic device (101) of FIG. 1) may be performed by an operation block of a processor implemented as an algorithm. For example, the operation of the electronic device illustrated in FIG. 7 (e.g., the electronic device (101) of FIG. 1) can be performed by the image statistical processing operation block (610) described with reference to FIG. 6.

[0140] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted.

[0141] Referring to operation 710, an electronic device according to one embodiment may generate a black and white value. For example, operation 710 may be performed by a black and white value generation block (620) described with reference to FIG. 6. For example, the electronic device may generate a black and white image from a first image (e.g., the first image (801) of FIG. 8) acquired using a camera. For example, the first image (e.g., the first image (801) of FIG. 8) may be image data including a defined color space. For example, the first image (e.g., the first image (801) of FIG. 8) may be image data formed in an RGB, HSV, or YUV color space. For example, the first image (e.g., the first image (801) of FIG. 8) may be image data processed from image data output from a camera (e.g., the camera module (180) of FIG. 1) (e.g., downsampled image data). If the first image (e.g., the first image (801) of FIG. 8) is a black and white image, operation 710 may be omitted. The specific operation of the electronic device related to operation 710 is described below with reference to FIG. 8.

[0142] Referring to operation 720, an electronic device according to one embodiment can generate a first histogram. For example, operation 720 can be performed by a histogram generation block (630) described with reference to FIG. 6.

[0143] For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by a grid. For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0144] For example, an electronic device can generate a first histogram based on information regarding the intervals (bins) of a histogram included in the first data (e.g., the first data (601) of FIG. 6). For example, the electronic device can generate the first histogram by plotting the counts of black and white values ​​corresponding to each of the intervals (bins) of the histogram.

[0145] The specific operation of the electronic device related to operation 720 is described below with reference to FIG. 9. Duplicate content is omitted.

[0146] Referring to operation 730, an electronic device according to one embodiment can generate a second histogram. For example, operation 730 can be performed by a histogram analysis block (640) described with reference to FIG. 6.

[0147] For example, the electronic device can generate a second histogram for each of the multiple regions divided in operation 720. For example, the electronic device can generate a second histogram for each of the multiple regions in which a black and white image is divided into mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0148] For example, an electronic device can generate a second histogram by redistributing the frequencies of a first histogram. For example, an electronic device can generate a second histogram by redistributing the frequencies of a first histogram using a clip limit value identified from the first histogram. For example, an electronic device can generate a second histogram by redistributing the frequencies of the first histogram that exceed the identified clip limit value to the intervals of the first histogram.

[0149] The specific operation of the electronic device related to operation 730 is described below with reference to FIGS. 10 and FIGS. 11. Duplicate content is omitted.

[0150] Referring to operation 740, an electronic device according to one embodiment may generate a cumulative distribution function and / or a gain map. For example, operation 740 may be performed by a gain map generation block (650) described with reference to FIG. 6.

[0151] For example, the electronic device may generate a cumulative distribution function and / or gain map for each of the multiple regions divided in operation 720. The cumulative distribution function may be referred to as a tone curve or a tone map.

[0152] For example, the electronic device may generate at least one of a gain map, a tone curve, or a tone map from the second histogram generated through operation 730. For example, the electronic device may obtain a cumulative distribution function (CDF) from the second histogram. For example, the electronic device may obtain a gain map using the cumulative distribution function.

[0153] The specific operation of the electronic device related to operation 740 is described below with reference to FIGS. 12 to 14. Duplicate content is omitted.

[0154] Referring to operation 750, the electronic device can blend the gain map. For example, operation 750 can be performed by the blending operation block (660) described with reference to FIG. 6.

[0155] For example, the electronic device can blend the gain map of a first region and the gain map of a second region. The second region may be at least one region among regions adjacent to the first region.

[0156] For example, an electronic device can blend a gain map using a mask. For example, an electronic device can blend a gain map using a mask in which weights corresponding to each of the first region and the regions adjacent to the first region are set. For example, an electronic device can blend a gain map using weights that are set as fixed values. For example, an electronic device can blend a gain map using dynamically set weights.

[0157] For example, an electronic device can blend gain maps by performing spatial 2D convolution operations on the gain map and the mask. For example, an electronic device can blend gain maps by performing 2D convolution operations on the x-axis and y-axis of the grid.

[0158] For example, an electronic device can blend gain maps by performing a 3D convolution operation on the gain map and the mask. For example, an electronic device can blend gain maps by performing a 3D convolution operation on the x-axis and y-axis of the grid and the bins of the gain map.

[0159] The specific operation of the electronic device related to operation 750 is described below with reference to FIGS. 15 to 17.

[0160] According to one embodiment, the electronic device may further perform an operation of blending the gain map of the first image and the gain map of the second image after operation 750. The second image may be an earlier frame of the first image. The operation of blending the gain map of the first image and the gain map of the second image may be performed by a blending operation block (660) described with reference to FIG. 6.

[0161] For example, the electronic device can blend the gain map of the first image and the gain map of the second image using an infinite impulse response (IIR) filter. For example, the electronic device can blend the gain map of the first image and the gain map of the second image based on the pseudo code of Table 1.

[0162] for (y=0; y < grid_cnt_y; y++):for (x=0; x < grid_cnt_x; x++):for (n=0; n <bins; n++):gainMap[y][x][n]=(1.0-α)×previous_gainMap[y][x][n]+α×gainMap[y][x][n]

[0163] In the pseudocode of Table 1, grid_cnt_x may be the number of multiple regions arranged along the x-axis. grid_cnt_y may be the number of multiple regions arranged along the y-axis. previous_gainMap may be the gain map of the second image. gainMap may be the gain map of the first image. α may be a weight for adjusting the application ratio of the gain map of the first image and the gain map of the second image.

[0164] The electronic device can suppress changes in local tone mapping that may occur due to different gain maps per frame by blending the gain map of the first image and the gain map of the second image.

[0165] According to the disclosed embodiment, an electronic device according to one embodiment can perform a local tone mapping operation on a first image using a gain map generated through the operation described with reference to FIG. 7. The electronic device can obtain an image in which local contrast is naturally improved without the occurrence of halo artifacts.

[0166] FIG. 8 is a diagram illustrating the operation of an electronic device generating a black and white image according to one embodiment. FIG. 8 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The black and white value generating block (810) of FIG. 8 may correspond to the black and white value generating block (620) described with reference to FIG. 6.

[0167] According to one embodiment, a black and white value generation block (810) may receive a first image (801). For example, the first image (801) may be image data formed in an RGB, HSV, or YUV color space. For example, the first image (801) may include an image with a resolution corresponding to an output mode supported by an image sensor included in a camera module. An image sensor included in a camera module may support various output modes related to various resolutions (e.g., number of pixels). For example, the first image (801) may include an image whose resolution has been changed by processing (e.g., cropping, downscaling) an image output from an image sensor.

[0168] According to one embodiment, the black and white value generation block (810) may receive a coefficient (820) used to generate a black and white image. For example, the black and white value generation block (810) may receive a coefficient (820) stored in memory (e.g., memory (130) of FIG. 1). For example, the black and white value generation block (810) may receive a coefficient (820) included in first data (e.g., first data (601) of FIG. 6). For example, the coefficient (820) received in the black and white value generation block (810) may be a value set by the manufacturer of the electronic device.

[0169] For example, the black and white value generation block (810) may receive a coefficient (820) for minimizing brightness and color distortion of the black and white image. For example, the black and white value generation block (810) may receive a coefficient for adjusting the luminance of the colors (e.g., red, green, blue) of the pixels of the first image (801). For example, the black and white value generation block (810) may receive a coefficient for adjusting the ratio of pixel values ​​to luminance values ​​of the first image (801).

[0170] According to one embodiment, the black and white value generation block (810) can obtain a black and white value corresponding to the first image (801) based on the first image (801) and the coefficient (820). For example, the black and white value generation block (810) can generate a black and white image (802) by obtaining a black and white value corresponding to each of the pixels of the first image (801). For example, the black and white value generation block (810) can generate a second image (802) from the first image (801) using mathematical formulas 1 to 6.

[0171]

[0172] Equation 1 may be a formula for an electronic device to obtain the luminance value of a first pixel of a first image (801). In Equation 1, luminance may mean the luminance value of a first pixel of a first image (801). R may mean the red pixel value of a first pixel of a first image (801). G may mean the green pixel value of a first pixel of a first image (801). B may mean the blue pixel value of a first pixel of a first image (801). c0, c1, and c2 may be coefficients (e.g., weights) for the red pixel value of a first pixel, the green pixel value of a first pixel, and the blue pixel value of a first pixel, respectively. c0, c1, and c2 may each be values ​​greater than or equal to 0 and less than or equal to 1. c0, c1, and c2 may be set by the manufacturer of the electronic device.

[0173]

[0174] Equation 2 may be a formula for an electronic device to identify the highest value among the color pixel values ​​of the first pixel of the first image (801). In Equation 2, R may represent the red pixel value of the first pixel of the first image (801). G may represent the green pixel value of the first pixel of the first image (801). B may represent the blue pixel value of the first pixel of the first image (801). M may represent the maximum value among the color pixel values ​​of the first pixel of the first image (801).

[0175]

[0176] Equation 3 may be a formula for an electronic device to identify the lowest value among the color pixel values ​​of the first pixel of the first image (801). In Equation 3, R may represent the red pixel value of the first pixel of the first image (801). G may represent the green pixel value of the first pixel of the first image (801). B may represent the blue pixel value of the first pixel of the first image (801). m may represent the minimum value among the color pixel values ​​of the first pixel of the first image (801).

[0177]

[0178] Equation 4 may be a formula for an electronic device to identify the saturation value of the first pixel of the first image (801). In Equation 4, M may represent the maximum value among the color pixel values ​​of the first pixel of the first image (801). m may represent the minimum value among the color pixel values ​​of the first pixel of the first image (801).

[0179]

[0180]

[0181] Mathematical formulas 5 and 6 may be formulas for obtaining the grayscale value of the first pixel of a grayscale image (802) corresponding to the first pixel of the first image (801). s may be a value for linking the maximum value among the color pixel values ​​of the first pixel with the saturation value of the first pixel. gray may be the grayscale value of the first pixel of the grayscale image. c3 and c4 may be coefficients for adjusting the ratio of the pixel value and the brightness value of the first image (801). c3 and c4 may each be a value between 0 and 1. c3 and c4 may be set by the manufacturer of the electronic device.

[0182] When c3 is 1, the electronic device according to one embodiment can generate a black and white image (802) based on a black and white value obtained based on mathematical formula 7.

[0183]

[0184] In mathematical formula 7, gray may be the gray value of the first pixel of the gray image (802). M may represent the maximum value among the color pixel values ​​of the first pixel of the first image (801).

[0185] When c4 is 0, the electronic device according to one embodiment can generate a black and white image (802) based on a black and white value obtained based on mathematical formula 8.

[0186]

[0187] In Equation 8, gray may be the gray value of the first pixel of the gray image (802). M may represent the maximum value among the color pixel values ​​of the first pixel of the first image (801). Luminance may represent the brightness value of the first pixel of the first image identified by Equation 1. According to Equation 8, an electronic device according to one embodiment may obtain the gray value of the first pixel of the gray image (802) based on the ratio of the maximum value among the color pixel values ​​of the first pixel, determined by the c3 value, and the brightness value of the first pixel.

[0188] When c3 is 0, the electronic device according to one embodiment can generate a black and white image (802) based on a black and white value obtained based on mathematical formula 9.

[0189]

[0190] In Equation 9, gray may be the gray value of the first pixel of the grayscale image (802). Luminance may mean the brightness value of the first pixel of the first image identified by Equation 1. Saturation may be the saturation value of the first pixel of the first image (801) identified by Equation 4. According to Equation 9, an electronic device according to one embodiment may obtain the gray value of the first pixel of the grayscale image (802) based on the ratio of the maximum value among the color pixel values ​​of the first pixel, determined by the c4 value and the saturation value of the first pixel of the first image (801), and the brightness value of the first pixel.

[0191] According to the disclosed embodiment, the electronic device can generate a black and white image (802) in which color distortion and brightness distortion caused by saturation of color channels are minimized in the pixel gain operation described below with reference to FIGS. 21 and 22. The black and white image (802) of FIG. 8 is provided to aid in understanding the present disclosure and is not limited thereto. Instead of generating a black and white image (802), the electronic device may output only a black and white value corresponding to the first image (801).

[0192] FIG. 9 is a drawing for illustrating a histogram generated by an electronic device according to one embodiment. FIG. 9 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. FIG. 9 may show a histogram generated by a histogram generation block (630) described with reference to FIG. 6.

[0193] The black and white image (900) of FIG. 9 is for the purpose of facilitating understanding of the present disclosure, and the electronic device generating the black and white image (900) may not be an essential operation of the present disclosure. Instead of generating the black and white image (900), the electronic device may generate only a histogram corresponding to a plurality of regions with black and white values.

[0194] According to one embodiment, an electronic device can divide a black and white image (900) into a plurality of regions. For example, the electronic device can divide a black and white image (802) described with reference to FIG. 8 into a plurality of regions.

[0195] For example, the electronic device can divide the black and white image (900) based on information regarding the grid included in the first data (e.g., the first data (601) of FIG. 6). For example, the electronic device can divide the black and white image (900) based on at least one of information regarding the number of grids, information regarding the size of the grids, or information regarding the number of pixels of each region divided by the grids.

[0196] For example, the electronic device can divide a black and white image (900) (e.g., the black and white image (802) of FIG. 8) into multiple regions by an mxn grid. m and n can be natural numbers greater than or equal to 1. Referring to FIG. 9, for example, the electronic device can be divided into multiple regions by a 12 x 9 grid. For example, the electronic device can set coordinates for each of the multiple regions. For example, the electronic device can set coordinates such as (0,0) for the first region (910).

[0197] According to one embodiment, an electronic device can generate a histogram. For example, the electronic device can generate a histogram based on information regarding a histogram included in the first data (e.g., the first data (601) of FIG. 6). For example, the electronic device can generate a histogram based on information regarding the bins of the histogram. For example, the electronic device can generate a histogram based on information including that the bins of the histogram included in the first data (e.g., the first data (601) of FIG. 6) are 64 (6 bits). For example, the electronic device can generate a histogram by plotting the counts of black and white values ​​corresponding to each bin of the histogram.

[0198] For example, the electronic device can generate a histogram for each of the multiple regions into which the black and white image (900) is divided. For example, the electronic device can generate a histogram for each of the multiple regions to be divided by a 12 x 9 grid. For example, the electronic device can generate a first histogram (911) of a first region (910).

[0199] FIG. 10 is a diagram illustrating the operation of an electronic device redistributing the frequencies of a histogram according to one embodiment, and FIG. 11 is a diagram illustrating a histogram in which the frequencies have been redistributed by an electronic device according to one embodiment. FIG. 10 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. FIG. 10 may be performed by the histogram analysis block (640) described with reference to FIG. 6. FIG. 11 may show a histogram generated by the histogram analysis block (640) described with reference to FIG. 6.

[0200] The black and white image (1100) of FIG. 11 is for the purpose of facilitating understanding of the present disclosure, and the electronic device generating the black and white image (1100) may not be an essential operation of the present disclosure. Instead of generating the black and white image (1100), the electronic device may generate only a histogram corresponding to a plurality of regions with black and white values.

[0201] The first histogram (1010) of FIG. 10 may correspond to the first histogram (911) described with reference to FIG. 9. For example, the first histogram (1010) of FIG. 10 may correspond to the first histogram (911) of the first region (910) corresponding to the (0,0) coordinate of the black and white image (900) of FIG. 9.

[0202] The second histogram (1020) of FIG. 10 may correspond to the second histogram (1111) described with reference to FIG. 11. For example, the second histogram (1020) of FIG. 10 may correspond to the first histogram (1111) of the first region (1110) corresponding to the (0,0) coordinates of the black and white image (1100) of FIG. 11. The first region (910) of FIG. 9 and the first region (1110) of FIG. 11 may be corresponding regions.

[0203] Referring to FIG. 10, an electronic device according to one embodiment can generate a second histogram (1020) by redistributing the frequencies of a first histogram (1010). For example, the electronic device can generate a second histogram (1020) by distributing frequencies (1012) that exceed a clip limit (1011) value to intervals of the first histogram (1010).

[0204] An electronic device according to one embodiment can identify a clip limit (1011) value from a first histogram (1010). For example, the electronic device can identify a clip limit (1011) value used to redistribute a frequency (1012) concentrated in a specific section of the first histogram (1010) to the sections (bins) of the first histogram (1010). For example, the electronic device can identify a clip limit (1011) value based on mathematical formulas 10 to 12.

[0205] According to one embodiment, the electronic device can identify a dynamic range used to identify a clip limit (1011) value based on mathematical formula 10.

[0206]

[0207] max_bin in Equation 10 may refer to the largest bin among the bins in which the frequency of the first histogram (1010) is one or more. histogram_bins may refer to the number of bins (64, 6 bits) of the first histogram (1010). According to one embodiment, the electronic device may identify the entropy used to identify the clip limit (1011) value by referring to Equation 11.

[0208]

[0209] In mathematical formula 11, stdev may mean the standard deviation of the first histogram (1010). mean may mean the average of the first histogram (1010).

[0210] According to one embodiment, the electronic device can identify the clip limit (1011) value by referring to Equation 12.

[0211]

[0212] w1 in Equation 12 may be a weight set based on the average value of the first histogram (1010). w2 may be a weight set based on the average value and standard deviation of the first histogram (1010). offset may be an adjustment value set based on the average value of the first histogram (1010). clip_limit_max may be the maximum value of the clip limit (1011) set based on the average value of the first histogram (1010). The electronic device may redistribute the frequency (1012) exceeding the clip limit (1011) value identified by referring to Equation 12 as the first part (1021) of the second histogram (1020).

[0213] According to one embodiment, the electronic device can redistribute the frequencies of each histogram of a plurality of regions. For example, the electronic device can redistribute the frequencies of each histogram of a plurality of regions divided by a grid of m x n (where m and n are natural numbers greater than or equal to 1). For example, the electronic device can redistribute the frequencies of each histogram of a plurality of regions divided by a grid of 12 x 9. For example, the electronic device can redistribute the frequencies of each histogram of regions corresponding to the (0,0) coordinate to the (11, 8) coordinate.

[0214] FIG. 12 is a diagram illustrating the operation of an electronic device according to one embodiment generating a cumulative distribution function, FIG. 13 is a diagram illustrating the operation of an electronic device according to one embodiment generating a gain map, and FIG. 14 is a diagram illustrating a tone curve or gain map generated by an electronic device according to one embodiment. FIG. 12 to FIG. 14 may be a cumulative distribution function or gain map generated by one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. FIG. 12 to FIG. 14 may be a cumulative distribution function or gain map generated by a gain map generating block (650) described with reference to FIG. 6.

[0215] FIG. 12 may show a cumulative distribution function (1210) corresponding to a first region (1110) corresponding to the (0,0) coordinates of the black and white image (1100) of FIG. 11. FIG. 13 may show a gain map (1310) corresponding to a first region (1110) corresponding to the (0,0) coordinates of the black and white image (1100) of FIG. 11.

[0216] The black and white image (1400) of FIG. 14 is for the purpose of facilitating understanding of the present disclosure, and the electronic device generating the black and white image (1400) may not be an essential operation of the present disclosure. Instead of generating the black and white image (1400), the electronic device may obtain a cumulative distribution function or a gain map function corresponding to multiple regions as black and white values.

[0217] According to one embodiment, an electronic device may generate a cumulative distribution function (CDF) using a histogram. For example, the electronic device may generate a cumulative distribution function using a histogram generated by a histogram generation block (630) or a histogram analysis block (640) described with reference to FIG. 6. The present disclosure describes, but is not limited to, an operation in which a gain map generation block (650) generates a cumulative distribution function using a second histogram generated by the histogram analysis block (640). The present disclosure may be applied by analogy to an operation in which a gain map generation block (650) generates a cumulative distribution function using a first histogram generated by the histogram generation block (630). The cumulative distribution function of the present disclosure may be referred to as a tone curve or a tone map.

[0218] According to one embodiment, the electronic device can obtain a cumulative distribution function based on Equation 13.

[0219]

[0220] In mathematical formula 13, bins can represent the number of intervals (bins) of the histogram. histogram(i) can represent the frequency for index (i) of an interval (bin) of the histogram. can mean the total frequency of the histogram.

[0221] Referring to FIG. 12, an electronic device according to one embodiment can obtain a cumulative distribution function corresponding to a first region (1110) corresponding to the (0,0) coordinate of the black and white image (1100) of FIG. 11 based on Equation 13. An electronic device according to one embodiment can obtain a cumulative distribution function for each of a plurality of regions of the black and white image (1400) of FIG. 14 based on Equation 13.

[0222] According to one embodiment, the electronic device can obtain a gain map function based on Equation 14.

[0223]

[0224] In Equation 14, bins may refer to the number of intervals (bins) of the histogram. cdf(n) may be a cumulative distribution function obtained based on Equation 13.

[0225] Referring to FIG. 13, an electronic device according to one embodiment can obtain a gain map function corresponding to a first region (1110) of FIG. 11 based on Equations 13 and 14. An electronic device according to one embodiment can obtain a gain map function or a tone curve function for each of a plurality of regions into which the black and white image (1100) of FIG. 11 is divided based on Equations 13 and 14. An electronic device according to one embodiment can obtain a gain map function or a tone curve function for each of a plurality of regions of the black and white image (1400) of FIG. 14 based on Equations 13 and 14.

[0226] FIG. 15 is a diagram illustrating the operation of an electronic device according to one embodiment performing a two-dimensional blending operation, and FIG. 16 is a diagram illustrating the operation of an electronic device according to one embodiment performing a three-dimensional blending operation. The blending operations of FIG. 15 and FIG. 16 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The blending operations described with reference to FIG. 15 and FIG. 16 may be performed by the blending operation block (660) described with reference to FIG. 6.

[0227] According to one embodiment, the electronic device can blend at least one of a histogram, a gain map, or a tone curve. For example, the electronic device can blend at least one of a histogram, a gain map, or a tone curve corresponding to each of a plurality of regions.

[0228] For example, the electronic device may perform a blending operation on a first histogram (e.g., the first histogram (911) of FIG. 9) generated in the histogram generation block (630) described with reference to FIG. 6. For example, the electronic device may perform a blending operation on a second histogram (e.g., the second histogram (1111) of FIG. 11) generated in the histogram analysis block (640) described with reference to FIG. 6.

[0229] For example, the electronic device can perform a blending operation on the tone curve generated in the gain map generation block (650) described with reference to FIG. 6.

[0230] For example, the electronic device can perform a blending operation on the gain map generated in the gain map generation block (650) described with reference to FIG. 6.

[0231] The following describes an embodiment in which a blending operation is performed on a gain map, but is not limited thereto. The present disclosure may be applied by analogy to blending operations using histograms and blending operations using tone curves.

[0232] According to one embodiment, an electronic device can perform a blending operation using a mask. For example, the electronic device can perform a blending operation using a mask (1530) in which weights are set for each of the first region (1511) and the regions (1520) adjacent to the first region (1511).

[0233] For example, the electronic device can blend the gain map using a mask (1530) in which the weights are set to a fixed value. For example, the electronic device can blend the gain map using a mask (1530) in which the weights are set dynamically.

[0234] For example, the electronic device can blend the gain map using a mask of an mxn array (where m and n are odd numbers greater than or equal to 3). For example, the electronic device can set the values ​​of m and n larger as the clip limit value identified in the histogram analysis block (640) increases. For example, the electronic device can set the values ​​of m and n based on the capacity of the memory (e.g., the memory (130) of FIG. 1). For example, the electronic device can set the values ​​of m and n larger than a predetermined value based on the capacity of the buffer being larger than a predetermined value.

[0235] According to one embodiment, the electronic device can perform a blending operation by performing a 2D convolution operation with respect to the spatial direction of the grid (e.g., the x-axis and y-axis of the grid). For example, the electronic device can blend the gain map of a first region (1511) of a first image (1510) and the gain map of regions (1520) adjacent to the first region (1511) using a mask (1530).

[0236] For example, an electronic device can blend gain maps based on Table 2. Table 2 may represent pseudo code for a blending operation using a 3 x 3 mask. The electronic device can perform a blending operation using an mxn mask based on an operation inferred and applied from the pseudo code included in Table 2.

[0237] for (n=0; n <bins; n++):out gainMap[y][x][n]=(gainMap[y-1][x-1][n]×w1 + gainMap[y-1][x][n]×w2 + gainMap[y-1][x+1][n]×w3 + gainMap[y][x-1][n]×w4 + gainMap[y][x][n]×w5 + gainMap[y][x+1][n]×w6 + gainMap[y+1][x-1][n]×w7 + gainMap[y+1][x][n]×w8 + gainMap[y+1][x+1][n]×w9) /

[0238] In the pseudocode of Table 2, n may represent a bin of the gain map. Referring to the pseudocode of Table 2, the electronic device can perform a 2D convolution operation between the gain map and the mask (1530) for all bins of the gain map. In the pseudocode of Table 2, x may be the x-axis coordinate of the first region (1511) in the grid, and y may be the y-axis coordinate of the first region (1511) in the grid. In the pseudocode of Table 2, w1 to w9 may represent weights applied to the gain map of the first region (1511) and the gain maps of the regions (1520) adjacent to the first region (1511), respectively. According to one embodiment, the electronic device can perform a blending operation using a mask (1530) in which the weights are fixed values. For example, the electronic device can perform a blending operation using a mask (1530) in which all weights are the same fixed value. For example, the electronic device may perform a blending operation using a mask (1530) containing weights determined based on a Gaussian distribution. For example, the electronic device may perform a blending operation using a mask (1530) containing weights determined to be inversely proportional to the distance between each of the regions (1520) adjacent to the first region (1511) and the first region (1511).

[0239] According to one embodiment, an electronic device can perform a blending operation using a mask (1530) in which at least one weight is dynamically assigned. For example, the electronic device can perform a blending operation using a mask (1530) in which a weight value corresponding to a first region (1511) is assigned as a fixed value, and a weight value corresponding to each of the regions (1520) adjacent to the first region (1511) is dynamically assigned.

[0240] According to one embodiment, the electronic device may assign weights to the mask (1530) based on a histogram corresponding to the first region (1511). For example, the electronic device may assign weights based on a histogram output from the histogram generation block (630) described above with reference to FIG. 6 and / or a histogram output from the histogram analysis block (640).

[0241] According to one embodiment, the electronic device may assign weights based on the brightness and / or complexity of the histogram corresponding to the first region (1511). For example, the electronic device may assign weights based on the mean spread value obtained based on Equation 15.

[0242]

[0243] In Equation 15, mean may mean the mean of the histogram. stdev may mean the standard deviation of the histogram. mean and stdev may each mean the mean and standard deviation of the first histogram output from the histogram generation block (630). The mean spread value may be a value for determining the similarity of the histogram.

[0244] According to one embodiment, the electronic device may determine the weight of the mask (1530) based on the similarity between the average spread of the first region (1511) and the average spread of the second region (1521). For example, if the ratio of the average spread of the first region (1511) to the average spread of the second region (1521) falls outside a predetermined reference range, the electronic device may assign a weight corresponding to the second region (1521) (e.g., w9) as a first weight that is less than or equal to the reference weight value. For example, if the ratio of the average spread of the first region (1511) to the average spread of the second region (1521) falls within a predetermined reference range, the electronic device may assign a weight corresponding to the second region (1521) (e.g., w9) as a second weight that is greater than or equal to the reference weight value. The predetermined reference range and reference weight value may be values ​​set by the manufacturer of the electronic device. For example, the electronic device can reduce the deviation of the gain map by giving a high weight to a second region (1521) that includes a histogram different from the first histogram of the first region (1511). As the deviation of the gain map is reduced, the electronic device can generate an image in which the occurrence of halo or ringing artifacts caused by tone mapping operations is suppressed.

[0245] According to one embodiment, the electronic device may determine weights based on clip limit values. For example, the electronic device may determine the weights of the second regions (1521) based on a first deviation between the clip limit value of the first region (1511) and the clip limit value of the second region (1521). For example, the electronic device may obtain the first deviation as in Equation 16.

[0246]

[0247] In mathematical formula 16, clip_limit[center] may be the clip limit value of the first region. clip_limit[n] may be the clip limit value of the second region.

[0248] According to one embodiment, the electronic device can determine the weights of the second regions (1521) based on the absolute value of the first deviation. For example, the electronic device can set the weight value smaller as the absolute value of the first deviation is smaller. For example, the electronic device can set the weight value larger as the absolute value of the first deviation is larger.

[0249] Referring to FIG. 16, an electronic device according to one embodiment can obtain a blended gain map (1630) by performing a 3D convolution operation on a gain map (1610) and a mask (1620). For example, the electronic device can blend the gain maps by performing a 3D convolution operation on spatial directions (e.g., the x-axis and y-axis of a grid) and sections (bins) of the gain map.

[0250] For example, the electronic device can blend gain maps based on Table 3. Table 3 may include pseudo code for blending operations using a 3 x 3 x 3 mask.

[0251] for (n=0; n <bins; n++):for (ky=-1;ky<2;ky++):for(kx=-1;kx<2;kx++):for(kbin=-1; kbin <2; kbin++):vsum = gainMap[y+ky][x+kx][n+kbin]×mask[ky+1][kx+1][kbin+1]wsum += mask[ky+1][kx+1][kbin+1]out gainMap[y][x][n] = vsum / wsum

[0252] In the pseudocode of Table 3, x may be the x-axis coordinate of the first region to be blended in the grid, and y may be the y-axis coordinate of the first region to be blended in the grid. n may represent a bin of the gain map. kx may be the x-axis coordinate of the mask, ky may be the y-axis coordinate of the mask, and kbin may be the bin-axis coordinate of the mask. mask[ky+1][kx+1][kbin+1] may be weights. wsum may be the sum of the weights. vsum may be the weighted sum of the gain map. According to one embodiment, the electronic device may perform a blending operation using a mask (1620) in which the weights are fixed values. For example, the electronic device may perform a blending operation using a mask (1620) in which all weights are the same fixed value. For example, the electronic device can perform a blending operation using a mask (1620) that includes weights in the x-axis direction, y-axis direction, and bin-axis direction determined based on a Gaussian distribution. For example, the electronic device can perform a blending operation using a mask (1620) that includes weights determined to be inversely proportional to the distance from the first region (1511).

[0253] According to one embodiment, an electronic device may perform a blending operation using a mask in which at least one weight is dynamically assigned. For example, the electronic device may perform a blending operation using a mask (1620) in which the weight in the bin axis direction is assigned as a fixed value, and the weights in the x-axis direction and the y-axis direction are dynamically assigned, respectively. For example, the electronic device may perform a blending operation using a mask (1620) in which weights in the x-axis direction and the y-axis direction are dynamically assigned based on a histogram corresponding to a first region. The operation of the electronic device assigning weights described with reference to Equation 15 may be inferred and applied to the mask (1620) in which weights in the x-axis direction and the y-axis direction are dynamically assigned. Redundant content is omitted.

[0254] According to the disclosed embodiment, the electronic device can suppress the occurrence of halo or ringing artifacts caused by tone mapping operations by performing a blending operation. The electronic device can suppress the occurrence of halo or ringing artifacts caused by tone mapping operations by assigning a high weight value to a second region (1521) that includes a histogram different from the first histogram of the first region (1511).

[0255] According to the disclosed embodiment, the degree to which the frequencies of the histogram are redistributed can be determined by the clip limit value. The weights of the second regions (1521) can be determined by the absolute value of the first deviation between the clip limit value of the first region (1511) and the clip limit value of the second region (1521). The second region with a large absolute value of the first deviation is set to a high weight, thereby reducing the deviation of the gain map. By reducing the deviation of the gain map, the occurrence of halo or ringing artifacts caused by the tone mapping operation can be suppressed. FIG. 17 is a diagram for explaining the comparison between an image before the gain map of the present disclosure is blended and an image after the gain map of the present disclosure is blended.

[0256] The first image (1710) may be an image to which a local tone mapping operation is applied. The first image (1710) may correspond to an image to which a local tone mapping operation is applied that does not include an operation in which the gain map of an electronic device (e.g., the electronic device (101) of FIG. 1) is blended, as described with reference to FIG. 7 through 16.

[0257] The second image (1720) may be an image to which a local tone mapping operation is applied. The second image (1720) may correspond to an image to which a local tone mapping operation is applied, which includes an operation in which the gain map of an electronic device (e.g., the electronic device (101) of FIG. 1) is blended, as described with reference to FIG. 7 through 16.

[0258] The first part (1711) of the first image (1710) and the first part (1721) of the second image (1720) may be the same part. When comparing the first part (1711) of the first image (1710) and the first part (1721) of the second image (1720), a difference in the boundary between the tree and the sky can be confirmed.

[0259] For example, it may be confirmed that a halo or ringing artifact has occurred in the first region (1712) of the first image (1710) at the boundary between the tree and the sky. When comparing the first gain map (1713) of the first region (1712) with the second gain map (1715) of the second region (1714), it may be confirmed that the middle section (bin) of the first gain map (1713) is higher than the middle section (bin) of the second gain map (1715). The halo or ringing artifact in the first region (1712) of the first image (1710) may have occurred because a gain value that brightens the pixel value corresponding to the darkly rendered tree is also applied to the sky surrounding the tree.

[0260] In contrast, it can be confirmed that the first region (1722) of the second image (1720) has a halo or ringing artifact suppressed at the boundary between the trees and the sky. It can be confirmed that the first gain map (1723) of the first region (1722) is similar to the gain maps of regions adjacent to the first region (1722). For example, it can be confirmed that the first gain map (1723) of the first region (1722) is similar to the second gain map (1725) of the second region (1724). It can be confirmed that halo or ringing artifacts are suppressed by blending the first gain map (1723) of the first region (1722) with the gain maps (e.g., 1725) of regions adjacent to the first region (1722) so that the deviation is reduced.

[0261] According to the disclosed embodiment, the electronic device can suppress the occurrence of a halo or ringing artifact caused by a tone mapping operation by performing a blending operation.

[0262] FIG. 18 is a flowchart relating to a method of statistically processing an image by an electronic device according to one embodiment.

[0263] FIG. 18 may correspond to an embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 18 may be performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3) performing operations or controlling components of the electronic device (e.g., the electronic device (101) of FIG. 1). For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 18 may be performed by an operation block of a processor implemented as an operation circuit. For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 18 may be performed by an operation block of a processor implemented as an algorithm. For example, the operation of the electronic device illustrated in FIG. 18 (e.g., the electronic device (101) of FIG. 1) can be performed by the image statistical processing operation block (610) described with reference to FIG. 6.

[0264] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted.

[0265] Referring to operation 1810, an electronic device according to one embodiment can generate a black and white value. Operation 1810 can be performed by the black and white value generation block (620) described with reference to FIG. 6. For example, the electronic device can obtain a black and white value from a first image (e.g., the first image (801) of FIG. 8) obtained using a camera. Operation 1810 can be applied by analogy to operation 710 of FIG. 7. Redundant content is omitted.

[0266] Referring to operation 1820, an electronic device according to one embodiment can generate a first histogram. For example, operation 1820 can be performed by a histogram generation block (630) described with reference to FIG. 6.

[0267] For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by a grid. For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0268] For example, an electronic device can generate a first histogram based on information regarding the intervals (bins) of a histogram included in the first data (e.g., the first data (601) of FIG. 6). For example, the electronic device can generate the first histogram by plotting the counts of black and white values ​​corresponding to each of the intervals (bins) of the histogram.

[0269] The specific operation of the electronic device related to operation 1820 may be applied by analogy to the embodiment described with reference to FIG. 9. Redundant content is omitted.

[0270] Referring to operation 1830, an electronic device according to one embodiment can generate a second histogram. For example, operation 1830 can be performed by a histogram analysis block (640) described with reference to FIG. 6.

[0271] For example, the electronic device can generate a second histogram for each of the multiple regions divided in operation 1820. For example, the electronic device can generate a second histogram for each of the multiple regions in which a black and white image is divided into mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0272] For example, an electronic device can generate a second histogram by redistributing the frequencies of a first histogram. For example, an electronic device can generate a second histogram by redistributing the frequencies of a first histogram using a clip limit value identified from the first histogram. For example, an electronic device can generate a second histogram by redistributing the frequencies of the first histogram that exceed the identified clip limit value to the intervals of the first histogram.

[0273] The specific operation of the electronic device related to operation 1830 may be applied by analogy to the embodiment described with reference to FIGS. 10 and FIGS. 11. Redundant content is omitted.

[0274] Referring to operation 1840, an electronic device according to one embodiment can blend a second histogram. For example, operation 1840 can be performed by a blending operation block (660) described with reference to FIG. 6.

[0275] For example, an electronic device can blend a second histogram of a first region and a second histogram of a second region. The second region may be at least one region among regions adjacent to the first region.

[0276] For example, an electronic device can blend a second histogram using a mask. For example, an electronic device can blend a second histogram using a mask in which weights corresponding to each of the first region and the regions adjacent to the first region are set. For example, an electronic device can blend a second histogram using weights that are set as fixed values. For example, an electronic device can blend a second histogram using dynamically set weights.

[0277] For example, the electronic device can blend the second histogram by performing a spatial 2D convolution operation on the second histogram and the mask. For example, the electronic device can blend the second histogram by performing a 2D convolution operation on the x-axis and y-axis of the grid.

[0278] For example, the electronic device can blend the second histogram by performing a 3D convolution operation on the second histogram and the mask. For example, the electronic device can blend the second histogram by performing a 3D convolution operation on the x-axis and y-axis of the grid and the bins of the second histogram.

[0279] The specific operation of the electronic device related to operation 1840 may be applied by analogy to the embodiments described with reference to FIGS. 15 to 17. Redundant content is omitted.

[0280] Referring to operation 1850, an electronic device according to one embodiment may generate a cumulative distribution function and / or a gain map. For example, operation 1850 may be performed by a gain map generation block (650) described with reference to FIG. 6.

[0281] For example, the electronic device can generate at least one of a gain map, a tone curve, or a tone map from a second histogram generated through operation 1840. For example, the electronic device can obtain a cumulative distribution function (CDF) from the second histogram. For example, the electronic device can obtain a gain map using the cumulative distribution function.

[0282] The specific operation of the electronic device related to operation 1850 may be applied by analogy to the embodiments described with reference to FIGS. 12 to 14. Redundant content is omitted.

[0283] According to one embodiment, the electronic device may further perform an operation of blending the gain map of the first image and the gain map of the second image after operation 1850. The second image may be an earlier frame of the first image. The operation of blending the gain map of the first image and the gain map of the second image may be performed by a blending operation block (660) described with reference to FIG. 6. For example, the electronic device may blend the gain map of the first image and the gain map of the second image using an infinite impulse response (IIR) filter.

[0284] The operation of the electronic device blending the gain map of the first image and the gain map of the second image can be applied by analogy to the embodiment described with reference to FIG. 7. Redundant content is omitted.

[0285] FIG. 19 is a flowchart relating to a method of statistically processing an image by an electronic device according to one embodiment.

[0286] FIG. 19 may correspond to an embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 19 may be performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3) performing operations or controlling components of the electronic device (e.g., the electronic device (101) of FIG. 1). For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 19 may be performed by an operation block of a processor implemented as an operation circuit. For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 19 may be performed by an operation block of a processor implemented as an algorithm. For example, the operation of the electronic device illustrated in FIG. 19 (e.g., the electronic device (101) of FIG. 1) can be performed by an image statistical processing operation block (610) described with reference to FIG. 6.

[0287] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted.

[0288] Referring to operation 1910, an electronic device according to one embodiment can generate a black and white value. Operation 1910 can be performed by the black and white value generation block (620) described with reference to FIG. 6. For example, the electronic device can obtain a black and white value from a first image (e.g., the first image (801) of FIG. 8) obtained using a camera. Operation 1910 can be applied by analogy to operation 710 of FIG. 7. Redundant content is omitted.

[0289] Referring to operation 1920, an electronic device according to one embodiment can generate a first histogram. For example, operation 1920 can be performed by a histogram generation block (630) described with reference to FIG. 6.

[0290] For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by a grid. For example, an electronic device can generate a first histogram for each of multiple regions in which a black-and-white image is divided by mxn. m and n may be natural numbers greater than or equal to 1. m and n may be values ​​set by the manufacturer of the electronic device.

[0291] For example, an electronic device can generate a first histogram based on information regarding the intervals (bins) of a histogram included in the first data (e.g., the first data (601) of FIG. 6). For example, the electronic device can generate the first histogram by plotting the counts of black and white values ​​corresponding to each of the intervals (bins) of the histogram.

[0292] The specific operation of the electronic device related to operation 1920 can be applied by analogy to the embodiment described with reference to FIG. 9. Redundant content is omitted.

[0293] Referring to operation 1930, an electronic device according to one embodiment can blend a first histogram. For example, operation 1930 can be performed by a blending operation block (660) described with reference to FIG. 6.

[0294] For example, an electronic device can blend a first histogram of a first region and a first histogram of a second region. The second region may be at least one region among regions adjacent to the first region.

[0295] For example, an electronic device can blend a first histogram using a mask. For example, an electronic device can blend a first histogram using a mask in which weights corresponding to each of the first region and the regions adjacent to the first region are set. For example, an electronic device can blend a first histogram using weights that are set as fixed values. For example, an electronic device can blend a first histogram using dynamically set weights.

[0296] For example, the electronic device can blend the first histogram by performing a spatial 2D convolution operation on the first histogram and the mask. For example, the electronic device can blend the first histogram by performing a 2D convolution operation on the x-axis and y-axis of the grid.

[0297] For example, the electronic device can blend the first histogram by performing a 3D convolution operation on the first histogram and the mask. For example, the electronic device can blend the first histogram by performing a 3D convolution operation on the x-axis and y-axis of the grid and the bins of the first histogram.

[0298] The specific operation of the electronic device related to operation 1930 may be applied by analogy to the embodiments described with reference to FIGS. 15 to 17. Redundant content is omitted.

[0299] Referring to operation 1940, an electronic device according to one embodiment can generate a second histogram. For example, operation 1940 can be performed by a histogram analysis block (640) described with reference to FIG. 6.

[0300] For example, the electronic device can generate a second histogram from a first histogram blended in operation 1930. For example, the electronic device can generate a second histogram for each of a plurality of regions. For example, the electronic device can generate a second histogram for each of a plurality of regions in which a black and white image is divided into mxn.

[0301] For example, the electronic device can generate a second histogram by redistributing the frequencies of a blended first histogram. For example, the electronic device can generate a second histogram by redistributing the frequencies of a blended first histogram using a clip limit value identified from the first histogram. For example, the electronic device can generate a second histogram by redistributing the frequencies of the blended first histogram that exceed the identified clip limit value to the intervals of the blended first histogram.

[0302] The specific operation of the electronic device related to operation 1940 may be applied by analogy to the embodiment described with reference to FIGS. 10 and FIGS. 11. Redundant content is omitted.

[0303] Referring to operation 1950, an electronic device according to one embodiment may generate a cumulative distribution function and / or a gain map. For example, operation 1950 may be performed by a gain map generation block (650) described with reference to FIG. 6.

[0304] For example, the electronic device can generate at least one of a gain map, a tone curve, or a tone map from a second histogram generated through operation 1940. For example, the electronic device can obtain a cumulative distribution function (CDF) from the second histogram. For example, the electronic device can obtain a gain map using the cumulative distribution function.

[0305] The specific operation of the electronic device related to operation 1950 may be applied by analogy to the embodiments described with reference to FIGS. 12 to 14. Redundant content is omitted.

[0306] According to one embodiment, the electronic device may further perform an operation of blending the gain map of the first image and the gain map of the second image after operation 1950. The second image may be an earlier frame of the first image. The operation of blending the gain map of the first image and the gain map of the second image may be performed by a blending operation block (660) described with reference to FIG. 6. For example, the electronic device may blend the gain map of the first image and the gain map of the second image using an infinite impulse response (IIR) filter.

[0307] The operation of the electronic device blending the gain map of the first image and the gain map of the second image can be applied by analogy to the embodiment described with reference to FIG. 7. Redundant content is omitted.

[0308] FIG. 20 is a drawing for explaining the comparison between an image before the histogram of the present disclosure is blended and an image after the histogram of the present disclosure is blended.

[0309] The first image (2010) may be an image to which a local tone mapping operation is applied. The first image (2010) may correspond to an image to which a local tone mapping operation is applied that does not include a blending operation of the histogram of an electronic device (e.g., the electronic device (101) of FIG. 1) as described with reference to FIG. 18 and 19.

[0310] The second image (2020) may be an image to which a local tone mapping operation is applied. The second image (2020) may correspond to an image to which a local tone mapping operation is applied, which includes a blending operation of the histogram of an electronic device (e.g., the electronic device (101) of FIG. 1) as described with reference to FIG. 18 and 19.

[0311] The first part (2011) of the first image (2010) and the second part (2021) of the second image (2020) may be the same part. When comparing the first part (2011) of the first image (2010) and the second part (2021) of the second image (2020), a difference in the boundary between the lift, the person, and the sky may be confirmed. For example, in the first part (2011) of the first image (2010), it may be confirmed that a halo or ringing artifact has occurred at the boundary between the lift, the person, and the sky. In the first part (2011) of the first image (2010), the halo or ringing artifact may have occurred because a gain value that brightens the pixel values ​​corresponding to the darkly rendered lift and person is also applied to the sky surrounding the lift.

[0312] In contrast, it can be confirmed that the second part (2021) of the second image (2020) has a halo or ringing artifact suppressed at the boundary between the lift, the person, and the sky. It can be confirmed that the halo or ringing artifact is suppressed by blending the histogram of the second part (2021) with the histograms of the regions adjacent to the second part (2021) so that the deviation is reduced.

[0313] FIG. 21 is a block diagram illustrating the configuration of a pixel processing block according to one embodiment. The electronic device (2100) of FIG. 21 may correspond to the electronic device (500) described with reference to FIG. 5. The pixel processing block (2110) of FIG. 21 may correspond to the pixel processing block (513) described with reference to FIG. 5.

[0314] According to one embodiment, the pixel processing block (2110) may receive the first data (2101). For example, the pixel processing block (2110) may obtain the first data (2101) from a memory (e.g., the memory (130) of FIG. 1).

[0315] For example, the pixel processing block (2110) may receive first data (2101) including a first image (e.g., the first image (501) of FIG. 5). For example, the first image (e.g., the first image (501) of FIG. 5) may be image data output from a camera (e.g., the camera module (180) of FIG. 1). For example, the first image (e.g., the first image (501) of FIG. 5) may be image data including a defined color space. For example, the first image (e.g., the first image (501) of FIG. 5) may be image data formed in an RGB, HSV, or YUV color space.

[0316] For example, the pixel processing block (2110) may receive first data (2101) including data output by an image statistical processing operation block (e.g., the image statistical processing operation block (512) of FIG. 5). For example, the pixel processing block (2110) may receive a tone curve (e.g., the tone curve (1210) of FIG. 12) and / or a gain map (e.g., the gain map (1310) of FIG. 13) generated by the image statistical processing operation block (e.g., the image statistical processing operation block (512) of FIG. 5).

[0317] For example, the pixel processing block (2110) may receive first data (2101) containing metadata required for image processing. For example, the metadata may include information regarding the size of the first image (e.g., the first image (501) of FIG. 5). For example, the metadata may include information regarding the number of pixels constituting the first image (e.g., the first image (501) of FIG. 5). For example, the metadata may include information regarding a grid that divides the first image (e.g., the first image (501) of FIG. 5). For example, the metadata may include information regarding the number of grids. For example, the metadata may include information regarding the size of the grids. For example, the metadata may include information regarding the number of pixels in each of a plurality of regions of the first image (e.g., the first image (501) of FIG. 5) divided by the grids. For example, the metadata may include information regarding the bins of the gain map. For example, metadata may include information regarding the size and / or number of bins in the gain map.

[0318] According to one embodiment, the pixel processing block (2110) may include, but is not limited to, a grey value generating block (2120), a gain map interpolating block (2130), and a pixel gain multiplying block (2140). To correspond to the use and / or function of the operation block, there may be more blocks than the operation block shown in FIG. 21, or two or more operation blocks may be implemented as a single operation block. At least one of the operation blocks disclosed in FIG. 21 may be implemented as an operation circuit to perform a defined logical operation. At least one of the operation blocks disclosed in FIG. 21 may be implemented as an algorithm for performing a defined logical operation.

[0319] According to one embodiment, a grey value generating block (2120) can obtain grey values ​​from an image included in input data. For example, the grey value generating block (2120) can convert a first image (e.g., the first image (501) of FIG. 5) included in the first data (2101) into a grey image. For example, the grey value generating block (2120) can convert each luminance value of a pixel of an image having an RGB or YUV color space into a grey value. For example, the grey value generating block (2120) can be bypassed if the first image (e.g., the first image (501) of FIG. 5) included in the first data (2101) is a grey image. The grey value generating block (2120) may be inferred to have the description of the grey value generating block (620) described with reference to FIG. 6 applied by analogy. The operation of the black and white value generation block (2120) can be applied by analogy to the embodiment described with reference to FIG. 8. Redundant content is omitted.

[0320] According to one embodiment, a gain map interpolating block (2130) can calculate the gain to be applied to each of the pixels of a first image (e.g., the first image (501) of FIG. 5). For example, the gain map interpolating block (2130) can obtain gain values ​​applied to each of the pixels using the gain map and metadata included in the first data (2101). For example, the gain map interpolating block (2130) can perform spatial bilinear interpolation. For example, the gain map interpolating block (2130) can perform intensity linear interpolation. For example, the operation of the gain map interpolating block (2130) can be inferred from the embodiment described below with reference to FIG. 23 and FIG. 24. Redundant content is omitted.

[0321] According to one embodiment, a pixel gain multiplying block (2140) can apply gain values ​​to a first image (e.g., the first image (501) of FIG. 5). For example, the pixel gain multiplying block (2140) can obtain a second image (e.g., the second image (502) of FIG. 5) by applying gain values ​​to the first image (e.g., the first image (501) of FIG. 5). For example, the pixel gain multiplying block (2140) can apply gain values ​​obtained by the gain map interpolation block (2130) to each of the pixels of the first image (e.g., the first image (501) of FIG. 5). For example, the pixel gain multiplying block (2140) can apply gain values ​​to each pixel channel.

[0322] According to one embodiment, the pixel processing block (2110) may output the second data (2102). For example, the pixel processing block (2110) may output the second data (2102) to a memory (e.g., the memory (130) of FIG. 1). For example, the pixel processing block (2110) may output the second data (2102) containing a second image to which a gain value is applied (e.g., the second image (502) of FIG. 5). For example, the second image (e.g., the second image (502) of FIG. 5) may be an image to which a gain value is applied to each of the pixels. For example, the second image (e.g., the second image (502) of FIG. 5) may be an image to which a local tone mapping operation is applied.

[0323] FIG. 22 is a flowchart relating to a method in which an electronic device processes pixels of an image. FIG. 22 may correspond to an embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 22 may be performed by at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the processor (320) of FIG. 3) performing operations or by controlling components of the electronic device (e.g., the electronic device (101) of FIG. 1). For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 22 may be performed by an operation block of a processor implemented as an operation circuit. For example, the operation of the electronic device (e.g., the electronic device (101) of FIG. 1) shown in FIG. 22 may be performed by an operation block of a processor implemented as an algorithm. For example, the operation of the electronic device illustrated in FIG. 22 (e.g., the electronic device (101) of FIG. 1) can be performed by the pixel processing block (2110) described with reference to FIG. 21.

[0324] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted.

[0325] Referring to operation 2210, an electronic device according to one embodiment may generate a black and white value. For example, operation 2210 may be performed by a black and white value generation block (2120) described with reference to FIG. 21. For example, the electronic device may obtain a black and white value from a first image (e.g., the first image (801) of FIG. 8) obtained using a camera. For example, the first image (e.g., the first image (801) of FIG. 8) may be image data containing a defined color space. For example, the first image (e.g., the first image (801) of FIG. 8) may be image data formed in an RGB, HSV, or YUV color space. The specific operation of the electronic device related to operation 2210 may be inferred from the embodiment described with reference to FIG. 8. Redundant content is omitted.

[0326] According to one embodiment, operation 2210 may be replaced with an operation of obtaining a black and white value generated by a black and white value generation block (620) of an image statistical processing operation block (e.g., image statistical processing operation block (610) of FIG. 6) from a memory (e.g., memory (130) of FIG. 1).

[0327] According to one embodiment, if the first image (e.g., the first image (801) of FIG. 8) is a black and white image, operation 2210 may be omitted.

[0328] Referring to operation 2220, an electronic device according to one embodiment can obtain gain values ​​for each pixel by interpolating a gain map. For example, operation 2220 can be performed by a gain map interpolation block (2130) described with reference to FIG. 21. For example, the electronic device can obtain gain values ​​for each pixel by interpolating a blended gain map or tone curve obtained by operation 750 of FIG. 7.

[0329] For example, the electronic device may perform a spatial bilinear interpolation operation. For example, the spatial bilinear interpolation operation may include an operation to calculate the gain value of a pixel using a plurality of gain maps corresponding to regions adjacent to a first pixel of a first image. The operation of the electronic device performing the spatial bilinear interpolation operation is described below with reference to FIG. 23. Redundant content is omitted.

[0330] For example, the electronic device may perform an intensity linear interpolation operation. The intensity linear interpolation operation may include an operation to obtain a gain value in which the fractional part of the interval value corresponding to the grayscale value of the first pixel is reflected. The fractional part of the interval value may occur due to the difference between the bit depth of the pixel value of the first pixel of the first image and the number of intervals of the gain map. The operation of the electronic device performing the intensity linear interpolation operation is described below with reference to FIG. 24. Redundant content is omitted.

[0331] Referring to operation 2230, an electronic device according to one embodiment may perform a pixel gain operation. For example, operation 2230 may be performed by a pixel gain operation block (2140) described with reference to FIG. 21. For example, the electronic device may apply gain values ​​obtained in operation 2220 to corresponding pixel values. For example, the electronic device may apply a gain value to a first pixel of a first image (e.g., the first image (501) of FIG. 1) based on Equation 17.

[0332]

[0333] In Equation 17, pixelGain may be the gain value of the first pixel obtained by operation 2220. R may be the pixel value of the red channel of the first pixel. G may be the pixel value of the green channel of the first pixel. B may be the pixel value of the blue channel of the first pixel. Rout may be the pixel value of the red channel of the first pixel to which the gain value is applied. Gout may be the pixel value of the green channel of the first pixel to which the gain value is applied. Bout may be the pixel value of the blue channel of the first pixel to which the gain value is applied.

[0334] An electronic device according to one embodiment can generate a second image (e.g., the second image (502) of FIG. 5) by applying a gain value to all pixels of a first image (e.g., the first image (501) of FIG. 5) using Equation 17. The second image (e.g., the second image (502) of FIG. 5) may be an image to which a local tone mapping operation has been applied.

[0335] For example, the electronic device may store the second image (e.g., the second image (502) of FIG. 5) in memory (e.g., the memory (130) of FIG. 1). For example, the electronic device may output the second image (e.g., the second image (502) of FIG. 5) through a display (e.g., the display module (160) of FIG. 1). For example, the electronic device may provide the second image (e.g., the second image (502) of FIG. 5) to an external device (e.g., the electronic device (102), electronic device (104), server (108)) through a communication module (e.g., the communication module (190) of FIG. 1).

[0336] FIG. 23 is a diagram illustrating the operation of an electronic device according to one embodiment performing a spatial bilinear interpolation operation. FIG. 23 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The spatial bilinear interpolation operation of FIG. 23 may be performed by the gain map interpolation block (2130) of FIG. 21.

[0337] Referring to FIG. 23, an image according to one embodiment may be divided by a grid to include a plurality of regions (2310, 2320, 2330, 2340). The plurality of regions may be divided such that the width is w and the height is h. The coordinates associated with the plurality of regions may be set as relative values. w and h may be set to 1.

[0338] Referring to FIG. 23, the first pixel (2301) may be spaced from the first center (2311) by a value x in the x-axis direction and by a value y in the y-axis direction. The first pixel (2301) may be spaced from the second center (2321) by a value 1-x in the x-axis direction and by a value y in the y-axis direction. The first pixel (2301) may be spaced from the third center (2331) by a value x in the x-axis direction and by a value 1-y in the y-axis direction. The first pixel (2301) may be spaced from the fourth center (2341) by a value 1-x in the x-axis direction and by a value 1-y in the y-axis direction. For example, the x value may be a relative value to the w value. For example, the y value may be a relative value to the h value. For example, the x value and the y value may be 0 to 1.

[0339] According to one embodiment, the electronic device can obtain a gain value of the first pixel (2301) by performing a spatial bilinear interpolation operation. For example, the electronic device can obtain a gain value of the first pixel (2301) by performing linear interpolation based on the centers (2311, 2321, 2331, 2341) of regions (2310, 2320, 2330, 2340) adjacent to the first pixel (2301). For example, the electronic device can obtain a gain value of the first pixel (2301) based on the position of the first pixel (2301) with respect to the first center (2311) of the first region (2310), the second center (2321) of the second region (2320), the third center (2331) of the third region (2330), and the fourth center (2341) of the fourth region (2340). For example, the electronic device may obtain a gain value of the first pixel (2301) such that the gain map of the region located close to the first pixel (2301) is higher than the gain map of the region located far from the first pixel (2301). For example, the electronic device may obtain a gain value of the first pixel (2301) based on a first difference between the width (w) of the region and the value at which the first pixel (2301) is offset in the x-axis direction from each of the centers (2311, 2321, 2331, 2341) and the second difference between the height (h) of the region and the value at which the first pixel (2301) is offset in the y-axis direction from each of the centers (2311, 2321, 2331, 2341). w and h may be 1. For example, the electronic device can obtain the gain value of the first pixel (2301) based on mathematical formula 18.

[0340]

[0341] In mathematical formula 18, N may be an index representing a bin of the gain map. x may be a value separated by the first pixel (2301) and the first center (2311) in the x-axis direction. y may be a value separated by the first pixel (2301) and the first center (2311) in the y-axis direction. gainMap1 may be a gain map of the first region (2310). gainMap2 may be a gain map of the second region (2320). gainMap3 may be a gain map of the third region (2330). gainMap4 may be a gain map of the fourth region (2340). pixelGain[N] may be a first gain value corresponding to the N+1th bin (N bin) of the first pixel (2301).

[0342] FIG. 24 is a diagram illustrating the operation of an electronic device according to one embodiment performing an intensity linear interpolation operation. FIG. 24 may correspond to one embodiment of the operation of the electronic device (101) described with reference to FIG. 1. The intensity linear interpolation operation of FIG. 24 may be performed by the gain map interpolation block (2130) of FIG. 21.

[0343] Due to the difference between the bit depth of the pixel value of the first pixel of the first image (e.g., the first pixel (2310) of FIG. 23) and the number of bins of the gain map, the bin value (2410) corresponding to the black and white value of the first pixel (2310) may not exist as an integer. For example, when the pixel value (intensity) 268 of the first pixel (e.g., the first pixel (2310) of FIG. 23) with a bit depth of 10 bits is applied to a gain map containing 64 (6 bits) bins, the bin value corresponding to the pixel value of the first pixel may be 16.75. That is, the bin value corresponding to the pixel value of the first pixel may have a fractional part of 0.75.

[0344] An electronic device according to one embodiment can obtain a second gain value by performing an intensity linear interpolation operation on a fractional part of a segment value based on a first gain value obtained by a spatial bilinear interpolation operation described with reference to FIG. 23. For example, the electronic device can obtain a second gain value in which the gain value of a first segment (e.g., N) located close to the first segment value (2410) is reflected as higher than the gain value of a segment (e.g., N+1) located far from the first segment value (2410). For example, the electronic device can obtain a second gain value of a first pixel (e.g., the first pixel (2310) of FIG. 23) using Equation 18.

[0345]

[0346] In Equation 19, N may be an index corresponding to a bin of the gain map. pixelGain[N] and pixelGain[N+1] may be first gain values ​​obtained by the spatial bilinear interpolation operation described with reference to FIG. 23. pixelGain[N] and pixelGain[N+1] may be first gain values ​​of the N+1th bin (N bin) and the N+2th bin (N+1 bin), respectively. pixelGain may be a second gain value obtained through intensity linear interpolation. x may be a fractional part of the bin value (2410) that is generated due to the difference between the bit depth of the pixel value and the number of bins of the gain map.

[0347] An electronic device according to one embodiment can perform a local tone mapping operation that applies a second gain value obtained through an intensity linear interpolation operation to a pixel value.

[0348] FIG. 25 is a drawing for explaining the comparison between an image before the local tone mapping of the present disclosure is applied and an image after the local tone mapping of the present disclosure is applied.

[0349] The first image (2510) may be an image to which a local tone mapping operation has not been applied. For example, the first image (2510) may be an image before the local tone mapping operation described with reference to FIGS. 5 through 24 is applied. The first image (2510) may correspond to the first image (501) of FIG. 5.

[0350] The second image (2520) may be an image to which a local tone mapping operation has been applied. For example, the second image (2520) may be an image after the local tone mapping operation described with reference to FIGS. 5 through 24 has been applied. The second image (2520) may correspond to the second image (502) of FIG. 5.

[0351] When comparing the first image (2510) and the second image (2520), it can be confirmed that the second image (2520) has improved overall brightness and contrast compared to the first image (2510). It can be confirmed that the second image (2520) has become clearer as the contrast between the bright areas (e.g., pillars of a building) and the dark areas (e.g., windows of a building) is increased.

[0352] It can be confirmed that the occurrence of halo or ringing artifacts is suppressed in the second image (2520) even though local tone mapping operations have been applied. For example, the boundary between the tree and the sky in the second image (2520) and the boundary between the building and the sky in the second image (2520) are areas prone to the occurrence of halo or ringing artifacts, but it can be confirmed that the occurrence of halo or ringing artifacts is suppressed in the second image (2520).

[0353] According to the disclosed embodiments, the electronic device can provide a natural image to the user by using the disclosed local tone mapping operation to enhance brightness and contrast and suppress the occurrence of halo or ringing artifacts.

[0354] A method of operation of an electronic device according to a disclosed embodiment (e.g., the electronic device (101) of FIG. 1) may include an operation of acquiring a first image using a camera of the electronic device (e.g., the camera module (180) of FIG. 1). A method of operation of the electronic device may include an operation of dividing the first image (e.g., the first image (501) of FIG. 1) into a plurality of regions. A method of operation of the electronic device may include an operation of acquiring a first histogram (e.g., the first histogram (911) of FIG. 9) of a first region (e.g., the first region (910) of FIG. 9) among the plurality of regions. A method of operation of the electronic device may include an operation of acquiring a second histogram (e.g., the second histogram (1111) of FIG. 11) of a first region (e.g., the first region (1110) of FIG. 11) by redistributing the frequencies of the first histogram (e.g., the first histogram (911) of FIG. 9). It may be possible. The method of operation of the electronic device may include the operation of obtaining a first gain map of a first region (e.g., a first gain map (1310) of FIG. 13) based on a second histogram of a first region (e.g., a second histogram (1111) of FIG. 11). The method of operation of the electronic device may include the operation of obtaining a second gain map of a first region (e.g., a first region (1511) of FIG. 15) and a first gain map of a second region (e.g., a second region (1521) of FIG. 15) among regions adjacent to the first region (e.g., a first region (1722) of FIG. 17) by weighted summing the first gain map of the first region (e.g., a first region (1723) of FIG. 17) of FIG. 17). The method of operation of the electronic device may include the operation of the first region The operation may include obtaining the gain value of a first pixel (e.g., the first pixel (2301) of FIG. 23) included in a first region using a second gain map (e.g., the second gain map (1724) of FIG. 17).The method of operation of the electronic device may include the operation of obtaining a second image (e.g., a second image (2520) of FIG. 25) by applying a gain value of a first pixel (e.g., a first pixel (2301) of FIG. 23) to a pixel value of a first pixel (e.g., a first pixel (2301) of FIG. 23).

[0355] According to one embodiment, the operation of obtaining a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10) may include the operation of identifying a first mean and a first standard deviation of a first histogram of a first region (e.g., the first histogram (1010) of FIG. 10). The operation of obtaining a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10) may include the operation of obtaining a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10) by redistributing the frequencies of the first histogram (e.g., the first histogram (1010) of FIG. 10) based on the first mean and the first standard deviation of the first histogram of a first region (e.g., the first histogram (1010) of FIG. 10).

[0356] According to one embodiment, the operation of obtaining a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10) may include the operation of obtaining a first clip limit value of the first histogram (e.g., the first histogram (1010) of FIG. 10) based on a first mean and a first standard deviation of the first histogram of the first region (e.g., the first histogram (1010) of FIG. 10). The operation of acquiring a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10)) may include the operation of redistributing the frequencies of the first histogram (e.g., the first histogram (1010) of FIG. 10) that exceed the first clip limit value of the first histogram (e.g., the first histogram (1010) of FIG. 10)) into the bins of the first histogram (e.g., the first histogram (1010) of FIG. 10).

[0357] According to one embodiment, the operation of obtaining a second gain map of a first region (e.g., the first region (1511) of FIG. 15) may include the operation of weighting the first gain map of the first region (e.g., the first region (1511) of FIG. 15) and the first gain map of the second region using a mask (e.g., the mask (1530) of FIG. 15) which includes weights corresponding to each of the first region (e.g., the first region (1511) of FIG. 15) and the regions adjacent to the first region (e.g., the first region (1520) of FIG. 15).

[0358] According to one embodiment, a first weight corresponding to a second region included in a mask (e.g., mask (1530) of FIG. 15) may be determined based on the similarity between a first histogram of a first region (e.g., first region (1511) of FIG. 15) and a first histogram of a second region.

[0359] According to one embodiment, the similarity between a first histogram of a first region and a first histogram of a second region can be determined based on the deviation between a first clip limit value of a first histogram of a first region and a second clip limit value of a first histogram of a second region.

[0360] According to one embodiment, the operation of acquiring a second histogram of a first region may include the operation of acquiring a first clip limit value of the first histogram based on a first mean and a first standard deviation of the first histogram of the first region. A first weight corresponding to the second region included in the mask may be determined based on the first clip limit value of the first histogram.

[0361] According to one embodiment, the operation of obtaining a first histogram of a first region may include the operation of obtaining a black and white value of a first pixel from a color pixel value of a first pixel using a first coefficient that adjusts the ratio of a maximum value among the color pixel values ​​of a first pixel to the brightness value of a first pixel.

[0362] According to one embodiment, the operation of obtaining a gain value of a first pixel may include obtaining a gain value of the first pixel by adding a weighted sum of a second gain map of a first region and a second gain map of a second region based on the coordinates of the first pixel, the coordinates of the center of a first region, and the coordinates of the center of a second region.

[0363] According to one embodiment, the operation of dividing a first image into a plurality of regions may include the operation of downsampling the first image. The operation of dividing a first image into a plurality of regions may include the operation of dividing the downsampled first image into a plurality of regions.

[0364] An electronic device according to one disclosed embodiment (e.g., the electronic device (101) of FIG. 1) may include a camera (e.g., the camera module (180) of FIG. 1). The electronic device (e.g., the electronic device (101) of FIG. 1) may include at least one processor including a processing circuit. The electronic device (e.g., the electronic device (101) of FIG. 1) may include a memory (e.g., the memory (130) of FIG. 1)) including instructions. The instructions may be executed by at least one processor (e.g., the processor (120) of FIG. 1). The electronic device (e.g., the electronic device (101) of FIG. 1) may acquire a first image (e.g., the first image (501) of FIG. 1) using a camera (e.g., the camera module (180) of FIG. 1). An electronic device (e.g., the electronic device (101) of FIG. 1) can divide a first image (e.g., the first image (501) of FIG. 1) into multiple regions. The electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a first histogram (e.g., the first histogram (911) of FIG. 9) of a first region (e.g., the first region (910) of FIG. 9) among the multiple regions. By redistributing the frequencies of the first histogram (e.g., the first histogram (911) of FIG. 9), the electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a second histogram (e.g., the second histogram (1111) of FIG. 11) of a first region (e.g., the first region (1110) of FIG. 11). An electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a first gain map of a first region (e.g., the first gain map (1310) of FIG. 13) based on a second histogram of a first region (e.g., the second histogram (1111) of FIG. 11).An electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a second gain map (e.g., the second gain map (1723) of FIG. 17) of a first region (e.g., the first region (1511) of FIG. 15)) by weighting the first gain map of a first region (e.g., the first region (1511) of FIG. 15) and the first gain map of a second region (e.g., the second region (1521) of FIG. 15) among regions adjacent to the first region (e.g., the first region (1511) of FIG. 15) (e.g., the regions (1520) of FIG. 15). An electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a gain value of a first pixel (e.g., the first pixel (2301) of FIG. 23) included in a first region by using a second gain map of a first region (e.g., the second gain map (1724) of FIG. 17). The electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a second image (e.g., the second image (2520) of FIG. 25) by applying the gain value of the first pixel (e.g., the first pixel (2301) of FIG. 23) to the pixel value of the first pixel (e.g., the first pixel (2301) of FIG. 23).

[0365] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can identify a first mean and a first standard deviation of a first histogram of a first region (e.g., the first histogram (1010) of FIG. 10). The electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a second histogram of a first region (e.g., the second histogram (1020) of FIG. 10)) by redistributing the frequencies of the first histogram (e.g., the first histogram (1010) of FIG. 10) based on the first mean and the first standard deviation of the first histogram of the first region (e.g., the first histogram (1010) of FIG. 10).

[0366] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a first clip limit value of a first histogram (e.g., the first histogram (1010) of FIG. 10) based on a first mean and a first standard deviation of a first histogram of a first region (e.g., the first histogram (1010) of FIG. 10). The electronic device (e.g., the electronic device (101) of FIG. 1) can redistribute the frequencies of the first histogram (e.g., the first histogram (1010) of FIG. 10) that exceed the first clip limit value of the first histogram (e.g., the first histogram (1010) of FIG. 10) into bins of the first histogram (e.g., the first histogram (1010) of FIG. 10).

[0367] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can perform a weighted sum of a first gain map of a first region (e.g., the first region (1511) of FIG. 15) and a first gain map of a second region by using a mask (e.g., the mask (1530) of FIG. 15) which includes weights corresponding to each of the first region (e.g., the first region (1511) of FIG. 15) and the region adjacent to the first region (e.g., the first region (1520) of FIG. 15).

[0368] According to one embodiment, a first weight corresponding to a second region included in a mask (e.g., mask (1530) of FIG. 15) may be determined based on the similarity between a first histogram of a first region (e.g., first region (1511) of FIG. 15) and a first histogram of a second region.

[0369] According to one embodiment, the similarity between a first histogram of a first region and a first histogram of a second region can be determined based on the deviation between a first clip limit value of a first histogram of a first region and a second clip limit value of a first histogram of a second region.

[0370] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a first clip limit value of a first histogram based on a first mean and a first standard deviation of a first histogram of a first region. A first weight corresponding to a second region included in a mask can be determined based on the first clip limit value of the first histogram.

[0371] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a black and white value of a first pixel from a color pixel value of a first pixel by using a first coefficient that adjusts the ratio of the maximum value among the color pixel values ​​of the first pixel to the brightness value of the first pixel.

[0372] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can obtain a gain value of a first pixel by adding a second gain map of a first region and a second gain map of a second region based on the coordinates of a first pixel, the coordinates of the center of a first region, and the coordinates of the center of a second region.

[0373] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can downsample a first image. The electronic device can divide the downsampled first image into a plurality of regions.

[0374] According to a disclosed embodiment, a method of operation of an electronic device may include an operation of acquiring a first image using a camera of the electronic device. The method of operation of the electronic device may include an operation of dividing the first image into a plurality of regions. The method of operation of the electronic device may include an operation of acquiring a first histogram of a first region among the plurality of regions. The method of operation of the electronic device may include an operation of acquiring a second histogram of a first region by weighting the first histogram of the first region and the first histogram of a second region among regions adjacent to the first region. The method of operation of the electronic device may include an operation of acquiring a third histogram of a first region by redistributing the frequencies of the second histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a first gain map of a first region based on the third histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a gain value of a first pixel included in the first region using the first gain map of the first region. The method of operation of the electronic device may include an operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0375] According to a disclosed embodiment, a method of operation of an electronic device may include an operation of acquiring a first image using a camera of the electronic device. The method of operation of the electronic device may include an operation of dividing the first image into a plurality of regions. The method of operation of the electronic device may include an operation of acquiring a first histogram of a first region among the plurality of regions. The method of operation of the electronic device may include an operation of acquiring a second histogram of the first region by redistributing the frequencies of the first histogram. The method of operation of the electronic device may include an operation of acquiring a third histogram of the first region by weighting the second histogram of the first region and the second histogram of the second region among regions adjacent to the first region. The method of operation of the electronic device may include an operation of acquiring a first gain map of the first region based on the third histogram of the first region. The method of operation of the electronic device may include an operation of acquiring a gain value of a first pixel included in the first region using the first gain map of the first region. The method of operation of the electronic device may include an operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel.

[0376] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0377] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0378] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0379] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0380] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' merely means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0381] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0382] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In the method of operating an electronic device, The operation of acquiring a first image using the camera of the electronic device; The operation of dividing the above first image into a plurality of regions; The operation of obtaining a first histogram of a first region among the plurality of regions above; The operation of obtaining a second histogram of the first region by redistributing the frequencies of the first histogram; The operation of obtaining a first gain map of the first region based on the second histogram of the first region; The operation of obtaining a second gain map of a first region by weighting the first gain map of the first region and the first gain map of a second region among regions adjacent to the first region; An operation of obtaining a gain value of a first pixel included in the first region using the second gain map of the first region; and The operation of acquiring a second image by applying the gain value of the first pixel to the pixel value of the first pixel, Method of operation. In paragraph 1, The operation of obtaining the second histogram of the first region above is, An operation to identify the first mean and the first standard deviation of the first histogram of the first region; and The operation of obtaining a second histogram of the first region by redistributing the frequencies of the first histogram based on the first mean and first standard deviation of the first histogram of the first region, Method of operation. In paragraph 2, The operation of obtaining the second histogram of the first region is, The operation of obtaining a first clip limit value of the first histogram based on the first mean and first standard deviation of the first histogram of the first region; and The operation of distributing the frequencies of the first histogram that exceed the first clip limit value of the first histogram to the intervals of the first histogram. Method of operation. In paragraph 1, The operation of obtaining the second gain map of the first region is, The operation of performing a weighted sum of a first gain map of the first region and a first gain map of the second region using a mask comprising weights corresponding to each of the first region and regions adjacent to the first region, Method of operation. In paragraph 1, The operation of obtaining the first histogram of the first region is, The method includes the operation of obtaining a black and white value of the first pixel from the color pixel value of the first pixel using a first coefficient that adjusts the ratio of the maximum value among the color pixel values ​​of the first pixel and the brightness value of the first pixel. Method of operation. In paragraph 1, The operation of obtaining the gain value of the first pixel above is, The operation of obtaining a gain value of the first pixel by adding the second gain map of the first region and the second gain map of the second region based on the coordinates of the first pixel, the coordinates of the center of the first region, and the coordinates of the center of the second region. Method of operation. In paragraph 1, The operation of dividing the above-mentioned first image into multiple regions is The operation of downsampling the first image above; and The operation of dividing the downsampled first image into the plurality of regions, Method of operation. In electronic devices, camera; At least one processor including a processing circuit; Memory containing instructions; The above instructions are executed by the at least one processor, thereby causing the electronic device: A first image is obtained using the camera of the electronic device, and The above first image is divided into a plurality of regions, and Among the plurality of regions above, a first histogram of a first region is obtained, and By redistributing the frequencies of the first histogram, a second histogram of the first region is obtained, and Based on the second histogram of the first region, a first gain map of the first region is obtained, and A second gain map of the first region is obtained by weighting the first gain map of the first region and the first gain map of the second region among regions adjacent to the first region. Using the second gain map of the first region, the gain value of the first pixel included in the first region is obtained, and By applying the gain value of the first pixel to the pixel value of the first pixel, a second image is obtained. Electronic device. In paragraph 8, The above instructions are executed by the at least one processor, thereby causing the electronic device: Identifying the first mean and first standard deviation of the first histogram of the first region, and A method for obtaining a second histogram of the first region by redistributing the frequencies of the first histogram based on the first mean and first standard deviation of the first histogram of the first region. Electronic device. In Paragraph 9, The above instructions are executed by the at least one processor, thereby causing the electronic device: A first clip limit value of the first histogram is obtained based on the first mean and first standard deviation of the first histogram of the first region, and Distributing the frequencies of the first histogram that exceed the first clip limit value of the first histogram to the intervals of the first histogram. Electronic device. In paragraph 8, The above instructions are executed by the at least one processor, thereby causing the electronic device: A method for performing a weighted sum of a first gain map of the first region and a first gain map of the second region using a mask comprising weights corresponding to each of the first region and regions adjacent to the first region. Electronic device. In Paragraph 11, A first weight corresponding to the second region included in the mask is determined based on the similarity between the first histogram of the first region and the first histogram of the second region. Electronic device. In Paragraph 12, The similarity between the first histogram of the first region and the first histogram of the second region is Determined based on the deviation between the first clip limit value of the first histogram of the first region and the second clip limit value of the first histogram of the second region. Electronic device. In Paragraph 11, The above instructions are executed by the at least one processor, thereby causing the electronic device: A first clip limit value of the first histogram is obtained based on the first mean and first standard deviation of the first histogram of the first region, and The first weight corresponding to the second region included in the mask is determined based on the first clip limit value of the first histogram, Electronic device. In paragraph 8, The above instructions are executed by the at least one processor, thereby causing the electronic device: A method for obtaining a black and white value of the first pixel from the color pixel value of the first pixel using a first coefficient that adjusts the ratio of the maximum value among the color pixel values ​​of the first pixel and the brightness value of the first pixel. Electronic device.

Images