Electronic device for image processing, image processing method thereof, and non-transitory computer-readable storage medium

Abstract

An electronic device according to various embodiments may comprise a camera, at least one processor, and a memory storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to: identify, by using segmentation, one or more regions included in a first image acquired by the camera; perform first image signal processing on a region associated with a designated category among the identified one or more regions; acquire statistical information about the region associated with the designated category on the basis of the result of the first image signal processing; determine at least one image processing operation on the basis of the statistical information and a database stored in the at least one memory; and acquire a second image by performing, on the first image, second image signal processing including the at least one image processing operation.

Description

Electronic device for performing image processing, image processing method thereof and non-transitory computer-readable storage medium

[0001] The present disclosure relates to an electronic device for performing image processing, an image processing method thereof, and a non-transitory computer-readable storage medium.

[0002] An electronic device, including a digital camera, can generate a digital image using an image sensor that detects light and outputs the detected light as an electrical signal. The electronic device can perform an operation to process the image signal output from the image sensor to improve image quality. For example, the electronic device can perform an image processing operation on the image signal depending on the shooting environment in which the image is being taken. For example, the electronic device can perform at least one of an operation to adjust white balance, an operation to correct color, an operation to remove noise from an image, or an operation to adjust the contrast of an image.

[0003] Electronic devices can determine the environment in which an image is captured based on pixel values ​​and brightness information contained in the image. For example, the electronic device can determine exposure-related parameters (e.g., shutter speed, sensor gain) for capturing an image based on the ratio of values ​​and brightness values ​​between each color channel (e.g., red channel, green channel, and blue channel) contained in the pixel values.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0005] An electronic device according to one embodiment may include a camera, at least one processor, and a memory storing instructions. The instructions, when executed by the at least one processor, may cause the electronic device to identify at least one area included in a first image acquired by the camera using segmentation. The instructions, when executed by the at least one processor, may cause the electronic device to perform first image signal processing on an area associated with a designated category among the at least one identified area. The instructions, when executed by the at least one processor, may cause the electronic device to obtain statistical information on an area associated with the designated category based on a result of performing the first image signal processing. The instructions, when executed by the at least one processor, may cause the electronic device to determine at least one image processing operation based on the statistical information and a database stored in the at least one memory. The above instructions, when executed by the at least one processor, may cause the electronic device to perform second image signal processing including the at least one image processing operation on the first image to obtain a second image.

[0006] A method performed by an electronic device according to one embodiment may include an operation of identifying at least one area included in a first image acquired by a camera of the electronic device using segmentation. The method may include an operation of performing first image signal processing on an area associated with a designated category among the at least one area identified. The method may include an operation of obtaining statistical information on the area associated with the designated category based on a result of performing the first image signal processing. The method may include an operation of determining at least one image processing operation based on the statistical information and a database stored in at least one memory of the electronic device. The method may include an operation of performing second image signal processing including the at least one image processing operation on the first image to obtain a second image.

[0007] A non-transitory storage medium recording a program according to one embodiment may record a program for executing the above-described method.

[0008] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0009] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0010] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0011] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0012] FIG. 4 is a flowchart illustrating a procedure for operating an electronic device according to one embodiment.

[0013] FIG. 5 illustrates an example of a segmentation map obtained from an image according to one embodiment.

[0014] Figure 6 illustrates an example of reference information and statistical information of a database according to one embodiment.

[0015] FIG. 7 is a flowchart illustrating a procedure in which an electronic device determines an image processing operation using statistical information according to one embodiment.

[0016] FIG. 8 is a flowchart illustrating a process by which an electronic device, according to one embodiment, classifies a scene and determines an image processing operation to perform.

[0017] FIG. 9 is a conceptual diagram illustrating an example of statistical information for determining scene classification by an electronic device according to one embodiment.

[0018] FIG. 10 is a flowchart illustrating a process by which an electronic device according to one embodiment determines whether a cloud exists in an image.

[0019] FIG. 11 is a flowchart illustrating a procedure for an electronic device to update reference information included in a database according to one embodiment.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0021] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0022] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The number of processors (120) may be one or more. For example, the processor (120) may have a multi-core processor structure such as a dual core, quad core, or hexa core.

[0025] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in the memory (130). For example, the processor (120) can correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In an embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0046] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0047] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0048] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0049] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0050] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0051] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0052] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0053] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjust the read-out timing) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0054] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, or read-out timing control) on at least one of the components included in the camera module (180) (e.g., the image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., the memory (130), the display module (160), the electronic device (102), the electronic device (104), or the server (108)). According to one embodiment, the image signal processor (260) may be configured as at least a part of the processor (120), It may be configured as a separate processor that operates independently from the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0055] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0056] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one embodiment.

[0057] An electronic device (101) according to one embodiment may include a display (310) (e.g., a display module (160) of FIG. 1), a camera (320) (e.g., a camera module (180) of FIG. 1 or FIG. 2), a memory (330) (e.g., a memory (130) of FIG. 1), and a processor (340) (e.g., a processor (120) of FIG. 1). FIG. 3 is provided to explain an example of a configuration of an electronic device (101) according to one embodiment, and the electronic device (101) may be configured by omitting some of the components illustrated in FIG. 3 or replacing them with other components. The electronic device (101) may further include other components in addition to the components illustrated in FIG. 3.

[0058] In one embodiment, the memory (330) may store instructions. The processor (340) may execute the instructions stored in the memory (330) to perform operations or control the operation of the electronic device (101). In the present disclosure, the operation of the electronic device (101) may be understood as being performed by the processor (340) executing the instructions stored in the memory (330).

[0059] In one embodiment, the electronic device (101) may control the camera (320) to acquire a first image. The electronic device (101) may identify at least one area included in the first image using segmentation, which classifies the area by category. Here, the category may refer to a category that classifies pixels according to the subject being photographed. The segmentation may refer to, for example, semantic segmentation, which classifies the image into a plurality of pixel sets using an artificial intelligence model. For example, the electronic device (101) may input the image into an artificial intelligence model that semantically segments the image to acquire a segmentation map. Alternatively, the electronic device (101) may transmit the image or information related to the image to an external device (e.g., the server (108) of FIG. 1) and receive a segmentation map from the external device. The electronic device (101) may perform first image signal processing on an area associated with a specified category (e.g., sky) among the identified at least one area. The first image signal processing may include an image processing operation of converting the first image into a color image format (e.g., a format similar to the Joint Photograph Expert Group (JPG) format) so as to obtain at least one of color information or brightness information from the first image. The first image signal processing may include software image signal processing corresponding to at least one of a demosaicing operation, an operation of applying an auto white balance gain, an operation of applying a color correction matrix (CCM), an operation of performing dynamic range correction, or an operation of performing gamma correction, for example. The first image signal processing may include a simple image processing operation performed so as to obtain statistical information from the first image.The electronic device (101) may calculate statistics on pixel values ​​of pixels based on data acquired as a result of performing the first image signal processing. For example, the statistics on pixel values ​​of pixels may include an average value of red channel values, an average value of green channel values, an average value of blue channel values, or an average value of luminance values. The statistics on pixel values ​​of pixels are not limited thereto and may be determined in various ways depending on the embodiment. The electronic device (101) may be configured to determine an environment for capturing an image based on statistical information including statistical values ​​calculated for an area associated with a specified category and a database stored in at least one memory (330), and to determine at least one image processing operation set to be applied to the determined environment. The image processing operation may include an operation for improving the image quality. The image processing operation may include an operation related to color expression, contrast, sharpness, or other image quality improvement elements. For example, the image processing operation may include at least one of an operation for adjusting white balance, an operation for correcting color, an operation for removing image noise, or an operation for adjusting image contrast. For example, the electronic device (101) may determine whether to perform auto white balance by applying an auto white balance gain value applied to an image captured in clear weather, or to perform auto white balance by applying an auto white balance gain value applied to an image captured in cloudy weather. Examples of the image processing operation are not limited thereto. For example, the image processing operation may include an operation for adjusting an exposure value or an operation for applying a color correction matrix. The electronic device (101) may be configured to perform second image signal processing including at least one image processing operation on the first image to obtain a second image.The second image signal processing may include image processing performed to obtain a second image from the first image.

[0060] In one embodiment, the electronic device (101) may be configured to divide an area associated with a specified category into a plurality of patches, and obtain statistical information for each of the plurality of patches based on data obtained as a result of performing the first image signal processing. For example, the electronic device (101) may perform first image signal processing on a first image or an area associated with a specified category included in the first image, divide the area associated with the specified category into a plurality of patches, and obtain statistical information for each of the plurality of patches including the area associated with the specified category based on data obtained as a result of performing the first image signal processing. Alternatively, for example, the electronic device (101) may divide an area associated with a specified category into a plurality of patches, perform first image signal processing on the plurality of patches, and obtain statistical information for each of the plurality of patches including the area associated with the specified category based on data obtained as a result of performing the first image signal processing. The plurality of patches may be included within an area associated with a specified category (e.g., the first area (510) of FIG. 5) or may include an area associated with a specified category (e.g., the first area (510) of FIG. 5) such as the second segmentation map of FIG. 5.

[0061] In one embodiment, the database may include reference information defining criteria for classifying statistical information. The electronic device (101) may be configured to compare statistical information for a plurality of patches with the reference information, classify the plurality of patches based on the comparison results, and determine at least one image processing operation based on the classification results. For example, the electronic device (101) may determine an environment for capturing an image based on the results of classifying the plurality of patches, and determine at least one image processing operation set to be applied to the determined environment. However, the present invention is not limited thereto.

[0062] In one embodiment, the electronic device (101) may be configured to determine a scene classification (or a characteristic of an environment in which the first image is captured) corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of the plurality of patches, and to determine at least one image processing operation based on the scene classification. For example, the reference information may define a criterion for classifying each patch into a color category based on statistical information acquired for each patch. The scene classification corresponding to the first image may mean a classification according to a characteristic of the environment in which the first image was captured. For example, the scene classification may include at least one of a clear daytime, a cloudy clear daytime, a clear daytime without clouds, a cloudy daytime, a cloudy daytime, a cloudy cloudy daytime, a cloudy cloudy daytime, a nighttime, a sunrise, or a sunset. For example, the electronic device (101) may classify a scene classification set for the first color category as a scene classification corresponding to the first image when the ratio of the number of patches belonging to a specified color category (e.g., a first color category) to the number of multiple patches is greater than or equal to a threshold set for the first color category.

[0063] In one embodiment, the reference information may include range information that specifies a range of values ​​stored in association with a color category. In this case, the electronic device (101) may be configured to classify a patch into a color category associated with the range information if a value for the patch among the values ​​included in the statistical information is within the range. For example, if the statistical information for a specific patch is within the range for the blue category, the electronic device (101) may classify the specific patch into the blue category. Alternatively, if the statistical information for a specific patch is within the range for the gray category, the electronic device (101) may classify the specific patch into the gray category. However, the present invention is not limited thereto.

[0064] In one embodiment, the electronic device (101) may determine whether the proportion of patches belonging to the blue category among a plurality of patches is greater than or equal to a threshold. If the proportion is greater than or equal to the threshold, the environment (e.g., weather) in which the first image was captured may have characteristics associated with the blue category (e.g., clear daytime). If the electronic device (101) determines that the proportion is greater than or equal to the threshold, the electronic device (101) may determine that the scene classification of the first image is clear daytime. In this way, the electronic device (101) may be configured to associate the blue category with the characteristics of clear daytime. If the electronic device (101) determines that the proportion is greater than or equal to the threshold or determines that the scene classification of the first image is clear daytime, the electronic device (101) may be configured to further determine whether clouds exist within an area associated with a specified category, and determine at least one image processing operation based on whether clouds are determined to exist. For example, if there is a patch belonging to the gray category among a plurality of patches including an area associated with a specified category, the electronic device (101) may determine that clouds exist within the area associated with the specified category (e.g., sky). If the electronic device (101) determines that clouds exist, the electronic device (101) may determine an image processing operation suitable for a cloudy, clear daytime, and if the electronic device (101) determines that clouds do not exist, the electronic device (101) may determine an image processing operation suitable for a cloudless, clear daytime. However, the present invention is not limited thereto.

[0065] In one embodiment, the electronic device (101) may be configured to determine an image processing setting including at least one of an exposure value for an image sensor included in the camera (320), a sensor gain value for the image sensor, a color correction matrix, or a noise reduction processing strength based on a comparison result of statistical information and reference information, and to determine at least one image processing operation based on the image processing setting. The electronic device (101) may perform second image signal processing including at least one image processing operation determined for the first image to obtain a second image.

[0066] FIG. 4 is a flowchart illustrating a procedure for operating an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) according to one embodiment.

[0067] The operation method supported by the electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to the flowchart illustrated in FIG. 4. The flowchart illustrated in FIG. 4 is merely a flowchart according to one embodiment of the operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 410 to 450 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 , the image signal processor (260) of FIG. 2 , or the processor (340) of FIG. 3 ).

[0068] Referring to FIG. 4, according to one embodiment, in operation 410, the electronic device may identify at least one area included in a first image acquired through a camera (e.g., the camera module (180) of FIGS. 1 and 2) through segmentation.

[0069] In one embodiment, the first image may include image data output by an image sensor included in a camera of the electronic device (e.g., the image sensor (230) of FIG. 2). The first image may include image data prior to at least a portion of image signal processing performed by the electronic device. For example, the first image may include raw image data having a Bayer pattern output from the image sensor. However, the present invention is not limited thereto. For example, the first image may include data that has been demosaiced from a raw image to be converted into a color image, but has not been subjected to other image signal processing (e.g., auto white balance, color correction, gamma correction, or dynamic range correction).

[0070] In one embodiment, at least one region may include regions classified by category by semantically segmenting pixels included in the first image. For example, the at least one region may include at least one of a region associated with a structure (e.g., a building), a region associated with the sky, a region associated with the ground (e.g., a road), a region associated with plants (e.g., street trees, flower beds), or a region associated with a person. The electronic device may determine (or identify) a location of the at least one region. The electronic device may store a location where the at least one region is identified within the first image.

[0071] In one embodiment, the electronic device may use a segmentation algorithm to classify the categories to which pixels included in the first image belong to identify at least one region. In the present disclosure, segmentation or semantic segmentation may refer to the process of dividing an image into multiple sets of pixels.

[0072] According to one embodiment, in operation 410, the electronic device can identify at least one area (e.g., an area associated with an object, an area associated with the sky, an area associated with the ground, an area associated with a plant, an area associated with a person) by categorizing or classifying pixels included in the first image into a plurality of categories (e.g., a set of pixels associated with an object, a set of pixels associated with the sky, a set of pixels associated with the ground, a set of pixels associated with a plant, a set of pixels associated with a person). At this time, the category or class may be predetermined, or may be determined based on a model that has learned the type of the subject classified based on a machine learning algorithm. However, the present invention is not limited thereto.

[0073] According to one embodiment, in operation 420, the electronic device may perform first image signal processing on an area associated with a specified category among the at least one identified area.

[0074] In one embodiment, the designated category may include a category or classification identified using a segmentation algorithm. In operation 420, the electronic device may perform first image signal processing on the first image. The electronic device may perform the first image signal processing on an area including an area associated with the designated category. For example, if the designated category includes a category indicating the sky, the electronic device may identify an area associated with the sky and perform the first image signal processing on the area associated with the sky. Alternatively, if the designated category includes a category indicating the ground, the electronic device may identify an area associated with the ground. The electronic device may perform the first image signal processing on the area associated with the ground. The operation of the electronic device identifying an area associated with the designated category may include an operation of storing a location of the area. In one embodiment, the electronic device may perform the first image signal processing on the first image. In one embodiment, the electronic device may also perform the first image signal processing on an area associated with the designated category among the first image.

[0075] In the present disclosure, the first image signal processing may refer to a simple image processing operation performed to obtain statistical information from the first image. The first image signal processing may include an image processing operation of converting the first image into a color image format (e.g., a format similar to the JPG format) to obtain at least one of color information or brightness information from the first image. The first image signal processing may include software image signal processing corresponding to at least one of a demosaicing operation, an operation of applying an auto white balance gain, an operation of applying a color correction matrix (CCM), an operation of performing dynamic range correction, or an operation of performing gamma correction. In the present disclosure, data obtained by performing the first image signal processing may be referred to as temporary image data for obtaining statistical information. In one embodiment, the first image signal processing may be configured to be performed based on a smaller amount of computation than the second image signal processing performed in operation 450.

[0076] According to one embodiment, in operation 430, the electronic device may obtain statistical information about an area associated with a specified category based on a result of performing the first image signal processing.

[0077] In the present disclosure, statistical information may refer to information including statistical values ​​calculated from temporary image data acquired as a result of performing first image signal processing. The electronic device may divide the temporary image data into a plurality of patches and obtain statistical information for each patch included in an area associated with the designated category. For example, the electronic device may classify a scene based on statistical information obtained from patches included in an area classified as having captured the sky, thereby performing a more accurate classification of the shooting environment (e.g., weather). The electronic device may calculate statistics on pixel values ​​of pixels included in a patch. For example, the statistical information may include an average value of red channel values ​​(hereinafter referred to as “R value”), an average value of green channel values ​​(hereinafter referred to as “G value”), an average value of blue channel values ​​(hereinafter referred to as “B value”), or an average value of luminance values ​​(hereinafter referred to as “Y value”) of pixels included in a patch. The method for calculating statistics is not limited thereto and may be determined in various ways depending on the embodiment. For example, an electronic device may calculate statistics based on the pixel value of at least one pixel among the pixels included in a patch. In the present disclosure, statistical information including R, G, B, and Y values ​​may be referred to as RGBY information. However, the present disclosure is not limited thereto. For example, the statistical information may include values ​​expressed as YUV values ​​including luminance values ​​and chrominance values ​​rather than RGB values.

[0078] According to one embodiment, at operation 440, the electronic device may determine at least one image processing operation based on statistical information and a database stored in memory.

[0079] In one embodiment, the image processing operation may include an operation for improving the image quality of the image. For example, the image processing operation may relate to color expression, contrast, sharpness, or other image quality improvement factors. In the present disclosure, determining the image processing operation may mean determining how to perform image signal processing on the first image. For example, determining the image processing operation may include determining whether to perform auto white balance by applying an auto white balance gain value applicable to an image captured in clear weather, or whether to perform auto white balance by applying an auto white balance gain value applicable to an image captured in cloudy weather. Examples of the image processing operation are not limited thereto. For example, the image processing operation may include an operation for adjusting an exposure value or an operation for applying a color correction matrix.

[0080] In one embodiment, a database stored in memory may include reference information. The reference information included in the database may be obtained based on information (e.g., at least one of brightness information or color information (RGB values)) about an area classified into a specified category within images classified by scene classification. For example, the reference information may include a range of average values ​​for the luminance values, red channel values, green channel values, and blue channel values ​​of pixels included in an area where the sky is captured in a plurality of images captured on clear days. In one embodiment, the electronic device may determine an image processing operation by comparing statistical information with the reference information. The reference information may define a criterion for determining the characteristics of the environment (e.g., weather, whether indoors or outdoors) in which the subject is captured from information about the subject (e.g., statistical information). The reference information may define a criterion for classifying patches into color categories based on the statistical information. Alternatively, the reference information may include range information that specifies a range of values ​​stored in association with a color category.

[0081] According to one embodiment, the reference information may include a range of R, G, B, and Y values. In this case, the electronic device may classify the statistical information by comparing statistical information (RGBY information) including R, G, B, and Y values ​​with the reference information. However, the present invention is not limited thereto. For example, the reference information may include a range of luminance values ​​and a range of chrominance values ​​rather than a range of RGB values. In this case, the electronic device may classify the statistical information by comparing statistical information including YUV values ​​with the reference information.

[0082] According to one embodiment, in operation 450, the electronic device may perform second image signal processing, including at least one image processing operation, on the first image to obtain a second image.

[0083] In the present disclosure, the second image signal processing may refer to image processing performed to obtain a second image from a first image. For example, if the number of patches classified as blue based on the comparison results between statistical information and reference information is greater than or equal to a specified ratio compared to the number of patches within an area associated with a specified category, the electronic device may obtain a second image based on at least one image processing operation set to be applied to clear weather. The second image may have image quality enhancement elements applied that are appropriate for the characteristics of the subject's shooting environment (e.g., location, time, weather).

[0084] FIG. 5 illustrates a segmentation map obtained from an image according to one embodiment.

[0085] In one embodiment, an electronic device may acquire a first image through a camera, and perform segmentation on the first image to obtain a segmentation map that classifies the first image into a plurality of regions. The electronic device may identify at least one region included in the first image through segmentation. Referring to FIG. 5, the first image may include a region (501) including the sky, a region (502) including plants, a region (503) including a road, and a region (504) including a building. The electronic device may perform segmentation on the first image to obtain a segmentation map that includes a first region (510) classified as the sky, a second region (520) classified as plants, a third region (530) classified as a road, and a fourth region (540) classified as a structure such as a building.

[0086] An electronic device can identify a first region, a second region, a third region, and a fourth region (e.g., operation 410 of FIG. 4 ). According to one embodiment, based on a result of performing the first image signal processing (e.g., operation 420 of FIG. 4 ), the electronic device can obtain statistical information (e.g., RGBY values) for a first region (510) classified into a specified category (e.g., sky) among a plurality of regions (e.g., operation 430 of FIG. 4 ). For example, the first image signal processing may include image processing performed to obtain color and / or brightness information for the first region (510). For example, the first image signal processing may include at least one of an auto white balance (AWB) operation, a color correction operation, a dynamic range correction operation, or a gamma correction operation. The electronic device can perform first image signal processing on the first region (510) to obtain color information and / or brightness information of at least one pixel included in the first region (510).

[0087] In this way, an electronic device according to one embodiment can identify an area associated with a specified category through segmentation in a first image acquired by a camera, and determine an appropriate image processing operation for the first image based on statistical information about the area associated with the specified category.

[0088] According to one embodiment, the electronic device may segment an area associated with a specified category included in the first image into a plurality of patches. Referring to FIG. 5, the electronic device may segment the entire area of ​​the first image into patches (555), such as a first segmentation map (AI Segmentation MAP 1), and obtain statistical information on patches associated with the first area (510) among the patches. The patches associated with the first area (510) may include patches included in the first area (510) or patches including the first area (510). Alternatively, the electronic device may segment only an area associated with a specified category among the first image (e.g., the first area (510)) into a plurality of patches (550), such as a second segmentation map (AI Segmentation MAP 2), and obtain statistical information on the plurality of patches (550). Referring to the second segmentation map of FIG. 5, a plurality of patches (550) may include an area (e.g., a first area (510)) associated with a specified category. A plurality of patches (550) may also be included within an area (e.g., a first area (510)) associated with a specified category.

[0089] Each patch may include at least one pixel. According to one embodiment, statistical information on patches or a plurality of patches (550) related to the first region (510) may include information (e.g., RGBY average value) obtained by calculating statistics from at least one pixel included in each patch. According to one embodiment, the electronic device may perform an image processing operation according to more accurate scene classification by determining an image processing operation based on statistical information on an area associated with a specified category.

[0090] Figure 6 illustrates an example of reference information and statistical information of a database according to one embodiment.

[0091] In one embodiment, the electronic device may determine at least one image processing operation based on statistical information and a database stored in memory (e.g., operation 440 of FIG. 4 ).

[0092] A database stored in the memory of an electronic device according to one embodiment may include reference information. In one embodiment, the electronic device may determine an image processing operation by comparing statistical information with the reference information. The reference information may define criteria for determining the characteristics of the environment (e.g., weather) in which the subject is photographed based on information about the subject (e.g., statistical information).

[0093] The reference information may define a criterion for classifying statistical information into color categories. The statistical information (602) of FIG. 6 may include a value for each patch among a plurality of patches including an area associated with a specified category. According to one embodiment, the statistical information (602) may include an R value, a G value, a B value, and a Y value. The reference information (600) may include values ​​of range information classified by color category (e.g., a range of R values, a range of G values, a range of B values, a range of Y values). The reference information (600) may define a criterion for determining which color category the statistical information (602) can be classified into.

[0094] In the present disclosure, a color category may refer to a category for classifying patches by color. Specifically, a patch within an area associated with a specified category of a first image may be classified into one color category based on statistical information of the patch. For example, an electronic device may classify each patch into a corresponding color category by comparing statistical information (602) about the patch with reference information (600). According to one embodiment, each color category may correspond to a characteristic of an environment (e.g., time, location, weather) in which the first image was captured. Referring to FIG. 6, the reference information (600) may include information about a plurality of color categories. The plurality of color categories may include a blue category corresponding to daytime on a clear day (a first color category), a gray category corresponding to daytime on a cloudy day (a second color category), a gray or black category corresponding to nighttime (a third color category), a red category corresponding to sunrise and sunset (a fourth color category), and a white category corresponding to clouds (a fifth color category). For example, if a plurality of patches among the plurality of patches including areas associated with a specified category are classified into the blue category, the electronic device may determine that the first image was captured during the daytime on a clear day. Furthermore, if a plurality of patches among the plurality of patches including areas associated with the specified category are classified into the gray category, the electronic device may determine that the first image was captured during the daytime or at night on a cloudy day. Furthermore, if some of the plurality of patches including areas associated with the specified category are classified into the white category, the electronic device may determine that the first image was captured during a cloudy day (e.g., cloudy clear weather or cloudy overcast weather).

[0095] In one embodiment, the reference information may include range information specifying a range of values ​​stored in association with a color category. The reference information (600) according to one embodiment may include range information of RGBY values ​​associated with each color category. The reference information (610) for the first color category may include range information of luminance values ​​(Y values) (1-1 range information in FIG. 6), range information of Red values ​​(1-2 range information in FIG. 6), range information of Green values ​​(1-3 range information in FIG. 6), and range information of Blue values ​​(1-4 range information in FIG. 6). When an image is captured during the day on a clear day, pixels including a sky area included in the image may have high Blue values ​​and high luminance values, and similar Red values ​​and Green values. Multiple images captured in an environment with similar characteristics (i.e., during the day on a clear day) may exhibit a tendency for high Blue values ​​and luminance values, and similar Red values ​​and Green values. Based on a plurality of reference images captured during the daytime on a clear day, a range of RGBY values ​​associated with the daytime on a clear day can be calculated. The reference information can include range information of the RGBY values ​​thus calculated. For example, the range information of RGBY values ​​included in the reference information (610) for the first color category can be determined based on the RGBY values ​​of several reference images captured during the daytime on a clear day. The electronic device can classify the statistical information into the first color category associated with the daytime on a clear day by comparing the statistical information with the range information included in the reference information. That is, the 1-1 range information and the 1-4 range information can include high luminance values ​​and high Blue values, and the 1-2 range information and the 1-3 range information can include similar Red values ​​and Green values. For example, the reference information can indicate range information by including a minimum value and / or a maximum value of values ​​stored in association with a color category.

[0096] For example, for any first patch including an area associated with a designated category, the electronic device may compare statistical information (602) of the first patch with reference information (600). If the electronic device determines that the statistical information (602) corresponds to reference information (610) for the first color category as a result of the comparison, the electronic device may classify the first patch into the first color category. Referring to FIG. 6, if the statistical information (602) of the first patch includes RGBY information of Y: 50, R: 20, G: 20, and B: 60, and includes 1-1 range information: 50 or more, 1-2 range information: 20 or more and 40 or less, 1-3 range information: 20 or more and 40 or less, and 1-4 range information: 50 or more, the electronic device may classify the first patch into the first color category. For any second patch including an area associated with the designated category, the electronic device may compare statistical information of the second patch with reference information (600). The electronic device may classify the second patch into the fifth color category if it determines that the comparison result statistical information corresponds to the reference information (650) for the fifth color category.

[0097] An electronic device can perform comparison and classification on a plurality of patches including areas associated with a specified category. The electronic device can then determine at least one image processing operation based on the classification result.

[0098] Specifically, according to one embodiment, the electronic device may determine a scene classification corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of patches as a result of performing classification on all of the plurality of patches. In the present disclosure, the scene classification corresponding to the first image may mean a classification according to characteristics of an environment in which the first image was captured. For example, the scene classification may include a clear daytime, a cloudy clear daytime, a clear daytime without clouds, a cloudy daytime, a cloudy daytime, a cloudy cloudy daytime, a cloudy cloudy daytime, a nighttime, a sunrise, or a sunset.

[0099] For example, if the ratio of the number of patches classified into the first color category to the number of patches is 80% or more, the electronic device may determine the scene classification of the first image as a clear daytime. For example, if the area associated with a specified category from the first image is divided into 100 patches, if the number of patches classified into the first color category is 80 or more, the electronic device may determine the scene classification of the first image as a clear daytime. Alternatively, the electronic device may be configured such that if the ratio of the number of patches classified into the second color category to the number of patches is greater than or equal to a threshold (e.g., 0.8), the scene classification of the first image is determined as a nighttime or a cloudy daytime. Alternatively, the electronic device may be configured such that if the ratio of the number of patches classified into the third color category to the number of patches is greater than or equal to a threshold (e.g., 0.8), the scene classification of the first image is determined as a nighttime. At this time, the reference information may define a criterion for determining the characteristics of the environment (e.g., time) in which the subject is photographed (e.g., nighttime) from statistical information. Alternatively, the electronic device may be configured such that, if the ratio of the number of patches classified into the fourth color category to the number of multiple patches is greater than a threshold (e.g., 0.8), the scene classification of the first image is determined to be sunrise or sunset.

[0100] In one embodiment, if there is a patch classified into a fifth color category, the electronic device may be configured to determine a scene classification of the first image as a weather with clouds.

[0101] Based on the scene classification determined in this manner, the electronic device can determine at least one image processing operation. In the present disclosure, the image processing operation may refer to an operation for improving the image quality. For example, the image processing operation may be related to color expression, contrast, sharpness, or other image quality improvement factors. For example, the image processing operation may include at least one of auto white balance (AWB), color correction, dynamic range correction, gamma correction, or other image processing operations.

[0102] In one embodiment, the database can be updated. The electronic device can determine to update the reference information based on user input including user feedback regarding scene classification or image processing operation determination. The electronic device can change the minimum or maximum value set for the range information to a larger or smaller value. The electronic device can modify the reference information based on statistical information of patches included in the first image. For example, if the ratio of the number of patches classified into the fourth color category to the number of multiple patches associated with a given category is less than a threshold (e.g., 0.8), the electronic device may not determine the scene classification of the first image as sunrise or sunset. In this case, the electronic device can receive a user input related to the scene classification of sunrise for the first image (e.g., a user input instructing to perform second image signal processing suitable for sunrise for the first image), and based on this, among the multiple patches that were not classified into the fourth color category, the statistical information of some patches can be used to modify the reference information for the fourth color category.

[0103] FIG. 7 is a flowchart illustrating a procedure in which an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) according to one embodiment determines an image processing operation using statistical information.

[0104] An operation method of an electronic device (101) according to one embodiment of the present disclosure for determining a retrieval processing operation using statistical information may be performed, for example, according to a flowchart illustrated in FIG. 7. The flowchart illustrated in FIG. 7 is merely a flowchart according to one embodiment of the operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 721 to 747 may be performed in at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 , the image signal processor (260) of FIG. 2 , or the processor (340) of FIG. 3 ).

[0105] In one embodiment, based on the operations illustrated in FIG. 7, the electronic device may perform an operation of performing first image signal processing on an area associated with a designated category among at least one identified area (e.g., operation 420 of FIG. 4), and an operation of obtaining statistical information on an area associated with the designated category based on a result of performing the first image signal processing (e.g., operation 430 of FIG. 4).

[0106] According to one embodiment, in operation 721, the electronic device may divide at least a portion of the first image into a plurality of patches. Specifically, the electronic device may divide the first image into a plurality of patches and obtain statistical information about the patches located in an area associated with a specified category. Alternatively, at least a portion of the first image may include an area associated with a specified category, in which case the electronic device may divide the area associated with the specified category into a plurality of patches and obtain statistical information about the patches located in the area associated with the specified category.

[0107] To this end, the electronic device can divide an area associated with a category specified in operation 721 into a plurality of patches. The electronic device can obtain statistical information for each patch in operation 731.

[0108] In the present disclosure, a patch may include at least one pixel. If the patch includes a plurality of pixels, statistical information about the patch acquired by the electronic device in operation 731 may include representative values ​​(e.g., average value, maximum value, minimum value, median value, or mode value) of the plurality of pixels constituting the patch. According to an embodiment, the statistical information about the patch may include at least one of an average value of R values, an average value of G values, an average value of B values, and an average value of Y values ​​of the pixels included in the patch. By using statistical information about the patch instead of pixel information in operations 731 and 741, the electronic device may reduce the amount of computation of the first image signal processing or the amount of computation of the comparison operation with reference information.

[0109] According to one embodiment, in operation 731, the electronic device may obtain statistical information for each of a plurality of patches based on the results of performing the first image signal processing. In operations 741 to 747, the electronic device may determine an image processing operation based on the statistical information for each patch and a database stored in the memory.

[0110] According to one embodiment, in operation 741, the electronic device may compare statistical information for a plurality of patches with reference information included in a database. According to one embodiment, the electronic device may determine whether values ​​included in the statistical information for each patch are included in the information range of a color category included in the database. The statistical information for each patch may refer to statistical information acquired for each patch with respect to the first image. The database may include reference information defining a criterion for classifying the statistical information. The reference information may include a plurality of color categories and range information for each color category. The range information for each color category may refer to a range of statistical information that a patch must have in order to be classified into the corresponding color category.

[0111] According to one embodiment, in operation 743, the electronic device may classify a plurality of patches based on the comparison result of operation 741. If the electronic device determines that the values ​​included in the statistical information of the patch fall within a reference information range associated with the color category, the electronic device may classify the patch into the corresponding color category. If the values ​​included in the statistical information do not fall within the information range of the color category defined in the reference information, some patches may not be classified into the color category defined in the reference information.

[0112] According to one embodiment, in operation 747, the electronic device may determine at least one image processing operation based on the classification result of operation 743. The electronic device according to one embodiment may determine a scene classification corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of a plurality of patches, and may determine at least one image processing operation based on the scene classification. If the ratio of the number of patches belonging to the specified color category to the number of a plurality of patches is greater than or equal to a threshold, a threshold may be set such that the electronic device determines a scene classification of the first image based on the specified color category. For example, if the threshold for the first color category is set to 0.8, if the number of patches belonging to the first color category is greater than or equal to 80% of the total number of patches, the electronic device may determine the scene classification of the first image as a clear daytime. In addition, the image processing operation to be performed by the electronic device based on the scene classification may be preset. For example, if the scene classification is determined to be a clear daytime, the electronic device may determine an image processing operation to emphasize color expression and contrast to express clear weather according to preset conditions.

[0113] In one embodiment, the electronic device may further perform secondary classification based on luminance information, time information, infrared (IR) information, flicker sensor information, or other information after determining a scene classification for the first image in operation 747. For example, if the electronic device classifies the first image as having been taken at night by comparing statistical information for the first image with reference information in a database, but the time information indicates daytime, the electronic device may determine a scene classification for the first image again.

[0114] FIG. 8 is a flowchart illustrating a process by which an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) classifies a scene and determines an image processing operation to be performed, according to one embodiment. FIG. 8 is provided solely for the purpose of illustrating the present disclosure and is not limiting, and one or more of the operations of FIG. 8 may be executed in a different order, omitted, or one or more other operations may be added.

[0115] According to one embodiment of the present disclosure, operations 810 to 850 may be performed in at least one processor (120) of the electronic device (101) (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, or processor (340) of FIG. 3).

[0116] According to one embodiment, in operation 810, the electronic device may identify at least one area included in a first image acquired by the camera through segmentation. Operation 810 may correspond to operation 410 of FIG. 4. Referring to FIG. 5, the electronic device may identify at least one area by classifying pixels included in the first image into a set of pixels associated with a workpiece, a set of pixels associated with the sky, a set of pixels associated with the ground, a set of pixels associated with a plant, and a set of pixels associated with a person.

[0117] According to one embodiment, in operation 815, the electronic device may determine that at least one area includes an area associated with a designated category. Since statistical information is obtained from a set of pixels including an area associated with the designated category, the electronic device may determine whether the first image includes an area associated with the designated category. If the electronic device determines that an area associated with the designated category is identified in the first image, the electronic device may store a position of the area associated with the designated category in operation 817. The position information may include a position of a patch including an area associated with the designated category. The position information may be used to divide an area associated with the designated category into a plurality of patches (operation 821), perform first image signal processing on patches including an area associated with the designated category (operation 820), obtain statistical information from patches including an area associated with the designated category (operation 831), compare statistical information on patches including an area associated with the designated category with reference information (841), determine a color category classification of each patch (operation 843), or determine a scene classification of the first image (operation 847). For example, the electronic device may store the location of an area associated with a specified category in a segmentation map such as FIG. 5, and obtain or store statistical information of pixels or patches matching this location.

[0118] According to one embodiment, the electronic device may divide an area associated with a specified category into a plurality of patches in operation 821. The plurality of patches may be included within an area associated with the specified category (e.g., the first area (510) of FIG. 5) or may include an area associated with the specified category (e.g., the first area (510) of FIG. 5) such as the second segmentation map of FIG. 5. Alternatively, the electronic device may divide a first image including an area associated with the specified category into patches (e.g., patches (555) of FIG. 5) and perform operation 831 on patches associated with the first area (510) among the patches. Operation 821 may correspond to operation 721 of FIG. 7.

[0119] According to one embodiment, the electronic device may perform first image signal processing on a first image including an area associated with a category specified in operation 820 or a specified category. Operation 820 may correspond to operation 420 of FIG. 4.

[0120] According to one embodiment, statistical information may be obtained for each of a plurality of patches based on the results of performing the first image signal processing in operation 831. The electronic device may calculate statistical information for each of a plurality of patches based on data obtained as a result of performing the first image signal processing on an area associated with a specified category. Operation 831 may correspond to operation 731 of FIG. 7.

[0121] According to one embodiment, in operation 841, the electronic device may compare statistical information for a plurality of patches with reference information contained in a database. The electronic device may determine whether values ​​contained in the statistical information for each patch fall within ranges specified by range information contained in the reference information. Operation 831 may correspond to operation 731 of FIG. 7.

[0122] According to one embodiment, a plurality of patches may be classified based on the comparison results in operation 843. If the values ​​included in the statistical information for a patch fall within ranges specified by the range information included in the reference information, the electronic device may classify the patch into a color category associated with the ranges. Operation 843 may correspond to operation 743 of FIG. 7.

[0123] According to one embodiment, based on the classification result of operation 843, the electronic device may determine at least one image processing operation in operation 847. For example, the electronic device may determine a scene classification (or a characteristic of an environment in which the first image is captured) corresponding to the first image based on the result of classifying the color categories for each of the plurality of patches in operation 843. The electronic device may determine an image processing operation corresponding to the determined scene classification. Operation 847 may correspond to operation 747 of FIG. 7.

[0124] According to one embodiment, the electronic device may determine whether the scene classification corresponding to the first image is a clear daytime in operation 871. Specifically, the electronic device may determine whether a ratio of patches belonging to the blue category among a plurality of patches including a sky area is greater than or equal to a first threshold. If it is determined that the ratio is greater than or equal to the first threshold, the electronic device may determine whether a patch belonging to the white category exists among the plurality of patches including the sky area in operation 873. If it is determined that a patch belonging to the white category exists, the electronic device may determine a first image processing operation in operation 875. The first image processing operation may include at least one image processing operation preset for a scene classification of a clear daytime in operation 871 and at least one image processing operation preset for a scene classification of a cloudy daytime in operation 872. For example, the first image processing operation may include at least one image processing operation related to color expression emphasis and contrast emphasis for expressing clear weather, and sharpness emphasis and contrast emphasis for expressing the gradation of clouds. In this case, in operation 850, the electronic device may perform second image signal processing including a first image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0125] In operation 873, the electronic device may determine that there are no clouds. Specifically, the electronic device may determine that there are no patches belonging to the white category. In this case, the electronic device may determine a second image processing operation in operation 877. The second image processing operation may include at least one image processing operation preset for scene classification during the daytime on a clear day. For example, the second image processing operation may include at least one image processing operation related to color expression enhancement and contrast enhancement for expressing clear weather. In this case, in operation 850, the electronic device may perform a second image signal processing operation including a second image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0126] In operation 871, if the first image determines that the corresponding scene classification is not a clear daytime or if the ratio of patches belonging to the blue category among the plurality of patches including the sky area is less than a first threshold, the electronic device may determine in operation 879 whether the corresponding scene classification of the first image is a cloudy daytime. Specifically, the electronic device may determine whether the ratio of patches belonging to the gray category among the plurality of patches including the sky area is greater than or equal to a second threshold. If the ratio is determined to be greater than or equal to the second threshold, the electronic device may determine the corresponding scene classification of the first image as a cloudy daytime or nighttime. If the ratio of patches belonging to the gray category is determined to be greater than or equal to the second threshold, the electronic device may perform a secondary classification to determine the scene classification as either a cloudy daytime or nighttime based on time information. If the time information indicates a daytime, the electronic device may determine the scene classification as a cloudy daytime or daytime based on the time information. In operation 881, the electronic device may determine whether there is a patch belonging to the white category among the plurality of patches including the sky area. If it is determined that a patch belonging to the white category exists, the electronic device may determine a third image processing operation in operation 883. The third image processing operation may include at least one image processing operation preset for classifying a scene during a daytime period on a cloudy day and at least one image processing operation preset for classifying a scene during a cloudy weather. For example, the third image processing operation may include at least one image processing operation related to brightness and contrast enhancement for improving low brightness and contrast characteristics of cloudy weather, and sharpness enhancement and contrast enhancement for expressing the gradation of clouds.In this case, in operation 850, the electronic device may perform a second image signal processing operation including a third image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0127] In operation 881, the electronic device may determine that clouds do not exist. Specifically, the electronic device may determine that no patches belonging to the white category exist. In this case, the electronic device may determine a fourth image processing operation in operation 885. The fourth image processing operation may include at least one image processing operation preset for scene classification during the daytime on a cloudy day. For example, the fourth image processing operation may include at least one image processing operation related to brightness and contrast enhancement to improve low brightness and contrast characteristics of cloudy weather. In this case, in operation 850, the electronic device may perform a second image signal processing operation including the fourth image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4 .

[0128] According to one embodiment, if the first image determines in operation 879 that the corresponding scene classification is not a cloudy daytime or that the ratio of patches belonging to the gray category among the plurality of patches including the sky area is less than a second threshold, the electronic device may determine in operation 887 whether the corresponding scene classification of the first image is nighttime. Specifically, the electronic device may determine whether the ratio of patches belonging to the gray category or the black category among the plurality of patches including the sky area is greater than or equal to a third threshold. If the ratio is determined to be greater than or equal to the third threshold, the electronic device may determine the corresponding scene classification of the first image as nighttime. In operation 889, the electronic device may determine a fifth image processing operation. The fifth image processing operation may include at least one image processing operation preset for the nighttime scene classification. For example, the fifth image processing operation may include at least one image processing operation related to brightness and contrast enhancement to improve the bright and hazy characteristics of the night, and sharpness and contrast enhancement to reduce the flare phenomenon caused by signs and lights. In this case, in operation 850, the electronic device may perform a second image signal processing operation including a fifth image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0129] In operation 879, if it is determined that the proportion of patches belonging to the gray category among the plurality of patches including the sky area is greater than or equal to the second threshold, the electronic device may perform a secondary classification to determine the scene classification as either daytime on a cloudy day or nighttime based on the time information. If the time information indicates nighttime, the electronic device may determine the scene classification corresponding to the first image as nighttime, and in this case, the fifth image processing operation may also be determined.

[0130] According to one embodiment, if the first image determines in operation 887 that the corresponding scene classification is not nighttime or if the proportion of patches belonging to the gray or black category among the plurality of patches including the sky area is less than a third threshold, the electronic device may determine in operation 891 whether the corresponding scene classification of the first image is sunset or sunrise. Specifically, the electronic device may determine whether the proportion of patches belonging to the red category among the plurality of patches including the sky area is greater than or equal to a fourth threshold. If the proportion is determined to be greater than or equal to the fourth threshold, the electronic device may determine the corresponding scene classification of the first image as sunset or sunrise. In operation 893, the electronic device may determine the corresponding scene classification of the first image as either sunset or sunrise based on time information. If the time information indicates between about 5:00 AM and 7:00 AM, the electronic device may determine the scene classification as sunrise and determine the sixth image processing operation in operation 895. The sixth image processing operation may include at least one image processing operation related to color expression enhancement for expressing a sunrise. In this case, in operation 850, the electronic device may perform a second image signal processing operation, including the sixth image processing operation, on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0131] For example, if the time information indicates between approximately 5 PM and 7 AM, the electronic device may determine the scene classification as sunset and determine a seventh image processing operation at operation 897. The seventh image processing operation may include at least one image processing operation related to color expression enhancement for expressing sunset. In this case, at operation 850, the electronic device may perform a second image signal processing operation including the seventh image processing operation on the first image to obtain a second image. Operation 850 may correspond to operation 450 of FIG. 4.

[0132] Operation 847, which determines a scene classification corresponding to the first image based on the classification result of operation 843, may include operations 871, 873, 879, 881, 887, and / or 891. The order of operations 871, 873, 879, 881, 887, and 891 is not limited to the order of FIG. 8, and one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added. According to one embodiment, operations that perform classification based on luminance information, time information, IR information, flicker sensor information, or other information may be added, and such additional classification may be performed before or after operations 871, 873, 879, 881, 887, and 891.

[0133] FIG. 9 is a conceptual diagram illustrating an example of statistical information for determining scene classification by an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) according to one embodiment.

[0134] Referring to FIG. 9, the first image is divided into 80 patches, and each of the patches (950) including an area (e.g., sky area) associated with a designated category among the 80 patches is shown with a color category determined based on statistical information. In FIG. 9, a capital letter B represents a blue category (e.g., the first color category of FIG. 6), a capital letter G represents a gray category (e.g., the second color category of FIG. 6), and a capital letter W represents a white category (e.g., the fifth color category of FIG. 6).

[0135] According to one embodiment, the electronic device may determine at least one image processing operation based on the classification result of FIG. 9 (operation 747). Specifically, the electronic device may determine whether a ratio of patches included in the blue category among patches (950) including an area associated with a specified category is greater than or equal to a first threshold. Referring to FIG. 9, since the number of a plurality of patches including an area associated with a specified category (e.g., sky) is 37 and the number of patches belonging to the blue category is 31, the electronic device may determine whether the ratio 31 / 37 is greater than or equal to the first threshold. For example, when the first threshold is set to 0.8, the electronic device may determine that the ratio 31 / 37 is greater than the first threshold, and based on this, may determine a scene classification corresponding to the first image as a clear daytime.

[0136] Additionally, according to one embodiment, the electronic device may determine whether a threshold number of patches belonging to the white category (or gray category) exist among the patches (950) that include an area associated with a specified category. Referring to FIG. 9 , since patches (905) belonging to the white category exist, the electronic device may determine that clouds exist within the area associated with the specified category. Based on this, the electronic device may determine the scene classification corresponding to the first image as cloudy weather.

[0137] Additionally, according to one embodiment, the electronic device may determine whether there is a patch included in the gray category among the patches (950) including an area associated with a specified category. Referring to FIG. 9, since the number of patches belonging to the gray category is three, the electronic device may determine whether the ratio 3 / 37 is greater than or equal to the second threshold. For example, when the second threshold is set to 0.8, the electronic device may determine that the ratio 3 / 37 is less than the first threshold, and based on this, may determine that the scene classification corresponding to the first image is not a cloudy day.

[0138] FIG. 10 is a flowchart illustrating a process by which an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) determines whether a cloud exists in an image according to one embodiment.

[0139] According to one embodiment of the present disclosure, operations 1041 to 1077 may be performed in at least one processor (120) of the electronic device (101) (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, or processor (340) of FIG. 3).

[0140] According to one embodiment, in operation 1041, the electronic device may compare statistical information on a plurality of patches with reference information included in a database. In operation 1043, the electronic device may classify the plurality of patches based on the comparison result. In operation 1071, the electronic device may determine that the proportion of patches belonging to the blue category among the plurality of patches is greater than or equal to a first threshold. Operations 1041, 1043, and 1071 may correspond to operations 741, 743, and 871.

[0141] In one embodiment, an electronic device may determine whether clouds exist within an area associated with a specified category (e.g., sky) in operation 1073. The electronic device may determine whether a number of patches belonging to the white category among a plurality of patches including an area associated with the specified category exceeds a threshold. If it is determined that a patch belonging to the white category exists, the electronic device may determine at least one image processing operation related to sharpness and contrast enhancement for expressing the gradation of clouds. Referring to FIG. 10 , since the electronic device determines that the scene classification of the first image is a clear daytime in operation 1071, the first image processing operation of operation 1075 may include at least one image processing operation related to color expression enhancement and contrast enhancement for expressing clear weather, and sharpness enhancement and contrast enhancement for expressing the gradation of clouds. In this case, the electronic device may perform a second image signal processing operation including the first image processing operation on the first image to obtain a second image (operation 450 of FIG. 4 ). If it is determined that there is no patch belonging to the white category, the electronic device may determine a second image processing operation including at least one image processing operation related to color expression emphasis and contrast emphasis for expressing clear weather based on the scene classification determined in operation 1071 (operation 1077). In this case, the electronic device may perform a second image signal processing including the second image processing operation on the first image to obtain a second image (operation 450 of FIG. 4).

[0142] According to one embodiment, the determination of the presence of clouds, such as operation 1073, can be determined independently from other scene classifications (clear day, cloudy day, night, sunset, sunrise). Therefore, the operation of determining whether a patch belonging to the white category exists among a plurality of patches including an area associated with a designated category can be performed even when the determination of operation 1071 is not made, unlike in FIG. 10, and can be performed before operation 871 or operation 879 in FIG. 8, or additionally performed after operations 887, 891, and 893. Operations 1073, 1075, and 1077 can correspond to operations 873, 875, and 877 in FIG. 8, in that order.

[0143] FIG. 10 is merely intended to illustrate a process for determining whether clouds exist within an area associated with a specified category, and does not limit the present disclosure. For example, according to one embodiment, even when the electronic device determines that the proportion of patches belonging to the gray category among a plurality of patches is equal to the second threshold, it can determine whether clouds exist within an area associated with a specified category, as in operation 1073. In this case, operation 1073 may correspond to operation 881 of determining the presence of clouds in FIG. 8.

[0144] FIG. 11 is a flowchart illustrating a procedure for an electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) to update reference information included in a database according to one embodiment.

[0145] A procedure for updating reference information included in a database by an electronic device (101) according to one embodiment of the present disclosure may be performed, for example, according to a flowchart illustrated in FIG. 11. The flowchart illustrated in FIG. 11 is merely a flowchart according to one embodiment of the operation of the electronic device (101), and the order of at least some operations may be changed, performed in parallel, performed as independent operations, or at least some other operations may be performed complementarily to at least some operations. According to one embodiment of the present disclosure, operations 1141 to 1180 may be performed by at least one processor (120) of the electronic device (101) (e.g., the processor (120) of FIG. 1 , the image signal processor (260) of FIG. 2 , or the processor (340) of FIG. 3 ).

[0146] Referring to FIG. 11, an electronic device according to an embodiment may compare statistical information acquired in operation 1141 with reference information included in a database. Operation 1141 may correspond to operation 741 of FIG. 7, operation 841 of FIG. 8, or operation 1041 of FIG. 10. In operation 1143, the electronic device may classify a plurality of patches based on the comparison result. Operation 1143 may correspond to operation 743 of FIG. 7, operation 843 of FIG. 8, or operation 1043 of FIG. 10. In operation 1147, at least one image processing operation may be determined based on the classification result. Operation 1147 may correspond to operation 747 of FIG. 7 or operation 847 of FIG. 8. In operation 1150, the electronic device may perform second image signal processing including at least one image processing operation on the first image to generate a second image. Action 1150 may correspond to action 450 of FIG. 4 or action 850 of FIG. 8.

[0147] In operation 1160, the electronic device according to one embodiment may display an object indicating the scene classification or image processing operation determined in operation 1147 through the display of the electronic device. For example, when a second image is acquired by performing second image signal processing including an image processing operation on an image captured on a cloudy day, the electronic device may display an icon indicating that the second image was acquired by determining that the image was captured on a cloudy day, together with the second image. For example, the electronic device may further display the first image and / or the second image, thereby guiding the user as to whether the scene classification or image processing operation has been appropriately determined for the first image, whether the second image signal processing has been appropriate, or whether a problem of the first image has been improved in the second image. The electronic device may further display a GUI (graphical user interface) for updating the database.

[0148] According to one embodiment, in operation 1170, the electronic device may receive user input associated with a scene classification. The user input for updating the database may include user feedback regarding the scene classification or the image processing operation determination. For example, the user input for updating the database may instruct to modify, delete, or redetermine the scene classification determined for the first image, instruct to cancel execution of one or more of the determined at least one image processing operation, or instruct to execute a new image processing operation.

[0149] In operation 1180, an electronic device according to an embodiment may update a database with acquired statistical information based on a user input. The electronic device may determine whether to update the reference information based on the received user input. The electronic device may store the changed reference information in a memory based on a scene classification or image processing operation associated with the user input. For example, after performing a second image signal processing operation including an image processing operation on an image captured on a cloudy day to acquire a second image, if a user input for selecting a clear day is received from the user and the second image is replaced with an image for which an image processing operation corresponding to a clear day is performed and stored, the electronic device may update the reference information associated with a clear day based on the statistical information on the corresponding image.

[0150] For example, if it is determined that the ratio of the number of patches classified into the gray category among the number of multiple patches associated with a given category is less than a second threshold and the ratio of the number of patches classified into the gray or black category is greater than a third threshold, in operation 1147, the electronic device may determine the scene classification of the first image as night, and in operation 1150, perform a second image signal processing operation including a fifth image processing operation on the first image to generate a second image with emphasized clarity and contrast. In operation 1160, the electronic device may display on the display the first image, the second image, information indicating that the determined scene classification is night, and / or information indicating that the determined at least one image processing operation is the fifth image processing operation. If, at operation 1170, the electronic device receives a user input to change the scene classification of the first image to a cloudy daytime, and since the number of patches classified into the gray category at operation 1143 is smaller than the number of patches actually belonging to the gray category, at operation 1180, the electronic device may modify the range information for the gray category to be wider. For example, the electronic device may change the minimum value set for the range information for the gray category to a smaller value or change the maximum value to a larger value.

[0151] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0152] An electronic device (e.g., the electronic device (101) of FIG. 1 or FIG. 3) according to one embodiment may include a camera (e.g., the camera module (180) of FIGS. 1 and 2, or the camera (320) of FIG. 3), at least one processor (e.g., the processor (120) of FIG. 1, the processor (340) of FIG. 3), and a memory (e.g., the memory (130) of FIG. 1, the memory (330) of FIG. 3)) that stores instructions. When the instructions are executed by the at least one processor (e.g., the processor (120) of FIG. 1, the processor (340) of FIG. 3), the instructions may cause the electronic device to identify at least one area included in a first image acquired by the camera using segmentation. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1, the processor 340 of FIG. 3), may cause the electronic device to perform first image signal processing on an area associated with a designated category among the at least one identified area. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1, the processor 340 of FIG. 3), may cause the electronic device to obtain statistical information on an area associated with the designated category based on a result of performing the first image signal processing. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1, the processor 340 of FIG. 3), may cause the electronic device to determine at least one image processing operation based on the statistical information and a database stored in the at least one memory. The above instructions, when executed by the at least one processor (e.g., the processor (120) of FIG. 1, the processor (340) of FIG. 3), may cause the electronic device to perform second image signal processing including the at least one image processing operation on the first image to obtain a second image.

[0153] In one embodiment, the instructions, when executed by the at least one processor (e.g., processor 120 of FIG. 1 , processor 340 of FIG. 3 ), may cause the electronic device to divide an area associated with the designated category into a plurality of patches. The instructions, when executed by the at least one processor (e.g., processor 120 of FIG. 1 , processor 340 of FIG. 3 ), may cause the electronic device to obtain statistical information for each of the plurality of patches based on a result of performing the first image signal processing.

[0154] In one embodiment, the database may include reference information defining a reference for classifying the statistical information. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1 , the processor 340 of FIG. 3 ), may cause the electronic device to compare statistical information about the plurality of patches with the reference information. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1 , the processor 340 of FIG. 3 ), may cause the electronic device to classify the plurality of patches based on a result of the comparison. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1 , the processor 340 of FIG. 3 ), may cause the electronic device to determine the at least one image processing operation based on a result of the classification.

[0155] In one embodiment, the instructions, when executed by the at least one processor (e.g., processor 120 of FIG. 1 , processor 340 of FIG. 3 ), may cause the electronic device to determine a scene classification corresponding to the first image based on a ratio of a number of patches belonging to a specified color category to a number of the plurality of patches. The instructions, when executed by the at least one processor (e.g., processor 120 of FIG. 1 , processor 340 of FIG. 3 ), may cause the electronic device to determine the at least one image processing operation based on the scene classification.

[0156] In one embodiment, the reference information may include range information specifying a range of values ​​stored in association with a color category. When the instructions are executed by the at least one processor (e.g., processor (120) of FIG. 1, processor (340) of FIG. 3), the electronic device may classify the patch into the color category associated with the range information if a value for the patch among the values ​​included in the statistical information falls within the range.

[0157] In one embodiment, the instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1 , the processor 340 of FIG. 3 ), may cause the electronic device to determine whether a cloud exists within an area associated with the designated category if a ratio of patches belonging to the blue category among the plurality of patches is greater than or equal to a threshold. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1 , the processor 340 of FIG. 3 ), may cause the electronic device to determine the at least one image processing operation based on whether it is determined that a cloud exists.

[0158] In one embodiment, the instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1, the processor 340 of FIG. 3), may cause the electronic device to determine an image processing setting including at least one of an exposure value for an image sensor included in the camera, a sensor gain value for the image sensor, a color correction matrix, or a noise reduction processing strength based on a comparison result of the statistical information and the reference information. The instructions, when executed by the at least one processor (e.g., the processor 120 of FIG. 1, the processor 340 of FIG. 3), may cause the electronic device to determine the at least one image processing operation based on the image processing setting.

[0159] In one embodiment, the computational amount of the first image signal processing may be less than the computational amount of the second image signal processing.

[0160] In one embodiment, the statistical information may include statistical values ​​for pixel values ​​of pixels included in at least a portion of an area associated with a specified category.

[0161] In one embodiment, the instructions, when executed by the at least one processor (e.g., processor (120) of FIG. 1, processor (340) of FIG. 3), may cause the electronic device to store in the memory the location of an area associated with the designated category.

[0162] In one embodiment, the specified category may include sky.

[0163] A method performed by an electronic device (e.g., the electronic device 101 of FIG. 1 or 3) according to one embodiment may include an operation of identifying at least one area included in a first image acquired by a camera of the electronic device using segmentation. The method may include an operation of performing first image signal processing on an area associated with a designated category among the at least one area identified. The method may include an operation of obtaining statistical information on the area associated with the designated category based on a result of performing the first image signal processing. The method may include an operation of determining at least one image processing operation based on the statistical information and a database stored in at least one memory of the electronic device. The method may include an operation of performing second image signal processing including the at least one image processing operation on the first image to obtain a second image.

[0164] In one embodiment, the operation of performing first image signal processing on an area associated with a designated category among the at least one identified area may include an operation of dividing the area associated with the designated category into a plurality of patches. The operation of obtaining statistical information on the area associated with the designated category may include an operation of obtaining statistical information for each of the plurality of patches.

[0165] In one embodiment, the database may include reference information defining a criterion for classifying the statistical information. The operation of determining the at least one image processing operation may include: comparing statistical information about the plurality of patches with the reference information; classifying the plurality of patches based on the comparison result; and determining the at least one image processing operation based on the classification result.

[0166] In one embodiment, the operation of determining the at least one image processing operation may include: determining a scene classification corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of the plurality of patches; and determining the at least one image processing operation based on the scene classification.

[0167] In one embodiment, the reference information may include range information specifying a range of values ​​stored in association with a color category. The operation of determining the at least one image processing operation may include an operation of classifying the patch into the color category associated with the range information if a value for the patch among the values ​​included in the statistical information falls within the range.

[0168] In one embodiment, the operation of determining the at least one image processing operation may include an operation of determining whether a cloud exists within an area associated with the designated category when a ratio of patches belonging to the blue category among the plurality of patches is greater than or equal to a threshold; and an operation of determining the at least one image processing operation based on whether the cloud is determined to exist.

[0169] In one embodiment, the operation of determining the at least one image processing operation may include: an operation of determining an image processing setting including at least one of an exposure value for an image sensor included in the camera, a sensor gain value for the image sensor, a color correction matrix, or a noise reduction processing strength based on a comparison result of the statistical information and the reference information; and an operation of determining the at least one image processing operation based on the image processing setting.

[0170] In one embodiment, the computational amount of the first image signal processing may be less than the computational amount of the second image signal processing.

[0171] A non-transitory storage medium recording a program according to one embodiment may record a program for executing the above-described method.

[0172] According to various embodiments, an electronic device and a method of operating the same can be provided that can determine an optimal image processing operation by directly considering a light source or weather factor that substantially affects an image.

[0173] According to various embodiments, an electronic device and an operating method thereof can be provided that reduce processing or computational load and save resources by utilizing statistical information of patches and / or utilizing first image signal processing while determining an optimal image processing operation.

[0174] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0175] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0176] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

[0177] In the present disclosure, the functions or operations performed by the electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The functions or operations of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include circuitry for performing calculations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operations of the electronic device described above.

[0178] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0179] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0180] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0181] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0182] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0183] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0184] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, including both a and b, including both b and c, including both a and c, or including all of a, b, and c.”

[0185] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In an electronic device (101), Camera (180, 320); At least one processor (120, 340); At least one memory (130, 330) for storing instructions; and When the above instructions are executed by the at least one processor, the electronic device: Identifying at least one area included in a first image acquired by the camera using segmentation, Performing first image signal processing on an area associated with a designated category among at least one of the above-identified areas, Obtain statistical information on an area associated with the specified category based on the result of performing the first image signal processing, Determine at least one image processing operation based on the statistical information and the database stored in the at least one memory, An electronic device that performs second image signal processing including at least one image processing operation on the first image to obtain a second image.

2. In claim 1, When the above instructions are executed by the at least one processor, the electronic device: Divide the area associated with the above-mentioned category into multiple patches, An electronic device configured to obtain statistical information for each of the plurality of patches based on the results of performing the first image signal processing.

3. In claim 2, The above database includes reference information defining criteria for classifying the above statistical information, When the above instructions are executed by the at least one processor, the electronic device: Compare statistical information on the above multiple patches with the above reference information, Classify the plurality of patches based on the above comparison results, An electronic device that determines at least one image processing operation based on the classification result.

4. In claim 3, When the above instructions are executed by the at least one processor, the electronic device: Determine a scene classification corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of the plurality of patches, An electronic device that determines at least one image processing operation based on the scene classification.

5. In claim 3, The above reference information includes range information that specifies the range of values ​​stored in association with the color category, When the above instructions are executed by the at least one processor, the electronic device: An electronic device that classifies the patch into the color category associated with the range information when the value for the patch among the values ​​included in the statistical information is within the range.

6. In claim 5, When the above instructions are executed by the at least one processor, the electronic device: If the ratio of patches belonging to the blue category among the above multiple patches is greater than or equal to a threshold, it is determined whether a cloud exists within the area associated with the above specified category, An electronic device that determines at least one image processing operation based on whether the cloud is determined to exist.

7. In claim 3, When the above instructions are executed by the at least one processor, the electronic device: Based on the comparison result of the statistical information and the reference information, an image processing setting including at least one of an exposure value for an image sensor included in the camera, a sensor gain value for the image sensor, a color correction matrix, or a noise reduction processing intensity is determined, An electronic device that determines at least one image processing operation based on the image processing settings.

8. In claim 1, An electronic device, wherein the statistical information includes statistical values ​​for pixel values ​​of pixels included in at least a portion of an area associated with a specified category.

9. In a method performed by an electronic device (101), An operation of identifying at least one area included in a first image acquired by a camera (180, 320) of the electronic device using segmentation; An operation of performing first image signal processing on an area associated with a designated category among at least one of the identified areas; An operation of obtaining statistical information about an area associated with the specified category based on a result of performing the first image signal processing; An operation of determining at least one image processing operation based on the statistical information and a database stored in at least one memory (130, 330) of the electronic device; and A method comprising an operation of obtaining a second image by performing a second image signal processing operation including at least one image processing operation on the first image.

10. In claim 9, The operation of performing first image signal processing on an area associated with a designated category among at least one of the identified areas includes an operation of dividing the area associated with the designated category into a plurality of patches, A method wherein the operation of obtaining statistical information for an area associated with the above-mentioned specified category includes an operation of obtaining statistical information for each of the plurality of patches. The above database includes reference information defining criteria for classifying the above statistical information, The operation of determining at least one image processing operation is: An operation of comparing statistical information for the plurality of patches with the reference information; An operation of classifying the plurality of patches based on the comparison results; and A method comprising an operation of determining at least one image processing operation based on the classification result.

11. In claim 10, The operation of determining at least one image processing operation is: An operation of determining a scene classification corresponding to the first image based on a ratio of the number of patches belonging to a specified color category to the number of the plurality of patches; and A method comprising an operation of determining at least one image processing operation based on the scene classification.

12. In claim 10, The above reference information includes range information that specifies the range of values ​​stored in association with the color category, A method wherein the operation of determining at least one image processing operation includes an operation of classifying the patch into the color category associated with the range information when a value for the patch among the values ​​included in the statistical information is included within the range.

13. In claim 12, The operation of determining at least one image processing operation is: An operation of determining whether a cloud exists within an area associated with the specified category when the proportion of patches belonging to the blue category among the plurality of patches is greater than or equal to a threshold; and A method comprising an operation of determining at least one image processing operation based on whether the cloud is determined to exist.

14. In claim 10, The operation of determining at least one image processing operation is: An operation of determining an image processing setting including at least one of an exposure value for an image sensor included in the camera, a sensor gain value for the image sensor, a color correction matrix, or a noise reduction processing intensity based on a comparison result of the statistical information and the reference information; and A method comprising an operation of determining at least one image processing operation based on the image processing settings.

15. In a non-transitory computer-readable recording medium, When an electronic device (101) including a camera is running: Identifying at least one area included in the first image acquired by the above camera (130, 330) using segmentation, Performing first image signal processing on an area associated with a designated category among at least one of the above-identified areas, Obtain statistical information on an area associated with the specified category based on the result of performing the first image signal processing, Determine at least one image processing operation based on the statistical information and the database stored in the at least one memory, A recording medium having recorded thereon a computer program for performing second image signal processing including at least one image processing operation on the first image to obtain a second image.

Images