Electronic device and method for acquiring processed image on basis of machine learning model

Abstract

In an embodiment, an electronic device may comprise: one or more processors; and a memory for storing one or more instructions and a neural network model generated by training a neural network to output an inference result for information of an input image. The one or more instructions are executed by the one or more processors and may cause the electronic device to, on the basis of image information related to an image to be processed or context information related to a state of the electronic device, determine at least one image processing step for acquiring a processed image from the image to be processed, among a plurality of image processing steps performed by the neural network model. The electronic device may acquire the processed image from the image to be processed, on the basis of the determined at least one image processing step. Each of the plurality of image processing steps may include an operation of inputting the image to be processed or an output of the neural network model into the neural network model, thereby acquiring an inferred result. Various embodiments other than the above-described embodiment are possible.

Description

Electronic device and method for acquiring images processed based on a machine learning model

[0001] The present disclosure relates to an electronic device and method for obtaining an image processed based on a machine learning model.

[0002] Deep learning can be used to train neural networks and enhance images using the resulting neural network model. Image super-resolution can generate high-resolution images corresponding to input low-resolution images. AI neural networks can learn from training data using deep learning algorithms. Training data can consist of pairs of data and labels (correct answers). Training an AI neural network can require a large amount of training data. For example, an AI neural network can be configured to predict enhanced images from degraded images by learning training data consisting of pairs of images (labels) and degraded images (data).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] In one embodiment, an electronic device may include one or more processors, a memory storing a neural network model generated by training a neural network to output an inference result for information of an input image, and one or more instructions. The one or more instructions may be executed by the one or more processors to cause the electronic device to determine, based on image information related to an image to be processed or context information related to a state of the electronic device, at least one image processing step for obtaining a processed image from the image to be processed among a plurality of image processing steps performed by the neural network model. The one or more instructions may be executed by the one or more processors to cause the electronic device to obtain a processed image from the image to be processed based on the determined at least one image processing step. Each of the plurality of image processing steps may include an operation of inputting the image to be processed or an output of the neural network model into the neural network model to obtain an inferred result.

[0005] According to one embodiment, a method for operating an electronic device may include an operation of determining at least one image processing step for obtaining a processed image from an image to be processed, among a plurality of image processing steps performed based on a neural network model generated by training a neural network to predict a processing result for an input image based on image information related to an image to be processed or context information related to a state of the electronic device. According to one embodiment, a method for operating an electronic device may include an operation of obtaining a processed image from the image to be processed based on the determined at least one image processing step. The operation of obtaining the processed image may include an operation of inputting the image to be processed into the neural network model so as to perform a first step of the at least one image processing step. The operation of obtaining the processed image may include an operation of inputting an output image output as a result of performing the first step on the image to be processed by the neural network model into the neural network model so as to perform a second step of the at least one image processing step.

[0006] In one embodiment, a non-transitory computer-readable recording medium storing one or more programs may record a computer program including instructions that are executed by an electronic device to cause the electronic device to perform the method described above.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0009] FIG. 3 conceptually illustrates a process by which an electronic device according to one embodiment performs an operation to obtain a processed image.

[0010] FIG. 4 is a flowchart illustrating a process by which an electronic device operates according to one embodiment.

[0011] FIG. 5 conceptually illustrates an example of an electronic device selecting at least one step from among a plurality of steps according to one embodiment.

[0012] FIG. 6 is a flowchart illustrating a process by which an electronic device performs operations to obtain a processed image according to one embodiment.

[0013] FIG. 7 illustrates a relationship between steps formed in a learning process and steps performed in a process of performing an operation to obtain a processed image based on a plurality of steps in one embodiment.

[0014] FIG. 8 illustrates a relationship between a step formed in a learning process and a step performed in a process of performing an operation to obtain a processed image based on at least one step among a plurality of steps in one embodiment.

[0015] FIG. 9 is a flowchart illustrating a process including a process of storing an image while an electronic device performs an operation to obtain a processed image according to one embodiment.

[0016] FIG. 10 is a flowchart illustrating a process including a process of suspending and resuming an operation for obtaining a processed image by an electronic device according to one embodiment.

[0017] Fig. 11 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0019] An electronic device and an operating method thereof according to various embodiments may be for obtaining high quality images through fast computation speed.

[0020] An electronic device and an operating method thereof according to various embodiments may be for reducing power consumption consumed to obtain high quality images.

[0021] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0022] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0023] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0024] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0025] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0026] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0027] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0028] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0029] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0030] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0031] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0032] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0033] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0034] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0035] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0036] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0037] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0038] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0039] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0040] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0041] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0042] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0045] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0046] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0047] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0048] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0049] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0050] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0051] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0052] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0053] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0054] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0055] In the present disclosure, the operation of an electronic device may be understood as being performed by one or more processors executing one or more instructions stored in a memory to perform operations or control components of the electronic device. Alternatively, one or more processors of the electronic device may include circuitry configured to perform the operation of the electronic device of the present disclosure.

[0056] FIG. 3 conceptually illustrates a process in which an electronic device (e.g., the electronic device (101) of FIG. 1 ) according to one embodiment performs an image processing operation (300) to obtain a processed image. In the present disclosure, the image processing operation (300) to obtain a processed image may include an inference process performed by a neural network model constructed by learning data.

[0057] In one embodiment, the electronic device may perform an image processing operation (300) to obtain a processed image that obtains an enhanced image (329) based on an iterative refinement model (310) from a processing target image (321). The iterative refinement model (310) may be generated by learning deteriorated images that are repeatedly deteriorated from a correct image as learning data. For example, the learning data may include a first deteriorated image in which noise is added to the correct image, a second deteriorated image in which noise is added to the first deteriorated image, and a third deteriorated image in which noise is added to the second deteriorated image, as data corresponding to the correct image. The iterative refinement model (310) may be generated by reversely learning the image deterioration process by repeatedly removing noise from the deteriorated image. The process of learning the neural network may be performed in the electronic device, but a neural network model obtained as a result of learning through another device may also be mounted in the electronic device. In the present disclosure, the process of learning by restoring a degraded image through multiple steps may be referred to as training a neural network based on a diffusion process. An iterative refinement model (310) trained based on N degraded images may include a process of obtaining an enhanced image (processed image) (329) from a target image through N steps.

[0058] FIG. 3 illustrates an example of performing an image processing operation (300) to obtain a processed image based on an iterative refinement model (310) having four stages. In one embodiment, the electronic device may perform a first image processing step (311) of inputting an image to be processed (321) into a neural network model and outputting an output image (323) from the image to be processed (321). In FIG. 3 , for convenience of explanation, an image is input into the neural network model and an image is output from the neural network model. However, it may be understood that feature information acquired from the image is input into the neural network model and feature information inferred through the neural network model is output. Alternatively, the neural network model may extract feature information from the input image and output a result inferred from the extracted feature information. The inferred result may be an enhanced image predicted to be obtained when image processing is performed or feature information of the enhanced image predicted. The electronic device may identify at least one image processing step to be performed through the neural network model based on a stage identification value. The image processing step referred to in the present disclosure may include an inference step included in a plurality of inference steps performed through a neural network model trained based on a diffusion process. In the present disclosure, the image processing step may be referred to as an operation of receiving an image to be processed or an output value of a previously performed image processing step while performing an operation to obtain a processed image, and outputting an image inferred through a neural network model.

[0059] Referring to FIG. 3, the electronic device can input the output image (323) of the first image processing step (311) into a neural network model to perform a second image processing step (313). The electronic device can input the output image (325) of the second image processing step (313) into the neural network model again to perform a third image processing step (315). The electronic device can input the output image (327) of the third image processing step (315) into the neural network model again to perform a fourth image processing step (317). The electronic device can obtain an enhanced image (329) as a result of performing the fourth image processing step (317). For convenience of explanation, FIG. 3 illustrates outputting an enhanced image (329) from a neural network model, but it can be understood that the enhanced image (329) is restored based on feature information output from the neural network model.

[0060] FIG. 4 is a flowchart (400) illustrating a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) operates according to one embodiment.

[0061] In operation 410, an electronic device according to an embodiment may obtain a processing target image (e.g., a processing target image (321) of FIG. 3). The processing target image (321) may include, for example, an image obtained through a camera module of the electronic device (e.g., a camera module (180) of FIGS. 1 and 2). The processing target image (321) may be, for example, an image stored in a memory of the electronic device (e.g., a memory (130) of FIG. 1). The processing target image (321) may also be, for example, an image received from an external device (e.g., an electronic device (102), an electronic device (104), a server (108) of FIG. 1).

[0062] In operation 420, an electronic device according to an embodiment may determine at least one image processing step for performing an image processing operation for obtaining a processed image (e.g., an image processing operation (300) for obtaining a processed image of FIG. 3) from among a plurality of image processing steps (e.g., image processing steps (311), 313, 315, and 317) of FIG. 3) included in an iterative refinement model (e.g., an iterative refinement model (310) of FIG. 3). In an embodiment, the electronic device may determine at least one image processing step for performing an image processing operation for obtaining a processed image based on image information or context information. The image information may refer to information related to an image to be processed. For example, the image information may include information about a category to which a subject (e.g., a person's face, the sky, text) or a scene (e.g., a landscape photo, a portrait photo) captured in the image to be processed belongs. For example, the image information may include information about characteristics of the image to be processed (e.g., image quality, noise level, amount of detail, dynamic range, brightness). For example, image information may be information included in the metadata of the image to be processed. Context information may refer to setting information related to the environment or conditions under which the electronic device performs an image processing operation to obtain a processed image, or a setting value for capturing an image through a camera. For example, context information may include illuminance information detected through an illuminance sensor of the electronic device (e.g., the sensor module (176) of FIG. 1, the image sensor (230) of FIG. 2). For example, context information may include a sensor gain value of an image sensor set to capture an image to be processed (e.g., the image sensor (230) of FIG. 2). For example, context information may include information on the remaining capacity of a battery of the electronic device (e.g., the battery (189) of FIG. 1) identified at the time of performing an image processing operation to obtain a processed image.For example, the context information may include temperature information detected by a temperature sensor included in the electronic device (e.g., the sensor module (176) of FIG. 1). For example, the context information may include information about the time (e.g., nighttime) at which the image to be processed is captured. For example, the context information may include information about the processor occupancy rate by processes. For example, the context information may include information related to an application running on the electronic device. However, the image information or the context information is not limited to the examples described above.

[0063] According to one embodiment, the electronic device may determine at least one image processing step by selecting at least some of the plurality of image processing steps. For example, the electronic device may determine at least one image processing step having a different number of steps than the number of the plurality of image processing steps included in an iterative refinement model (e.g., the iterative refinement model (310) of FIG. 3) based on image information or context information. For example, the electronic device may select 200 image processing steps from 1,000 image processing steps. For example, when the image to be processed is a high dynamic range (HDR) image, the electronic device may determine at least one image processing step including a large number of steps. For example, when the sensor gain value of the image sensor is high, the electronic device may determine at least one image processing step including a large number of steps. For example, when the illuminance information indicates a high illuminance, the electronic device may determine a smaller number of steps as at least one image processing step.

[0064] In one embodiment, the electronic device may divide the image to be processed into a plurality of regions and perform a different number of image processing steps on each region. For example, the electronic device may perform a large number of image processing steps on a region containing a person's face or a region displaying text. The method for dividing the image to be processed into a plurality of regions may be implemented in various ways. For example, the electronic device may divide the image to be processed into a plurality of regions based on regions in which the object appears within the image through semantic segmentation. However, the present invention is not limited thereto. For example, the electronic device may divide the image to be processed into a plurality of regions based on information obtained as a result of performing face detection on the image to be processed, skin color information detected from the image to be processed, or frequency information (e.g., values ​​of high-frequency components or values ​​of low-frequency components) obtained from the image to be processed.

[0065] In operation 430, an electronic device according to an embodiment may perform an image processing operation to obtain a processed image from an image to be processed based on the determined at least one image processing step. The electronic device may select and operate in either an operation mode that prioritizes the image quality of an improved image obtained as a result or an operation mode that prioritizes processing speed, as needed, while performing the image processing operation to obtain a processed image based on the selected at least one image processing step. For example, when the electronic device performs an image processing operation to obtain a processed image from an image to be processed that does not require a large number of steps, the electronic device may obtain a processed image at a relatively fast speed through a small number of image processing steps.

[0066] In one embodiment, at least some of the operations illustrated in flowchart (400) may be performed by different devices. For example, at least some of operations 410 and 420 may be performed by an application processor (AP) or a central processing unit (CPU) of the electronic device, and at least some of operation 430 may be performed by a neural processing unit (NPU) or a graphic processing unit (GPU). For example, operations 410 and 420 may be performed by the electronic device, and operation 430 may be performed by an external device that receives feature information of an image to be processed and information about at least one determined image processing step. For example, operation 430 may be performed by an artificial neural network implemented by at least one processing circuit included in the electronic device.

[0067] FIG. 5 conceptually illustrates an example of an electronic device (e.g., the electronic device (101) of FIG. 1) selecting at least one step from among a plurality of steps according to one embodiment.

[0068] In one embodiment, an iterative refinement model (510) trained using stepwise degraded images as learning data may be configured to provide an enhanced image by having the electronic device perform multiple steps of inputting feature information into the neural network model. For example, referring to FIG. 5 , the iterative refinement model (510) may be configured to enable the electronic device to provide an enhanced image through N image processing steps. The N image processing steps included in the iterative refinement model (510) may be identified based on a step identification value.

[0069] In one embodiment, the electronic device may determine at least one image processing step (520) of M (M may be a natural number) for performing an image processing operation for obtaining a processed image from an image to be processed among N (N may be a natural number) image processing steps included in an iterative refinement model (510) (e.g., operation 420 of FIG. 4 ). For example, the image processing step (520) of FIG. 5 illustrates an example configured to execute a plurality of image processing steps included in the iterative refinement model (510) by skipping three steps at a time. M may have a different value from N. For example, when the electronic device performs an image processing operation for obtaining a processed image for an HDR image, the electronic device may determine M to be a large value in order to perform the image processing operation for obtaining a processed image through many steps. The electronic device may determine M to be a small value as the dynamic range of the image to be processed is smaller. For example, the electronic device can obtain a processed image based on a large number of M values ​​as the sensor gain value applied to detect light and output the detected value by the image sensor (e.g., the image sensor (230) of FIG. 2) of the camera module (e.g., the camera module (180) of FIGS. 1 and 2) is large. The electronic device can obtain a processed image based on a small number of M values ​​as the sensor gain value is small. For example, the electronic device can obtain a processed image based on a small number of M values ​​for an image to be processed that is taken outdoors during the daytime.

[0070] In one embodiment, the electronic device may determine a plurality of image patches, each of which divides an image to be processed into a plurality of regions, and determine at least one image processing step to be applied to each of the plurality of image patches. For example, the electronic device may perform a greater number of image processing steps on an image patch containing a designated object (e.g., an object classified as a primary subject, such as a person's face or text).

[0071] At least one image processing step (520) of FIG. 5 is determined to be performed by skipping a plurality of image processing steps included in the iterative refinement model (510) at regular intervals, but the electronic device may also perform an image processing operation to obtain a processed image based on at least one image processing step (530) of which the intervals are not regular within the iterative refinement model (510). Referring to FIG. 5, the electronic device may perform an image processing operation to obtain a processed image based on image processing steps having narrow intervals (e.g., no steps are skipped) in a first section (531) in which there is a lot of noise and a large amount of change in feature information according to performing the image processing steps. The electronic device may perform an image processing operation to obtain a processed image based on image processing steps having wide intervals (e.g., nine steps are skipped) in a second section (533) in which there is little noise and a small amount of change in feature information according to performing the image processing steps.

[0072] FIG. 6 is a flowchart (600) illustrating a process in which an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment performs an image processing operation to obtain a processed image.

[0073] The flowchart (600) of FIG. 6 illustrates a process of performing an image processing operation to obtain a processed image based on at least one image processing step determined from among a plurality of image processing steps. In operation 610, an electronic device according to an embodiment may obtain a processing target image (e.g., a processing target image (321) of FIG. 3). For example, the electronic device may capture an image through a camera module (e.g., a camera module (180) of FIG. 1 or 2) to obtain image data. For example, the electronic device may obtain an image stored in a memory (e.g., a memory (130) of FIG. 2). For example, the electronic device may receive an image from a peripheral device (e.g., an electronic device (102), an electronic device (104), an electronic device (108) of FIG. 1) through a communication module (a communication module (190) of FIG. 1).

[0074] In one embodiment, upon initiating an image processing operation to obtain a processed image for an image to be processed, the electronic device may perform operation 620 of extracting feature information from the image to be processed. The electronic device may perform operation 630 of setting the current stage to an initial stage.

[0075] In one embodiment, the electronic device may perform operation 640 of inputting the feature information extracted in operation 620 into a neural network model to perform a neural network operation based on a current stage (initial stage) among at least one image processing stage. The electronic device may input the feature information and an identification value for the current image processing stage into the neural network model.

[0076] In one embodiment, the electronic device may perform operation 650 of determining whether the current step is the final step. If the current step is not the final step, the electronic device may perform operation 660 of changing the current step to the next step among at least one image processing step (e.g., at least one image processing step determined in operation 420 of FIG. 4). In operation 670, the electronic device may input the output of the previous step into a neural network model to perform a neural network operation based on the current step. In operation 670, the electronic device may input a step identification value indicating the current step into the neural network model to perform a neural network operation corresponding to the current step on the output of the previous step. The electronic device may determine whether the final step included in at least one image processing step (e.g., t in FIG. 5) N ) can be repeated by re-inputting the output of the neural network operation into the neural network model until the output is performed.

[0077] In operation 650, an electronic device according to one embodiment may obtain an enhanced image based on the output of a neural network model, if the current stage is the final stage. For example, the electronic device may obtain an enhanced image by restoring the image based on feature information included in the output of the neural network model.

[0078] FIG. 7 illustrates a relationship between steps formed in a learning process and steps performed in a process of performing an image processing operation to obtain a processed image based on a plurality of steps in one embodiment.

[0079] FIG. 7 illustrates a relationship for determining at least one image processing step for performing image processing based on an iterative refinement model (e.g., the iterative refinement model (310) of FIG. 3, the iterative refinement model (510) of FIG. 5) trained to output a processed image based on 1000 image processing steps.

[0080] In one embodiment, the electronic device may determine at least one image processing step to be performed in an inference process for image processing based on a linear relationship (710). If the electronic device has learned an inference process based on 1000 image processing steps (training), the iterative refinement model may include 1000 image processing steps (inference steps). If the electronic device performs an image processing operation to obtain a processed image based on 1000 steps (inference), the image processing steps of steps 1 to 200 of the first section (711) among the at least one image processing step may correspond to steps 1 to 200 of the iterative refinement model. The image processing steps of steps 801 to 1000 of the second section (712) among the at least one image processing step may correspond to steps 801 to 1000 of the iterative refinement model.

[0081] In one embodiment, the electronic device may determine at least one image processing step to be performed in the inference process based on a nonlinear relationship (720). When the electronic device performs an image processing operation to obtain a processed image based on 1000 steps, the image processing steps of steps 1 to 200 of the third section (721) among the at least one image processing step may correspond to fewer steps than 200 steps (1 to 200) of the iterative refinement model. The image processing steps of steps 801 to 1000 of the fourth section (722) among the at least one image processing step may correspond to more steps than 200 steps (801 to 1000) of the iterative refinement model.

[0082] FIG. 8 illustrates a relationship between a step formed in a learning process and a step performed in a process of performing an image processing operation to obtain a processed image based on at least one step among a plurality of steps in one embodiment.

[0083] FIG. 8 illustrates a relationship for determining 200 image processing steps for performing image processing operations for obtaining a processed image based on an iterative refinement model (e.g., the iterative refinement model (310) of FIG. 3, the iterative refinement model (510) of FIG. 5) trained to perform image processing operations for obtaining a processed image based on 1000 image processing steps.

[0084] In one embodiment, the electronic device may determine at least one image processing step to be performed in an inference process for image processing based on a linear relationship (810). If the electronic device has learned the inference process based on 1000 image processing steps, the iterative refinement model may include 1000 image processing steps (inference steps). If the electronic device performs an image processing operation to obtain a processed image based on 200 steps, the image processing steps of steps 1 to 40 of the fifth section (811) among the at least one image processing step may correspond to steps 1 to 200 of the iterative refinement model. The image processing steps of steps 161 to 200 of the sixth section (812) among the at least one image processing step may correspond to steps 801 to 1000 of the iterative refinement model. For example, each time the electronic device performs an image processing step, the next step may increase by 5 steps (e.g., 1, 6, 11, 16, ...) of the iterative refinement model.

[0085] In one embodiment, the electronic device may determine at least one image processing step to be performed in the inference process based on a nonlinear relationship (820). When the electronic device performs an image processing operation to obtain a processed image based on 200 steps, the image processing steps 1 to 40 of the seventh section (821) among the at least one image processing step may correspond to fewer steps than 200 steps (1 to 200 steps) of the iterative refinement model. The image processing steps 161 to 200 of the eighth section (822) among the at least one image processing step may correspond to more steps than 200 steps (801 to 1000 steps) of the iterative refinement model.

[0086] FIG. 9 is a flowchart (900) illustrating a process including a process of storing an image while an electronic device (e.g., the electronic device (101) of FIG. 1) performs an image processing operation to obtain a processed image according to one embodiment.

[0087] The flowchart (900) of FIG. 9 illustrates a process of performing an image processing operation to obtain a processed image based on at least one image processing step determined from among a plurality of image processing steps. In operation 910, an electronic device according to an embodiment may obtain a processing target image (e.g., a processing target image (321) of FIG. 3). For example, the electronic device may capture an image through a camera module (e.g., a camera module (180) of FIG. 1 or 2) to obtain image data. For example, the electronic device may obtain an image stored in a memory (e.g., a memory (130) of FIG. 2). For example, the electronic device may receive an image from a peripheral device (e.g., an electronic device (102), an electronic device (104), an electronic device (108) of FIG. 1) through a communication module (a communication module (190) of FIG. 1).

[0088] In one embodiment, upon initiating an image processing operation to obtain a processed image for an image to be processed, the electronic device may perform operation 920 of extracting feature information from the image to be processed. The electronic device may perform operation 930 of setting a current stage of the image processing to an initial stage.

[0089] In one embodiment, the electronic device may perform operation 940 of inputting the feature information extracted in operation 920 into a neural network model to perform a neural network operation based on a current stage (initial stage) of at least one image processing stage. The electronic device may input the feature information and an identification value for the current image processing stage into the neural network model.

[0090] In one embodiment, the electronic device may perform operation 950 of determining whether the current step is the final step. If the current step is not the final step, the electronic device may perform operation 960 of changing the current step to a next step among at least one image processing step (e.g., at least one image processing step determined in operation 420 of FIG. 4). In operation 970, the electronic device may input the output of the previous step into a neural network model to perform a neural network operation based on the current step. In operation 970, the electronic device may input a step identification value, which instructs the current step to perform a neural network operation corresponding to the current step on the output of the previous step, into the neural network model. The electronic device may repeat the operation of re-inputting the output of the neural network operation into the neural network model until a final step included in at least one image processing step is performed.

[0091] In operation 980, an electronic device according to an embodiment may determine whether an intermediate storage condition is satisfied. The intermediate storage condition may refer to a condition for storing an image corresponding to an output of an image processing step other than the final step. If it is determined that the intermediate storage condition is satisfied, the electronic device may perform operation 990 of storing an output image corresponding to the output of the current step of the neural network operation. The electronic device may store the output image in a memory of the electronic device or an external electronic device capable of communicating with the electronic device in operation 990. The electronic device may store the output image in operation 990 and perform operation 950. If it is determined that the intermediate storage condition is not satisfied, the electronic device may perform operation 950. The electronic device may further store metadata for the output image in operation 990.

[0092] For example, an electronic device may identify that a function of an application (e.g., a gallery application) that displays at least one image acquired through a camera has been executed while performing an image processing operation to acquire a processed image from an image to be processed acquired through a camera module. The electronic device may determine that an intermediate storage condition has been satisfied based on the identification of the execution of the application. The electronic device may store an image restored based on the output of a previous step based on the execution of the application, and display the stored image through a display of the electronic device (e.g., the display module (160) of FIG. 1).

[0093] In operation 950, the electronic device according to one embodiment can obtain an enhanced image based on the output of the neural network model if the current stage is the final stage. For example, the electronic device can obtain an enhanced image by restoring the image based on feature information included in the output of the neural network model. In operation 990, if a stored output image exists, the electronic device can replace the stored output image with the enhanced image.

[0094] FIG. 10 is a flowchart (1000) illustrating a process including a process of stopping and resuming an image processing operation for obtaining a processed image by an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment.

[0095] The flowchart (1000) of FIG. 10 illustrates image processing operations for obtaining a processed image based on at least one image processing step determined from among a plurality of image processing steps. In operation 1010, an electronic device according to an embodiment may obtain a processing target image (e.g., a processing target image (321) of FIG. 3). For example, the electronic device may capture an image through a camera module (e.g., a camera module (180) of FIG. 1 or 2) to obtain image data. For example, the electronic device may obtain an image stored in a memory (e.g., a memory (130) of FIG. 2). For example, the electronic device may receive an image from a peripheral device (e.g., an electronic device (102), an electronic device (104), an electronic device (108) of FIG. 1) through a communication module (a communication module (190) of FIG. 1).

[0096] In one embodiment, upon initiating an image processing operation to obtain a processed image for an image to be processed, the electronic device may perform operation 1020 of extracting feature information from the image to be processed. The electronic device may perform operation 1030 of determining a current stage as an initial stage.

[0097] In one embodiment, the electronic device may perform operation 1040 of inputting feature information extracted in operation 1020 into a neural network model to perform a neural network operation based on a current stage (initial stage) of at least one image processing stage. The electronic device may input the feature information and an identification value for the current image processing stage into the neural network model.

[0098] In one embodiment, the electronic device may perform operation 1050 of determining whether the current step is the final step. If the current step is not the final step, the electronic device may perform operation 1060 of changing to a next step among at least one image processing step (e.g., at least one image processing step determined in operation 420 of FIG. 4). In operation 1070, the electronic device may input the output of the previous step into a neural network model to perform a neural network operation based on the current step. In operation 1070, the electronic device may input a step identification value, which instructs the current step to perform a neural network operation corresponding to the current step on the output of the previous step, into the neural network model. The electronic device may repeat the operation of re-inputting the output of the neural network operation into the neural network model while performing the at least one image processing step.

[0099] In operation 1080, the electronic device according to one embodiment may determine whether an intermediate storage condition is satisfied. The intermediate storage condition may refer to a condition for storing an image corresponding to the output of an image processing stage other than the final stage. If it is determined that the intermediate storage condition is satisfied, the electronic device may perform operation 1090 of storing an output image corresponding to the output of the current stage of the neural network operation and a stage identification value indicating the current stage. In operation 1090, the electronic device may store the output image and the stage identification value in a memory of the electronic device or an external electronic device capable of communicating with the electronic device. In operation 1090, the electronic device may store the output image and the stage identification value and stop (or pause) an operation of performing at least one image processing stage for an image to be processed. If it is determined that the intermediate storage condition is not satisfied, the electronic device may perform operation 1050. The electronic device may further store metadata for the output image in operation 1090.

[0100] In operation 1095, the electronic device according to one embodiment may determine whether to resume an image processing operation for obtaining a processed image. For example, the electronic device may store an output image (or feature information output from a neural network model) and a step identification value in operation 1090 based on determining that the remaining capacity of a battery (e.g., battery (189) of FIG. 1) of the electronic device is less than or equal to a reference value in operation 1080. The electronic device may stop performing at least one image processing step until a condition for resuming the operation in operation 1090 is satisfied. If the operation for performing at least one image processing step is stopped based on the fact that the remaining capacity of the battery is less than or equal to the reference value, the electronic device may determine in operation 1095 whether the remaining capacity of the battery exceeds the reference value or whether external power is supplied. When the remaining capacity of the battery exceeds a reference value or external power is supplied, the electronic device may perform operation 1050 to resume an operation of performing at least one image processing step based on a stored output image (or feature information output from a neural network model) and a step identification value. The electronic device may resume an image processing operation for obtaining a processed image from a stopped image processing step (from a step following the image processing step that was last performed) based on the step identification value. When a condition for resuming the image processing operation for obtaining a processed image is not satisfied, the electronic device may monitor whether the condition for resuming the image processing operation for obtaining a processed image is satisfied. For example, the electronic device may periodically perform operation 1095 until the condition for resuming the image processing operation for obtaining a processed image is satisfied.

[0101] Additionally, for example, the electronic device may determine that the intermediate storage condition is satisfied if the temperature acquired through the temperature sensor of the electronic device in operation 1080 is greater than or equal to the first threshold, and may perform operation 1090. The electronic device may store the output image and the step identification value in operation 1090, and determine whether the temperature detected through the temperature sensor is less than the second threshold in operation 1095. If the temperature is less than the second threshold, the electronic device may resume the image processing operation to acquire the processed image by performing operation 1050. The second threshold may indicate a temperature lower than the first threshold.

[0102] In operation 1050, the electronic device according to one embodiment can obtain an enhanced image based on the output of the neural network model if the current stage is the final stage. For example, the electronic device can obtain an enhanced image by restoring the image based on feature information included in the output of the neural network model. If a stored output image exists in output 1090, the electronic device can replace the stored output image with the enhanced image.

[0103] FIG. 11 is a block diagram illustrating the configuration of an electronic device (101) (e.g., the electronic device (101) of FIG. 1) according to one embodiment.

[0104] In one embodiment, the system may include at least one processor (1120) (e.g., processor (120) of FIG. 1) and memory (1130) (e.g., memory (130) of FIG. 2). The memory (1130) may store a neural network model (1131) and one or more instructions (1133) for performing an image processing operation to obtain a processed image. The at least one processor (1120) may perform an image processing operation to obtain a processed image for an image to be processed by executing one or more instructions (1133) to perform one or more image processing steps using the neural network model (1131).

[0105] In one embodiment, the neural network model (1131) may include a deep learning model generated through training based on learning data so as to perform an image processing operation for obtaining a processed image through a plurality of image processing steps. One or more commands (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to determine at least one image processing step for performing an image processing operation for obtaining a processed image from an image to be processed among the plurality of image processing steps.

[0106] In one embodiment, the electronic device (101) may further include a camera module (1180) (e.g., the camera module (180) of FIG. 1 or 2). The electronic device (101) may acquire an image to be processed through the camera module (1180). For example, by operating the camera module (1180) based on a camera application executed by at least one processor (1120), the electronic device (101) may acquire image data based on a signal detected through an image sensor (e.g., the image sensor (230) of FIG. 2) of the camera module (180). The electronic device (101) may determine at least one image processing step for acquiring a processed image from the acquired image data based on information related to the acquired image data.

[0107] In one embodiment, the electronic device (101) may further include a communication module (1190) (e.g., the communication module (190) of FIG. 1). The electronic device (101) may also receive an image to be processed from an external device (1104) (e.g., the electronic device (102), the electronic device (104), and the server (108) of FIG. 1) through the communication module (1190). Although FIG. 11 illustrates that the electronic device (101) includes a neural network model (1131) for performing an image processing operation to obtain a processed image from the image to be processed, in one embodiment, the electronic device (101) may transmit feature information of the image to be processed to the external device (1104) and receive feature information corresponding to a result of image processing performed based on the neural network model from the external device (1104).

[0108] In one embodiment, one or more instructions (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to determine a policy for determining at least one image processing step for performing an image processing operation to obtain a processed image from an image to be processed. One or more instructions (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to determine a schedule for performing at least one image processing step based on the determined policy. For example, the neural network model (1131) may be a deep learning model trained based on N image processing steps, or the electronic device (101) may obtain a processed image based on M image processing steps. For example, in performing an image processing operation to obtain a processed image based on a neural network model (1131) trained based on 1000 image processing steps, the electronic device (101) may perform the image processing operation to obtain a processed image based on 100 steps, and may perform the image processing operation to obtain a processed image while increasing the identification value of the image processing step to be performed by 10 steps among the 1000 image processing steps. In one embodiment, the image processing steps may be performed in small steps in some sections of the image processing steps, and the image processing steps may be performed in many steps in other sections. For example, in the early stages of performing image processing, the inference steps may be densely arranged, and in the latter stages of image processing, the image processing steps (inference processes within the diffusion process according to the neural network model) may be widely arranged.

[0109] In one embodiment, one or more commands (1133) may be executed by at least one processor (1120) to determine at least one image processing step based on at least one of metadata about a situation in which the electronic device (101) captures an image, region information obtained by analyzing a region of an image to be processed, information related to a user of the electronic device (101), or system information of the electronic device (101). The metadata about the situation in which the image is captured may include, for example, at least one of whether the electronic device (101) captured the image based on an HDR mode, a noise reduction operation mode, a sensor gain value for an image sensor, or the number of images to be synthesized in the case of an image synthesis operation mode. The region information may include, for example, information about a region in which a subject is captured within an image and information classifying the subject captured in the region, or information classifying whether the captured image is an image of a certain scene (e.g., a person, a landscape, a document). User-related information may include, for example, whether a preview is being provided to the user, or information related to settings stored in image processing operations for obtaining processed images. System information may include, for example, information regarding the remaining battery capacity of the electronic device, the temperature detected by a temperature sensor (e.g., the temperature of the processor), or the status of the network to which the electronic device is connected.

[0110] For example, one or more commands (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to perform an image processing operation to obtain a processed image based on a larger number of steps than the image processing steps applied during learning of the neural network model (1131) when the electronic device (101) obtains an image to be processed through the camera module (1180) based on the HDR mode. Since an HDR image requires processing for a wide dynamic range from dark parts with a lot of noise to bright parts, the electronic device (101) may perform a larger number of image processing steps for the HDR image.

[0111] For example, one or more instructions (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to analyze frequency information of an image, perform a large number of image processing steps when there is a lot of information in a high frequency band, and perform a small number of image processing steps when there is less information in a high frequency band.

[0112] For example, one or more commands (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to perform a greater number of image processing steps as the noise in the image to be processed increases, and to perform a fewer number of image processing steps as the noise decreases. For example, one or more commands (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to determine at least one image processing step based on saturation or hue information of the image.

[0113] For example, one or more instructions (1133) may be executed by at least one processor (1120) to cause the electronic device (101) to determine at least one image processing step based on a sensor gain value set for the camera module (1180) to capture an image. The electronic device (101) may capture an image based on a high sensor gain value at night or in a dark room where there is insufficient ambient light. When the sensor gain value is high, the electronic device (101) may perform an image processing operation to obtain a processed image based on a large number of image processing steps. Conversely, when capturing an image outdoors during the daytime when there is a lot of light, the electronic device (101) may capture an image based on a low sensor gain value. When the sensor gain value is low, the electronic device (101) may perform an image processing operation to obtain a processed image based on a small number of image processing steps.

[0114] For example, the electronic device (101) can perform an image processing operation to obtain a processed image based on a large number of image processing steps for a part that greatly affects the quality of the photo, such as a person's face, the sky, or an area where text is captured. The electronic device (101) can perform an image processing operation to obtain a processed image based on a small number of image processing steps for an image captured of a scene with little noise, such as an outdoor mountain or the sea. The electronic device (101) can perform an image processing operation to obtain a processed image based on a large number of image processing steps for an area with a lot of detail in the image to be processed (e.g., an area captured of hair or a furry animal). If the image to be processed is an image captured of the moon, the electronic device (101) can perform an image processing operation to obtain a processed image based on a large number of image processing steps because the image is captured at night when there is insufficient light.

[0115] For example, the electronic device (101) may perform an image processing operation to obtain a processed image based on a relatively small number of image processing steps while displaying a preview image (live preview) based on an image stream output from the camera module (1180). The electronic device (101) may obtain a processed image based on a larger number of image processing steps for an image captured based on a night shooting mode or a quality priority mode. The electronic device (101) may obtain a processed image based on an image processing mode set by a user (e.g., quality priority mode or speed priority mode). For example, if the user sets the speed priority mode, the processed image may be obtained based on a small number of image processing steps, and if the user sets the quality priority mode, the processed image may be obtained based on a large number of image processing steps.

[0116] The examples described above can be composed of two or more combinations, or vice versa.

[0117] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 11) may include one or more processors (e.g., processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) and a memory (e.g., memory (130) of FIG. 1, memory (1130) of FIG. 11) that stores a neural network model and one or more instructions generated by training a neural network to output an inference result for information of an input image. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to determine at least one image processing step for performing image processing on the image to be processed from among a plurality of image processing steps performed by the neural network model based on image information related to the image to be processed or context information related to a state of the electronic device. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to obtain a processed image from the image to be processed based on the determined at least one image processing step. Each of the plurality of image processing steps may include an operation of inputting the image to be processed or the output of the neural network model into the neural network model to obtain an inferred result.

[0118] In one embodiment, the one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to input an output obtained as a result of inputting feature information of the image to be processed into the neural network model, together with a step identification value indicating a next step among the determined at least one image processing step, into the neural network model.

[0119] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may further include a camera (e.g., the camera module (180) of FIG. 1, the camera module (1180) of FIG. 11) for acquiring the image to be processed. The context information may include setting information related to setting values ​​or environments for the camera (e.g., the camera module (180) of FIG. 1, the camera module (1180) of FIG. 11) for acquiring the image to be processed.

[0120] In one embodiment, the setting information may include information on a dynamic range of the image to be processed obtained through the camera (e.g., the camera module (180) of FIG. 1, the camera module (1180) of FIG. 11). The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, the at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to determine a smaller number of steps as the at least one image processing step as the dynamic range becomes smaller.

[0121] In one embodiment, the setting information may include illuminance information obtained through an image sensor (e.g., an image sensor (230) of FIG. 2) of the camera (e.g., a camera module (180) of FIG. 1, a camera module (1180) of FIG. 11)) or an illuminance sensor of the electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 11). The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 11)) to determine a smaller number of steps as the at least one image processing step as the illuminance information indicates a higher illuminance.

[0122] In one embodiment, the configuration information may include a sensor gain value of an image sensor (e.g., an image sensor (230) of FIG. 2) included in the camera (e.g., a camera module (180) of FIG. 1, a camera module (1180) of FIG. 11). The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to determine a greater number of steps as the at least one image processing step as the sensor gain value increases.

[0123] In one embodiment, the one or more instructions may be executed by the one or more processors (e.g., the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to determine whether an intermediate storage condition regarding whether to store the output of any one of the at least one image processing steps is satisfied. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to store the output image of the one image processing step and the step identification value for the image processing step in the memory (e.g., the memory (130) of FIG. 1, the memory (1130) of FIG. 11)) or an external electronic device capable of communicating with the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)) based on the determination that the intermediate storage condition is satisfied.

[0124] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may further include a display (160, 1160). The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, the at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)) to determine that the intermediate storage condition is satisfied based on the execution of a function of an application that causes the electronic device to display an image while performing the image processing, and to cause the electronic device to display a screen including the output image stored in the memory (e.g., the memory (130) of FIG. 1, the memory (1130) of FIG. 11)) or the external electronic device on the display (180, 1180).

[0125] In one embodiment, the one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to further store metadata for the image to be processed in the memory (e.g., the memory (130) of FIG. 1, the memory (1130) of FIG. 11)) or the external electronic device, based on the determination that the intermediate storage condition is satisfied.

[0126] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may further include a battery (e.g., the battery (189) of FIG. 1). The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, the at least one processor (1120) of FIG. 11) so that the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)) may determine that the intermediate storage condition is satisfied based on a remaining capacity of the battery (e.g., the battery (189) of FIG. 1) being less than or equal to a reference value. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to store an output image of the one image processing step and a step identification value for the image processing step in the memory (e.g., the memory (130) of FIG. 1, the memory (1130) of FIG. 11)) or an external electronic device capable of communicating with the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)) when the remaining capacity of the battery (e.g., the battery (189) of FIG. 1) is below a reference value. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to stop the operation of the image processing when the remaining capacity of the battery (e.g., the battery (189) of FIG. 1) is below a reference value.

[0127] In one embodiment, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may include a temperature sensor. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) so that the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)) determines that the intermediate storage condition is satisfied when the temperature acquired through the temperature sensor is equal to or higher than a first threshold value, stores the output image and the step identification value in the memory (e.g., the memory (130) of FIG. 1, the memory (1130) of FIG. 11) or an external electronic device capable of communicating with the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11)), and stops the operation of the image processing. The one or more instructions may be executed by the one or more processors (the processor (120) of FIG. 1, at least one processor (1120) of FIG. 11) to cause the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) to restart the operation of the image processing from the interrupted step based on the stored output image and the step identification value when the temperature acquired through the temperature sensor is less than the second threshold after storing the output image.

[0128] According to one embodiment, a method for operating an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may include an operation of determining at least one image processing step for performing image processing on an image to be processed from among a plurality of image processing steps performed based on a neural network model generated by training a neural network to predict a processing result for an input image based on image information related to the image to be processed or context information related to a state of the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11). According to one embodiment, a method for operating an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) may include an operation of performing image processing on the image to be processed based on the determined at least one image processing step. The operation of performing the image processing may include an operation of inputting the image to be processed into the neural network model so as to perform a first step of the at least one image processing step. The operation of performing the image processing may include an operation of inputting an output image output as a result of performing the first step on the image to be processed by the neural network model into the neural network model so as to perform a second step among the at least one image processing step.

[0129] In one embodiment, the context information may include setting information related to a setting value or environment for capturing an image by a camera (e.g., a camera module (180) of FIG. 1, a camera module (1180) of FIG. 11) of the electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 11).

[0130] In one embodiment, the setting information may include information on the dynamic range of the image to be processed obtained through the camera (e.g., the camera module (180) of FIG. 1, the camera module (1180) of FIG. 11). The operation of determining the at least one image processing step may include an operation of determining a smaller number of steps as the at least one image processing step as the dynamic range becomes smaller.

[0131] In one embodiment, the setting information may include illuminance information obtained through an image sensor (e.g., an image sensor (230) of FIG. 2) of the camera (e.g., a camera module (180) of FIG. 1, a camera module (1180) of FIG. 11) or an illuminance sensor of the electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (101) of FIG. 11). The operation of determining the at least one image processing step may include an operation of determining a smaller number of steps as the at least one image processing step as the illuminance information indicates a higher illuminance.

[0132] In one embodiment, the configuration information may include a sensor gain value of an image sensor (e.g., an image sensor (230) of FIG. 2) included in the camera (e.g., a camera module (180) of FIG. 1, a camera module (1180) of FIG. 11). The operation of determining the at least one image processing step may include an operation of determining a greater number of steps as the at least one image processing step as the sensor gain value increases.

[0133] In one embodiment, the operation of performing the image processing may include an operation of determining whether an intermediate storage condition regarding whether to store an output of any one of the at least one image processing steps is satisfied. The operation of performing the image processing may include an operation of storing an output image of the one image processing step and a step identification value for the image processing step in an external electronic device capable of communicating with the memory (e.g., memory (130) of FIG. 1, memory (1130) of FIG. 11) or the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 11) based on determining that the intermediate storage condition is satisfied.

[0134] In one embodiment, the operation of determining whether the intermediate storage condition is satisfied may include an operation of identifying that a function of an application that displays the image has been executed while performing the image processing. The method may further include an operation of displaying the output image stored in the memory (e.g., memory (130) of FIG. 1, memory (1130) of FIG. 11) or the external electronic device on a display (160) of the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 11) based on identifying that the function of the application that displays the image has been executed while performing the image processing.

[0135] In one embodiment, the operation of determining whether the intermediate storage condition is satisfied may include an operation of determining that the intermediate storage condition is satisfied based on the remaining battery capacity of the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 11) being less than or equal to a reference value. The method may include an operation of stopping the operation of performing the image processing based on determining that the intermediate storage condition is satisfied.

[0136] In one embodiment, a non-transitory computer-readable recording medium may have recorded thereon a computer program that is executed by an electronic device to cause the electronic device to perform the method described above.

[0137] The components of the embodiments described above are not inseparably combined. To the extent that they provide a means to solve the problem to be solved in the present disclosure, at least some of the components of one embodiment may be combined with other components of another embodiment to form an embodiment.

[0138] According to various embodiments of the present disclosure, an image having high quality (e.g., an image with less noise or good detail) can be obtained using a neural network model while reducing the time required for computation.

[0139] According to various embodiments of the present disclosure, high-quality images can be obtained using a neural network model while reducing the required power consumption.

[0140] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0141] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0142] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

[0143] In the present disclosure, the functions or operations performed by the electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The functions or operations of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include circuitry for performing calculations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operations of the electronic device described above.

[0144] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0145] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0146] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0147] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0148] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0149] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0150] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, or including a combination of two or more (including a and b, including b and c, including a and c, or including all of a, b, and c).

[0151] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In electronic devices, one or more processors; and A memory for storing a neural network model and one or more instructions generated by training a neural network to output inference results for information of an input image, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: Based on image information related to the image to be processed or context information related to the state of the electronic device, at least one image processing step for obtaining a processed image from the image to be processed is determined among a plurality of image processing steps performed by the neural network model, To obtain a processed image from the image to be processed based on at least one image processing step determined above, An electronic device, wherein each of the plurality of image processing steps includes an operation of inputting the image to be processed or the output of the neural network model into the neural network model to obtain an inferred result.

2. In claim 1, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: An electronic device that inputs an output obtained as a result of inputting feature information of an image to be processed into the neural network model, together with a step identification value indicating a next step among at least one of the determined image processing steps, into the neural network model.

3. In either claim 1 or 2, The electronic device further includes a camera for acquiring the image to be processed, An electronic device wherein the context information includes setting information related to settings or environments for the camera to acquire the image to be processed.

4. In any of the preceding claims, The above setting information includes information on the dynamic range of the image to be processed obtained through the camera, An electronic device, wherein said one or more instructions are collectively or individually executed by said one or more processors, such that said electronic device determines a smaller number of steps as said at least one image processing step as the dynamic range becomes smaller.

5. In any of the preceding claims, The above setting information includes illumination information obtained through the image sensor of the camera or the illumination sensor of the electronic device, An electronic device, wherein said one or more instructions are collectively or individually executed by said one or more processors to cause said electronic device to determine a smaller number of steps as said at least one image processing step as the illuminance information indicates a higher illuminance.

6. In any of the preceding claims, The above setting information includes the sensor gain value of the image sensor included in the camera, An electronic device, wherein said one or more instructions are collectively or individually executed by said one or more processors, such that said electronic device determines a greater number of steps as said at least one image processing step as the sensor gain value increases.

7. In any of the preceding claims, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: Determining whether an intermediate storage condition regarding whether to store the output of at least one of the image processing steps above is satisfied, An electronic device, which stores an output image of said one image processing step and a step identification value for said image processing step in said memory or in an external electronic device capable of communicating with said electronic device, based on the determination that said intermediate storage condition is satisfied.

8. In any of the preceding claims, The electronic device further comprises a display, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: An electronic device that determines that the intermediate storage condition is satisfied based on the execution of a function of an application that causes an image to be displayed while performing at least one of the image processing steps, and causes a screen including the output image stored in the memory or the external electronic device to be displayed on the display.

9. In any of the preceding claims, An electronic device, wherein the one or more instructions are collectively or individually executed by the one or more processors to cause the electronic device to further store metadata for the image to be processed in the memory or the external electronic device based on determining that the intermediate storage condition is satisfied.

10. In any of the preceding claims, The above electronic device further comprises a battery, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: Based on the fact that the remaining capacity of the above battery is below the reference value, it is determined that the above intermediate storage condition is satisfied: Store the output image of the above one image processing step and the step identification value for the image processing step in the memory or an external electronic device capable of communicating with the electronic device, An electronic device that causes an operation of performing at least one of the image processing steps to be stopped.

11. In any of the preceding claims, The electronic device comprises a temperature sensor, The one or more instructions are collectively or individually executed by the one or more processors so that the electronic device: If the temperature acquired through the temperature sensor is greater than or equal to the first threshold, the intermediate storage condition is determined to be satisfied, and the output image and the step identification value are stored in the memory or an external electronic device capable of communicating with the electronic device, and the operation of performing the at least one image processing step is stopped. An electronic device, which, after storing the output image, restarts an operation of performing the at least one image processing step from a stopped step based on the stored output image and the step identification value if the temperature acquired through the temperature sensor is less than a second threshold.

12. In the method of operating an electronic device, An operation of determining at least one image processing step for obtaining a processed image from the image to be processed among a plurality of image processing steps performed based on a neural network model generated by training a neural network to predict a processing result for an input image based on image information related to the image to be processed or context information related to the state of the electronic device; and Comprising an operation of obtaining a processed image from the processing target image based on at least one image processing step determined above, The operation of obtaining the processed image is: An operation of inputting the image to be processed into the neural network model to perform the first step of at least one of the image processing steps, and A method comprising an action of inputting an output image output as a result of performing the first step on the image to be processed by the neural network model into the neural network model so as to perform a second step among the at least one image processing step.

13. In claim 12, A method wherein the context information includes setting information related to settings or environments for capturing images by a camera of the electronic device.

14. In claim 13, The above setting information includes information on the dynamic range of the image to be processed obtained through the camera, A method according to claim 1, wherein the operation of determining at least one image processing step includes an operation of determining a smaller number of steps as the at least one image processing step as the dynamic range is smaller.

15. A non-transitory computer-readable recording medium storing one or more programs, which are executed by an electronic device, causing the electronic device to: An operation of determining at least one image processing step for obtaining a processed image from the image to be processed among a plurality of image processing steps performed based on a neural network model generated by training a neural network to predict a processing result for an input image based on image information related to the image to be processed or context information related to the state of the electronic device; and A computer program comprising instructions for causing an electronic device to perform an operation method including an operation of obtaining a processed image from the image to be processed based on at least one image processing step determined above, The operation of obtaining the processed image is: An operation of inputting the image to be processed into the neural network model to perform the first step of at least one of the image processing steps, and A recording medium including an operation of inputting an output image output as a result of performing the first step on the processing target image by the neural network model into the neural network model so as to perform a second step among the at least one image processing step.

Images