Electronic device and control method of electronic device

Abstract

An electronic device and a control method thereof are provided. An electronic device includes a memory configured to store at least one instruction and a processor configured to run the at least one instruction and operate according to the instructions of the at least one instruction. The processor is configured to: obtain a first image; based on receiving a first user command to correct the first image, obtain a second image by correcting the first image; based on the first image and the second image, train a neural network model; and based on receiving a second user command to correct the third image, obtain a fourth image by correcting the third image using the trained neural network model.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a method of controlling the electronic device, and more particularly, to an electronic device capable of correcting an image and a method of controlling the electronic device. Background Art

[0002] With the development of portable electronic devices and social media, it has become a daily routine to obtain images by using various portable electronic devices including smartphones and to modify the obtained images to suit the user's own taste.

[0003] In order to solve the problem that users of portable electronic devices need to spend a lot of effort to correct a large number of images, many applications are being developed so that even if the user does not set relevant correction parameters to correct the pattern, a function that can automatically correct the image is provided by the application for image correction.

[0004] However, the correction mode preferred by each user may be different, and thus there is a limitation that it is difficult to satisfy all users having a wide variety of aesthetic preferences and tastes using only a function of automatically correcting an image by applying a generalized correction mode.

[0005] Therefore, it is necessary to automatically correct the original image to improve the convenience of the user, and at the same time automatically reflect the correction mode preferred by the user when correcting the image, so as to provide an image that satisfies each individual user in a preferred mode. Summary of the invention

[0006] Technical issues

[0007] Provided are an electronic device capable of obtaining a corrected image by reflecting a correction mode preferred by a user into an original image and a control method thereof.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0009] Technical Solution

[0010] According to one aspect of the present disclosure, an electronic device is provided, comprising: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction and operate according to the instructions of the at least one instruction. The processor is configured to: obtain a first image; based on receiving a first user command to correct the first image, obtain a second image by correcting the first image; based on the first image and the second image, train a neural network model; and based on receiving a second user command to correct the third image, obtain a fourth image by correcting the third image using the trained neural network model.

[0011] The processor is also configured to: obtain first type information including a type of the first image; and train a neural network model based on the first image, the second image, and the first type information.

[0012] The processor is also configured to: based on obtaining the third image, obtain second type information including the type of the third image; and based on receiving a second user command to correct the third image, obtain a fourth image by correcting the third image using a trained neural network model based on the third image and the second type information.

[0013] The first type information includes at least one of information on an object included in the first image and the second image, information on a location where the first image and the second image are obtained, and a time when the first image and the second image are obtained.

[0014] The neural network model includes a first neural network model, which includes a generator configured to generate an image and a comparator configured to compare multiple images, and wherein the processor is further configured to: obtain a fifth image by inputting the first image into the generator, the fifth image being obtained by correcting the first image; obtain first feedback information based on the difference between the pixel value of the second image and the pixel value of the fifth image by inputting the second image and the fifth image into the comparator; and train the generator based on the first feedback information.

[0015] The processor is also configured to obtain a fourth image by correcting the third image by inputting the third image into the first neural network model based on receiving a second user command.

[0016] The first neural network model is a generative adversarial network (GAN) model, and the generator and the comparator are trained in an adversarial manner with each other.

[0017] The neural network model includes a second neural network model, which includes an implementer configured to obtain information about at least one correction parameter associated with the correction of the image and a comparator configured to compare multiple images, and the processor is further configured to: obtain information about at least one correction parameter associated with the correction of the first image by inputting the first image into the implementer; obtain a sixth image by correcting the first image based on the information about the at least one correction parameter; obtain second feedback information based on the difference between the pixel value of the second image and the pixel value of the sixth image by inputting the second image and the sixth image into the comparator; and train the implementer based on the second feedback information.

[0018] The processor is also configured to: input the third image to the second neural network model based on receiving the second user command; and obtain the fourth image by correcting the third image based on the information about the at least one correction parameter associated with the correction of the first image.

[0019] The electronic device also includes: a display, and the processor is further configured to: based on obtaining the fourth image, control the display to display a first user interface (UI) element to select whether to correct the fourth image based on user settings regarding correction parameters; based on receiving a third user command for selecting to correct the fourth image through the first UI element, control the display to display a second UI element to select at least one parameter associated with correction of the fourth image; and based on receiving a fourth user command for selecting the at least one parameter associated with correction of the fourth image through the second UI element, obtain a seventh image by correcting the fourth image.

[0020] The processor is further configured to train a neural network model based on the third image and the seventh image based on obtaining the seventh image by modifying the fourth image.

[0021] The processor is also configured to: based on obtaining the third image, identify at least one object included in the third image; and based on the at least one identified object including a preset object, determine the at least one identified object as a correction target, and obtain an eighth image by correcting the correction target among the at least one object included in the third image by inputting the third image into the trained neural network model.

[0022] The first image and the second image are compared based on first metadata and second metadata respectively corresponding to each of the first image and the second image, and the first metadata is input to the generator and the second metadata is input to the comparator, wherein the first metadata includes information about at least one of a first generation time, a first generation device, and a first correction time of the first image, and wherein the second metadata includes information about at least one of a second generation time, a second generation device, and a second correction time of the second image.

[0023] According to one aspect of the present disclosure, a method for controlling an electronic device is provided, the method comprising: obtaining a first image; obtaining a second image by correcting the first image based on receiving a first user command to correct the first image; training a neural network model based on the first image and the second image; and obtaining a fourth image by correcting the third image using the trained neural network model based on receiving a second user command to correct the third image.

[0024] Training the neural network model includes: obtaining first type information including the type of the first image; and training the neural network model based on the first image, the second image, and the first type information.

[0025] Training the neural network model also includes: based on obtaining the third image, obtaining second type information associated with the type of the third image; and based on receiving a second user command to correct the third image, obtaining a fourth image by correcting the third image based on the third image and the second type information using the trained neural network model.

[0026] The first type information includes at least one of information on an object included in the first image and the second image, information on a location where the first image and the second image are obtained, and a time when the first image and the second image are obtained.

[0027] The method also includes: based on obtaining the fourth image, displaying a first user interface (UI) element to select whether to correct the fourth image based on user settings regarding correction parameters; based on receiving a third user command for selecting to correct the fourth image through the first UI element, displaying a second UI element to select at least one parameter associated with correction of the fourth image; and based on receiving a fourth user command for selecting the at least one parameter associated with correction of the fourth image through the second UI element, obtaining a seventh image by correcting the fourth image.

[0028] Training the neural network model also includes: obtaining a seventh image based on correcting the fourth image, and training the neural network model based on the third image and the seventh image.

[0029] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium is provided, which includes a program for running a control method of an electronic device, the method including: obtaining a first image; obtaining a second image by correcting the first image based on receiving a first user command to correct the first image; training a neural network model based on the first image and the second image; and obtaining a fourth image by correcting the third image using the trained neural network model based on receiving a second user command to correct the third image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a flowchart illustrating a control method of an electronic device according to an embodiment;

[0032] 2A is a diagram illustrating a process of training a first neural network model and obtaining a corrected image using the trained first neural network model according to an embodiment;

[0033] 2B is a diagram illustrating a process of training a first neural network model and obtaining a corrected image using the trained first neural network model according to an embodiment;

[0034] 3A is a diagram illustrating a process of training a second neural network model and a process of obtaining a corrected image according to an embodiment;

[0035] 3B is a diagram illustrating a process of training a second neural network model and a process of obtaining a corrected image according to an embodiment;

[0036] 4A, 4B, 4C, and 4D are diagrams illustrating user interfaces according to various embodiments;

[0037] Figure 5 is a diagram showing a user interface according to an embodiment;

[0038] 6A and 6B are diagrams illustrating user interfaces according to various embodiments;

[0039] Figure 7 is a flowchart showing a process of performing image correction by type of image according to an embodiment;

[0040] 8A is a flowchart illustrating a process of performing image correction by recognizing an object included in an image according to an embodiment;

[0041] FIG. 8B is a diagram illustrating a process of performing image correction by recognizing an object included in an image according to an embodiment; and

[0042] Fig. 9 and Fig.10 is a block diagram illustrating an electronic device according to various embodiments. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it will be appreciated that the present disclosure is not limited to the embodiments described hereinafter, but also includes various modifications, equivalents and / or alternatives of these embodiments. With regard to the description of the accompanying drawings, similar reference numerals may be used for similar constituent elements.

[0044] In the following description, when it is determined that a detailed description of the related art may obscure the gist of the present disclosure, such description may be omitted.

[0045] In addition, the following embodiments may be combined and modified in many different forms, and the scope of the technical spirit of the present disclosure is not limited to the following examples. Rather, these embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the technical spirit to those skilled in the art.

[0046] The terms used herein are for the purpose of describing certain embodiments and are not intended to limit the scope of the claims. A singular expression includes a plural expression unless otherwise specified.

[0047] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (for example, components such as numbers, functions, operations, or parts), and do not exclude the existence of additional features.

[0048] In the present disclosure, expressions such as "at least one of A [and / or] B" or "one or more of A [and / or] B" include all possible combinations of the listed items. For example, "at least one of A and B" or "at least one of A or B" includes any of (1) only A, (2) only B, or (3) both A and B.

[0049] As used herein, the terms “first,” “second,” and the like may refer to various components, regardless of order and / or importance, and may be used to distinguish one component from another and not to limit the components in other ways.

[0050] If an element (e.g., a first element) is described as being “operably or communicatively coupled to / operably or communicatively coupled to another element (e.g., the second element)” or “connected to another element (e.g., the second element)”, it should be understood that the element may be connected to the other element directly or through another element (e.g., a third element).

[0051] On the other hand, if it is described that an element (for example, a first element) is “directly coupled to” or “directly connected to” another element (for example, a second element), it can be understood that there is no element (for example, a third element) between the element and the other element.

[0052] Furthermore, the expression "configured to" used in the present disclosure may be used interchangeably with other expressions such as "suitable for", "capable of", "designed to", "adapted to", "made to", and "capable of" as appropriate. The term "configured to" does not necessarily mean that a device is "specifically designed to" in terms of hardware.

[0053] In some cases, the expression "a device configured to..." may mean that the device is "capable" of performing an operation together with another device or component. For example, the phrase "a processor configured to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by running one or more software programs stored in a memory device.

[0054] In an embodiment, a "module" or "part" performs at least one function or operation and can be implemented in hardware or software or in a combination of hardware and software. In addition, multiple "modules" or multiple "parts" other than the "module" or "part" that needs to be implemented in specific hardware can be integrated into at least one module and can be implemented as at least one processor.

[0055] It will be understood that the various elements and regions in the drawings may not be to scale. The scope of the present disclosure is not limited by the relative sizes or spacings shown in the drawings.

[0056] The electronic device according to various embodiments may include, for example, at least one of a smart phone, a tablet personal computer (PC), a desktop PC, a laptop PC, or a wearable device. The wearable device may include at least one of the following: an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, a pair of glasses, a contact lens, or a head mounted device (HMD)); a fabric or clothing embedded type (e.g., electronic cloth); a skin attached clothing (e.g., a skin pad or a tattoo); or a bio-implantable circuit.

[0057] In some embodiments, the electronic device may include, for example, at least one of the following: a television, a digital video disc (DVD) player, an audio system, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (such as SAMSUNG HOMESYNCM, APPLE TVTM or GOOGLETVTM), a game console (such as XBOXTM, PLAYSTATIONTM), an electronic dictionary, an electronic key, a camera or an electronic photo frame.

[0058] In other embodiments, the electronic device may include at least one of the following: various medical devices (e.g., various portable medical measurement devices (such as blood glucose meters, heart rate meters, blood pressure meters, or temperature measurement devices), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), or ultrasound devices, etc.), navigation systems, global navigation satellite systems (GNSS), event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, maritime electronic facilities (e.g., maritime navigation equipment, gyrocompasses, etc.), avionics equipment, security equipment, automotive head units, industrial or household robots, drones, automated teller machines (ATMs), points of sale in stores, or Internet of Things (IoT) devices (e.g., light bulbs, sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, boilers, etc.).

[0059] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily make and use the embodiments.

[0060] Figure 1 is a flowchart illustrating a control method of an electronic device according to an embodiment.

[0061] The electronic device may be a portable electronic device such as a smart phone, a tablet computer, etc., and there is no limitation on the type of the electronic device. Hereinafter, an electronic device according to an embodiment of the present disclosure will be described as an electronic device 100.

[0062] Reference Figure 1 , the electronic device 100 according to the embodiment may obtain the first image in operation S110. Specifically, the electronic device 100 may obtain the first image through a camera included in the electronic device 100. The electronic device 100 may also obtain the first image from an external device connected to the electronic device 100. ( Fig. 9 and Fig.10 The electronic device 100 as shown may store the obtained first image in the memory 110 of the electronic device 100.

[0063] If a first user command for correcting the first image is received, the electronic device 100 may obtain a second image obtained by correcting the first image in operation S120. Specifically, when a first user command for correcting the first image is received after obtaining the first image, the electronic device 100 may change at least one correction parameter related to the correction of the first image based on the user's setting and obtain the second image by correcting the first image. The electronic device 100 may store the obtained second image in a memory of the electronic device 100.

[0064] In the present disclosure, the term "correction parameter" may refer to various types of parameters related to correction of an image, such as hue, brightness, saturation, contrast, exposure, highlight, shadow, brightness, color temperature, noise, vignette, and black spot, but the correction parameter is not limited to the above examples.

[0065] "First user command" may refer to a user command for obtaining a corrected image based on the user's setting of the correction parameters. Specifically, the first user command may set how the user of the electronic device 100 will change at least one correction parameter related to the correction of the first image, and apply the set correction parameter to the first image to generate a second image. For example, if the user of the electronic device 100 inputs a first user command for changing the brightness and saturation of the first image to +13 and +8, respectively, through an image correction application, the electronic device 100 may obtain a second image obtained from the first image, the brightness and saturation of the second image being changed to +13 and +8, respectively. When a user interface (UI) such as a control bar is provided in the image correction application without providing a numerical value for each correction parameter, adjusting correction parameters such as exposure, sharpness, vignette, etc. through the adjustment bar may be a first user instruction, and the electronic device 100 may obtain a second image with changes in exposure, sharpness, vignette, etc., according to the first user instruction.

[0066] The "first image" may refer to an original image before obtaining the second image, and the "second image" may refer to a corrected image of the first image. The first image and the second image may be used for learning by a neural network model as described below. In the present disclosure, the "original image" does not only mean an image that is not corrected after the image is obtained by the electronic device 100 or an external device. In other words, the original image may be determined relative to an image obtained by performing correction on the original image.

[0067] The original image and the modified image corresponding to the original image can be distinguished from each other based on metadata. Specifically, the first image and the second image can be distinguished based on first metadata and second metadata corresponding to each of the first image and the second image.

[0068] First, when obtaining each of the first image and the second image, the electronic device 100 may obtain first metadata corresponding to the first image and second metadata corresponding to the second image. For example, the first metadata and the second metadata may respectively include information about at least one of the generation time, generation device, and correction time of the first image and the second image. At least one of the first metadata and the second metadata may include various information of the image, such as generation location, image quality, storage format, etc. In particular, at least one of the first metadata and the second metadata may be obtained in an image file format such as the Exchangeable Image File Format (EXIF).

[0069] When the first metadata and the second metadata are obtained, the electronic device 100 can identify whether the first image and / or the second image is an original image or a corrected image based on the first metadata and the second metadata. For example, the electronic device 100 can identify the image with the earliest correction time among multiple images with the same shooting time and shooting device in the learning data as the original image, and identify the image with a correction time later than the earliest correction time associated with the original image as the corrected image. As another example, the electronic device 100 can identify the image with the earliest correction time among multiple images with the same generation location in the learning data as the original image, and identify the image with a correction time later than the original image as the corrected image.

[0070] As described above, when the original image and the modified image are distinguished or identified, the electronic device 100 can input the original image and the modified image into the neural network model and train the neural network model.

[0071] When the first image and the second image are obtained, the electronic device 100 may train a neural network model based on the first image and the second image in operation S130 for obtaining an output image obtained by correcting the input image. Here, the "neural network model" may refer to an artificial intelligence model including a neural network. The term neural network model may be used interchangeably with the term artificial intelligence model.

[0072] Specifically, the electronic device 100 can perform training based on the first image and the second image so that the neural network model outputs a corrected image from the original image, or train the neural network model to output information for at least one correction parameter related to the correction of the original image.

[0073] In the present disclosure, for convenience, it will be described that the neural network model is trained mainly based on a pair of images of a first image and a second image, but the neural network model according to one or more embodiments may be trained based on learning data including images corrected by user settings and various pairs of original images, and the learning effect of the neural network model may be improved as the number of original images and corrected images included in the learning data increases. That is, in the learning process of the neural network model, the first image and the second image may refer to an image set randomly selected from a plurality of image sets included in the learning data.

[0074] The learning of the neural network model according to one or more embodiments may be performed under various conditions. Specifically, the learning process of the neural network model may be performed under the condition that the original image and the corrected image corresponding to the original image are obtained by the electronic device. In other words, once the original image and the corrected image corresponding to the original image are obtained, the electronic device 100 may input the obtained original image and corrected image into the neural network model to train the neural network model.

[0075] In addition, the learning process of the neural network model can be performed under certain conditions. Specifically, the electronic device 100 can train the neural network model based on the power supply of the electronic device 100 and the frequency of the user using the electronic device 100. For example, when the power supply of the electronic device 100 is sufficient (for example, greater than or equal to 50% of the battery), the electronic device 100 may be able to provide enough power to train the neural network model. As another example, when the user uses the electronic device 100 at a low frequency, the learning process of the neural network model can be performed to reduce the load of the electronic device. The frequency of use can be identified, for example, based on information such as movement of the electronic device sensed by a gyroscope sensor.

[0076] In operation S140, when a second user command for correcting the third image is received, a fourth image may be obtained by correcting the third image using the trained neural network model in operation S130.

[0077] Here, the "third image" may refer to a new original image that is different from the first image and the second image, and may be an image newly taken by a user or an image captured by a screen displayed on the electronic device 100. The third image refers to an original image before obtaining the fourth image, and the "fourth image" refers to an image obtained by correcting the third image. That is, similar to the first image and the second image, the terms third image and fourth image are also used in relative terms between the original image and the image obtained by performing correction on the original image. However, the first image and the second image may be used to train a neural network model according to an embodiment, and the third image is an input image to the trained neural network model after the neural network model is trained based on the first image and the second image, and the fourth image is an output image obtained by using the trained neural network model.

[0078] As described above, the "first user command" is a user command for obtaining a corrected image based on the user's correction parameter settings, and the "second user command" may refer to a user command for obtaining a corrected image through a neural network model according to an embodiment. For example, the second user command may be received based on a user input for selecting a specific user interface (UI) in a user interface provided by an application for image correction. The second user command may be provided based on a user input for selecting one of a plurality of thumbnail images, wherein a plurality of thumbnail images corresponding to a plurality of original images are provided by an application for reading an image. In addition, when information indicating whether correction of an original image corresponding to a thumbnail among a plurality of thumbnails is possible, a second user command may be received to select one of the thumbnails displayed as being correctable. An example of a specific user interface for inputting a second user command will be described in more detail with reference to FIG. 4A.

[0079] The neural network model according to the embodiment may include a first neural network model including a generator trained to generate an image and a comparator trained to distinguish or identify multiple images. The neural network model may include a second neural network model including an implementer trained to obtain information about at least one correction parameter related to the correction of the image and a comparator trained to distinguish or identify multiple images. For example, the first neural network model may be a generative adversarial network (GAN) model, but the neural network model according to the embodiment of the present disclosure is not limited thereto. Hereinafter, an example of implementing the neural network model as a first neural network model and an example of implementing the neural network model as a second neural network will be described in more detail.

[0080] First, in the case where the neural network model is implemented as a first neural network model, when a second user command for correcting a third image different from the first image and the second image is received, the electronic device 100 can input the third image into the trained first neural network model to obtain a fourth image obtained by correcting the third image.

[0081] Specifically, after performing the learning process for the first neural network model, when receiving a second user command for correcting the third image, the electronic device 100 can obtain a fourth image by correcting the third image using the trained first neural network model. The first neural network model according to an embodiment will be described with reference to FIGS. 2A and 2B.

[0082] Secondly, when the neural network model is implemented as a second neural network model, if a second user command for correcting a third image different from the first image and the second image is received, the electronic device 100 can obtain information about at least one correction parameter related to the correction of the third image by the neural network model in the trained neural network model, and obtain a fourth image by correcting the third image based on the information of the at least one correction parameter.

[0083] Specifically, when a second user command for correcting the third image is received after performing the learning process for the first neural network model, the electronic device 100 may obtain information about at least one correction parameter related to the correction of the third image by using the trained second neural network model. When the information about at least one correction parameter related to the correction of the third image is obtained, the electronic device 100 may obtain a fourth image by correcting the third image based on the at least one correction parameter. The second neural network model according to the embodiment will be described with reference to FIGS. 3A and 3B.

[0084] The embodiment of obtaining the fourth image based on the second user command is described above, but the present disclosure is not limited thereto. According to another embodiment of the present disclosure, the electronic device 100 can obtain the fourth image by correcting the third image when the third image is obtained by the electronic device. In other words, if the third image is obtained, the electronic device 100 can automatically obtain the fourth image even if the second user command is not received from the user.

[0085] As a result, according to the above reference Figure 1 In the various embodiments of the present disclosure described, the electronic device 100 may train a neural network model based on learning data, the learning data including an original image and a corrected image based on user settings. The electronic device 100 may use a trained first neural network model to obtain a corrected image that matches a correction mode preferred by the user, or may use a trained second neural network model to obtain information about correction parameters corresponding to a correction mode preferred by the user. In addition, the first neural network model and the second neural network model may be stored in the memory 110 and may be iteratively trained by the processor 120 that accesses the memory 110.

[0086] Therefore, the electronic device 100 can automatically correct the original image to improve user convenience, and can automatically apply the correction mode of user preference learned through the neural network model, thereby providing a corrected image in the mode preferred by each individual user.

[0087] A specific embodiment of the neural network model will be described in detail with reference to FIGS. 2A , 2B , 3A , and 3B .

[0088] 2A and 2B are diagrams illustrating a process of training a first neural network model 10 and obtaining a corrected image using the trained first neural network model 10 according to an embodiment.

[0089] 2A , the neural network model according to an embodiment of the present disclosure may be a first neural network model 10 including a generator 11 and a comparator 12. The first neural network model 10 may be a GAN model, but the embodiment is not limited thereto.

[0090] The generator 11 is a neural network model configured to generate an image. Specifically, when an original image among a plurality of images is randomly selected from learning data and input, the generator 11 can generate and output a corrected image based on the input original image. Here, the image being input to the generator 11 may mean that image data is input, and a vector corresponding to the information included in the image can be obtained by a neural network such as an embedder. By generating a corrected image by changing the pixel value of the image or by changing at least one parameter related to the correction of the image, a process of outputting a corrected image based on the input image can be performed.

[0091] The comparator 12 may refer to a neural network model configured to distinguish between multiple images. Specifically, if the generator 11 outputs a corrected image, the comparator 12 may compare the image generated by the generator 11 with the corrected image of the learning data to output feedback information. Here, the corrected image of the learning data may refer to a corrected image corresponding to the original image input to the generator 11 among the image set (i.e., the first image and the second image) included in the learning data. The feedback information may include probability information about whether the image generated by the generator 11 is similar to the corrected image of the learning data.

[0092] Hereinafter, the process of training the first neural network model 10 constructed using the above-mentioned architecture will be described. First, in the case of the comparator 12, a probability value close to 1 may mean that the image generated by the generator 11 is substantially similar to the corrected image of the learning data, and a probability value close to 0 may mean that the image generated by the generator 11 is not similar to the corrected image of the learning data. The value obtained by adding the probability value when the image generated by the generator 11 is input to the comparator 12 and the probability value when the corrected image of the learning data is input to the comparator 12 may be the loss function of the comparator 12.

[0093] By updating the weight of the comparator 12, the loss function can be minimized and the learning process of the comparator 12 can be performed. Specifically, the loss function value can be passed to the weight of each layer included in the generator 11 and the comparator 12 by back propagation to determine the direction and size of the update. The method of optimizing the weight in this way is called the gradient descent method. However, the optimization method of the weight is not limited to this, and other optimization methods may be included.

[0094] The comparator 12 may be trained to obtain a probability value close to 1 when the image generated by the generator 11 is input to the comparator 12. That is, the difference between the probability value when the image generated by the generator 11 is input to the comparator 12 and 1 may be the loss function of the generator 11, and the weight of the generator 11 is updated in a manner that minimizes the loss function value, so that the learning process of the generator 11 may be performed.

[0095] The generator 11 according to an embodiment of the present disclosure may be trained to generate an image similar to the modified image of the learning data, and the comparator 12 may be trained to distinguish or recognize the image generated by the generator 11 and the modified image of the learning data, so that the generator 11 and the comparator 12 may be trained by adversarial learning. As a result, the generator 11 may generate and output an image substantially similar to the modified image of the learning data.

[0096] In addition, the generator 11 can be trained based on several pairs of original images and corrected images included in the pre-constructed learning data. Here, the corrected images included in the pre-constructed learning data can be images corrected by experts in the field of image correction. When the generator 11 is trained based on the pre-constructed learning data, the generator can generate corrected images by reflecting effects commonly used in the field of image correction.

[0097] The generator 11 may be trained based on several pairs of original images and corrected images included in the learning data added by the user. Here, the learning data added by the user may include original images obtained by the user of the electronic device 100 and images corrected by user settings. When the generator 11 is trained based on the learning data added by the user, the generator may generate corrected images in a mode preferred by the user.

[0098] Hereinafter, the process of implementing the first neural network by the electronic device 100 according to an embodiment of the present disclosure will be described. First, the electronic device 100 may input the first image to the generator 11 of the first neural network model 10 to obtain a fifth image obtained by correcting the first image. Here, the "first image" may refer to an original image among a plurality of images randomly selected from the learning data as described above, and the "fifth image" may refer to an image generated and output by the generator 11 during the learning process of the first neural network model 10.

[0099] When the fifth image is obtained, the electronic device 100 may input the second image and the fifth image into the comparator 12 of the first neural network model 10 to obtain first feedback information related to the difference between the pixel value of the second image and the pixel value of the fifth image. Here, the "second image" may refer to an original image among a plurality of images randomly selected from the learning data as described above, and the "first feedback information" may include information about the loss function value obtained based on the difference between the pixel value of the second image and the pixel value of the fifth image.

[0100] In addition, the generator 11 can be trained based on the first feedback information. Specifically, the electronic device 100 can obtain a loss function value based on the difference between the pixel values ​​of the second image and the pixel values ​​of the fifth image, and back-propagate the loss function value to train the generator 11, thereby minimizing the loss function value. That is, the generator 11 can be trained to minimize the difference between the second image obtained based on the user setting for the correction parameter and the fifth image output by the generator 11. The comparator 12 can be trained to distinguish between the second image obtained based on the user setting for the correction parameter and the fifth image output by the generator 11.

[0101] As described above, when the first neural network model 10 is trained based on learning data including a first image and a second image (i.e., learning data including several pairs of original images and corrected images based on user settings), the generator 11 included in the first neural network model 10 can generate and output an image that is substantially similar to the image corrected based on the user settings.

[0102] After performing the above learning process, the electronic device 100 can obtain a fourth image obtained by correcting the third image using the trained neural network model. Specifically, referring to FIG. 2B , when a second user command for correcting a third image different from the first image and the second image is received, the electronic device 100 can input the third image into the training generator 11 to obtain a fourth image obtained by correcting the third image.

[0103] An embodiment in which the neural network model is a first neural network model 10 including a generator 11 and a comparator 12 has been described, and an embodiment in which the neural network model is a second neural network model including an implementer and a comparator 12 will be described below.

[0104] 3A and 3B are diagrams illustrating a process of training the second neural network model 20 and a process of obtaining a corrected image using the trained second neural network model 20 according to an embodiment.

[0105] 3A , the neural network model according to an embodiment of the present disclosure may be a second neural network model 20 including an implementer 21 and a comparator 23. In addition, as shown in FIG3A , in the process of obtaining a corrected image using a learning process of the second neural network model 20 and a trained second neural network model 20, an editor 22 may be used together with the implementer 21 and the comparator 23.

[0106] The implementer 21 refers to a neural network model configured to obtain information about at least one correction parameter related to the correction of the image. When an original image among a plurality of images randomly selected from the learning data is input, the implementer 21 can obtain and output information about at least one correction parameter related to the correction of the image based on the input original image. Here, inputting the image to the implementer 21 may refer to obtaining a vector corresponding to the information included in the image through a neural network such as an embedder and inputting the obtained vector and the input image itself in the same manner as inputting the image to the generator 11. The process of obtaining information about at least one correction parameter related to the correction of the image can be performed by a process of identifying which correction parameter should be applied to the original image to generate the corrected image or by identifying a certain pixel value to generate the corrected image.

[0107] The editor 22 refers to a module configured to generate a corrected image based on information about at least one correction parameter. Specifically, when information about at least one correction parameter related to correction of an image is output through the implementer 21, the editor 22 may receive information about at least one correction parameter from the implementer 21. In addition, based on the information about at least one correction parameter being input to the editor 22, the editor 22 may generate and output a corrected image based on the information about at least one correction parameter.

[0108] In addition, referring to FIG3A, although it is pointed out that only information about at least one correction parameter is input to the editor 22, the original image may be input to the editor 22 together with the information about at least one correction parameter. The implementer 21 may generate and output a corrected image by reflecting the information about at least one correction parameter to the original image. The editor 22 may be implemented as a separate module distinguished from the second neural network model 20 as shown in FIG3A, but may also be implemented as a module included in the second neural network model 20.

[0109] The comparator 23 refers to a neural network model configured to distinguish or recognize multiple images, as in the case of the first neural network model. Specifically, if the editor 22 outputs a corrected image, the comparator 23 may compare the image generated by the editor 22 with the corrected image of the learning data to output feedback information. As described above, the corrected image of the learning data refers to a corrected image corresponding to the original image input to the generator among the image set included in the learning data. The feedback information may include probability information as to whether the image generated by the editor 22 is similar to the corrected image of the learning data.

[0110] Hereinafter, the process of training the second neural network model 20 constructed using the above-mentioned architecture will be described. First, in the case of the comparator 23, a probability value close to 1 may indicate that the image generated by the editor 22 is substantially similar to the corrected image of the learning data, and a probability value close to 0 may indicate that the image generated by the editor 22 is not similar to the corrected image of the learning data. The value obtained by adding the probability value when the image generated by the editor 22 is input to the comparator 23 and the probability value when the corrected image of the learning data is input to the comparator 23 may be the loss function of the comparator 23. By updating the weight of the comparator 23 in a manner that minimizes the value of the loss function, the learning process of the comparator 23 may be performed.

[0111] The comparator 23 may be trained to obtain a probability value close to 1 when information about at least one correction parameter obtained by the generator is input to the comparator 23. That is, the difference between the probability value when the image generated by the editor 22 is input to the comparator 23 and the value 1 may be the loss function of the generator, and the weight of the generator may be updated to minimize the value of the loss function, so that the learning process of the generator may be performed.

[0112] In particular, the implementer 21 may be trained based on feedback information obtained by the comparator 23. The feedback information may include information about correction parameter values ​​that will be modified so that the image generated by the editor 22 is an image similar to the correction image of the learning data. More specifically, if an image is generated by the editor 22 based on the correction parameters output by the implementer 21, the correction image of the image and the learning data generated by the editor 22 may be input to the comparator 23. The comparator 23 may compare the image generated by the editor 22 with the correction image of the learning data to obtain feedback information related to the difference between the pixel values ​​of the image generated by the editor 22 and the pixel values ​​of the correction image of the learning data.

[0113] Regarding the learning method of the second neural network model 20, the learning method of the first neural network model described above can be applied with reference to FIG. 3A, and some or all of its repeated descriptions will be omitted.

[0114] Hereinafter, a process of implementing an embodiment using a second neural network by the electronic device 100 will be described. First, the electronic device 100 may input a first image into a generator of the second neural network model 20 to obtain at least one correction parameter related to correction of the first image. As described above, the at least one correction parameter may include at least one of hue, brightness, saturation, and contrast. In addition, the at least one correction parameter may include various correction parameters related to correction of an image, such as exposure, highlight, shadow, brightness, color temperature, noise, vignette, black spot, etc.

[0115] When at least one correction parameter related to the correction of the first image is obtained, the electronic device 100 may input at least one correction parameter related to the correction of the first image into the editor 22 to obtain a sixth image obtained by correcting the first image. The term "sixth image" may refer to an image output by the editor 22 based on the correction parameter output by the implementer 21 during the learning process of the second neural network model 20. The editor 22 may be implemented by an application for image correction and may be embodied in a form included in the second neural network model 20.

[0116] When obtaining the sixth image, the electronic device 100 may input the second image and the sixth image into the comparator 23 to obtain second feedback information related to the difference between the pixel values ​​of the second image and the pixel values ​​of the sixth image. The “second feedback information” may include information about a loss function value obtained based on the difference between the pixel values ​​of the second image and the pixel values ​​of the sixth image.

[0117] The implementer 21 can be trained based on the second feedback information. As described above, the learning method for the first neural network model can also be applied to the learning method for the second neural network model 20, so its detailed description will be omitted.

[0118] If the learning process as described above is performed based on the image pairs included in the learning data, the implementer 21 can obtain correction parameters similar to the correction parameters set by the user. The editor 22 can generate and output an image similar to the correction image based on the user setting based on the information about at least one correction parameter obtained by the implementer 21.

[0119] In addition, after performing the learning process as described above, the electronic device 100 can obtain a fourth image by correcting the third image using the trained neural network model. Specifically, referring to FIG. 3B , when a second user command for correcting a third image different from the first image and the second image is received, the electronic device 100 can obtain information about at least one correction parameter related to the correction of the first image in the trained neural network model, and correct the third image based on the information about the at least one correction parameter to obtain the fourth image.

[0120] 4A to 4D are diagrams illustrating user interfaces according to various embodiments.

[0121] 4A shows a first user interface 41 according to an embodiment. The first user interface (UI) 41 may include a third image as an original image and a first user interface element 410. In the third image shown in FIG4A , “face of a person” and “hand of a person” are shaded to indicate portions having a lower brightness in the original image than in the modified image.

[0122] A second user command for correcting the third image by the neural network model according to an embodiment of the present disclosure may be input based on a user interaction of touching first UI element 410 of FIG. 4A . When the second user command is received based on a user interaction of touching first UI element 410, electronic device 100 may use the trained neural network model to obtain a fourth image by correcting the third image.

[0123] FIG4B shows a second user interface 42, which includes a fourth image obtained according to the process described with reference to FIG4A. Specifically, the second user interface 42 may include a fourth image and a second UI element for selecting whether to correct the fourth image. In the fourth image shown in FIG4B, the "human hand" is shaded, but the "human face" is not shaded, indicating that the brightness of the "human face" is corrected to be higher through correction via the neural network model. The electronic device 100 can obtain a fourth image in which the brightness of the "human face" of the third image as the original image is corrected.

[0124] The second user interface 42 may include a second UI element 420 for selecting whether to additionally correct the fourth image based on the user settings for the correction parameters. For example, if a user input is received on the "OK" interface displayed on the second UI element 420 as shown in FIG. 4B , the electronic device 100 may determine that the fourth image does not require additional correction based on the user settings for the correction parameters, and may store the fourth image. If a user input is received on the "Edit More" interface of the second UI 420 as shown in FIG. 4B , the electronic device 100 may additionally correct the fourth image based on the user settings.

[0125] FIG. 4C illustrates a third user interface 43 displayed when it is determined that the fourth image is to be additionally corrected based on the user's settings according to the process described with reference to FIG. 4B .

[0126] The third user interface 43 may include a third UI element 430 as shown in FIG. 4C . Specifically, if it is determined that the fourth image is to be additionally corrected based on the user settings, the electronic device 100 may display the third user interface 43, which includes a third UI element 430 for selecting parameters such as "shadow", "saturation" and "brightness". The third UI element 430 may include at least one of the parameters such as the user's frequency of use. That is, a parameter indicating the frequency with which the user uses one or more of the parameters displayed on the third UI element 430. In addition, when a fourth user command for selecting at least one parameter related to the correction of the fourth image is received through the second UI element 430, the electronic device 100 may obtain and display a seventh image obtained by correcting the fourth image.

[0127] FIG. 4D illustrates a fourth user interface 44 displayed when user setting for a correction parameter is performed through the third UI element 430 described with reference to FIG. 4C .

[0128] The fourth user interface 44 may include a seventh image obtained by correcting the fourth image. For example, in the seventh image shown in FIG. 4D , shading processing is not performed on both the “face of the person” and the “hand of the person”, indicating that after the brightness of the “face of the person” has been corrected to be high through correction via the neural network model, the brightness of the “hand of the person” has been corrected to be high through user settings. The electronic device 100 may obtain the seventh image by correcting the brightness of the “hand of the person” of the fourth image that may be used as an original image.

[0129] If the seventh image is obtained by correcting the fourth image, the electronic device 100 can train the neural network model based on the third image and the seventh image. In other words, the third image as the original image and the seventh image as the image corrected according to the embodiment described with reference to FIGS. 4A to 4D can also be included in the learning data for training the neural network model according to the present disclosure.

[0130] Figure 5 , FIG. 6A and FIG. 6B are diagrams illustrating user interfaces according to various embodiments.

[0131] 4B , when a second user command for correcting the original image is received through the neural network model, the electronic device 100 obtains a corrected image corresponding to the original image through the trained neural network model, and further corrects the corrected image through user settings, but the embodiment is not limited thereto.

[0132] like Figure 5 As shown, when a second user command for correcting an image is received through the neural network model, the electronic device 100 may obtain a plurality of corrected images corresponding to the original image through the neural network model, and display a fifth user interface 510 including the obtained plurality of corrected images. When a user command for selecting one of the plurality of corrected images is received through the fifth user interface 510, the electronic device 100 may store the selected image as the corrected image and train the neural network model based on the original image and the selected corrected image.

[0133] According to an embodiment, the electronic device 100 may display the sixth user interface 610 and 620 for selecting whether to change the correction parameters in a predefined manner to obtain a corrected image or to obtain a corrected image through a neural network model. For example, as shown in FIGS. 6A and 6B , the sixth user interfaces 610 and 620 may include a recommendation list including UI elements (“contrast”, “brighten”, and “soften”) for selecting a predefined correction scheme and a UI element (“AI recommendation”) for selecting to obtain a corrected image through a neural network model.

[0134] In this example, the order in which multiple UI elements are displayed can change as the corrected image is obtained by user settings. In particular, as shown in FIG6A, if the number of corrected images is not large in the learning data, the UI element for selecting the correction method frequently used by the user can be displayed as the upper item of the recommendation list, and the UI element for selecting the correction method through the neural network model can be displayed as the lower item of the recommendation list displayed on the user interface 610. As shown in FIG6B, when the number of corrected images increases in the learning data, the neural network model can provide the corrected image in a mode preferred by each individual user. Here, the electronic device 100 can display a UI element (e.g., "AI recommendation") for selecting one of the correction methods to obtain the corrected image through the neural network model in the recommendation list on the sixth user interface 620. Specifically, the UI element recommended by AI can be displayed on the sixth user interface 620, so that the AI ​​recommendation is displayed on the recommendation list at a higher position than other UI elements (e.g., "contrast", "brighten" and "soften") for selecting a predetermined correction method. Here, the electronic device 100 can determine whether the number of corrected images is large based on a predetermined value.

[0135] Figure 7 is a flowchart illustrating a method of performing image correction according to image type according to an embodiment.

[0136] A method of training a neural network model based on a first image and a second image and obtaining a fourth image obtained by correcting a third image using the trained neural network model is described, but the correction mode preferred by the user may vary depending on the image type. For example, if a person is included in the image, the user may prefer to correct (e.g., increase the brightness value) an area of ​​the image corresponding to the "face" of the person, and if a plant is included in the image, the user may prefer to correct (e.g., increase the sharpness value) another area of ​​the image corresponding to the "leaves of the plant".

[0137] Therefore, the electronic device 100 may perform various corrections on the image depending on the type of the image, which will be referred to as Figure 7 Describe in detail.

[0138] As described above, the electronic device 100 may obtain a first image as an original image and obtain a second image by correcting the first image in operation S710. In operation S720, the electronic device 100 may obtain first type information related to a type of the first image and a type of the second image.

[0139] Here, the term first type information may refer to all information that can be used to classify the type of the first image. For example, the first type information may include at least one of information about an object included in the first image, information about a location where the first image was taken, and information about a time when the first image was taken.

[0140] In particular, information about the object included in the first image can be obtained through the following process. First, the electronic device 100 can extract the boundary in the first image to identify the existence of the object and the position of the object. For example, the electronic device 100 can identify at least one object included in the first image through 2D image matching, optical character recognition (OCR), an artificial intelligence model for object recognition, etc. More specifically, the electronic device 100 can use various methods such as edge detection, corner detection, histogram feature detection, image high frequency analysis, image variance analysis, etc. to extract features of the object included in the first image. The electronic device 100 can obtain the probability that the object included in the first image corresponds to each of the multiple categories for classifying the object based on the extracted features, and identify at least one object included in the third image.

[0141] The electronic device 100 can identify at least one object included in the first image through a trained object recognition model. Specifically, the electronic device 100 can input the first image into a trained artificial intelligence model to identify at least one object included in the image. Here, the object recognition model can be an artificial intelligence model trained using at least one of artificial intelligence algorithms (such as machine learning, neural networks, genes, deep learning, and classification algorithms), and can include at least one artificial neural network between a convolutional neural network (CNN) and a recurrent neural network (RNN). However, there is no limitation on the type of the object recognition model and the type of the artificial neural network included in the object recognition model.

[0142] The object recognition model according to an embodiment of the present disclosure may include a detailed model implemented for each type of object, such as a user recognition model, a food recognition model, and a user recognition model. A face recognition model for recognizing a user's face at a finer level than the user recognition model and an emotion recognition model for respectively recognizing a user and the user's emotions based on the user's facial expression, etc. At least some of the above models may operate based on predefined rules that may not be artificial intelligence.

[0143] Information about the location where the first image was taken and the time when the first image was taken can be obtained in the form of metadata as described above by acquiring the first image. In particular, the information about the location where the first image was taken can be obtained based on information about the background among the objects included in the first image, and can be obtained based on global positioning system (GPS) information obtained based on a sensor included in the electronic device 100.

[0144] Once the first type information is obtained, the electronic device 100 may train a neural network model based on the first image, the second image, and the first type information in operation S730. Specifically, an example of implementing the neural network model as a single neural network model is described, but the neural network model according to an embodiment of the present disclosure may include a plurality of neural network models divided according to the type of the input image. The electronic device 100 may identify a neural network model corresponding to the type of the first image among the plurality of neural network models based on the first type information as described above, and perform learning via each of the plurality of neural network models by inputting the first image and the second image to the neural network model identified as the type corresponding to the first image. For example, if it is identified based on the first type information that the first image includes an object "person", the electronic device 100 may identify a neural network model for correcting the "person" among the plurality of neural network models, and train the identified neural network model based on the first image and the second image. As another example, if it is identified based on the first type information that the shooting space of the first image is an "area", the electronic device 100 may identify a neural network model for correcting an image in which the shooting space of the plurality of neural network models is an "area", and train the identified neural network model based on the first image and the second image.

[0145] As described above, the electronic device 100 may obtain a third image as a new original image different from the first image and the second image in operation S740. If the third image is obtained, the electronic device 100 may obtain second type information related to the type of the third image in operation S750. The term second type information is used here to refer to all information that can be used to classify the type of the third image. Specifically, the second type of information may include at least one of information about an object included in the third image, information about a location where the third image was taken, and information about a time when the third image was taken.

[0146] When receiving the second user command for correcting the third image, the electronic device 100 may obtain a fourth image obtained by correcting the third image using the trained neural network model based on the third image and the second type information in operation S760. The electronic device 100 may identify a neural network model corresponding to the type of the third image among a plurality of neural network models trained by the type of the image based on the second type information as described above, and input the third image to the identified neural network model corresponding to the type of the third image to obtain the fourth image by correcting the third image.

[0147] For example, if it is identified based on the second type of information that the third image includes an object "person", the electronic device 100 can identify a neural network model for correcting the "person" among the trained multiple neural network models, and input the third image to the identified neural network model to obtain the fourth image by correcting the third image. As another example, if it is identified based on the second type of information that the shooting location is "Area A", the electronic device 100 can identify a neural network model for correcting the "Area A" among the trained multiple neural network models, and can obtain a fourth image that corrects the third image input to the identified neural network model.

[0148] An example has been described in which the first and third images include the object "person", but the type of the image may be classified in various ways, such as "specific person X." Furthermore, the type and breadth / narrowness of the category for classifying the type of the image may be implemented in various ways depending on the design.

[0149] An example is described in which the location where the first image and the third image were taken is "Area A", but after a neural network model for correcting images taken at "Area A" (for example, Area A famous for the aurora) is used among multiple neural network models, if a third image taken at "Area B similar to Area A (for example, Area B famous for the aurora like Area A)" is obtained based on the first image, the third image can be corrected using the neural network model for correcting images taken at "Area A".

[0150] An example is described in which a neural network model corresponding to the type of a first image among multiple neural network models is identified based on first type information, and learning is performed by the multiple neural network models by inputting the first image and the second image into the neural network model identified as the type corresponding to the first image, but the process of identifying the neural network model corresponding to the type of the first image among multiple neural network models and performing learning based on the first image and the second image can be implemented by one neural network model based on information about the first type.

[0151] Although the neural network model according to an embodiment of the present disclosure may be implemented in a plurality of neural network models depending on the type of an image, the neural network model may be implemented as an integrated neural network model, and the electronic device 100 may input the first type information together with the first image and the second image into the neural network model to train the neural network model.

[0152] 8A and 8B are diagrams illustrating a method of performing image correction by recognizing an object included in an image.

[0153] FIG. 8A is a flowchart illustrating a method of performing image correction by recognizing an object included in an image, and FIG. 8B is a diagram illustrating recognizing an object included in an image.

[0154] If the third image is obtained in operation S810, the electronic device 100 may recognize at least one object included in the third image in operation S820. Figure 7 A method for recognizing an object included in an image is described in detail, so a description thereof will be omitted.

[0155] For example, the electronic device 100 may recognize the object 510 including the face, head, and hair portion of a person in the third image including the person by the method described above, as shown in FIG. 8B .

[0156] If at least one object included in the third image is identified, the electronic device 100 may determine whether to perform correction on the third image based on a preset object that substantially matches the target object of correction in the at least one identified object. For example, the preset object may be obtained based on learning data for training a neural network model and may be stored in a memory of the electronic device.

[0157] Specifically, if the preset object substantially matches the target object among at least one identified object on which correction is to be performed (operation S830-yes), the electronic device 100 may input the third image into the trained neural network model in operation S840 to obtain an eighth image in which the target object among at least one object included in the third image is corrected.

[0158] The term "correction" refers to correction performed on an image using a neural network model according to an embodiment of the present disclosure. Here, matching a preset object with a target object for correction may mean that if a preset object (e.g., a plant) is included in at least one object identified in another image or substantially matches a target object among the at least one object, correction is performed using a neural network model in the electronic device 100. In addition, if it is determined that the identified target object is frequently corrected by the user, correction may be performed using a neural network model based on at least one parameter previously used by the user to correct an image including the preset object.

[0159] 8B, if the electronic device 100 is preset to perform correction using the neural network model when an object 810 (i.e., a person) including a face, a head, and hair is included in the third image, For example, if the object 810 is recognized as including a face, a head, and a hair portion of a person, the electronic device 100 may input the third image into the trained neural network model to obtain an eighth image in which the object 810 is corrected among at least one object included in the third image.

[0160] If the preset object is not included as the object that is the correction target among the recognized at least one object in operation S830-No, the electronic device 100 may not perform correction on the third image using the trained neural network model.

[0161] According to an embodiment described above with reference to FIGS. 8A and 8B , the electronic device 100 may provide higher user satisfaction by performing correction using a neural network model only on an image including an object that the user wishes to correct.

[0162] Fig. 9 and Fig.10 is a block diagram showing the electronic device 100 .

[0163] Fig. 9 is a block diagram showing an electronic device 100 according to an embodiment, Fig.10 is a block diagram showing an electronic device 100 according to an embodiment.

[0164] like Fig. 9 As shown, the electronic device 100 includes a memory 110 and a processor 120. Fig.10 As shown, the electronic device 100 may further include a communicator 130, an inputter 150, and an outputter 140. However, the embodiment of the electronic device 100 is only an example, and new configurations may be added or some configurations may be omitted in addition to the embodiment described herein.

[0165] At least one instruction regarding the electronic device 100 may be stored in the memory 110. In addition, an operating system (O / S) for driving the electronic device 100 may be stored in the memory 110. According to various embodiments, the memory 110 may store various software programs or applications for operating the electronic device 100. The memory 110 may include a semiconductor memory such as a flash memory, a magnetic storage medium such as a hard disk, and the like.

[0166] Specifically, the memory 110 may store various software modules for operating the electronic device 100, and the processor 120 may control the operation of the electronic device 100 by running the various software modules stored in the memory 110. That is, the memory 110 may be accessed by the processor 120, and the processor 120 may perform reading, recording, modification, deletion, updating, etc. of data.

[0167] The memory 110 may refer to any volatile or nonvolatile memory, a read-only memory (ROM), a random access memory (RAM) communicatively coupled to or in the processor 120, or a memory card (e.g., a micro SD card, a memory stick) connectable to the electronic device 100.

[0168] In various embodiments according to the present disclosure, the memory 110 may store learning data for training a neural network model, information about the first to eighth images, first feedback information and second feedback information, first metadata information and second metadata information, etc. In addition, various information necessary to achieve the purpose of the present disclosure may be stored in the memory 110, and the information stored in the memory 110 may be received from a server or an external device or updated by a user's input.

[0169] The processor 120 controls the overall operation of the electronic device 100. Specifically, the processor 120 may be connected to and configured to control the operations of the memory 110, the communicator 130, the outputter 140, and the inputter 150, and may control the operation of the electronic device 100 by executing at least one command stored in the memory 110.

[0170] The processor 120 may be implemented in various ways. For example, the processor 120 may be implemented as at least one of an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), etc. The term processor 120 in the present disclosure may be used in the sense of including a central processing unit (CPU), a graphics processing unit (GPU), a main processing unit (MPU), etc.

[0171] In particular, in various embodiments, the processor 120 may: obtain a first image by executing at least one instruction; obtain a second image obtained by correcting the first image based on receiving a first user command to correct the first image; train a neural network model based on the first image and the second image to obtain an output image obtained by correcting the input image; and obtain a fourth image obtained by correcting the third image using the trained neural network model based on receiving a second user command to correct the third image. As has been mentioned above, Figure 1 Various embodiments according to the present disclosure based on the control of the processor 120 are described in detail through FIG. 8B , so a description thereof will be omitted.

[0172] The communicator 130 includes a circuit or an interface and can perform communication with a server or an external device. Specifically, the processor 120 can receive various data or information from a server or an external device connected through the communicator 130, and can send various data or information to the server or the external device.

[0173] The communicator 130 may include at least one of a Wi-Fi module, a Bluetooth module, a wireless communication module, and a near field communication (NFC) module. Specifically, the Wi-Fi module may communicate via a Wi-Fi method, and the Bluetooth module may communicate via a Bluetooth method. When using a Wi-Fi module or a Bluetooth module, various connection information such as a service set identifier (SSID) may be sent and received for communication connection, and then various information may be sent and received.

[0174] The wireless communication module may communicate according to various communication specifications such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), 5th Generation (5G), etc. The NFC module may communicate by an NFC method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860-960 MHz, 2.45 GHz, etc.

[0175] In various embodiments, a plurality of images included in learning data for training a neural network model may be received from an external device through the communicator 130 .

[0176] The neural network model may be included in the electronic device 100 in the form of an on-device device, or may be included in a server outside the electronic device 100. When the neural network model is included in a server outside the electronic device 100, the electronic device 100 may use the neural network model of the server to implement various embodiments. For example, if a second user command for correcting a third image is received, the electronic device 100 may control the communicator 130 to send a control command corresponding to the second user command together with the third image to the server, and receive and obtain, through the communicator 130, a fourth image obtained by correcting the third image from the server.

[0177] The outputter 140 includes a circuit, and the processor 120 may output various functions executable by the electronic device 100 through the outputter 140. The outputter 140 may include at least one of a display, a speaker, and an indicator.

[0178] The display may output image data under the control of the processor 120. Specifically, the display may output an image pre-stored in the memory 110 under the control of the processor 120. In particular, the display according to an embodiment may display a user interface stored in the memory 110.

[0179] The display may be implemented as a liquid crystal display (LCD) panel, an organic light emitting diode (OLED), etc., and the display may also be implemented as a flexible display, a transparent display, etc. However, the display is not limited thereto. The speaker may output audio data under the control of the processor 120, and the indicator may be turned on by the control of the processor 120.

[0180] According to an embodiment of the present disclosure, the display can display various types of images, and can display images as shown in Figures 4A-4D, Figure 5 and various types of user interfaces shown in FIGS. 6A-6B .

[0181] The input device 150 includes a circuit configured to receive input from a user or other devices, and the processor 120 may receive a user command for controlling the operation of the electronic device 100 through the input device 150. Specifically, the input device 150 may include a microphone, a camera, a remote control signal receiver, etc. The input device 150 may be embodied as a touch screen in the form of a display.

[0182] According to various embodiments of the present disclosure, a first user command and a second user command for correcting an image may be input through the inputter 150. For example, at least one of the first user command and the second user command may be input in the form of user interaction through a touch screen, and may be input in the form of a voice signal through a microphone. The processor 120 may obtain an original image according to the present disclosure through a camera.

[0183] The control method of the electronic device 100 according to the aforementioned embodiment may be implemented as a program and provided to the electronic device 100. In particular, the program including the control method of the electronic device 100 may be stored in a non-transitory computer-readable medium and provided.

[0184] Specifically, a computer-readable recording medium including a program for running a control method of the electronic device 100 includes: obtaining a first image; based on receiving a first user command to correct the first image, obtaining a second image obtained by correcting the first image; training a neural network model to obtain an output image obtained by correcting the input image; and based on receiving a second user command to correct the third image, obtaining a fourth image obtained by correcting the third image using the trained neural network model.

[0185] The non-transitory computer-readable medium refers to a medium that stores data semi-permanently rather than for a short period of time, such as a register, a cache, the memory 110, etc., and can be read by a device. In detail, the aforementioned various applications or programs may be stored in a non-transitory computer-readable medium (e.g., a compact disk (CD), a digital versatile disk (DVD), a hard disk, a Blu-ray disk, a universal serial bus (USB), a memory card, a read-only memory (ROM), etc.), and may be provided.

[0186] According to the various embodiments described above, the electronic device 100 can automatically correct the original image to improve user convenience and image processing accuracy by automatically reflecting the correction mode preferred by the user when correcting the image, thereby providing an image corrected to a mode preferred by each individual user.

[0187] In addition, according to various embodiments, unlike the related art of merely converting the style of an image, a neural network model can be trained by detecting slight differences between the original image and the corrected image, thereby enabling users to apply more natural correction effects that users frequently use for correcting people, landscapes, etc.

[0188] Functions associated with artificial intelligence according to an embodiment of the present disclosure are operated by a processor and a memory. The processor may be configured with one or more processors. One or more processors may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), etc.), a pure graphics processor (such as a graphics processor (GPU), a visual processing unit (VPU)), a pure AI processor (such as a neural network processor (NPU)), etc., but the processor is not limited thereto. One or more processors may control the processing of input data according to predefined operating rules or AI models stored in the memory. If one or more processors are pure AI processors, the pure AI processor may be designed with a hardware structure specifically for processing a specific AI model.

[0189] The predefined operating rules or artificial intelligence models are formed by learning. Here, formation by learning may refer to training the basic artificial intelligence model according to the learning algorithm by using multiple learning data, thereby generating predefined operating rules or artificial intelligence models that are set to perform the desired characteristics (or purposes). Learning can be performed by the device itself in which the artificial intelligence according to the embodiments of the present disclosure is executed, and can be implemented by a separate server and / or system. Examples of training algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0190] The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network processing operation may be performed by iterative operations using the results of the previous layer and multiple weight values. The multiple weight values ​​included in the multiple neural network layers may be optimized by the training results of the AI ​​model. For example, multiple weight values ​​may be updated so that the loss value or cost value obtained in the AI ​​model during the training process is reduced or minimized. The artificial neural network may include a deep neural network (DNN) and may include, for example, but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc. The learning algorithm is a method for training a predetermined target device (e.g., a robot) by using multiple learning data to determine or predict the predetermined target device by itself. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is not limited to the above examples unless otherwise stated.

[0191] A machine-readable storage medium may be provided in the form of a non-transitory storage medium that is a tangible device and may not include signals (e.g., electromagnetic waves). The term does not distinguish whether data is permanently or temporarily stored in a storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0192] According to an embodiment, the method disclosed herein may be provided in the software of a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or published online through an application store (e.g., PlayStore) or distributed online (e.g., downloaded or directly transferred) directly between two user devices (e.g., smart phones). In the case of online publishing, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily stored or at least temporarily stored in a storage medium (such as a manufacturer's server, a server in an application store, or a memory in a relay server).

[0193] Each component (e.g., module or program) according to one or more embodiments may be composed of one or more objects, and some subcomponents in the above components may be omitted, or other subcomponents may be further included in an embodiment. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding component before integration.

[0194] According to an embodiment, operations performed by a module, program or other component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or at least some operations may be performed in a different order, omitted, or other operations may be added.

[0195] The term "unit" or "module" used in the present disclosure includes a unit including hardware, software or firmware, and can be used interchangeably with terms such as, for example, logic, logic block, part or circuit. A "unit" or "module" can be a component of an integral structure that performs one or more functions or its smallest unit or part. For example, a module can be configured as an application-specific integrated circuit (ASIC).

[0196] The embodiments of the present disclosure may be implemented as software, which includes instructions stored in a machine-readable storage medium readable by a machine (e.g., a computer). The device may call instructions from the storage medium and may operate according to the called instructions, including an electronic device (e.g., an electronic device).

[0197] When the instruction is executed by the processor, the processor may use other components to perform the function corresponding to the instruction directly or under the control of the processor. The instruction may include a code generated or executed by a compiler or an interpreter.

[0198] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined at least by the appended claims and their equivalents.

Claims

1. An electronic device, include: a memory configured to store at least one instruction; as well as at least one processor configured to execute the at least one instruction to: Obtaining the original image and the corrected image; obtaining information about at least one modification parameter by identifying at least one modification parameter to be applied to the original image to generate the modified image using an artificial intelligence (AI) model; receiving a user command to modify the first image; as well as Based on the information about the at least one correction parameter, a second image is obtained in which the first image is corrected.

2. The electronic device according to claim 1, in, The corrected image is an image obtained by correcting the original image.

3. The electronic device according to claim 1, in, Information about at least one correction parameter is obtained by identifying pixel values ​​to generate a corrected image.

4. The electronic device according to claim 1, in, At least one processor includes an AI processor for controlling the operation of the AI ​​model, wherein the AI ​​processor is designed using a hardware structure dedicated to processing the trained AI model.

5. The electronic device according to claim 1, in, The AI ​​model is a second AI model including an implementer configured to obtain information about at least one correction parameter and a comparator configured to compare a plurality of images, and Wherein, at least one processor is further configured to: obtaining information about at least one correction parameter associated with correction of the original image by inputting the original image into the implementer; obtaining a third image by correcting the original image based on the information about the at least one correction parameter; obtaining feedback information based on a difference between a pixel value of the corrected image and a pixel value of the third image by inputting the corrected image and the third image into a comparator; and The implementer is trained based on the feedback information.

6. The electronic device according to claim 5, in, The at least one processor is further configured to: Based on receiving the user command, inputting the first image to the second AI model; and The second image is obtained by correcting the first image based on the information about the at least one correction parameter associated with the correction of the original image.

7. The electronic device according to claim 1, further comprising: include: monitor, The processor is further configured as follows: Based on obtaining the second image, controlling the display to display a first user interface (UI) element for selecting whether to correct the second image based on a user setting regarding a correction parameter; Based on receiving a second user command through the first UI element for selecting to modify the second image, controlling the display to display a second UI element for selecting at least one parameter associated with modification of the second image; as well as Based on receiving a third user command for selecting the at least one parameter associated with correction of the second image through the second UI element, a fourth image is obtained by correcting the second image.

8. The electronic device according to claim 1, in, The at least one processor is further configured to execute the at least one instruction to: obtaining first type information associated with a type of the original image; and The AI ​​model is trained based on the original image, the corrected image and the first type of information. The first type of information includes at least one of information about an object included in the original image, information about a location where the original image was photographed, and information about a time when the original image was photographed.

9. The electronic device according to claim 8, in, The at least one processor is further configured to execute at least one instruction to: Based on obtaining the first image, obtaining second type information associated with the type of the first image; and Based on receiving a first user command to correct the first image, obtaining a second image by correcting the first image based on the first image and the second type of information using the trained AI model, The first type of information includes at least one of information about an object included in the first image, information about a location where the first image was photographed, and information about a time when the first image was photographed.

10. A method for controlling an electronic device, the method include: Obtaining the original image and the corrected image; obtaining information about at least one modification parameter by identifying at least one modification parameter to be applied to the original image to generate the modified image using an artificial intelligence (AI) model; receiving a user command to modify the first image; as well as Based on the information about the at least one correction parameter, a second image is obtained in which the first image is corrected.

11. The method according to claim 10, in, The corrected image is an image obtained by correcting the original image.

12. The method according to claim 10, in, Information about at least one correction parameter is obtained by identifying at least one correction parameter to be applied to the original image to generate the corrected image.

13. The method according to claim 10, in, Information about at least one correction parameter is obtained by identifying pixel values ​​to generate a corrected image.

14. The method according to claim 10, in, The AI ​​model is a second AI model including an implementer configured to obtain information about at least one correction parameter and a comparator configured to compare a plurality of images, and The method further comprises: obtaining information about at least one correction parameter associated with correction of the original image by inputting the original image into the implementer; obtaining a third image by correcting the original image based on the information about the at least one correction parameter; obtaining feedback information based on a difference between a pixel value of the corrected image and a pixel value of the third image by inputting the corrected image and the third image into a comparator; and The implementer is trained based on the feedback information.

Images